What is Database Sharding? Architecture & Horizontal Scaling Explained
Database Sharding is an architectural pattern in which a single logical dataset is horizontally partitioned across multiple physical database servers (called shards). Each shard runs on independent hardware and holds a distinct subset of the overall data.
Why It Matters in Production
While read replicas can scale read queries, traditional databases hit a hard physical ceiling on write throughput and disk IOPS on a single primary node. Sharding breaks past single-machine CPU, memory, and disk bottlenecks to scale write operations linearly.
How It Works Under the Hood
Incoming write and read queries pass through a Shard Router or application-level proxy. The router hashes the Shard Key (such as tenant_id or user_uuid) to calculate which specific physical database instance holds the target records.
Code Architecture Example
// Consistent Hashing / Modulo Shard Selection Concept
function getShardConnection(userId, shardPool) {
const hash = crypto.createHash('md5').update(userId).digest();
const shardIndex = hash.readUInt32BE(0) % shardPool.length;
return shardPool[shardIndex];
}Common Architectural Pitfalls
- Choosing a low-cardinality or time-based shard key causes write hotspots on a single shard.
- Cross-shard joins require costly multi-node distributed transactions and two-phase commits.
- Re-sharding existing live production databases without downtime requires complex dual-writing strategies.
Sharding, Replication and System Design: High-Scale Architecture
Explore the complete, step-by-step tutorial published on Aditya's engineering blog.