Database Sharding

Sharding addresses the limitations of vertical scaling by distributing data across multiple database instances. Each shard contains a subset of the total data, determined by a shard key (e.g., user ID, geographic region, or time range).

Choosing the right shard key is critical: it should evenly distribute data (avoiding hotspots), support the most common query patterns, and minimize cross-shard queries which are expensive and complex.

Common sharding strategies include range-based sharding (shard by ranges of the shard key), hash-based sharding (apply a hash function to the shard key), and directory-based sharding (maintain a lookup table mapping keys to shards).

Challenges include cross-shard joins, maintaining global uniqueness of IDs, rebalancing shards as data grows unevenly, and managing schema changes across all shards. Despite these complexities, sharding is essential for systems handling massive datasets.