Scalability
Sharding
Partitioning one dataset across multiple databases by a shard key (user id, region) so each node holds a slice, letting writes and storage scale beyond a single machine. It is the answer when one database can no longer keep up. The costs are steep: cross-shard queries and transactions get hard, and a poor key creates hot spots and painful re-sharding.
Purpose
Sharding partitions one logical dataset across multiple database nodes by a shard key — user ID, tenant, region — so each node stores and serves only a slice. It is how writes and storage scale past what any single machine can hold, when replication (which only scales reads) is no longer enough.
When to Use It
Genuinely large systems: multi-tenant SaaS sharded by tenant, social platforms sharded by user, telemetry sharded by time. Distributed databases (DynamoDB, Cassandra, Vitess) shard internally — using one is usually wiser than hand-rolling shards over PostgreSQL.
Trade-offs
Everything crossing shard boundaries gets hard: cross-shard queries need scatter-gather, transactions lose atomicity, unique constraints weaken. A skewed key creates hot shards (one celebrity user melting node 7), and re-sharding live data is among the most feared migrations in engineering.
Implementation
Choose the key by access pattern so the vast majority of queries touch exactly one shard, and verify its distribution against real data. Use consistent hashing or a directory service for placement, route in a data-access layer that hides topology from application code, and plan the re-shard story (virtual shards, split-friendly design) before you need it.