31. Design a globally distributed key-value store.
Use the Google system-design prompt and keep the scope to core requirements, storage topology, consistency choices, and how the service behaves across regions.
At a high level, the goal is to store key-value data safely across several regions. The main challenge is keeping local reads and writes fast while handling failures correctly. I would explain the global routing path, the regional storage path, and cross-region replication. Requests go through the Edge & Access Layer and Smart Router to stateless .NET services. Each shard has a primary, replicas, quorum operations, and a WAL. Cross-region replication is asynchronous, so regional failover can expose stale data or lose recent unreplicated writes.
The goal is to store a value under a key and find that value from different parts of the world. The service should stay available when a server, storage node, or region has trouble. The hard part is keeping normal requests fast without pretending every region always has the newest copy. The diagram solves this by routing requests to regional .NET services, splitting data into shards, keeping several copies of each shard, and copying changes between regions in the background. It also lets each operation choose how strict its read and write rules should be.
- Which operations need strong consistency within the leader region?
- Can some reads accept slightly old data from another region?
- Can we accept losing recent writes that were not copied before a full regional failure?
- Should quorum read and write settings be configurable per operation?
I would start with the global request path. A client sends a request to the Edge & Access Layer. Global DNS is geo-aware, and the Anycast / CDN Edge helps send traffic toward a suitable region.
This layer also shows DDoS protection, a WAF, authentication, authorization, rate limiting, and validation. The Global Routing & Request Coordinator then handles the routing decision. Its Smart Router uses the partition lookup, leader lookup, latency information, consistency policy, and retry policy before choosing the regional Data Plane.
Each region contains stateless .NET 8/10 API services. Stateless means the API process does not own the durable key-value data. More API nodes can therefore be added horizontally when traffic grows.
The regional API layer can also use a Local Cache. The diagram marks this cache as optional, per API node, and write-through. It is a speed layer only. The durable data still belongs to the Storage Nodes.
The Data Plane sends reads and writes to the Storage Nodes for the selected shard. Each shard group contains one Primary and two Replicas. The three copies are placed across failure domains.
The storage engine uses an LSM-tree design with SSTables on NVMe. Writes also use a Write-Ahead Log, or WAL, on the storage node. The WAL records a durable write before normal storage processing completes.
Quorum writes and reads are configurable per operation. The diagram supports strong consistency within the leader region when the required quorum is used. It also shows bounded staleness and an optional weaker mode for operations that can accept older data.
The Control Plane is globally replicated. It contains the Metadata Service, Config & Feature Flags, Cluster Manager, Placement & Failover Controller, and a Raft-based Consensus Layer across regions. These components keep the sharding map, replica map, membership, placement, and configuration coordinated.
Cross-region data replication is asynchronous. That means a remote region may receive a change slightly later. The Change Stream is the replication pipeline. The Dead Letter Queue holds replication failures that need later handling.
If a storage node fails, another replica can take over. Raft supports automatic leader election. If an entire region becomes unhealthy, traffic can be routed to another healthy region.
The important limitation is that asynchronous cross-region replicas may lag. After regional failover, a read may therefore return stale data. The system may also lose the newest writes that had not reached another region yet.
Backup & Snapshot, Compaction, Audit & Logging, and Metrics & Tracing support operations. Security includes TLS in transit, encryption at rest, RBAC / IAM, and audit logs. The main trade-off is clear: stronger consistency needs more coordination and usually increases latency.
The benefit is that users can usually reach a nearby region, which keeps normal requests fast. Each shard also has a Primary and two Replicas, so one storage-node failure does not stop the shard. Quorum writes and the WAL protect durable writes inside the leader region. The downside is cross-region replication happens in the background. A remote region may therefore be slightly behind. If the leader region fails before its newest writes are copied, failover can show old data or lose those recent writes. Stronger consistency can reduce that risk, but it needs more coordination and usually adds latency. The system also has more operational work because routing, replicas, backups, monitoring, and failover all need care.
Interviewers use this question to see how you break a global storage problem into understandable parts. They want to see how you think about routing, sharding, replicas, consistency choices, and failures. They also want to know whether you can explain trade-offs instead of promising perfect availability or zero data loss. A good answer shows practical judgment about what must be fast, what must be durable, and what can be slightly delayed.








