Dynamo was introduced in a 2007 paper by Amazon engineers, and remains one of the most influential papers in distributed systems. Much of what we now take for granted — consistent hashing, eventual consistency, sloppy quorums — was either introduced or popularised here. This article covers the technical mechanics. For why the paper is worth reading, see the Reads companion.
What Dynamo Is
A key-value store with a minimal interface: get(key) and put(key, value). No joins, no secondary indexes, no cross-key transactions. Built for applications that store relatively small objects (typically under 1MB) and need storage that is always writable.
It’s an AP system in CAP terms. See CAP Theorem if you need the background.
Design Principles
Four principles underpin every decision in Dynamo:
| Principle | What it means |
|---|---|
| Incremental scalability | Scale out one node at a time with minimal impact to the system |
| Symmetry | Every node has the same responsibilities — no special roles |
| Decentralisation | Peer-to-peer rather than centralised control |
| Heterogeneity | Workloads are distributed proportionally to each node’s capacity |
Conflict Resolution: When and Who
For systems prone to server and network failures, Dynamo uses optimistic replication — changes propagate to replicas in the background while concurrent work continues. This introduces conflicts that must be resolved.
When: conflict resolution happens on read, not write. Writes must never be rejected — that’s the whole point. Conflicts surface when a client reads divergent versions.
Who: by default, the application handles reconciliation. The application understands the business context better than the data store does. The shopping cart is the paper’s example — merging two diverged carts makes more sense than discarding one. For simpler cases, Dynamo can be configured to use “last write wins” instead.
Partitioning: Consistent Hashing + Vnodes
Keys are hashed onto a ring. A key is assigned to the first node clockwise from its hash position. Each node owns the keys between itself and its Nth predecessor — those nodes form the key’s preference list.
Standard consistent hashing has two problems: random node placement leads to non-uniform distribution, and it doesn’t account for node heterogeneity. Dynamo solves both with virtual nodes (vnodes).
Each physical node is assigned multiple positions on the ring — the number proportional to its capacity. A powerful node can hold more vnodes and absorb more load.
Physical nodes: A (high capacity), B, C
Ring positions: A1, C1, B1, A2, A3, B2, C2, A4 ...
↑ Node A holds more positions due to higher capacity
hash("cart:user_99") → position → first vnode clockwise → Node BBenefits of vnodes: even key distribution, heterogeneity-aware load balancing, and when a node fails its load spreads across many neighbours rather than one successor.
See Partitioning for the full consistent hashing deep dive.
Routing: Zero-Hop DHT
To minimize latency variance, Dynamo uses a zero-hop DHT approach. Each node maintains enough routing information locally to forward a request directly to the correct node — no intermediate hops.
Multi-hop routing increases latency variance, which is unacceptable under a 99.9th percentile SLA. One hop, every time.
Replication: N/W/R Quorums
Every key is replicated to N nodes — its preference list. Read and write operations involve a coordinator node, which is typically the first node in the preference list. If that node is unavailable, selection moves down the list.
Reads and writes are quorum-controlled:
- W — nodes that must acknowledge a write
- R — nodes that must respond to a read
The consistency invariant: W + R > N guarantees at least one node in the read set saw the latest write. Latency is dictated by the slowest R or W replica.
N=3, W=2, R=2 → W + R = 4 > 3 ✓| W | R | Trade-off |
|---|---|---|
| 1 | 3 | Fast writes, fresher reads |
| 3 | 1 | Slow writes (all replicas confirm), fast reads |
| 1 | 1 | Maximum availability, weakest consistency |
Amazon’s default: W=2, R=2, N=3.
Sloppy Quorum and Hinted Handoff
Standard quorum fails when target nodes are down. Dynamo’s solution: sloppy quorum — read and write operations use the first N healthy nodes from the preference list, not necessarily the intended N.
The substitute node stores the write with a hint pointing to the intended recipient. When that node recovers, the substitute delivers the write. This is hinted handoff.
Node C is down. Key belongs to Node C.
→ Write goes to Node D with hint: "deliver to C on recovery"
→ Node C recovers
→ Node D hands off, deletes its copyData Versioning: Vector Clocks
Modifications to data are treated as new immutable versions. When a client reads a stale version and makes changes, those changes are reconciled later — not blocked at write time.
Dynamo tracks versions using vector clocks: a list of (node, counter) pairs. To update an object, the client must specify the version it’s updating. Each write increments the writing node’s counter.
Write on Node A: value="[Shoes]" clock=[(A,1)]
Update on Node A: value="[Shoes, Hat]" clock=[(A,2)]
Concurrent write on B: value="[Shoes, Jacket]" clock=[(A,1),(B,1)]When the partition heals, neither clock dominates — they’re concurrent. Dynamo returns both versions to the client. The application merges them. For the shopping cart, that means both carts are merged — deleted items may resurface, but no additions are silently lost.
Replica Repair: Anti-Entropy with Merkle Trees
Hinted handoff handles short outages. For longer divergence, Dynamo runs background repair using Merkle trees — hash trees where each leaf is a key-value hash and each parent is a hash of its children.
To compare two replicas: compare root hashes. If they match, replicas are identical. If not, walk the tree to find the diverging subtree — only transfer the differing keys.
[root]
/ \
[A–M] [N–Z] ← hashes differ here
/ \ / \
[A–F] [G–M] [N–S] [T–Z] ← narrow down to the diverging rangeMembership and Failure Detection: Gossip
Nodes discover each other and propagate cluster state via a gossip protocol. Each node periodically picks a random peer and exchanges membership information. No central coordinator — consistent with the symmetry and decentralisation principles.
The Full Picture
| Scenario | Mechanism |
|---|---|
| Distributing keys across nodes | Consistent hashing + vnodes |
| Tolerating node failure during reads/writes | Sloppy quorum |
| Recovering writes made to substitute nodes | Hinted handoff |
| Detecting and resolving concurrent writes | Vector clocks |
| Repairing diverged replicas | Anti-entropy + Merkle trees |
| Cluster membership and failure detection | Gossip protocol |
Dynamo vs. DynamoDB
Dynamo is Amazon-internal. DynamoDB is its managed AWS descendant — same principles, significantly different in security and infrastructure:
- Storage separated from execution — data is validated and cryptographically signed
- mTLS encryption authenticates internal packets between nodes
- Nodes run specialised Linux VMs with zero persistent terminal access
- Transparent encryption-at-rest via AWS KMS — nodes require valid keys to read data
When to Use It
Good fit:
- Primary access by a single key
- Always-writable requirement — availability over correctness
- Briefly stale reads acceptable
- Small values (under 1MB), no complex querying
- Business logic that can handle conflict reconciliation
Poor fit:
- Need joins, secondary indexes, or cross-key transactions
- Consistency is non-negotiable — use a CP system
- Adversarial or untrusted environments — Dynamo has no integrity guarantees
Dynamo sacrifices consistency (linearizability — all nodes agreeing on the same value) No isolation guarantees, single-key updates only. It also provides no data integrity or security guarantees — it’s built for trusted internal environments. For adversarial contexts, look at Byzantine Fault Tolerant (BFT) storage systems.
Born out of Influence — Cassandra
Dynamo’s ideas didn’t stay inside Amazon. Apache Cassandra, built by Facebook engineers in 2008, drew directly from the Dynamo paper — adopting consistent hashing, vnodes, gossip-based membership, and tunable consistency — almost a mirror image but with full open-source benefit. Learn more about Cassandra and how it redefines data-at-scale in my next article.