I’ve sat through a fair share of assessments where you’re asked to design a cache. Funnily enough, what I practiced in interviews took me years to truly appreciate in production. A cache is a barrier — it resolves requests before they ever reach your database. Get it right and you shave latency by orders of magnitude. Get it wrong and you introduce subtle bugs that are infuriating to reproduce.
Cache Strategies
How and when you populate — and invalidate — a cache is a design decision, not a default.
Cache-Aside (Lazy Loading)
The most common pattern. The application is responsible for all cache interaction.
public async Task<Product?> GetProductAsync(int id)
{
if (_cache.TryGetValue($"product:{id}", out Product? cached))
return cached;
var product = await _db.Products.FindAsync(id);
if (product is not null)
_cache.Set($"product:{id}", product, TimeSpan.FromMinutes(5));
return product;
}On a cache miss, the app fetches from the DB and populates the cache. Reads are lazy — data only enters the cache when it’s actually requested.
Pros: Only caches what’s needed; resilient to cache restarts (DB is the source of truth).
Cons: First request always misses; risk of stale data if the DB is updated outside the cache path.
Write-Through
On every write, update the cache and the DB together — synchronously.
public async Task UpdateProductAsync(Product product)
{
await _db.SaveChangesAsync();
_cache.Set($"product:{product.Id}", product, TimeSpan.FromMinutes(5));
}Pros: Cache is always consistent with the DB; no stale reads.
Cons: Every write pays double the latency; cache fills with data that may never be read.
Write-Back (Write-Behind)
Write to the cache only. Flush to the DB asynchronously in the background.
Pros: Write latency is minimal — the caller is done as soon as the cache acknowledges.
Cons: If the cache crashes before flushing, writes are lost. Harder to implement correctly; requires durable intermediate storage (e.g. a queue).
Read-Through
Like cache-aside, but the cache itself fetches from the DB on a miss — the application doesn’t have to. The cache is the single interface; the DB is abstracted away.
Pros: Simpler application code; cache is self-maintaining.
Cons: Less control; not all caching libraries support it natively.
| Strategy | Who populates cache | Write behaviour | Staleness risk | Complexity |
|---|---|---|---|---|
| Cache-aside | Application | Cache not updated on write | Medium | Low |
| Write-through | Application | Cache + DB updated together | Low | Medium |
| Write-back | Application | Cache only; DB async | Low (until crash) | High |
| Read-through | Cache layer | Cache not updated on write | Medium | Low |
What’s Under the Hood
Modern libraries like .NET’s IMemoryCache abstract away the internals. You don’t have to implement anything — but knowing the underlying data structure matters. It tells you the CPU and memory overhead of each eviction policy, and it’s what interviewers ask you to build from scratch.
Eviction Policies
When the cache is full, something has to go. The policy determines what.
FIFO (First In, First Out) — evict the oldest entry regardless of how frequently it’s accessed. Simple to implement; poor hit rate for uneven access patterns.
LIFO (Last In, First Out) — evict the most recently inserted entry. Rarely useful in practice; mostly academic.
LRU (Least Recently Used) — evict the entry that hasn’t been accessed for the longest time. The most common policy for general-purpose caches. High hit rate for workloads with temporal locality (recently accessed data is likely to be accessed again).
Internals: a hashmap for O(1) lookup + a doubly linked list to track access order. On each access, move the node to the head of the list. On eviction, remove from the tail.
public class LruCache<TKey, TValue>(int capacity)
{
private readonly Dictionary<TKey, LinkedListNode<(TKey, TValue)>> _map = new();
private readonly LinkedList<(TKey Key, TValue Value)> _list = new();
public bool TryGet(TKey key, out TValue value)
{
if (_map.TryGetValue(key, out var node))
{
_list.Remove(node);
_list.AddFirst(node);
value = node.Value.Value;
return true;
}
value = default!;
return false;
}
public void Set(TKey key, TValue value)
{
if (_map.TryGetValue(key, out var existing))
_list.Remove(existing);
else if (_map.Count >= capacity)
{
var lru = _list.Last!;
_list.RemoveLast();
_map.Remove(lru.Value.Key);
}
var node = _list.AddFirst((key, value));
_map[key] = node;
}
}LFU (Least Frequently Used) — evict the entry accessed the fewest times. Better than LRU for workloads where popular data stays popular. More expensive to implement (requires frequency counters per key).
TTL (Time-To-Live) — entries expire after a fixed duration, regardless of access frequency. Not strictly an eviction policy — it’s expiry — but it’s how most production caches bound staleness. Often combined with LRU.
In practice, no single policy wins universally. Real workloads have both recency bias (recently accessed data tends to be hot) and frequency patterns (some data is perpetually popular). Modern hybrid policies exploit both signals — keeping a recency window for new entries and a frequency-protected segment for proven hot data. Most production cache implementations use some variant of this rather than a pure LRU or LFU. Modern implementations like W-TinyLFU layer frequency sketches over recency windows to get the best of both.
The Thundering Herd Problem
A hot cache key expires. In the moment between expiry and repopulation, 10,000 concurrent requests all miss the cache and simultaneously fire the same DB query. Your database absorbs a spike it was never designed to handle.
This is the thundering herd (or cache stampede).
Mutex / single-flight — the first request to miss acquires a lock and fetches from the DB. All other requests wait. Once the lock is released, the cache is warm and the rest are served. Simple, but introduces lock contention.
TTL jitter — add random noise to TTL values so entries don’t expire simultaneously.
var ttl = TimeSpan.FromMinutes(5) + TimeSpan.FromSeconds(Random.Shared.Next(0, 60));
_cache.Set(key, value, ttl);Spreads expiry across time. No coordination required.
Probabilistic early expiration — before a key actually expires, start returning a cache miss to a small fraction of requests. Those requests refresh the cache in the background while the majority still gets a cache hit.
Background refresh — a background service proactively refreshes popular keys before they expire. The cache is never cold. Requires knowing which keys are “hot”.
Concurrency and Consistency
Multiple threads reading and writing a cache simultaneously creates race conditions unless handled carefully.
Segmented locking — divide the cache into independent segments, each with its own lock. Reduces contention compared to a single global lock. ConcurrentDictionary<K, V> in .NET uses this approach internally.
Lock-free collections — use CPU-level atomic instructions (compare-and-swap, fetch-and-add) instead of locks. Avoid thread blocking entirely. System.Collections.Concurrent exposes these in .NET.
Cache coherence protocols — at the hardware level, when multiple CPU cores each have a local L1/L2 cache holding a copy of the same memory address, changes on one core must propagate to the others. Modern CPUs handle this via MESI or similar protocols. This is why volatile and Interlocked exist in C# — they signal to the compiler and CPU not to cache stale values in registers.
Hardware-Level Caching
Worth knowing, even if you never configure it. Every CPU has multiple levels of cache:
| Cache | Size (typical) | Latency |
|---|---|---|
| L1 | 32–64 KB | ~1 ns |
| L2 | 256 KB – 1 MB | ~4 ns |
| L3 | 4–32 MB | ~10 ns |
| RAM | GBs | ~100 ns |
This hierarchy is why cache-friendly data structures (contiguous arrays, structs) outperform pointer-chasing ones (linked lists, class heaps) in hot paths. When the CPU prefetcher can predict your access pattern, data arrives before you ask for it. When it can’t, you pay a cache miss penalty on every read.
Distributed Caching
An in-process cache (IMemoryCache) is local to a single instance. In a distributed system with multiple replicas, each instance has its own cache — and they can diverge.
When shared state needs to live globally, use a distributed cache:
Redis — in-memory data structure store. Supports strings, hashes, sorted sets, lists, and more. TTL on every key, pub/sub, atomic operations. The de facto standard for distributed caching. See Redis for the full deep dive.
Memcached — simpler, key-value only. Faster for pure caching workloads but no persistence, no rich data structures, no replication. Mostly outcompeted by Redis unless you need extreme read throughput with minimal overhead.
When Caching Hurts
Caching is a bet. You’re trading consistency risk and operational complexity for latency and throughput. It doesn’t always pay off.
Write-heavy workloads — if data changes faster than it’s read, you’re paying invalidation overhead for little benefit. Every write potentially invalidates a cache entry; at high write rates, the cache is cold more often than warm.
Stale reads are a bug, not a feature — for financial data, inventory counts, or anything requiring strong consistency, a stale cache read is a correctness error. A long TTL that serves a user an outdated account balance isn’t acceptable. Cache carefully or don’t cache at all.
Debugging complexity — a cache layer between your service and DB makes bugs harder to reproduce. “Works in dev, broken in prod” often means the local environment has no cache or a different warm state. Cache misses are nearly impossible to deterministically replicate.
Key Takeaway
The cache strategy you choose is an engineering decision — not a default. Cache-aside is the safe starting point for most read-heavy workloads. Write-through for data where staleness is unacceptable. Write-back only when write latency is the bottleneck and you can tolerate loss risk.
The hard part isn’t caching data — it’s knowing when to invalidate it.
There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton