Cache — Remembering Expensive Answers

What is it? #

A cache stores the result of expensive work so the next request can reuse it instead of repeating it.

Expensive means slow or costly: a complex query, an external API call, a rendered page, a computed report.

Caches exist at many levels — browser, CDN, application memory, a shared store like Redis, and inside the database itself. Each one removes work from everything behind it.

The difficulty is never storing the value. It is knowing when the stored value stops being true, and what to do about it.

Think of it like this #

Keeping your most-used tools on the bench rather than fetching them from the shed each time.

It works brilliantly until someone replaces the tool in the shed and you carry on using the old one on the bench. That is stale cache, and it is the entire problem in one sentence.

Simple example #

A dashboard runs a query taking 800 milliseconds and is loaded by every user. Caching the result for sixty seconds means one user pays the cost and everyone else gets an instant response.

Code #

import json
import time
import redis

cache = redis.Redis(host="localhost", port=6379, decode_responses=True)


# --- Cache-aside: the most common pattern ---
def get_dashboard(user_id: int) -> dict:
    key = f"dashboard:v2:{user_id}"          # version in the key, see below

    cached = cache.get(key)
    if cached is not None:
        return json.loads(cached)            # hit

    data = run_expensive_query(user_id)      # miss: do the work
    cache.setex(key, 60, json.dumps(data))   # store with a 60 second TTL
    return data


# --- Write-through: update the cache when the data changes ---
def update_profile(user_id: int, fields: dict) -> None:
    save_to_database(user_id, fields)
    cache.delete(f"profile:{user_id}")       # simplest correct option: remove it


# --- Stampede protection: stop 500 requests recomputing at once ---
def get_with_lock(key: str, ttl: int, compute):
    cached = cache.get(key)
    if cached is not None:
        return json.loads(cached)

    lock_key = f"lock:{key}"
    if cache.set(lock_key, "1", nx=True, ex=10):     # only one winner
        try:
            value = compute()
            cache.setex(key, ttl, json.dumps(value))
            return value
        finally:
            cache.delete(lock_key)

    time.sleep(0.1)                                   # others wait briefly
    cached = cache.get(key)
    return json.loads(cached) if cached else compute()


# --- In-process cache for small, hot, rarely-changing data ---
from functools import lru_cache

@lru_cache(maxsize=512)
def country_name(code: str) -> str:
    return lookup_country(code)

Strategies

cache-aside      app checks cache, computes on miss, stores  — most common
write-through    write to cache and database together        — consistent, slower writes
write-behind     write to cache, flush to database later     — fast, risk of loss
refresh-ahead    refresh before expiry                        — avoids user-visible misses

Eviction when full
LRU   drop least recently used   — good default
LFU   drop least frequently used — good for stable hot sets
TTL   drop on expiry             — combine with the above

How it works #

Cache-aside is the pattern you will use most. Check, compute on miss, store. It is simple and the cache never needs to know how the value is produced.

setex stores with an expiry. A TTL is the cheapest form of invalidation: the value is wrong for at most sixty seconds, and then it corrects itself. For many read-heavy features that is an acceptable trade.

Deleting on write is the simplest correct invalidation. Updating the cache with the new value is tempting but introduces races where two writers store values in the wrong order.

The version in the key — dashboard:v2: — is a practical trick. When the shape of the cached data changes, bump the version and every old entry is ignored instantly, with no purge needed.

The lock function addresses the cache stampede. When a popular key expires, hundreds of requests can miss simultaneously and all run the expensive query. The lock lets one do the work while the others wait briefly.

lru_cache is an in-process cache: extremely fast, but each server has its own copy, and it cannot be invalidated across servers. Use it only for small, stable data such as lookup tables.

The eviction policy matters when memory fills. LRU is a sensible default; without a policy, a full Redis instance starts rejecting writes.

Real-world use #

Caching is the single most effective performance tool in most applications. Sessions, rendered fragments, API responses, permission lookups, feature flags and rate limit counters typically live in Redis or a similar store.

What not to cache matters as much. Anything user-specific must be keyed by user; anything security-sensitive needs careful thought; anything that must be immediately correct — account balances, stock levels at checkout — should be read fresh.

Cache-related bugs share a signature: the problem disappears after a restart or reappears only for some users. That pattern almost always points at a stale or wrongly-keyed entry.

Measure hit rate. A cache with a 20% hit rate is adding complexity and latency for little gain, and the fix is usually the key design or the TTL rather than the cache itself.

Common mistakes #

• Caching user-specific data under a shared key, leaking one user’s data to another.
• No TTL and no invalidation, so stale values live forever.
• Updating the cache on write instead of deleting, creating ordering races.
• No eviction policy, so the cache fills and starts rejecting writes.
• Caching values that must be exactly correct, such as stock at the moment of purchase.

Practice #

Add cache-aside caching to a slow function with a 60-second TTL and a versioned key. Measure the time for a miss and a hit. Then add invalidation on update, and finally add the lock to prevent a stampede and test it with concurrent calls.