A Small Caching Layer for a Slow API

A dashboard I maintain calls a third-party API for exchange rates on every page load. The endpoint is slow — 800 to 900 milliseconds — and the data changes maybe once an hour. There was no reason to pay that latency tax on every request.

const TTL_SECONDS = 3600;

export async function getRates() {
  const cached = await redis.get('rates:latest');
  if (cached) return JSON.parse(cached);

  const fresh = await fetchRatesFromUpstream();
  await redis.set('rates:latest', JSON.stringify(fresh), 'EX', TTL_SECONDS);
  return fresh;
}

p95 on that route dropped from 910ms to 4ms. The cache miss — once an hour, when the TTL expires — still costs the full round trip, but every other request rides on data that is at most sixty minutes stale, which the dashboard’s users have never once noticed.

The miss is not free

The version above has a stampede problem. When the key expires, every concurrent request misses at once and they all call upstream together. At low traffic you will never see it. At the moment you most want the cache working, you will.

A single-flight lock fixes it:

export async function getRates() {
  const cached = await redis.get('rates:latest');
  if (cached) return JSON.parse(cached);

  // Whoever wins the lock refreshes; everyone else serves the stale copy
  // rather than piling onto an already-slow upstream.
  const won = await redis.set('rates:lock', '1', 'NX', 'EX', 30);
  if (!won) {
    const stale = await redis.get('rates:stale');
    if (stale) return JSON.parse(stale);
  }

  const fresh = await fetchRatesFromUpstream();
  await redis.set('rates:latest', JSON.stringify(fresh), 'EX', TTL_SECONDS);
  await redis.set('rates:stale', JSON.stringify(fresh));
  return fresh;
}

The second key has no expiry. It exists purely so there is always something to serve while the refresh is in flight. Serving data that is seventy minutes old beats serving a spinner for nine hundred milliseconds, and it beats hammering an upstream that is already struggling.

When not to bother

If the upstream is fast, caching adds a failure mode and buys nothing. If the data must be current to the second, a cache is a correctness bug wearing a performance costume.

This one was worth it because the numbers were lopsided: a slow call, data that changes hourly, and users who genuinely cannot tell. That combination is rarer than the number of caches in the average codebase would suggest.