Notes on Idempotency

Every outage post-mortem I have ever written has the same villain: a retry that was not safe to retry. A payment fires twice. A webhook double-processes. A queue redelivers a message the consumer already handled, and the consumer handles it again with enthusiasm.

The fix is rarely cleverness. It is making the operation idempotent, so that retrying it is a no-op instead of a hazard.

The shape of the fix

An idempotency key is just a promise from the caller: this request and the one I sent nine seconds ago are the same request, and you should only honour one of them. The server’s side of the bargain is to remember what it answered.

async function handlePayment(req, res) {
  const key = req.headers['idempotency-key'];
  const seen = await store.get(key);
  if (seen) return res.json(seen);

  const result = await charge(req.body);
  await store.set(key, result, { ttl: '24h' });
  return res.json(result);
}

That costs one extra lookup per request. In exchange, every client, proxy and load balancer between you and the caller is allowed to be a little bit sloppy about delivery — which, in practice, they always are.

Where it gets subtle

The naive version above has a race: two concurrent requests with the same key both miss the cache, and both charge. The store needs to be the thing that serialises them, not an afterthought.

create table idempotency (
  key         text primary key,
  response    jsonb,
  created_at  timestamptz not null default now()
);

-- The insert is the lock. Whoever wins goes on to charge; the loser waits
-- and reads the winner's response.
insert into idempotency (key) values ($1)
on conflict (key) do nothing
returning key;

If the insert returns no row, someone else is already working. The correct behaviour then is to poll briefly and return their answer — not to charge, and not to error.

Write it while you still might be wrong. That is when it is useful to someone else who is wrong the same way.

The part nobody writes down

Idempotency keys need a retention policy. Keep them forever and the table becomes the largest thing in your database; expire them too eagerly and a client retrying after a long network partition gets charged twice, which is the exact failure you built this to prevent.

Twenty-four hours is the number I keep landing on. It is longer than any sane client retry window and short enough that the table stays small. It is not a principled answer. It is just one that has not bitten me yet.