A retry is not a second try

The bug did not look like a retry problem

An integration token expires. A worker notices it while syncing. At almost the same moment, a webhook job and a scheduled job notice it too. Each job is doing the reasonable thing: refresh the token, save the new value, and continue.

That is exactly how a small race becomes a production problem. One refresh can invalidate another. One worker can save an older value after another worker has already saved the new one. The jobs do not know they are collaborating, so they behave like rivals.

Retries need an owner

My first instinct with background work is usually to ask about backoff, retry count, and dead-letter queues. Those matter. They were not the first question here.

The first question was: which worker owns the right to refresh this credential? Once that was clear, the shape became simpler. One worker acquires a short-lived lock. It refreshes and stores the credential. The others wait briefly, then read the result instead of starting another refresh.

flowchart LR
  A[Sync worker] --> L{Refresh lock}
  B[Webhook worker] --> L
  C[Scheduled worker] --> L
  L -->|one owner| R[Refresh and persist]
  R --> T[Updated credential]
  L -->|other workers| T

The lock is only half the design

A lock without a timeout can turn one crashed worker into a permanent outage. A lock with no re-read path just moves the race somewhere else. And a refreshed credential should only replace the current value when the write is still valid for that connection.

The practical pattern is small: bounded lock, refresh once, persist deliberately, release, and let waiters load the resulting state. If the owner fails, the next job can try after the lease ends. No worker has to guess whether another one succeeded.

Why this matters beyond OAuth

The same shape appears everywhere. A payment callback, a webhook delivery, a scheduled sync, an AI action, and a manual retry can all try to update the same record. Retrying the HTTP request is the easy part. Deciding whether the state transition is still safe is the engineering work.

That is why I like queues, locks, idempotency keys, and version checks together. None of them is a magic reliability feature. They are ways to make the system answer one question clearly: who gets to make this change now?

The lesson I kept

A retry is not a second try. It is another actor arriving later, with partial knowledge of what happened before. Treat it that way and the design gets calmer: make ownership explicit, make state transitions conditional, and give failed work a safe path back in.