Your webhook endpoint went down. Where did the deposit go?

Here is the scenario every operator eventually lives through. A player sends 500 USDT. The chain confirms it. The gateway calls your backend to say “credit this player” — and your backend is mid-deploy, or the database is failing over, or a firewall rule went out wrong ten minutes ago. The call fails.

The money is real and it has arrived. The question is what happens next, and the answer is a property of the gateway, not of your luck.

The default answer is worse than you think

The naive implementation — and it is extremely common, including in the first version of Coinflux — is a single HTTP POST wrapped in a try/catch. If it throws, log the error and move on. The deposit is marked credited in the gateway’s ledger, so from its point of view the job is done. Your player’s balance never moves.

What makes this failure mode nasty is that it is silent on both sides. You do not know a notification was attempted, because nothing durable recorded the attempt. The gateway does not know anything is wrong, because it did what it was told. The first signal is a player in your live chat asking where their deposit went, and by then you are reconciling by hand against a block explorer.

What “at least once” actually requires

Reliable delivery is not a retry loop bolted onto the send call. It needs three things, and skipping any of them puts you back where you started.

First, the notification has to be written down before it is sent. If it lives only in memory, a process restart between the credit and the send erases it. Coinflux writes a row — event, payload, target URL, status — and a separate worker delivers from that queue. A crash loses nothing because the intent was persisted before the attempt.

Second, retries need a schedule that survives a real outage, not a blip. Three tries over thirty seconds covers a dropped packet and nothing else. Coinflux retries eight times on a widening backoff — 30 seconds, 2 minutes, 10 minutes, 30 minutes, then 2, 6, 12 and 24 hours. That spans an overnight incident without hammering an endpoint that is already struggling.

Third, there has to be a floor. After the last attempt the delivery is not deleted and not retried forever; it is parked in a dead-letter state that a human can see. In the dashboard that is a list of undelivered events with the last HTTP status and error against each one, and a button that puts a delivery back in the queue against your current URL. An operator who fixes a wrong webhook URL can replay everything that failed while it was wrong.

Retries create a new problem: duplicates

Once you retry, your backend will eventually receive the same notification twice — the classic case being a request that timed out on our side but succeeded on yours. If your handler naively adds to a player balance, retries have converted a lost deposit into a double credit. That is not an improvement.

This is why every Coinflux attempt carries the same X-Coinflux-Delivery id in its headers. The id belongs to the delivery, not the attempt: retry number four carries the id that attempt number one carried. Store it, and “have I seen this id before?” is a single indexed lookup that makes your handler idempotent. The signature header is unchanged and still HMAC-SHA256 over the raw body, so nothing about verification changes.

On our side of the line, double-crediting is prevented structurally rather than carefully: every credit carries a unique key built from the network, transaction hash and receiving address, enforced by a database UNIQUE constraint inside a transaction. Reprocessing the same on-chain transfer writes nothing the second time — not because the code checked first, but because the database will not allow the row.

What to ask your current provider

You do not need to take anyone’s architecture on faith. Three questions separate a durable pipeline from a hopeful one. Can I see a list of notifications you failed to deliver to me? Can I replay one? Does every retry of the same event carry a stable identifier I can deduplicate on?

If the answer to the first is “check your logs”, the money is already relying on your reflexes. A payment system’s quality shows up on the day something breaks, and something always breaks.