Idempotency is a product decision, not an engineering one

An idempotency key is a small thing. A string the client sends with a request so that if the request is sent twice, the server does the thing once. Engineers treat it as hygiene, like input validation. It isn't. It is the moment your product decides who pays when the network lies.

Here is the situation it exists for. A customer taps "pay." The request reaches your server, the ledger entries commit, the response is on its way back, and the connection drops. The customer's phone sees a timeout. It does not know whether the payment happened. Neither does the customer. Something has to happen next, and every option is a policy about uncertainty.

Option one: the client retries and the server has no idempotency. The customer is charged twice. The cost of uncertainty lands on the customer, who now has to notice, complain, and wait for a refund. This is the default state of most early payments products, and it is a decision even when nobody made it.

Option two: the client does not retry. The payment may or may not have happened. The customer is told "something went wrong, please check your balance." The cost lands on the customer again, this time as anxiety and a support ticket instead of a double charge.

Option three: the client retries with a key, and the server honours it. The second request returns the result of the first. The customer sees one charge and a clear confirmation. The cost of uncertainty is absorbed by you, in the form of a key store, a lookup on every write, and a rule about how long keys live.

Notice that only option three costs the operator anything. The other two are free to build, which is why they get built. But "free to build" means the price was paid somewhere else, by someone who didn't agree to it.

This is why the retry policy is a product decision. It answers a question no engineer can answer alone: when the system doesn't know what happened, whose problem is that? A bank that double-charges and refunds on request has decided its customers' time is cheaper than its engineering hours. A wallet that makes retries safe has decided the opposite.

The decision spreads further than the endpoint. Once retries are safe, the client can be aggressive on flaky mobile networks, which is most networks where the product is used. Once they are not, the client must be timid, and timid clients produce "pending" states, and pending states produce the invisible-balance bugs that generate the support tickets that the idempotency key would have prevented.

If I were reviewing a payments product for the first time, I would not ask "do you have idempotency keys?" I would ask "when a request times out, who eats the loss?" The keys are just the implementation of the honest answer.