Quick Summary — TL;DR
Exactly-once processing means every message in a system is processed one time and only one time. No messages are lost, and no messages cause duplicate side effects. It's the ideal guarantee for any system that handles payments, state changes, or business-critical events.
Here's the hard truth: exactly-once delivery over a network is theoretically impossible. The reason is the Two Generals Problem — a fundamental result in distributed computing.
Imagine a sender delivers a message and the receiver processes it successfully. The receiver sends an acknowledgment back. But the sender has no way to know if the ack was lost in transit. If it was, the sender retries, and now the receiver has the message twice. If the sender doesn't retry, and the original message was the one that was lost, the receiver has it zero times.
You can add more acks (ack the ack), but each additional step has the same problem. At some point, a network message is in flight with no confirmation, and either side must make a decision without certainty. This is not a bug in any particular system — it's a mathematical impossibility when communicating over an unreliable channel.
When a system claims "exactly-once semantics," what it actually provides is:
At-least-once delivery + idempotent processing = exactly-once semantics
The transport layer guarantees every message is delivered at least once (retrying as needed). The application layer ensures that processing the same message multiple times has the same effect as processing it once. The result looks like exactly-once from the outside, even though duplicates may have been delivered and silently discarded.
Assign a unique key to each message (a UUID, event ID, or hash of the payload). Before processing, check a deduplication store. If the key exists, skip processing and return the cached result. If not, process the message and store the key. This is the pattern Stripe uses for payment idempotency.
Maintain a database table of processed message IDs. Before any side effect, check the table in the same transaction as the business logic. This prevents both duplicates and race conditions when two copies of the same message arrive simultaneously.
When a service needs to update its database and publish an event, write both the database change and the outbound event to the same database in a single transaction. A separate process reads the outbox table and publishes the events. This ensures the event is published if and only if the database change is committed — no orphaned events, no lost events.
For state changes, use INSERT ... ON CONFLICT UPDATE keyed on a natural identifier. Running the same upsert twice produces the same final state. This is the simplest form of idempotent processing for database writes.
Apache Kafka is one of the few systems that explicitly advertises exactly-once semantics (EOS). It achieves this through a combination of:
Even Kafka's EOS is ultimately at-least-once delivery with built-in deduplication. The "exactly-once" is at the processing semantics level, not the network delivery level.
Don't try to build a transport layer that never sends duplicates — it's impossible. Instead, accept that duplicates will happen and make them harmless:
No. Exactly-once delivery over an unreliable network is mathematically impossible (Two Generals Problem). What systems achieve is exactly-once processing — at-least-once delivery combined with idempotent consumers that discard duplicates. The end result is the same: every message is processed once and only once.
Use at-least-once delivery (retries until acknowledged) and make your consumers idempotent. Before processing each message, check a deduplication store using the message's unique ID. If it's already been processed, skip it. Wrap the check and your business logic in a database transaction to prevent race conditions.
The Two Generals Problem is a thought experiment that proves two parties cannot reach agreement over an unreliable communication channel with guaranteed certainty. In messaging terms: a sender can never be 100% sure the receiver got the message, because the acknowledgment itself can be lost. This is why at-least-once (with retries) or at-most-once (without retries) are the only achievable delivery guarantees.
Exactly-once processing is built on at-least-once delivery — the transport retries to prevent data loss — combined with idempotent consumers that make duplicates harmless. It's the opposite of at-most-once delivery, which avoids duplicates by accepting data loss. In practice, every reliable webhook handler and background job processor should implement exactly-once semantics.
Recuro handles cron scheduling, retries, alerts, and execution logs -- so you can focus on building your product.
No credit card required