Duplicate Messages
Overview
In a system with at-least-once delivery, the same message will arrive twice. It is not a hypothesis — it is a statistical certainty over time.
What the architecture decides is not whether the duplication happens. It is whether it produces an effect.
Problem
Duplication has several origins, and none of them is avoidable:
Producer retry. The acknowledgment was lost; it resends.
Broker redelivery. The consumer did not acknowledge in time — because it was slow, not because it failed.
Rebalancing. Consumers swap partitions and reprocess in-flight messages.
Deliberate reprocessing. Someone repositions the read to fix a defect, and already-processed messages come back.
The last is frequently forgotten in the design and is the most common in operations: every defect fix in a consumer implies reprocessing.
Core Concepts
Two strategies
Operation idempotency. The effect is the same whether executed once or N times. See idempotency. It is the most robust solution, because it does not depend on detecting the duplicate.
Explicit deduplication. Recording already-processed identifiers and discarding repetitions.
The first is preferable where possible — it works even with a lost identifier or an expired window. The second is necessary when the effect cannot be made idempotent.
In practice, mature systems use both: idempotency where they can, deduplication as a safety net.
Message identifier versus business identifier
A detail that decides the correctness of the deduplication.
The message identifier, generated by the broker, changes when the producer resends — because it is a new message, with the same content. Deduplicating by it does not detect producer duplication.
The business identifier — the order, the transaction, the operation — identifies the intent and is stable across resends.
Deduplicate by the business identifier. It is the same principle as the idempotency key: the key identifies the intent, not the attempt.
Persisted or windowed
| Persisted key | Cached window | |
|---|---|---|
| Guarantee | Complete | Within the window |
| Cost | A write per message | Memory |
| High volume | Expensive | Viable |
| Old reprocessing | Detected | Not detected |
The window is suitable for high volume where realistic duplication happens within seconds or minutes. The persisted key is necessary when reprocessing can be from days ago.
The common error is using a window and forgetting that deliberate reprocessing exceeds any reasonable window.
The check and the effect have to be atomic
If the consumer checks the key, processes, and then records — there is a window in which two consumers check at the same time and both process.
The correct form is for the key's insertion to be part of the same transaction as the effect, with a uniqueness constraint doing the work:
BEGIN
INSERT INTO processed (key) VALUES (:key) -- fails if it already exists
... apply the effect ...
COMMIT
The uniqueness violation indicates a duplicate, and the whole transaction is discarded. No race window.
Deduplication does not solve everything
If the effect leaves the system — a call to an external service — local deduplication does not prevent the call from happening twice if the process dies between the call and the acknowledgment.
There the idempotency has to be on the other side, with a key sent in the call.
The window size comes from real behavior
Choosing the deduplication window by intuition — "an hour seems reasonable" — is how the mechanism fails in production.
The window has to cover the largest possible interval between two deliveries of the same message, which is the sum of three measurable things:
window ≥ the broker's maximum redelivery delay
+ the consumer's maximum delay
+ margin for operational reprocessing
The third term is the most surprising. If operations include reprocessing a period after an incident, the window has to cover that whole period — and then it stops being a window and becomes persistent deduplication.
The rule of thumb: if the system has any reprocessing procedure, the window does not serve.
Mental Model
The message will arrive twice. The question is whether the world notices.
When to Use
- Every message consumer, without exception.
- Every operation with an external side effect.
- Especially where the effect is irreversible — a charge, a shipment, issuing a document.
When Not to Use
When the operation is naturally idempotent. Setting an absolute value, marking a state. It is worth checking whether it still is — natural idempotency breaks over time.
When the duplicated effect is harmless and cheap. A duplicated log line. It is worth recognizing explicitly, not assuming.
Windowed deduplication when reprocessing is common. The window does not cover it.
Deduplication by the message identifier. It does not detect producer resends.
Alternatives
- Operation idempotency — the preferable one.
- A uniqueness constraint in the database — the storage itself rejects the duplicate, with no deduplication code.
- Commutative and absolute operations — reformulating from "add" to "set".
- Reconciliation — accept and fix later, when real-time detection is expensive.
Trade-offs
| Deduplicate | Do not deduplicate |
|---|---|
| Single effect guaranteed | Duplication possible |
| An additional write per message | None |
| Key storage to maintain | Nothing |
| Slightly higher latency | Lower |
| Persisted | Window |
|---|---|
| Complete guarantee | Within the window |
| Cost per message | Memory cost |
| Detects old reprocessing | Does not |
Failure Modes
Non-atomic deduplication. Two consumers process simultaneously.
Wrong key. The message identifier instead of the business one.
A short window. Reprocessing exceeds it.
Keys with no expiry. A leak in storage.
An unprotected external effect. Local deduplication does not cover the call that already went out.
Common Mistakes
Assuming the broker solves it. The duplicate that matters is born from the producer's retry and from resends after a consumer failure — the two points outside the broker's reach.
Deduplicating by the message identifier. A producer that resends generates a new identifier for the same business intent. The key has to come from the domain — the order's identifier, not the envelope's.
Checking and processing in separate transactions. Between the "have I processed this?" query and the write there is a window in which the second copy also queries and also gets no. Two concurrent copies pass the check.
Not considering deliberate reprocessing when sizing the window. A 24-hour deduplication window blocks the seven-day reprocessing a defect fix requires — or lets it duplicate everything.
Not expiring the keys. The deduplication store grows without limit and becomes, itself, the bottleneck of the consumer it was supposed to protect.
Real-World Example
A sales event consumer recorded salespeople's commissions.
The deduplication used the message identifier, kept in a cache with a one-hour window.
Two incidents.
The first was a defect in the commission calculation for one product type. With the code fixed, the team repositioned the read to reprocess the month's sales. The messages had new identifiers — they were new deliveries — and the one-hour window was irrelevant for events from weeks earlier.
All the month's commissions were posted again. The salespeople received double, and the reversal generated a conversation with employment counsel.
The second was subtler. Under load, two consumers processed the same message: one checked the cache, and before writing the key, the other checked too. Both passed.
The redesign changed both decisions.
A business key — the sale's identifier — instead of the message identifier. Reprocessing the same sale came to be detected regardless of how many times the message is redelivered or resent.
Persistence with uniqueness, in the same transaction as the commission posting. The key's insertion and the posting happen together or neither happens. The race stopped being possible.
And the keys got a 90-day expiry — a period longer than any plausible reprocessing, and short enough not to accumulate indefinitely.
The next reprocessing, six months later, ran with no incident: the already-commissioned sales were discarded silently, and only the new ones were posted.
Related Concepts
- Idempotency — the preferable strategy.
- Delivery Guarantees — why duplication is certain.
- Ordering — the sibling problem.
- Poison Messages.
Practical Exercise
For each consumer in your system, answer: how does it detect a duplicate? By the business key or the message key? Are the check and the effect atomic?
Then ask the operational question: if you had to reprocess the last month, what would happen?
Interview Questions
- Why is deduplicating by the message identifier insufficient?
- Why do the check and the effect have to be atomic?
- When is a cached window sufficient and when is it not?
Further Reading
- Helland, Pat. Idempotence Is Not a Medical Condition. ACM Queue, 2012.
- Kleppmann, Martin. Designing Data-Intensive Applications. O'Reilly, 2017.