Missing records between systems: recover webhook and API failures
Record what happened to each event, so staff can tell whether it needs a retry, has already been handled or requires investigation.
The workflow illustrates a possible design and is not a client case study.
- Record event
- Process and retry safely
- Reconcile regularly
A timeout may happen after the write
Both systems may be running normally while a transfer fails. A timeout can occur after the receiving system has already saved the record. Retrying blindly risks duplicates; never retrying risks missing data. Keep receipt, processing and the confirmed result separate.
Check retry and recovery features
Check the connector’s logs and retry or replay functions. Stripe’s webhook documentation describes repeated notifications and events arriving out of order. Each provider has its own behaviour, so the integration must not assume that every event arrives once and in sequence.
Record each event’s progress
Protect one important process first. Verify events, save their IDs, queue the work and show failed items. Before retrying, check whether the action already took effect. Give staff a safe recovery procedure and periodically compare the records with the source system.
Does replay repeat the charge?
Try duplicate and reversed events, rate limits, outages, timeouts and partly completed work. A replay must not charge or ship twice. Confirm that genuinely missing records can be recovered and alerts tell staff which failures need action.
Bring a failed event’s history
Bring event timelines and failure logs with personal details removed, the API documentation and the result expected for each case.