System Resilient to Retries

About This Topic

The most troublesome aspect of business systems is recovering from operational errors.
Problems like "two identical invoices were created" or "an error occurred midway and I don't know how much was processed" create real confusion on the ground.

This project was designed to handle various "oops" moments gracefully.
That kind of safety net makes day-to-day operations much less stressful.

Sending the Same Invoice Twice is OK

A mechanism called "idempotency" prevents duplicate processing.
In simple terms, the same request should not create different outcomes.

Duplicate Prevention Mechanism

First Request

Send with key="invoice:001"

Cache Check

Not registered → Execute as new process

Save Result

key="invoice:001" → Success, ID: 12345

Second Request (Same Key)

Second Request

Send again with key="invoice:001"

Cache Check

Already registered → Check previous result

Return Previous Result

Skip processing, return ID: 12345

Idempotency Key Design

Each request is assigned a unique key. If a second submission uses the same key, it's treated as "already created" and processing is skipped.

Send Request

Execute API call

Error Occurs

429 (rate limit) or 5xx (server error)

Retry 1

Wait 300ms + random (0-199ms)

Retry 2

Wait 600ms + random

Retry 3

Wait 1200ms + random

Up to 5 Retries

Return error if limit reached

Retryable Errors

Not all errors trigger retries. Only temporary errors are retried.

Detailed Logs for Problem Identification

Records when, who, what was executed, and the result. Even when errors occur, it's clear which row had what problem.

Structured Log Example

{
  "event": "row_processed_error",
  "docType": "invoice",
  "rowNumber": 5,
  "idempotencyKey": "comp1:invoice:001",
  "httpStatus": 422,
  "error": "partner_code not found",
  "timestamp": "2025-01-24T10:30:00.000Z"
}