PortfolioAbout This Topic
The most troublesome aspect of business systems is recovering from operational errors. Problems like "two identical invoices were created" or "an error occurred midway and I don't know how much was processed" cause workplace confusion.
This project was designed to handle various "oops" moments gracefully.
Sending the Same Invoice Twice is OK
A mechanism called "idempotency" prevents duplicate processing.
Send with key="invoice:001"
Not registered → Execute as new process
key="invoice:001" → Success, ID: 12345
Send again with key="invoice:001"
Already registered → Check previous result
Skip processing, return ID: 12345
Idempotency Key Design
Each request is assigned a unique key. If a second submission uses the same key, it's treated as "already created" and processing is skipped.
| Key Example | Components | Purpose |
|---|---|---|
| comp1:invoice:001 | Company ID + doc type + doc number | Invoice duplicate prevention |
| comp1:delivery:2024-001 | Company ID + doc type + doc number | Delivery slip duplicate prevention |
7-Day Memory
Results are cached for 7 days. Within a week, you can verify "has this invoice been created or not."
The cache has a two-tier structure: Redis (distributed cache) and memory. Even if Redis is unreachable, the memory cache serves as a backup.
Auto-Retry on Communication Errors
For temporary network issues or server congestion, the system automatically waits and retries.
Exponential Backoff with Jitter
"Exponential backoff" gradually lengthens retry intervals. "Jitter" adds random delay to prevent multiple clients from retrying simultaneously and causing congestion again.
Execute API call
429 (rate limit) or 5xx (server error)
Wait 300ms + random (0-199ms)
Wait 600ms + random
Wait 1200ms + random
Return error if limit reached
Retryable Errors
Not all errors trigger retries. Only temporary errors are retried.
| Status | Meaning | Retry |
|---|---|---|
| 429 | Too many requests (rate limit) | Yes |
| 500-599 | Temporary server error | Yes |
| 400 | Invalid request | No (needs fixing) |
| 401 | Authentication error | No (check token) |
| 404 | Resource not found | No |
Detailed Logs for Problem Identification
Records when, who, what was executed, and the result. Even when errors occur, it's clear which row had what problem.
Structured Log Example
{
"event": "row_processed_error",
"docType": "invoice",
"rowNumber": 5,
"idempotencyKey": "comp1:invoice:001",
"httpStatus": 422,
"error": "partner_code not found",
"timestamp": "2025-01-24T10:30:00.000Z"
}
What Logs Reveal
| Field | Content | Use |
|---|---|---|
| event | What happened | Identify event type |
| rowNumber | Which row it occurred on | Find row in spreadsheet |
| httpStatus | API response code | Understand error type |
| error | Error message | Identify specific cause |
| timestamp | When it occurred | Chronological tracking |
Error Handling Flow
Status column in spreadsheet shows "Error"
Check error content in logs. Find row using rowNumber
Correct problematic data in spreadsheet
Select fixed row and resend. New idempotency key means new processing
What This Design Achieves
For Operations
- Prevent duplicate invoices: Same invoice is never created twice
- Resilience to temporary failures: Auto-retry handles recovery
- Easy cause identification: Structured logs pinpoint problem areas instantly
For Field Staff
- Operate with confidence: Peace of mind knowing mistakes are OK
- Immediate status awareness: Result confirmation via status display
- Easy re-execution: Just fix and resend