Notification Delivery Pipeline Design (Queue, Throttling, Retry)

Building delivery around a pending send list so large alert volumes don't clog, and deciding retry rules up front

Notification DeliveryJob QueueThrottlingRetryDead Letter Queue
7 min read

Sending Everything at Once Is How Alerts Stop Arriving

When a popular item restocks, the number of people waiting for the alert can be in the hundreds or thousands. Try to email all of them simultaneously and you run into the delivery service's rate limits, or worse, the messages get scored as spam and never land. The more alerts you have to send, the more "arriving reliably" outweighs "arriving fast."

This article covers the order and pace in which alerts go out, and what happens to the ones that fail. The delivery system hasn't been built yet, so this isn't a tour of a finished feature — it's a record of the decisions worth making before starting. Failures in this layer affect a wide area, so the design prioritizes repeatable results and the ability to recover over raw speed.

Separate Accepting the Work From Doing the Work

What happens when you send in the same step you decide

If the email goes out in the same operation that decides stock has returned, any slowness on the delivery service's side stalls the whole process. With one recipient that's fine. Partway through several hundred, an external service slowing down means the operation times out and nobody knows how far it got.

So split the deciding from the sending. The deciding side finishes once it has written "who gets what" into the pending send list, and a separate process picks entries off that list and works through them. Drawing that line between responsibilities is the starting point for a design you can extend later.

Put a pending send list in the middle

The pending send list is the delivery schedule: who gets an alert about which item, and when. With that list in place, a burst of alerts stops being alarming. You work through however many are queued, in order, which converts a sudden pile-up into something moving at your own pace.

Failures are handled on the same list. Anything that couldn't be sent stays there with its state updated, ready to be picked up again if it meets the conditions. Making the send itself repeatable means a temporary wobble in the external service doesn't cost you anything permanent.

Delivery With a Pending Send List in the Middle
Confirm the recipients

Match the restocked item against the people who asked to be notified

Line them up

Write who gets what into the pending send list

Send in order

Hand them to the email delivery service at a controlled pace

Write the result back

Record on the list whether it sent, failed, or should be retried

Instead of calling the send directly, write the work out to a list first and pick it up from there. That single extra step is what keeps things from clogging when alerts arrive all at once.

Show How Far Each Alert Has Got Using States

Four states are enough

Give every individual alert a state showing which stage it is at. Four will do: pending, sending, sent, and failed. That distinction alone means you can answer "where is this one stuck?" the moment a question comes in.

The table just divides an alert's life into four segments. With those segments in place, a person can look at the system and tell where things are backing up.

Vague states leave you with nothing to investigate

Manage alerts with nothing but "sent" and "not sent" and anything that stalls mid-process goes missing. Nobody can tell whether it's safe to send again or whether it already arrived, so nobody decides, and it sits there untouched.

Clear states also let the development side express retry conditions plainly: which states get retried, and under what circumstances. When features get added later, they can be reasoned about against the existing sequence of states, so the structure holds up.

Pacing the Sends, and Retrying the Failures

Blasting everything at once means fewer alerts arrive

Set two kinds of limits on delivery pace. One is a global cap — how many messages per second overall. The other is a per-destination cap, which keeps too many messages from hitting the same mail provider in a short window.

Neither exists to finish faster; both exist to keep messages arriving. A sudden concentration of mail toward one type of destination is exactly what gets flagged as spam on the receiving end. Delivering reliably is worth more than delivering everything at once, and that's worth agreeing on early.

Failures worth retrying, and failures that must not be

Lump all failures together and retry them all, and the retries stack up until the list is congested. Split failures into two kinds and treat them differently: the ones that clear up with time, and the ones that never will.

The first kind is temporary congestion or a slow response on the other end. Wait, then try again — and make the wait longer with each attempt. The second kind is something like an address that doesn't exist. Don't retry those; mark them as permanently failed and stop processing them.

How Each Kind of Failure Is Handled
Clears up with time

Congestion, slow responses Retry with growing intervals

AFTER
Never clears up

Address doesn't exist No retry, mark as failed

Sort failures into two buckets and retry only one of them. Without that line, pointless retries fill up the list.

Decide How Records Are Kept, Up Front

Don't write contact details into the pending send list

The pending send list holds entries for the people about to be notified. Store email addresses there directly and you've added one more place where personal data lives. Give the list only a reference to the sign-up record, and pull the actual contact details out immediately before sending — far safer to handle.

The same thinking applies to operational records. Keep what's needed for investigation, and leave out contact details and authentication values. Set that policy at the start and you avoid going back through every record to fix how it was written.

Keep a record of who retried what, and when

Alerts touch users directly, so every manual action gets recorded: when, by whom, and which alert was retried. That history is what lets you reconstruct events when an unexpected message goes out.

An accurate history does more for day-to-day confidence than another convenience feature. Record-keeping requirements tend to be expensive to bolt on later, so folding them into the initial spec is usually the faster route.

What to Agree On Before Building

Before starting on the delivery system, confirm that four things are settled: the sequence of states for each alert, the conditions for retrying, the pace limits, and the classification of failures. Begin development with these vague and you'll spend the following months reshaping the system around requests that arrive after launch.

Outcome metrics are out of scope for now. The notification system itself doesn't exist, so there are no real figures for revenue attributed to alerts. Build delivery that works reliably first, then add measurement in stages. That order also makes the eventual numbers mean something.

Summary

A delivery system isn't a mechanism for sending fast — it's a mechanism for sending safely, continuously. Put a pending send list in the middle, split each alert's life into four states, treat failures differently by cause, and decide how records are kept before you write them. Lock those four down and you cut both the rework during implementation and the risk once operations begin.