Feed Caching and Health-Check Operations

Introduction

A product feed isn't the kind of thing that's finished once you've built it. Delivering it properly every single day takes considerably more effort than building it did.

There are two problems. The first is weight: rebuilding several thousand SKUs — the unit you get when you count products down to individual colours and sizes — from scratch on every request costs both time and server capacity. The second is that failure never announces itself. When the feed stops generating, no red error appears in any console. You find out when products vanish from Shopping and sales start dropping.

This article covers the cache that keeps feed generation light (a cache being simply a place where a finished result is kept and reused), and the health-check routine that confirms delivery hasn't stopped. Neither is exciting, but leaving them out causes trouble later.

Not rebuilding thousands of items every time

Google comes to collect the URL on its own schedule

Google Merchant Center (GMC for short — the console where you register product information with Google) fetches the feed URL you registered at regular intervals. You don't push anything; it comes and collects.

If every one of those visits meant retrieving and converting thousands of SKUs, each round would be genuinely heavy. Take too long and GMC times out, so the feed never loads at all. Nothing loaded means nothing updated.

Keep the finished result for a few hours and reuse it

Feed contents don't shift minute by minute. Even when prices and stock move, reflecting them into GMC every few hours is fine in practice. So keep the result you already built and hand out the same one for a while. That's the whole idea behind caching.

In other words, we stopped rebuilding the same thing over and over and started handing out something prepared in advance.

Match the expiry time to how fast stock moves

The cache gets an expiry time. On this project we set it to a few hours. When it expires, the next fetch triggers a rebuild, and that result is reused for another set period, over and over.

Too short and generation runs constantly, pushing up load. Too long and sold-out products stay listed well past the point they should have gone. Fast-moving stock wants a shorter expiry; slow-moving stock can take a longer one. There's no universal right answer, so the practical approach was to agree with the operations side how much listing lag was acceptable and set it from there.

One more thing worth knowing: shortening the expiry doesn't make Google update any faster. How often GMC comes to collect the URL, and how long it takes to reflect what it collected into live listings, are both outside your control. Cut your expiry to an hour and, if GMC only fetches once a day, listing freshness doesn't change at all — you've simply added load. Keeping the expiry roughly in step with how often GMC actually fetches turned out to be the sensible setting.

Cap how many items are handled at once

Once a catalogue reaches tens of thousands of SKUs, packing everything into a single file becomes difficult in itself — the build runs out of memory partway through.

So the work is split, with a cap on how many items are handled in one pass. With that limit in place, catalogue growth doesn't break the process. Product counts almost always move in one direction, so building this in from the start saves you from revisiting it.

A separate mechanism to tell you it stopped

When the feed stops, the ads stop too, with no warning

The tricky thing about feeds is that failures never surface on screen. A change to the specification on the product-data side, a network error, an inconsistent cache. The causes vary, but they all show up the same way: the listings had already stopped by the time anyone noticed.

Which is why you need something separate from the feed itself, checking from the outside that it's still being built.

A dedicated URL that confirms generation works

So we set up a dedicated URL — a health check — that confirms the feed can still be generated. When called, it actually attempts a build and reports back on whether the product count falls within the expected range and whether the required fields are present.

Effectively, this is the manual morning check somebody used to do by hand, handed to a machine and run on a timer.

Treat a sudden drop in the count as a failure

"Did it build?" isn't the only question worth asking. The change in product count matters just as much. If a feed that normally carries several thousand items shows a few dozen one morning, something is wrong with the contents even though generation succeeded.

Product retrieval stopped partway. An exclusion condition matched far more broadly than intended. "Succeeded, but nearly empty" really does happen. Widening the test from "is it non-zero" to "is it far below normal" means you catch it while the listing is thinning out, not after it's gone.

Decide where alerts go, and who moves when they arrive

Monitoring achieves nothing if the alert lands somewhere nobody looks. Route it to the chat tool the team already has open, or to email, and settle in advance who responds first when one comes in.

Just as important is how sensitive the alert threshold should be. Fire on every small fluctuation and people stop reading the messages entirely. Set it too loosely and it stays silent when it matters. Narrowing it to conditions genuinely worth investigating — the count fell below half of normal, or generation failed twice in a row — worked better in practice than trying to catch everything.

Summary

Feed operations are easier to think about as two pillars: build it light, and know when it stops.

• Caching: reuse the finished result for a few hours to balance processing weight against freshness
• Expiry time: set it against how fast your stock moves; there's no universal right answer
• Item cap: limit how much is handled per pass so catalogue growth doesn't break the build
• Health check: confirm generation and count on a schedule to catch failures that never surface

The feed is a revenue-critical mechanism: when it stops, Shopping ads and free listings stop with it. Watching that it keeps running deserves as much attention as building it did.

How the feed itself gets built is covered in automated product feeds and mapping design; the safeguards that protect listing quality are in managing feed quality.