Generating and Caching Dynamic Sitemaps and robots.txt

Introduction

A sitemap.xml is a map file that hands search engines a list saying "this site has these pages." A robots.txt is the note telling them which parts of the site they're welcome to visit. Neither gets a moment's thought normally — until you go headless and have to provide them yourself.

EC platforms like Shopify generate a sitemap automatically, but that's a list of the platform's own URLs; not one URL from your custom frontend appears in it. The same goes for robots.txt: whatever is served on your custom frontend's domain, you return yourself.

With thousands to tens of thousands of SKUs turning over daily, maintaining those two by hand just isn't realistic. This article explains how, on a headless EC site selling motorcycle gear, I auto-generated the sitemap and robots, kept the generation cost down, and built in the checks that prevent quiet failures.

Collecting Only the URLs That Belong to This Site

Using the Sales Channel as the Basis

The URLs that belong in a sitemap are only the pages that actually exist on that site. Obvious enough, but when one backend runs several EC sites, how you build this really matters.

Deciding which products appear on which site is handled by sales channels (a feature where a checkbox in the admin controls where each product is listed), so sitemap generation uses the same basis. The admin API is asked for "products and categories published to this channel," and URLs are built from the returned handles (the product-specific strings used in URLs) and their update timestamps.

The nice consequence is that flipping a product's publishing target in the admin changes the sitemap too. No code changes, no deployment — the shop team handles it end to end. That's the part that cut the most operational work.

Static Pages Live in a List

Pages like the company profile, FAQ, and terms can't be derived from product data, so their paths, update frequency, and priority are kept as a list in the code. Adding a page means adding a line, but there are few enough of them that doing it by hand is fine.

Those four steps are one decision: use what's already built, or fetch again and rebuild.

Keeping Generation Cost Down with Two Layers of Cache

Full Generation Every Time Piles Up Requests

Fetching tens of thousands of records from an external source and assembling them into XML is not a light operation. Crawlers (the programs search engines use to read pages) come for the sitemap more often than you'd guess, so full generation every time makes both the external request count and the server load hard to ignore.

Different Expiries for Delivery and for the Material

So I made the cache two-tiered. The delivery layer holds the assembled sitemap for a day; the material store holds the fetched product and category data for a few hours to half a day.

Retention in the material store varies by type — six hours for products and blog posts, twelve for categories — on the simple principle that whatever changes more often expires sooner. The delivery layer is set to "when the cache expires, serve the old copy first and rebuild in the background," so a crawler arriving right after expiry never waits for generation to finish.

It amounts to not rebuilding the same thing over and over.

Returning a Valid Sitemap Even When Generation Fails

Repeated Errors Reduce Crawl Frequency Itself

Sitemap generation depends on several moving parts — external APIs, the cache store — and if any one of them doesn't respond, you get an error. My first mistake was returning a plain server error (500) in that case.

When a sitemap repeatedly returns errors, the search side records it as a fetch failure and lowers how often it re-crawls the site at all. New products take longer to appear, and the revisit interval for already-known URLs stretches out too. A few hours of trouble keeps affecting things well afterwards.

A Minimal Sitemap with a Short Expiry

Now, whatever goes wrong, the response is always a normal one (200). If product data can't be fetched, it assembles a minimal sitemap of static pages only and returns that. In that case alone the cache expiry drops to ten minutes, so the full version comes back soon after recovery.

Fewer URLs listed is fine; delivering something valid takes priority. It's not a visible decision, but it ties directly to keeping crawl frequency up.

"Don't Crawl" and "Don't Show in Search" Are Different Instructions

Block the Crawl and the "Don't Show" Instruction Never Gets Read

robots.txt is generated dynamically too. The content is straightforward: allow crawling overall, exclude the admin, the API, and the account pages, and finish with the location of the sitemap.

I got this backwards once. For a site I didn't want appearing in search results, I blocked crawling wholesale in robots.txt. But blocking the crawl means the search engine can't fetch the page — and if it can't fetch the page, it can't read the "please don't show this in search results" instruction written inside it either. The result was that URLs linked from elsewhere stayed in the index and kept appearing in search results with no readable content.

Allow the Crawl, Then Exclude on the Page

The correct approach is the opposite. Allow crawling, and return the "don't show in search results" instruction (noindex) on the page itself. Only when a crawler fetches the page and reads that instruction does it drop out of search results.

The third row is the combination you'll actually use most. Leaving crawling allowed also lets product feed validation for advertising pass, so you get "invisible in search, visible to machines."

Monitoring the Count and Standardizing the Format

Watching the URL Count to Catch Problems Early

The weakness of anything automatic is that nobody notices when it breaks. A sitemap can lose half its contents with zero visible change on screen. You tend to start investigating only after rankings drop, and by then weeks have passed.

So I built an admin API that returns the product, category, and article counts held in the cache, the total URL count, and the timestamp of the last generation. A count well below the previous one is enough to suspect a fetch failure or a misconfiguration. The same API also has an endpoint for clearing the cache by hand, for moments like a large product swap where you don't want to wait for expiry.

Folding URL Formatting into a Single 301

The URLs listed in the sitemap need consistent formatting too. If the same product page appears with mixed-case letters, or with a parameter marking a selected color, search engines treat those as separate pages and the evaluation that should concentrate on one page gets split.

This is handled by a lightweight process that runs at the site's entry point (Edge middleware), folding lowercasing, removal of unnecessary parameters, and trailing-slash adjustment into a single 301 redirect. A 301 means "moved permanently," so the evaluation carries over to the canonical URL.

Handling it at the entry point means normalization happens before the request reaches the app itself, and the same rule applies to every page.

Conclusion

Four things to get right when auto-generating sitemaps and robots: collect only the URLs belonging to that site, using the sales channel as the basis; use two layers of cache to keep generation cost down; always return a valid, minimal sitemap even on error; and keep a way to monitor the URL count.

On top of that, pages you don't want in search results shouldn't be blocked in robots.txt — allow the crawl and exclude them with an instruction on the page. This is the one people get backwards.

With SSR output, structured data, and dynamic sitemaps plus URL normalization all in place, the SEO foundation that stopped working when you went headless is broadly back.