Two Next.js rewrites that serve Markdown to agents without breaking the cache

On 25 August 2026 I ran an agentic-readiness audit against this site, saw it fail Markdown content negotiation, and refused to build it. Seven weeks later, on 12 October, I reversed half of that decision and shipped it in next.config.ts. The audit moved from 82/100 to 100/100.

What changed in between was not the audit. It was finding a way to serve text/markdown on / and on 404s without paying any of the four costs that made me refuse it the first time.

#The standard middleware pattern breaks either static caching or client navigation

Search for how to handle Accept: text/markdown in Next.js today and you will find two patterns repeated across almost every tutorial. The first calls headers() from next/headers inside page.tsx, checks for text/markdown, and branches. That one fails immediately on a static site: reading request headers inside a page component opts the route out of full-route static rendering and turns every human visit into a dynamic render.

The second pattern intercepts the request in middleware.ts — renamed to `proxy.ts` in Next.js 16 — rewrites to a Markdown endpoint, and calls response.headers.set("Vary", "Accept"). That looks cleaner and causes two worse problems.

• Overwriting `Vary` breaks App Router client navigation. Next.js already sets Vary: rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch on page responses so the browser and CDN do not mix full HTML documents with React Server Component flight payloads. Replacing that header with Vary: Accept strips all four tokens.
• Appending `Accept` on the HTML response fragments the CDN cache. Chrome sends a 133-character Accept string ending in application/signed-exchange;v=b3;q=0.7. Firefox and Safari send different strings. Putting Vary: Accept on the HTML page splits one cached document into several.
• A Markdown 404 in `proxy.ts` needs a manual list of valid routes. Because not-found.tsx renders a React component and cannot change its own Content-Type to text/markdown, a proxy can only return a Markdown 404 if it already knows which URLs exist. The moment you add a page and forget that list, live pages start returning 404 to Markdown requests while looking fine in every browser.

That third failure is why I refused the feature in August. A bug that only fires when an agent sends a header no browser ever sends is invisible by construction.

#Next.js's own rewrite example silently fails on static pages like the homepage

Next.js documents content negotiation in its own bundled guide (01-app/02-guides/backend-for-frontend.md). Its example returns a flat array from rewrites() in next.config.js, matching Accept: (.*)text/markdown(.*) on /docs/:slug* and rewriting to /docs/md/:slug*.

Copy that pattern onto / or /about and nothing happens. A curl -H "Accept: text/markdown" against the homepage comes back 200 text/html every time.

The reason is buried in the routing order documented in rewrites.md. When rewrites() returns a flat array, Next.js places every rule into the afterFiles phase. And Next.js evaluates routes in eight steps:

The documentation example works only because /docs/:slug* is a dynamic route resolved at step 7, after afterFiles has had its turn at step 6. A non-dynamic page like app/page.tsx resolves at step 5. By the time afterFiles runs, the HTML homepage has already won.

#beforeFiles and fallback let Next.js route both cases without a route list

Look at steps 4 and 8 in that table and both halves of the problem solve themselves at once. Returning { beforeFiles, afterFiles, fallback } from rewrites() puts the homepage rule before step 5 and the 404 rule after step 7:

async rewrites() {
  const wantsMarkdown = [
    { type: "header" as const, key: "accept", value: "(.*)text/markdown(.*)" },
  ];
  return {
    beforeFiles: [{ source: "/", has: wantsMarkdown, destination: "/index.md" }],
    afterFiles: [],
    fallback: [{ source: "/:path*", has: wantsMarkdown, destination: "/missing.md" }],
  };
},

Both destinations are ordinary static Route Handlers (export const dynamic = "force-static"). app/index.md/route.ts calls llmsTxt() from lib/llms-txt.ts — the exact same function that generates `/llms.txt` — and returns it with Content-Type: text/markdown; charset=utf-8 and Vary: Accept. app/missing.md/route.ts returns a static Markdown 404 body with status: 404, Vary: Accept, and links to /llms.txt, /sitemap.xml, /llms-full.txt and /.

Because fallback only runs after every static file, static page and dynamic route has failed to match, Next.js's own router decides what is a 404. There is no list of valid paths to keep in sync. And because Vercel evaluates next.config.ts rewrites before looking up the cache entry, Vary: Accept lives on /index.md and /missing.md alone. A browser asking for / never matches the rewrite, receives the static HTML page with Next's four router Vary tokens intact, and never shares a cache key with Accept: text/markdown.

One deliberate edge case remains. A missing slug inside a dynamic route family — such as /notebook/no-such-slug — matches app/notebook/[slug]/page.tsx at step 7 before fallback ever runs, calls notFound(), and returns the HTML 404. The HTTP status is still 404, and fixing that one case would bring back the manual route list.

#A site-wide Link header in next.config.ts silently drops the handler's canonical tag

Serving the homepage's content from /index.md creates a duplicate URL that a search crawler could index on its own. You cannot fix that by putting X-Robots-Tag: noindex on /index.md, because / itself serves that exact response whenever a crawler asks for text/markdown — a noindex header there would tell a Markdown-capable crawler to drop the homepage from its index.

The clean answer is an HTTP Link: </>; rel="canonical" header on /index.md, pointing back at the HTML homepage. I set that header inside app/index.md/route.ts, built the site, ran the check script, and watched the canonical header vanish.

Nothing had errored. What happened is that next.config.ts already sets a site-wide RFC 8288 discovery Link header on all documents (/((?!_next/|api/).*)), advertising /llms.txt, /sitemap.xml and /.well-known/ai-catalog.json. When a route handler and next.config.ts both set the same header key, the config rule wins and overwrites the handler's header.

{
  source: "/((?!_next/|api/).*)",
  headers: [{ key: "Link", value: discovery }],
},
{
  source: "/index.md",
  headers: [
    { key: "Link", value: `${discovery}, </>; rel="canonical"` },
  ],
},

Adding a second rule for /index.md below the document rule in next.config.ts solved it: in headers(), a later rule for the same key replaces the earlier one, so /index.md carries both the nine discovery relations and </>; rel="canonical".

#next start emits two Vary lines where Vercel serves one

When I tested /index.md locally against next start, the response headers came back with two separate Vary lines: Next's router Vary on the first line, and the handler's Vary: Accept on the second. Under RFC 9110 §5.3 multiple header lines with the same field name are semantically identical to one comma-separated line, and Node's fetch joins them automatically, so res.headers.get("vary") passes.

A naive scanner reading only the first raw header line would miss Accept. Before writing code to work around setVaryHeader in server/base-server.js, I deployed and checked the raw lines against Vercel's edge.

On Vercel, both /index.md and /missing.md return a single Vary: Accept line. The two-line output is an artefact of next start running the App Router server locally rather than serving the prerendered static metadata at the edge. Had I tried to strip Next's Vary in code before measuring production, I would have added machinery to fix something that never happens live.

#Every other route on this site still refuses Markdown on purpose

By October 2026 Cloudflare's *Markdown for Agents* report documented that coding agents such as Claude Code and OpenCode send Accept: text/markdown when fetching URLs, and Next.js's own documentation site serves Markdown twins across its guide pages. Even so, only / and the 404 negotiate on this domain. Asking /about, /resume or /banking/rm-copilot for text/markdown still returns the HTML page.

The reason is the fifth objection I wrote down in August, and it is the one that a better routing trick does not change: on a product dossier, a generated Markdown twin is thinner than the page it replaces. Converting a case study with interactive architecture diagrams, comparison tables and structured JSON-LD into a stripped Markdown summary hands an agent *less* evidence than the server-rendered HTML already contains.

The homepage is the one exception because /llms.txt was already built to be the complete, structured index of the site — every project, every route, every article and what question each URL answers. Serving that file when an agent asks / for Markdown gives it the map it actually needs, and forty-five checks in scripts/markdown-check.mjs make sure no human browser ever sees it.