Content Negotiation in Web API for AI Agents

Most product APIs were designed for applications, not language models. An agent can scrape an HTML page, but navigation, cookie banners, repeated links, and layout noise waste context and make extraction brittle. A better pattern is Markdown negotiation: use content negotiation in a web API to let the client ask for the representation it can use best.

The contract

Use the standard Accept header to select a representation of the same resource:

GET /docs/agent-tools HTTP/1.1
Accept: text/markdown

The server returns focused Markdown with the same semantic content as the browser page:

Content-Type: text/markdown; charset=utf-8
Vary: Accept

For a browser request, return normal HTML:

Accept: text/html
Content-Type: text/html; charset=utf-8

This is not a second API. It is one resource with two useful representations.

What belongs in the Markdown response

Keep the Markdown version short, direct, and task-oriented. Include the title, a one-paragraph summary, prerequisites, instructions, examples, and important limitations. Omit site navigation, promotional modules, unrelated internal links, visual-only captions, and duplicated calls to action.

The goal is to reduce ambiguity, not to recreate the page in another syntax. A model should be able to decide what to do after reading the response once.

A simple server implementation

At the route boundary, parse Accept and prefer Markdown only when the client explicitly requests it. A sensible fallback is HTML because browsers and link previews expect it.

const accept = request.headers.get('accept') ?? '';
const wantsMarkdown = accept.includes('text/markdown');

return wantsMarkdown
  ? new Response(markdown, { headers: { 'Content-Type': 'text/markdown; charset=utf-8', Vary: 'Accept' } })
  : renderHtmlPage();

Avoid returning a Markdown response that contains tracking links, hidden prompts, or unrelated instructions. Treat it as a public document: stable, cacheable, and safe to quote.

Why this helps AI agents

Markdown negotiation lowers token cost, improves retrieval quality, and gives tool-using agents a predictable input shape. It also keeps your human experience intact: people still get the designed HTML page, while machines receive the cleanest version of the information.

For documentation, changelogs, pricing, product specs, and knowledge bases, this small protocol decision can make an interface dramatically easier for agents to use.