llms-txt-generator

Generate spec-compliant llms.txt and llms-full.txt files from any website or sitemap — so LLMs and AI agents read your content the way you intend, instead of guessing from raw HTML.

Hosted, no-install version: https://rapidwombat.com/llms-txt-generator This CLI is the open-source core of that tool. Run it locally, wire it into CI, or use the hosted UI — same pipeline either way.

llms.txt is a small, predictable Markdown file at the root of your site (/llms.txt) that gives language models a clean, curated map of your content. Think of it as robots.txt and sitemap.xml, but written for reading comprehension rather than crawling:

• llms.txt — a curated index: title, one-line summary, and a sectioned list of your most important URLs with short descriptions.
• llms-full.txt — the same pages expanded to clean Markdown, concatenated into one file an agent can load wholesale.

As discovery shifts from blue links to AI answers (ChatGPT, Claude, Perplexity, Google AI Overviews), the practice of structuring content for machine readers has picked up names — Generative Engine Optimization (GEO), Answer Engine Optimization (AEO), AI-readiness. llms.txt is the most concrete, standards-track piece of that puzzle. This tool builds one for you.

Want the bigger picture — schema, llms.txt, AI-crawler access, and the rest of a 2026 AI-readiness checklist scored against your live site? Run a free scan with the RapidWombat site audit.

# Run without installing
npx llms-txt https://example.com

# Or add to a project
npm install --save-dev llms-txt-generator

Requires Node.js ≥ 18.17 (uses the built-in fetch).

# Simplest: discover via sitemap.xml, write ./llms.txt and ./llms-full.txt
npx llms-txt https://example.com

# Curate sections and mark low-priority paths as "Optional"
npx llms-txt https://example.com \
  --out ./public \
  --title "Example Docs" \
  --summary "Everything an AI agent needs to use Example." \
  --optional /blog,/changelog,/tag

# Print the index to stdout (good for piping / diffing in CI)
npx llms-txt https://example.com --stdout

import { generate } from "llms-txt-generator";

const { llmsTxt, llmsFullTxt, pages, skipped } = await generate({
  site: "https://example.com",
  title: "Example Docs",
  summary: "Everything an AI agent needs to use Example.",
  optionalPrefixes: ["/blog", "/changelog"],
  maxPages: 100,
});

// llmsTxt        -> write to /public/llms.txt
// llmsFullTxt    -> write to /public/llms-full.txt
// pages[].words  -> per-page word counts for token budgeting
// skipped[]      -> { url, reason } for anything excluded (robots, cross-origin, ...)

You can also use the building blocks directly: extract(html, url), buildLlmsTxt(...), buildLlmsFullTxt(...).

llms.txt follows the specification exactly:

# Example Docs

> Everything an AI agent needs to use Example.

## Docs

- [Getting started](https://example.com/docs/start): Install and first run.
- [API reference](https://example.com/docs/api): Every endpoint and type.

## Optional

- [Launch week recap](https://example.com/blog/launch): Narrative blog post.

The ## Optional section is special: agents on a tight token budget can drop everything in it and still have the core of your site.

sitemap.xml ──► fetch (polite, robots-aware) ──► extract readable Markdown ──► assemble
   discover            src/fetch.ts                    src/extract.ts          src/generator.ts

1. Discover — read sitemap URLs from robots.txt / /sitemap.xml, recursing through sitemap-index files (src/sitemap.ts).
2. Fetch — same-origin, rate-limited, and respecting robots.txt by default (src/fetch.ts).
3. Extract — strip nav/chrome, pick the densest content container, and convert it to faithful Markdown (src/extract.ts).
4. Assemble — group by URL section, render spec-compliant link lists, and build the full-content companion file (src/generator.ts).

See docs/METHODOLOGY.md for the design rationale, the parts of the llms.txt spec we implement, and how we keep extraction faithful and low-noise.

npm install
npm run build      # tsc -> dist/
npm test           # node:test, no network required
node examples/generate.mjs https://example.com

• Respects robots.txt unless you pass --no-robots for a site you own.
• Rate-limited and same-origin only.
• Sends an honest, identifiable User-Agent (llms-txt-generator/0.1 (+https://rapidwombat.com/llms-txt-generator)).

• RapidWombat llms.txt generator — hosted version of this tool with a one-click UI.
• RapidWombat AI-readiness audit — scores your live site on llms.txt, structured data, crawler access, and more.
• The llms.txt specification — the upstream standard.

Issues and PRs welcome. Run npm test and npm run lint before submitting.

MIT © RapidWombat