llms.txt & AI Search Visibility
Publish an llms.txt so ChatGPT, Claude, and Perplexity read and cite your site correctly. Generate both files, then verify what an AI crawler reaches.
Quick Answer
Run generate_llms_txt (5 credits) against your domain to produce llms.txt and llms-full.txt -- a curated map of your key pages plus usage guidelines for AI crawlers -- then map_site (2 credits) to confirm what those crawlers can reach. About 7 credits per site, and you re-run it whenever the site changes.
The Problem
AI answer engines now send real traffic, but they see a different site than Google does. Without a curated map of your important pages and usage terms, crawlers guess -- and summarize the wrong pages, or skip you entirely.
The Solution
CrawlForge generate_llms_txt crawls your site and writes both llms.txt and the longer llms-full.txt, with your organization name, contact, and custom guidelines. map_site then shows what an AI crawler can actually reach before you publish.
Code Example
// Generate llms.txt and llms-full.txt for your site
const llms = await mcp.generate_llms_txt({
url: "https://example.com",
format: "both",
complianceLevel: "standard",
analysisOptions: { maxPages: 200, maxDepth: 3, respectRobots: true },
outputOptions: {
organizationName: "Example Inc",
contactEmail: "ai@example.com",
customGuidelines: ["Cite the canonical URL", "Pricing changes monthly"],
},
});
// Confirm what an AI crawler can actually reach
const map = await mcp.map_site({
url: "https://example.com",
include_sitemap: true,
max_urls: 500,
});
console.log(llms.llmsTxt);
console.log(`Discoverable pages: ${map.urls.length}`);Tools Used
Estimated cost: ~7 credits per site
Frequently Asked Questions
What is llms.txt and do I need one?
llms.txt is a plain-text file at your site root that points AI crawlers at your important pages and states how the content may be used. It is a proposed standard rather than a ranking guarantee, but it is cheap to publish and replaces guesswork with a curated map.
How do I generate llms.txt for my site?
Run generate_llms_txt against your domain. It crawls up to 500 pages, respects robots directives, and writes llms.txt plus the longer llms-full.txt, including your organization name, contact email, and any custom guidelines you pass.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the short index — sections and key links. llms-full.txt inlines far more detail for models that can take it. Pass format "both" to get the pair, or pick one with "llms-txt" or "llms-full-txt".
How do I check what AI crawlers can actually see?
Run map_site with include_sitemap to list every reachable URL, then compare that against the pages listed in your llms.txt. Gaps show where a crawler will miss content, or where your sitemap has gone stale.
Ready to Get Started?
Every new account gets 1,000 free credits. No credit card required.
Start Free with 1,000 Credits