The Problem
AI answer engines now send real traffic, but they see a different site than Google does. Without a curated map of your important pages and usage terms, crawlers guess -- and summarize the wrong pages, or skip you entirely.
Use Cases
Publish an llms.txt so ChatGPT, Claude, and Perplexity read and cite your site correctly. Generate both files, then verify what an AI crawler reaches.
Estimated cost: ~7 credits per site
01Quick Answer
Run generate_llms_txt (5 credits) against your domain to produce llms.txt and llms-full.txt -- a curated map of your key pages plus usage guidelines for AI crawlers -- then map_site (2 credits) to confirm what those crawlers can reach. About 7 credits per site, and you re-run it whenever the site changes.
02The brief
The Solution
CrawlForge generate_llms_txt crawls your site and writes both llms.txt and the longer llms-full.txt, with your organization name, contact, and custom guidelines. map_site then shows what an AI crawler can actually reach before you publish.
03In code
Code Example
llms-txt-ai-visibility.js
// Generate llms.txt and llms-full.txt for your siteconst llms = await mcp.generate_llms_txt({ url: "https://example.com", format: "both", complianceLevel: "standard", analysisOptions: { maxPages: 200, maxDepth: 3, respectRobots: true }, outputOptions: { organizationName: "Example Inc", contactEmail: "ai@example.com", customGuidelines: ["Cite the canonical URL", "Pricing changes monthly"], },}); // Confirm what an AI crawler can actually reachconst map = await mcp.map_site({ url: "https://example.com", include_sitemap: true, max_urls: 500,}); console.log(llms.llmsTxt);console.log(`Discoverable pages: ${map.urls.length}`);04The pipeline
Tools Used
7◆ credits
Estimated cost: ~7 credits per site
- https://…
- 01generate_
llms_ txt 5 credits - 02map_
site 2 credits - JSON · markdown
05Questions
Frequently Asked Questions
01What is llms.txt and do I need one?
llms.txt is a plain-text file at your site root that points AI crawlers at your important pages and states how the content may be used. It is a proposed standard rather than a ranking guarantee, but it is cheap to publish and replaces guesswork with a curated map.
02How do I generate llms.txt for my site?
Run generate_llms_txt against your domain. It crawls up to 500 pages, respects robots directives, and writes llms.txt plus the longer llms-full.txt, including your organization name, contact email, and any custom guidelines you pass.
03What is the difference between llms.txt and llms-full.txt?
llms.txt is the short index — sections and key links. llms-full.txt inlines far more detail for models that can take it. Pass format "both" to get the pair, or pick one with "llms-txt" or "llms-full-txt".
04How do I check what AI crawlers can actually see?
Run map_site with include_sitemap to list every reachable URL, then compare that against the pages listed in your llms.txt. Gaps show where a crawler will miss content, or where your sitemap has gone stale.
06Keep exploring
Related Use Cases
SEO Analysis & Auditing
Audit your site and competitors for metadata, broken links, content gaps, and ranking opportunities. Returns a structured report across every crawled page.
extract_metadata 1 creditmap_site 2 creditsanalyze_content 3 credits
RAG Knowledge Base Ingestion
Turn documentation sites into clean, chunk-ready markdown for retrieval-augmented generation. Crawl, strip the boilerplate, and summarize before you embed.
map_site 2 creditsscrape 2 creditssummarize_content 4 credits
Start forging
Ready to Get Started?
Every new account gets 1,000 free credits. No credit card required.