generate_llms_txt

Crawl a site, analyze its structure, and emit a standard-compliant llms.txt (and optional llms-full.txt) file defining how AI models should interact with your content. Compliance levels from permissive to strict.

Use Cases

Ship AI-Ready Documentation

Publish llms.txt alongside your docs so Claude, ChatGPT, and other crawlers read clean guidelines.

AI Compliance Publishing

Use strict compliance to set training-data, caching, and attribution rules in one place.

Bot Policy Generation

Add custom guidelines and restrictions for specific AI user agents on your domain.

Endpoint

POST/api/v1/tools/generate_llms_txt

Auth Required

1 req/s on Free plan

5 credits

Parameters

Heavy operation: This tool may crawl up to 500 pages. It uses the reservation system so credits are held for the duration of the job.

Name	Type	Required	Default	Description
url	string	Required	-	The website URL to generate llms.txt for Example: https://example.com
format	string	Optional	both	Output format: "both" \| "llms-txt" \| "llms-full-txt" Example: both
complianceLevel	string	Optional	standard	Compliance level for generated guidelines: "basic" \| "standard" \| "strict" Example: standard
analysisOptions	object	Optional	-	Website analysis options (maxDepth 1-5, maxPages 10-500, respectRobots, detectAPIs, analyzeContent, checkSecurity) Example: {"maxDepth": 3, "maxPages": 100, "detectAPIs": true}
outputOptions	object	Optional	-	Output customization (organizationName, contactEmail, customGuidelines, customRestrictions, includeDetailed, includeAnalysis) Example: {"organizationName": "Example Inc.", "contactEmail": "ai@example.com"}
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. `analysisOptions.respectRobots` is an alias; either one set to `false` disables the gate. Example: true

Request Examples

cURL — both formats, standard compliance

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/generate_llms_txt \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "both",
    "complianceLevel": "standard",
    "outputOptions": {
      "organizationName": "Example Inc.",
      "contactEmail": "ai@example.com"
    }
  }'

TypeScript — strict with custom guidelines

generateLlmsTxt.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/generate_llms_txt', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://docs.example.com',
    format: 'both',
    complianceLevel: 'strict',
    analysisOptions: {
      maxDepth: 4,
      maxPages: 250,
      detectAPIs: true,
      analyzeContent: true,
    },
    outputOptions: {
      organizationName: 'Example Inc.',
      contactEmail: 'ai@example.com',
      customGuidelines: [
        'AI crawlers must respect robots.txt',
        'Cache responses for up to 24 hours',
      ],
      customRestrictions: [
        'No training on user-submitted content',
      ],
      includeAnalysis: true,
    },
  }),
});

const { data } = await response.json();
await fs.writeFile('public/llms.txt', data.files['llms.txt']);
await fs.writeFile('public/llms-full.txt', data.files['llms-full.txt']);

Python

generate_llms_txt.pyPython

import requests, os

response = requests.post(
    'https://crawlforge.dev/api/v1/tools/generate_llms_txt',
    headers={
        'X-API-Key': os.environ['CRAWLFORGE_API_KEY'],
        'Content-Type': 'application/json',
    },
    json={
        'url': 'https://example.com',
        'format': 'llms-txt',
        'complianceLevel': 'basic',
    },
)

data = response.json()['data']
with open('public/llms.txt', 'w') as f:
    f.write(data['files']['llms.txt'])

Response Example

200 OK4.1s

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "llms_txt": "# Example Inc.\n\n> Tools for building things.\n\n## Docs\n\n- [Quickstart](https://example.com/docs/quickstart): Get started in five minutes\n",
    "llms_full_txt": "# Example Inc.\n\n## Quickstart\n\nURL: https://example.com/docs/quickstart\n\nGet started in five minutes...",
    "pages_analyzed": 12,
    "pages": [
      {
        "url": "https://example.com",
        "title": "Example Inc."
      },
      {
        "url": "https://example.com/docs/quickstart",
        "title": "Quickstart"
      }
    ],
    "compliance": {
      "robots_txt_found": true,
      "disallow_count": 3
    },
    "api_endpoints_detected": [
      "https://example.com/api/v1"
    ],
    "clamps_applied": [
      "maxPages clamped from 100 to 25"
    ],
    "generated_at": "2026-08-26T14:30:00.000Z"
  },
  "credits_used": 5,
  "credits_remaining": 995,
  "processing_time": 4100
}

Field Descriptions

data.llms_txtThe generated llms.txt. Present unless `format` was set to llms-full-txt.

data.llms_full_txtllms.txt with the text of each analyzed page inlined. Present only when `format` is both or llms-full-txt.

data.pages_analyzedRoot page plus every page successfully fetched.

data.pagesThe pages that went into the output, each with its title.

data.compliance.robots_txt_foundWhether a robots.txt was retrieved; disallow_count is how many Disallow rules it carried.

data.api_endpoints_detectedUp to 10 API-looking URLs spotted in the fetched HTML. A heuristic on the path, not a verified API listing.

data.clamps_appliedHuman-readable notes when your requested limits were reduced to fit the serverless budget. Empty when nothing was clamped.

data.generated_atISO 8601 timestamp of generation.

Error Handling

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to. Only the target URL 403s: pages discovered during the analysis that robots.txt disallows are left out of it instead, and counted in warnings.

Credit Cost