AI 工具5 credits

generate_llms_txt

爬取站点、分析其结构，并生成符合标准的 llms.txt（及可选的 llms-full.txt）文件，用于定义 AI 模型应如何与你的内容交互。合规级别从宽松到严格可选。

使用场景

发布 AI 就绪的文档

在文档旁发布 llms.txt，让 Claude、ChatGPT 等爬虫读取清晰的准则。

AI 合规发布

使用严格合规级别，在一处统一设置训练数据、缓存和署名规则。

机器人策略生成

为你域名上特定的 AI user agent 添加自定义准则和限制。

Endpoint

POST/api/v1/tools/generate_llms_txt

Auth Required

Free 计划 2 req/s

5 credits

Parameters

重负载操作： 此工具可能爬取多达 500 个页面。它使用预留系统，因此 credits 会在任务期间被占用。

Name	Type	Required	Default	Description
url	string	Required	-	要为其生成 llms.txt 的网站 URL Example: https://example.com
format	string	Optional	both	输出格式："both" \| "llms-txt" \| "llms-full-txt" Example: both
complianceLevel	string	Optional	standard	生成准则的合规级别："basic" \| "standard" \| "strict" Example: standard
analysisOptions	object	Optional	-	网站分析选项（maxDepth 1-5、maxPages 10-500、respectRobots、detectAPIs、analyzeContent、checkSecurity） Example: {"maxDepth": 3, "maxPages": 100, "detectAPIs": true}
outputOptions	object	Optional	-	输出定制（organizationName、contactEmail、customGuidelines、customRestrictions、includeDetailed、includeAnalysis） Example: {"organizationName": "Example Inc.", "contactEmail": "ai@example.com"}

请求示例

cURL — 两种格式，标准合规

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/generate_llms_txt \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "both",
    "complianceLevel": "standard",
    "outputOptions": {
      "organizationName": "Example Inc.",
      "contactEmail": "ai@example.com"
    }
  }'

TypeScript — 带自定义准则的严格模式

generateLlmsTxt.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/generate_llms_txt', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://docs.example.com',
    format: 'both',
    complianceLevel: 'strict',
    analysisOptions: {
      maxDepth: 4,
      maxPages: 250,
      detectAPIs: true,
      analyzeContent: true,
    },
    outputOptions: {
      organizationName: 'Example Inc.',
      contactEmail: 'ai@example.com',
      customGuidelines: [
        'AI crawlers must respect robots.txt',
        'Cache responses for up to 24 hours',
      ],
      customRestrictions: [
        'No training on user-submitted content',
      ],
      includeAnalysis: true,
    },
  }),
});

const { data } = await response.json();
await fs.writeFile('public/llms.txt', data.files['llms.txt']);
await fs.writeFile('public/llms-full.txt', data.files['llms-full.txt']);

Python

generate_llms_txt.pyPython

import requests, os

response = requests.post(
    'https://crawlforge.dev/api/v1/tools/generate_llms_txt',
    headers={
        'X-API-Key': os.environ['CRAWLFORGE_API_KEY'],
        'Content-Type': 'application/json',
    },
    json={
        'url': 'https://example.com',
        'format': 'llms-txt',
        'complianceLevel': 'basic',
    },
)

data = response.json()['data']
with open('public/llms.txt', 'w') as f:
    f.write(data['files']['llms.txt'])

响应示例

200 OK4.1s

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "hostname": "example.com",
    "compliance_level": "standard",
    "files": {
      "llms.txt": "# llms.txt for Example Inc.\n# Generated by CrawlForge — compliance: standard\n\nUser-Agent: *\nAllow: /\n\nContact: ai@example.com",
      "llms-full.txt": "# llms.txt for Example Inc.\n..."
    }
  },
  "credits_used": 5,
  "credits_remaining": 995,
  "processing_time": 4100
}

Field Descriptions

data.files可直接发布的各文件文本内容

data.compliance_level回显你请求的级别

credits_used无论爬取多少页面，每次调用固定 5 credits

credit 费用