map_site
Enumerate a site's pages the cheap way first: if the origin serves a sitemap.xml it is read directly, and only when there isn't a usable one does the tool fall back to crawling. The response tells you which path it took.
Use Cases
Get a URL list before scraping
Enumerate first, then feed the list to batch_scrape — far cheaper than crawling with extraction turned on.
Check what a site publishes to search engines
Sitemap mode reports exactly what the site advertises, which is often not the same as what is reachable by following links.
Find orphaned or unlinked pages
Compare the sitemap list against a crawl_deep run: pages in the sitemap that the crawl never reached are unlinked.
Size a site before committing credits
total_pages tells you how large a job would be before you start paying per-page for extraction.
Endpoint
/api/v1/tools/map_siteParameters
max_depth and include_external apply only to the crawl fallback. When a usable sitemap.xml is found they are ignored, because no crawling happens.| Name | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | Required | - | Any URL on the site. Its origin is used to look for `/sitemap.xml`, and it is the starting point if the crawl fallback runs. Example: https://example.com |
max_depth | number | Optional | 2 | Crawl depth for the fallback, 1-5. Ignored when a sitemap is found. Example: 2 |
include_external | boolean | Optional | false | Include external links in the per-page link lists. Crawl mode only — external pages are counted but never visited. Example: false |
timeout | number | Optional | 15000 | Overall budget in milliseconds, 1000-30000. Capped near 18000 in practice to fit the serverless execution limit. Example: 15000 |
respect_robots | boolean | Optional | true | Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true |
Two ways it maps a site
source in the response tells you which one ran — the two modes return the same keys but populate them differently.
{origin}/sitemap.xml, following up to 3 child sitemaps, capped at 500 URLs. Fast and complete. max_depth_reached and sitemap come back null because nothing was crawled.sitemap is then populated with the links found on each page.sitemap field is not a sitemap. In crawl mode it holds a map of each crawled page to the links found on it; in sitemap mode it is null. The enumerated URLs are always in pages./sitemap.xml with a 200 status is detected and rejected, so it falls through to crawl mode rather than reporting a single bogus page.Request Examples
curl -X POST https://crawlforge.dev/api/v1/tools/map_site \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"max_depth": 2,
"include_external": false,
"timeout": 15000
}'Response Example
{ "success": true, "data": { "base_url": "https://example.com", "source": "sitemap", "total_pages": 128, "internal_links": 128, "external_links": 0, "pages": [ "https://example.com/", "https://example.com/pricing", "https://example.com/docs" ], "max_depth_reached": null, "sitemap": null }, "credits_used": 2, "credits_remaining": 998, "processing_time": 1240}data.sourceEither "sitemap" or "crawl" — which strategy produced this result.data.total_pagesNumber of URLs in the pages array.data.internal_linksDistinct internal links found. In sitemap mode this mirrors the page count.data.external_linksDistinct external links. Always 0 in sitemap mode, since no page bodies are read.data.pagesThe enumerated URLs — this is the list you want.data.max_depth_reachedDeepest crawl level reached, or null in sitemap mode where nothing was crawled.data.sitemapCrawl mode only: each crawled page mapped to the links found on it. Null in sitemap mode.Error Handling
Nothing could be mapped (422 NO_PAGES_MAPPED)
There was no usable sitemap and the start URL did not return an HTML page to crawl. Common when the URL points at a PDF, an image, or an API endpoint. No credits are charged.
Target returned an error (502 TARGET_HTTP_ERROR)
The site responded with a non-2xx status. Sites behind bot protection usually land here — try stealth_mode instead.
Invalid URL (400 VALIDATION_ERROR)
The url was malformed, used a scheme other than http/https, or resolved to a private address. max_depth must be 1-5 and timeout 1000-30000.
Target too slow (504 FETCH_TIMEOUT)
The site did not respond within the budget. Raise timeout, though it is capped near 18000ms.
Blocked by robots.txt (403 Forbidden)
The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to. Only the entry URL 403s: a URL discovered during the walk that robots.txt disallows is left out of the map instead, and named in warnings.
crawl_deep and compare the two lists.Credit Cost
Cost Breakdown:
Any map, sitemap or crawl mode: 2 credits
Plan Recommendations:
Free Plan: 1,000 one-time trial credits = 500 site maps
Hobby Plan: 5,000 credits/mo = 2,500 site maps ($19/mo)
Professional Plan: 50,000 credits/mo = 25,000 site maps ($99/mo)
Mapping is the cheapest way to size a job — enumerate first, then spend per-page credits only on the URLs you actually want.