Web Scraping Terms

Web scraping is the foundation of what CrawlForge provides through the Model Context Protocol. Instead of manually copying data from websites, CrawlForge tools like fetch_url and extract_content handle the entire process -- fetching pages, handling JavaScript rendering, and returning clean structured data.

For AI agents, web scraping is essential for accessing real-time information that falls outside their training data. CrawlForge makes this accessible through a single MCP interface, eliminating the need to build and maintain custom scraping infrastructure.

Web crawlers are distinct from scrapers in that they focus on discovery -- finding all the pages on a site rather than extracting specific data from a single page. CrawlForge provides crawl_deep for following internal links to a specified depth and map_site for generating a complete URL inventory of a domain.

These tools are critical for use cases like content migration, SEO auditing, and building comprehensive datasets where you need to process every page on a site rather than just known URLs.

When CrawlForge fetches a web page, it parses the DOM to understand the page structure before extracting content. This is what enables tools like extract_structured to pull specific data fields based on CSS selectors or schema definitions.

DOM parsing is particularly important for dynamic content where the initial HTML differs from what you see in a browser. CrawlForge handles this by rendering pages in headless browsers when needed, ensuring the parsed DOM matches what a real user would see.

CSS selectors are the primary way to tell a scraper which elements to extract. For example, ".product-price" targets elements with the class "product-price", while "h1.title" targets h1 elements with the "title" class. CrawlForge tools like scrape_structured accept CSS selectors to pinpoint exactly the data you need.

Using precise selectors is key to building reliable scraping pipelines. CrawlForge supports the full range of CSS selector syntax, making it possible to target deeply nested or dynamically generated elements.

XPath expressions can navigate up, down, and across the document tree, making them useful for complex extraction scenarios. For example, you can select a price element based on its sibling text content -- something CSS selectors cannot do.

CrawlForge supports XPath alongside CSS selectors in its extraction tools. XPath is particularly valuable when scraping legacy sites with poorly structured HTML or when you need to extract data based on text content rather than class names.

Many modern websites rely heavily on JavaScript to render content. A simple HTTP request only retrieves the initial HTML, missing any content loaded dynamically. Headless browsers solve this by fully rendering the page, including executing JavaScript, loading AJAX requests, and processing CSS.

CrawlForge uses headless browsers behind the scenes for tools like stealth_mode and scrape_with_actions. This means you get fully rendered page content without managing browser infrastructure yourself.

Websites detect and block scrapers by monitoring IP addresses. If too many requests come from a single IP, the site blocks it. Proxy rotation solves this by routing each request through a different IP address, making the traffic appear to come from many different users.

CrawlForge rotates through the proxies you supply: pass a list in stealth_mode's proxyRotation setting and each stealth context takes the next address. CrawlForge does not operate a proxy pool of its own. This is essential for competitive intelligence and large-scale data collection where sites actively defend against scraping.

Responsible web scraping requires respecting rate limits. Making too many requests too quickly can overwhelm a server and get your IP permanently banned. Rate limiting is also a common anti-bot measure that returns 429 (Too Many Requests) HTTP status codes.

CrawlForge tools automatically handle rate limiting by throttling requests and implementing exponential backoff when limits are hit. This means your scraping jobs complete reliably without manual intervention to manage request timing.

CAPTCHAs are one of the most common anti-bot defenses. They range from simple text challenges to complex image puzzles and invisible behavioral analysis. Bypassing them is often necessary for legitimate scraping use cases like price monitoring and competitive analysis.

CrawlForge does not solve CAPTCHAs or forge challenge tokens. stealth_mode renders in a real browser with fingerprint randomization, which lowers the chance of a challenge appearing at all, and a challenge page is reported as blocked rather than bypassed.

The robots.txt file acts as a set of guidelines for crawlers. While not legally binding, respecting it is considered standard practice for ethical scraping. It specifies which paths are off-limits, crawl delay preferences, and links to XML sitemaps.

CrawlForge tools respect robots.txt directives by default. When using crawl_deep or map_site, the crawler checks robots.txt before accessing pages, ensuring your scraping activity stays within the site owner's stated preferences.

Sitemaps provide a complete inventory of a website's pages without needing to discover them by following links. This makes them invaluable for comprehensive scraping, SEO audits, and content migration where you need to process every page.

CrawlForge map_site generates sitemaps for any domain, discovering URLs through both link-following and existing sitemap files. This gives you a reliable starting point for batch operations with batch_scrape.

Every HTTP request includes a User-Agent header. Websites analyze this header to serve different content to different clients and to identify automated traffic. Using a default scraping library user agent is a quick way to get blocked.

CrawlForge rotates user agent strings automatically, matching them to real browser profiles. In stealth_mode, user agents are paired with consistent browser fingerprints to avoid detection by advanced anti-bot systems.

Proper HTTP headers make the difference between a successful scrape and a blocked request. Anti-bot systems check for missing or inconsistent headers as a signal of automated traffic. A real browser sends dozens of headers; a naive scraper might send only a few.

CrawlForge automatically sends realistic header sets with every request. Tools like fetch_url and stealth_mode include complete header profiles that match real browser behavior, reducing the chance of detection.

Most websites break large datasets into paginated lists -- search results, product catalogs, article archives. A scraper that only fetches the first page misses the majority of the data. Pagination can be URL-based (?page=2), button-based, or use infinite scroll.

CrawlForge crawl_deep follows pagination links automatically, and scrape_with_actions can handle JavaScript-based pagination like infinite scroll and "Load More" buttons. This ensures complete data collection without manual page management.

A growing number of websites use JavaScript frameworks like React, Vue, or Angular to render content on the client side. A simple HTTP GET request to these pages returns an empty HTML shell, because the actual content is injected by JavaScript after the page loads.

CrawlForge handles dynamic content by rendering pages in headless browsers. Tools like extract_content and scrape_with_actions wait for JavaScript execution to complete before extracting data, ensuring you get the same content a real user would see.

A scraping API moves the hard parts -- fetching, JavaScript rendering, retries and parsing -- off your servers. You send a URL and get back markdown, HTML, links or structured fields, usually billed per request.

CrawlForge works this way over both REST and MCP. fetch_url (1 credit) returns the raw response; scrape (2 credits) returns markdown, HTML, text, links and metadata from a single page load; batch_scrape (5 credits) handles up to 50 URLs in one call. AI agents reach the same 31 tools through the MCP server, so an agent and your backend code make identical calls.

Detection runs in layers. The network layer checks the IP address and its reputation before any page loads, and services like Cloudflare and DataDome score datacenter addresses harshly. The browser layer inspects the fingerprint -- headers, fonts, WebGL, the navigator.webdriver flag -- and how the page is driven. A failed check returns a 403 or 429, a CAPTCHA, or an empty shell page.

CrawlForge's stealth_mode (5 credits) renders pages with a randomized browser fingerprint, and scrape can retry a blocked fetch in the stealth browser when you set escalate: true. It does not solve CAPTCHAs or get past logins and paywalls, and the hosted API exits from a datacenter address, so sites that score the network first may still challenge it. robots.txt is honoured on every request.

Sites trust residential addresses more than datacenter ranges, so residential proxies are used to reach pages that block cloud servers, or to view a page as it appears in another country. They cost more than datacenter proxies and are usually billed by bandwidth.

CrawlForge does not run a residential proxy pool. If a target needs one, you can pass your own residential or mobile proxy to stealth_mode (5 credits) through stealthConfig.proxyRotation on the default Chromium engine, and every stealth context in that call uses it. Without a proxy, CrawlForge's requests leave from its own hosted addresses.

What a page shows can depend on the visitor's IP location, the Accept-Language header, or a country chosen on the site. Search results vary most: the same keyword ranks differently in London and New York, and on mobile and desktop.

CrawlForge's serp_rank (5 credits) checks Google rankings for a named location and device, and search_web (5 credits) can restrict results to one language. To request a page in a given language, pass an Accept-Language header to fetch_url. The localization tool (2 credits) audits a page's hreflang, declared language and geo tags, but it does not route the request through another country: a block based on the visitor's IP still applies unless you supply your own proxy to stealth_mode.