Web Scraping
Definition
Web scraping is the automated extraction of data from websites. It involves programmatically fetching web pages and parsing their content to collect structured information.
Web scraping is the foundation of what CrawlForge provides through the Model Context Protocol. Instead of manually copying data from websites, CrawlForge tools like fetch_url and extract_content handle the entire process -- fetching pages, handling JavaScript rendering, and returning clean structured data.
For AI agents, web scraping is essential for accessing real-time information that falls outside their training data. CrawlForge makes this accessible through a single MCP interface, eliminating the need to build and maintain custom scraping infrastructure.
Related Terms: Web Crawler, DOM Parsing, Headless Browser, Dynamic Content
Web Crawler
Definition
A web crawler is a program that systematically browses the web by following links from page to page. Crawlers discover and index content across entire websites or domains.
Web crawlers are distinct from scrapers in that they focus on discovery -- finding all the pages on a site rather than extracting specific data from a single page. CrawlForge provides crawl_deep for following internal links to a specified depth and map_site for generating a complete URL inventory of a domain.
These tools are critical for use cases like content migration, SEO auditing, and building comprehensive datasets where you need to process every page on a site rather than just known URLs.
Related Terms: Web Scraping, Sitemap, Robots.txt, Pagination
DOM Parsing
Definition
DOM parsing is the process of converting raw HTML into a structured Document Object Model tree. This tree representation allows programs to navigate and extract specific elements from a web page.
When CrawlForge fetches a web page, it parses the DOM to understand the page structure before extracting content. This is what enables tools like extract_structured to pull specific data fields based on CSS selectors or schema definitions.
DOM parsing is particularly important for dynamic content where the initial HTML differs from what you see in a browser. CrawlForge handles this by rendering pages in headless browsers when needed, ensuring the parsed DOM matches what a real user would see.
Related Terms: CSS Selector, XPath, HTML Parsing, Dynamic Content
CSS Selector
Definition
A CSS selector is a pattern used to select and target specific HTML elements on a web page. In web scraping, selectors identify exactly which data to extract from a page's structure.
CSS selectors are the primary way to tell a scraper which elements to extract. For example, ".product-price" targets elements with the class "product-price", while "h1.title" targets h1 elements with the "title" class. CrawlForge tools like scrape_structured accept CSS selectors to pinpoint exactly the data you need.
Using precise selectors is key to building reliable scraping pipelines. CrawlForge supports the full range of CSS selector syntax, making it possible to target deeply nested or dynamically generated elements.
Related Terms: XPath, DOM Parsing, HTML Parsing, Structured Data
XPath
Definition
XPath (XML Path Language) is a query language for selecting nodes from an XML or HTML document. It provides a more powerful and flexible way to navigate document trees than CSS selectors alone.
XPath expressions can navigate up, down, and across the document tree, making them useful for complex extraction scenarios. For example, you can select a price element based on its sibling text content -- something CSS selectors cannot do.
CrawlForge supports XPath alongside CSS selectors in its extraction tools. XPath is particularly valuable when scraping legacy sites with poorly structured HTML or when you need to extract data based on text content rather than class names.
Related Terms: CSS Selector, DOM Parsing, HTML Parsing, Structured Data
Headless Browser
Definition
A headless browser is a web browser without a graphical user interface that can be controlled programmatically. It executes JavaScript and renders pages exactly like a regular browser, but runs in the background.
Many modern websites rely heavily on JavaScript to render content. A simple HTTP request only retrieves the initial HTML, missing any content loaded dynamically. Headless browsers solve this by fully rendering the page, including executing JavaScript, loading AJAX requests, and processing CSS.
CrawlForge uses headless browsers behind the scenes for tools like stealth_mode and scrape_with_actions. This means you get fully rendered page content without managing browser infrastructure yourself.
Related Terms: Dynamic Content, Web Scraping, CAPTCHA Solving, User Agent
Proxy Rotation
Definition
Proxy rotation is the practice of cycling through multiple proxy IP addresses when making web requests. This distributes requests across different IPs to avoid rate limits and IP-based blocking.
Websites detect and block scrapers by monitoring IP addresses. If too many requests come from a single IP, the site blocks it. Proxy rotation solves this by routing each request through a different IP address, making the traffic appear to come from many different users.
CrawlForge handles proxy rotation automatically through stealth_mode, which uses residential proxies and intelligent rotation strategies. This is essential for competitive intelligence and large-scale data collection where sites actively defend against scraping.
Related Terms: Rate Limiting, CAPTCHA Solving, User Agent
Rate Limiting
Definition
Rate limiting is a technique used by websites and APIs to control the number of requests a client can make within a given time period. It prevents server overload and defends against abusive scraping.
Responsible web scraping requires respecting rate limits. Making too many requests too quickly can overwhelm a server and get your IP permanently banned. Rate limiting is also a common anti-bot measure that returns 429 (Too Many Requests) HTTP status codes.
CrawlForge tools automatically handle rate limiting by throttling requests and implementing exponential backoff when limits are hit. This means your scraping jobs complete reliably without manual intervention to manage request timing.
Related Terms: Proxy Rotation, Robots.txt, HTTP Headers, CAPTCHA Solving
CAPTCHA Solving
Definition
CAPTCHA solving refers to automated techniques for bypassing CAPTCHA challenges that websites use to distinguish humans from bots. This includes image recognition, token-based solving, and browser fingerprint emulation.
CAPTCHAs are one of the most common anti-bot defenses. They range from simple text challenges to complex image puzzles and invisible behavioral analysis. Bypassing them is often necessary for legitimate scraping use cases like price monitoring and competitive analysis.
CrawlForge stealth_mode includes CAPTCHA handling as part of its anti-detection capabilities. By combining browser fingerprint randomization with residential proxies, it reduces the likelihood of triggering CAPTCHAs in the first place.
Related Terms: Headless Browser, Proxy Rotation, User Agent
Robots.txt
Definition
Robots.txt is a standard text file placed at the root of a website that tells web crawlers which pages they are allowed or disallowed from accessing. It is part of the Robots Exclusion Protocol.
The robots.txt file acts as a set of guidelines for crawlers. While not legally binding, respecting it is considered standard practice for ethical scraping. It specifies which paths are off-limits, crawl delay preferences, and links to XML sitemaps.
CrawlForge tools respect robots.txt directives by default. When using crawl_deep or map_site, the crawler checks robots.txt before accessing pages, ensuring your scraping activity stays within the site owner's stated preferences.
Related Terms: Web Crawler, Sitemap, Rate Limiting, User Agent
Sitemap
Definition
A sitemap is an XML file that lists all the URLs on a website, along with metadata like last modification date and priority. It helps search engines and crawlers discover and index all pages efficiently.
Sitemaps provide a complete inventory of a website's pages without needing to discover them by following links. This makes them invaluable for comprehensive scraping, SEO audits, and content migration where you need to process every page.
CrawlForge map_site generates sitemaps for any domain, discovering URLs through both link-following and existing sitemap files. This gives you a reliable starting point for batch operations with batch_scrape.
Related Terms: Web Crawler, Robots.txt, SEO Audit, Schema Markup
User Agent
Definition
A user agent is a string sent in HTTP request headers that identifies the client software making the request. Websites use it to detect browsers, bots, and scrapers.
Every HTTP request includes a User-Agent header. Websites analyze this header to serve different content to different clients and to identify automated traffic. Using a default scraping library user agent is a quick way to get blocked.
CrawlForge rotates user agent strings automatically, matching them to real browser profiles. In stealth_mode, user agents are paired with consistent browser fingerprints to avoid detection by advanced anti-bot systems.
Related Terms: HTTP Headers, Headless Browser, Proxy Rotation, CAPTCHA Solving
Dynamic Content
Definition
Dynamic content is web content that is loaded or generated by JavaScript after the initial page load. This includes single-page applications, AJAX-loaded data, and client-side rendered content.
A growing number of websites use JavaScript frameworks like React, Vue, or Angular to render content on the client side. A simple HTTP GET request to these pages returns an empty HTML shell, because the actual content is injected by JavaScript after the page loads.
CrawlForge handles dynamic content by rendering pages in headless browsers. Tools like extract_content and scrape_with_actions wait for JavaScript execution to complete before extracting data, ensuring you get the same content a real user would see.
Related Terms: Headless Browser, DOM Parsing, Web Scraping, Pagination
Start Scraping with 1,000 Free Credits
Get started with CrawlForge today. No credit card required.
Start scraping with 1,000 free credits