Data and API Terms

JSON-LD is embedded in web pages as a script tag and contains structured metadata about the page content -- product details, article information, organization data, and more. It is a rich source of clean, structured data for scrapers because it is already machine-readable.

CrawlForge extract_metadata detects and extracts JSON-LD from pages, giving you access to structured data that the site owner has already organized. This is often more reliable than parsing visual elements because JSON-LD is designed for machine consumption.

Structured data on websites includes JSON-LD, microdata, and RDFa annotations that describe the content in machine-readable format. Search engines use it for rich snippets, and scrapers can leverage it for reliable data extraction.

CrawlForge extract_metadata automatically detects and parses structured data from any page. This provides a shortcut for data extraction -- instead of writing CSS selectors for each field, you can often get product prices, ratings, availability, and more directly from the page's structured data.

Schema markup tells search engines exactly what your content means, not just what it says. A page with Product schema markup explicitly declares the name, price, availability, and reviews of a product, enabling rich search results.

For web scraping, schema markup is a goldmine. CrawlForge extract_metadata pulls schema markup from pages, providing structured product data, article metadata, and business information without the fragility of CSS selector-based extraction.

Many websites have APIs that serve the same data displayed on their pages, often in cleaner JSON format. Discovering and using these API endpoints can be more efficient and reliable than scraping the rendered HTML.

CrawlForge operates as a set of API endpoints itself -- each of the 31 tools is accessible via a REST endpoint. When scraping, fetch_url can access both web pages and API endpoints, returning the raw JSON response for programmatic processing.

Webhooks are commonly used to trigger actions when something changes -- a new order, a price update, or a content modification. They are the push-based counterpart to the pull-based approach of web scraping.

CrawlForge track_changes bridges the gap between scraping and webhooks. It monitors pages for changes and can notify your systems when content updates, giving you webhook-like real-time awareness without the target site needing to support webhooks.

REST APIs use HTTP methods like GET, POST, PUT, and DELETE to interact with resources identified by URLs. Most modern web services, including CrawlForge, expose functionality through REST endpoints.

CrawlForge's 31 tools are accessible via REST API endpoints in addition to MCP. This means you can integrate CrawlForge into any programming language or platform that can make HTTP requests, not just MCP-compatible AI clients.

Some websites use GraphQL APIs to power their frontend, which can be an efficient scraping target. GraphQL responses are already structured JSON, and you can request only the fields you need, reducing data transfer.

CrawlForge fetch_url can access GraphQL endpoints directly, sending queries and receiving structured responses. For sites that expose GraphQL, this approach is more efficient than scraping the rendered HTML.

JSON is the lingua franca of web data. API responses, configuration files, and structured data are almost universally encoded in JSON. When scraping, the goal is often to transform messy HTML into clean JSON that downstream systems can process.

CrawlForge tools return data in JSON format by default. Tools like scrape_structured and extract_structured let you define a JSON schema for the output, ensuring the extracted data matches your expected structure exactly.

Markdown preserves content structure (headings, lists, links, code blocks) while stripping away HTML complexity. This makes it an ideal output format for web scraping when you need readable, structured text rather than raw HTML or plain text.

CrawlForge extract_content supports markdown as an output format, converting web pages into clean markdown that preserves the document structure. This is particularly useful for content migration, documentation scraping, and feeding content to AI models that process markdown well.

HTML parsing is the core technical operation behind web scraping. Raw HTML from a web page must be parsed into a structured representation before any data can be extracted. The quality of the parser determines how well it handles malformed HTML, which is common on the web.

CrawlForge handles HTML parsing internally across all its tools, using robust parsers that handle real-world HTML gracefully. You never need to deal with parsing quirks yourself -- just specify what data you need and the tools return clean results.

SERP stands for search engine results page. A SERP API lets you check where a site ranks for a keyword, watch competitors, or feed search results into a pipeline without scraping the search engine yourself, which search engines actively block.

CrawlForge has two tools for this. serp_rank (5 credits) reports where a domain ranks in Google's organic results for a keyword, with the ranking URL and every position it holds, scanning up to 200 results for a chosen location and device. search_web (5 credits) returns the results themselves -- titles, URLs and snippets -- as JSON.

Modern JavaScript frameworks send a page's data to the browser twice: once as visible HTML and once as a JSON payload the framework uses to make the page interactive. That payload often holds exact values -- prices, stock levels, IDs, ratings -- in cleaner form than the rendered text.

CrawlForge's extract_embedded_state (2 credits) finds and returns this state from a single fetch: Next.js and React Server Components payloads, Nuxt, Apollo and Redux stores, and inline JSON blocks. It needs no LLM and runs none of the page's code, so values come back exactly as the site stores them.

Instead of asking for the whole page, you describe the fields you want -- product name, price, rating, availability -- with their types. The extractor returns exactly that shape for every URL, which makes the output easy to validate, store and compare across sites.

CrawlForge offers this two ways. When you know where the data sits, scrape_structured (2 credits) extracts it with CSS selectors. When you can describe a field but not locate it, extract_structured (3 credits) uses a language model to fill your JSON Schema.

When you already have the URLs -- a product list, search results, a sitemap -- sending them together lets the service fetch them in parallel and return the results as a set. Large batches usually run asynchronously, with results collected later or delivered by webhook.

CrawlForge's batch_scrape (5 credits) takes 2 to 50 URLs per call in sync or async mode, and get_batch_results (1 credit) pages through the output. To find the URLs first, map_site (2 credits) lists a site's pages.