User Agent

Web Scraping

Definition

A user agent is a string sent in HTTP request headers that identifies the client software making the request. Websites use it to detect browsers, bots, and scrapers.

How It Relates to CrawlForge

Every HTTP request includes a User-Agent header. Websites analyze this header to serve different content to different clients and to identify automated traffic. Using a default scraping library user agent is a quick way to get blocked.

CrawlForge rotates user agent strings automatically, matching them to real browser profiles. In stealth_mode, user agents are paired with consistent browser fingerprints to avoid detection by advanced anti-bot systems.

Related CrawlForge Tools

Related Terms

HTTP Headers

HTTP headers are key-value pairs sent with HTTP requests and responses that provide metadata about the communication. In scraping, headers like User-Agent, Accept, and Cookie are critical for successful requests.

Headless Browser

A headless browser is a web browser without a graphical user interface that can be controlled programmatically. It executes JavaScript and renders pages exactly like a regular browser, but runs in the background.

Proxy Rotation

Proxy rotation is the practice of cycling through multiple proxy IP addresses when making web requests. This distributes requests across different IPs to avoid rate limits and IP-based blocking.

CAPTCHA Solving

CAPTCHA solving refers to automated techniques for bypassing CAPTCHA challenges that websites use to distinguish humans from bots. This includes image recognition, token-based solving, and browser fingerprint emulation.

Start Scraping with 1,000 Free Credits

Get started with CrawlForge today. No credit card required.

Start scraping with 1,000 free credits