Security and Data Handling
What leaves your machine, what we keep, for how long, and who we share it with. Every period on this page is the one the code enforces.
Last Updated: September 2026
Quick Navigation
How Your Data Flows
CrawlForge has two ways in. The MCP server runs on your machine and talks to websites directly. The REST API runs on our servers. They handle your data differently, so they are described separately.
MCP path: results stay on your machine
- •You install crawlforge-mcp-server on your machine. Your MCP client (Claude Desktop, Claude Code, Cursor and others) talks to it over stdio.
- •The server fetches target sites from your network. Page content, extracted data and screenshots are returned to your client and never pass through CrawlForge.
- •The server calls our API for three things: to validate your API key when it starts, to read your credit balance, and to report each call after it runs.
- •A usage report contains the tool name, credits used, status, duration, the server version and the call parameters with secrets masked. It never contains the result.
- •Tools that use a language model run it where you configure it: a local Ollama instance by default, or a provider you set an API key for. That traffic goes from your machine to that provider, not to us.
REST path: results pass through our servers and are returned, not kept
- •Your request reaches a serverless function on Vercel. The function fetches the target site, identified and signed as CrawlForge, builds the response and returns it to you.
- •Five tools (stealth_mode, scrape_with_actions, agent, extract_with_llm and list_ollama_models) run on a browser worker we operate on Render. The page content passes through that worker while the call runs.
- •Responses are not stored, with three time-limited exceptions listed in the retention table: asynchronous batch results, change-tracking baselines and oversized results held for read_result.
- •One usage row is written per call, with the same masked parameters as the MCP path, plus the endpoint and method.
- •The REST API runs no language model of its own. extract_with_llm and agent send text to the model endpoint configured on the Render worker.
What We Keep, and for How Long
This table is the retention schedule for CrawlForge. The privacy policy shows the same table. Every period comes from the code path that enforces it; where a period is set by a provider rather than by us, the table says so.
| Data | Where it is held | Kept for | Notes |
|---|---|---|---|
| Tool results on the MCP path (page content, extracted data, screenshots) | Your machine only | Never sent to CrawlForge | The MCP server fetches sites directly and reports only usage metadata to our API. |
| Tool results on the REST path | Returned in the HTTP response | Not stored | The three time-limited stores below are the only exceptions. |
| Batch results (asynchronous batch_scrape jobs, read back with get_batch_results) | Redis (Upstash) | 24 hours | Expire on their own. The job id is the only way to read them. |
| Change-tracking baselines (the page text captured by track_changes) | Redis (Upstash) | 90 days | Creating a new baseline for the same URL and selector replaces the old one and restarts the period. |
| Hosted monitor baselines (the last captured text of each monitored page) | PostgreSQL (Neon) | Until the monitor is deleted | Every check replaces the previous capture, so only the latest text of each target is held. Deleting the monitor deletes its baselines. |
| Hosted monitor checks (per-check status, counts, credits and the added and removed lines of each target) | PostgreSQL (Neon) | 30 days by default, 1 to 365 days set per monitor | The scheduled run deletes older checks; deleting the monitor deletes its history. |
| Oversized results held for read_result (a result too large to return inline) | Redis (Upstash) | 1 hour | Stored under your account. The result handle is the only way to read it. |
| Usage logs: tool name, masked parameters (including the target URL or query), status, duration, credits, API key id and client version | PostgreSQL (Neon) | 12 months | A daily job deletes older rows. Secrets are masked before a row is written and again before it is shown in the dashboard. |
| Account data: email, name, password hash, sign-in provider ids, settings, credit balance and feedback-survey answers | PostgreSQL (Neon) | While your account exists | Deleted with the account. See Account Deletion below. |
| API keys | PostgreSQL (Neon), as a salted SHA-256 hash plus the key prefix | Until you revoke them | The full key is shown once, when it is created. It is not stored and cannot be shown again. |
| Sign-in sessions | httpOnly cookies, with a verification cache in Redis (Upstash) | Access token 15 minutes, refresh token 7 days | Signing out clears both. The verification cache expires after 5 minutes. |
| Password-reset and email-verification tokens | PostgreSQL (Neon), hashed | Reset links 1 hour, verification links 24 hours | Single use. The link in the email is the only copy of the token. |
| Fraud-prevention data: device fingerprint, risk score and sign-up rate-limit counters | PostgreSQL (Neon) and Redis (Upstash) | Fingerprints and risk scores until account deletion; rate-limit counters 1 hour to 30 days | Used only to block abusive sign-ups and key creation. Never used for profiling or advertising. |
| Payment records | Stripe | As long as tax and accounting law requires | We store the Stripe customer and subscription ids, the plan and the billing period. Card numbers never reach our servers. |
| Newsletter subscription (email address) | PostgreSQL (Neon) | Until you unsubscribe | Every newsletter carries an unsubscribe link. |
| Deleted accounts | PostgreSQL (Neon) | Removed within 30 days of your request | Account, keys, usage logs, subscriptions, fingerprints, risk scores and tokens are deleted together. Redis entries expire on the periods above. Stripe keeps payment records as the law requires. |
- Data
- Tool results on the MCP path (page content, extracted data, screenshots)
- Where it is held
- Your machine only
- Kept for
- Never sent to CrawlForge
- Notes
- The MCP server fetches sites directly and reports only usage metadata to our API.
- Data
- Tool results on the REST path
- Where it is held
- Returned in the HTTP response
- Kept for
- Not stored
- Notes
- The three time-limited stores below are the only exceptions.
- Data
- Batch results (asynchronous batch_scrape jobs, read back with get_batch_results)
- Where it is held
- Redis (Upstash)
- Kept for
- 24 hours
- Notes
- Expire on their own. The job id is the only way to read them.
- Data
- Change-tracking baselines (the page text captured by track_changes)
- Where it is held
- Redis (Upstash)
- Kept for
- 90 days
- Notes
- Creating a new baseline for the same URL and selector replaces the old one and restarts the period.
- Data
- Hosted monitor baselines (the last captured text of each monitored page)
- Where it is held
- PostgreSQL (Neon)
- Kept for
- Until the monitor is deleted
- Notes
- Every check replaces the previous capture, so only the latest text of each target is held. Deleting the monitor deletes its baselines.
- Data
- Hosted monitor checks (per-check status, counts, credits and the added and removed lines of each target)
- Where it is held
- PostgreSQL (Neon)
- Kept for
- 30 days by default, 1 to 365 days set per monitor
- Notes
- The scheduled run deletes older checks; deleting the monitor deletes its history.
- Data
- Oversized results held for read_result (a result too large to return inline)
- Where it is held
- Redis (Upstash)
- Kept for
- 1 hour
- Notes
- Stored under your account. The result handle is the only way to read it.
- Data
- Usage logs: tool name, masked parameters (including the target URL or query), status, duration, credits, API key id and client version
- Where it is held
- PostgreSQL (Neon)
- Kept for
- 12 months
- Notes
- A daily job deletes older rows. Secrets are masked before a row is written and again before it is shown in the dashboard.
- Data
- Account data: email, name, password hash, sign-in provider ids, settings, credit balance and feedback-survey answers
- Where it is held
- PostgreSQL (Neon)
- Kept for
- While your account exists
- Notes
- Deleted with the account. See Account Deletion below.
- Data
- API keys
- Where it is held
- PostgreSQL (Neon), as a salted SHA-256 hash plus the key prefix
- Kept for
- Until you revoke them
- Notes
- The full key is shown once, when it is created. It is not stored and cannot be shown again.
- Data
- Sign-in sessions
- Where it is held
- httpOnly cookies, with a verification cache in Redis (Upstash)
- Kept for
- Access token 15 minutes, refresh token 7 days
- Notes
- Signing out clears both. The verification cache expires after 5 minutes.
- Data
- Password-reset and email-verification tokens
- Where it is held
- PostgreSQL (Neon), hashed
- Kept for
- Reset links 1 hour, verification links 24 hours
- Notes
- Single use. The link in the email is the only copy of the token.
- Data
- Fraud-prevention data: device fingerprint, risk score and sign-up rate-limit counters
- Where it is held
- PostgreSQL (Neon) and Redis (Upstash)
- Kept for
- Fingerprints and risk scores until account deletion; rate-limit counters 1 hour to 30 days
- Notes
- Used only to block abusive sign-ups and key creation. Never used for profiling or advertising.
- Data
- Payment records
- Where it is held
- Stripe
- Kept for
- As long as tax and accounting law requires
- Notes
- We store the Stripe customer and subscription ids, the plan and the billing period. Card numbers never reach our servers.
- Data
- Newsletter subscription (email address)
- Where it is held
- PostgreSQL (Neon)
- Kept for
- Until you unsubscribe
- Notes
- Every newsletter carries an unsubscribe link.
- Data
- Deleted accounts
- Where it is held
- PostgreSQL (Neon)
- Kept for
- Removed within 30 days of your request
- Notes
- Account, keys, usage logs, subscriptions, fingerprints, risk scores and tokens are deleted together. Redis entries expire on the periods above. Stripe keeps payment records as the law requires.
The two hosted-monitor rows apply only once you create a monitor. An account with no monitors holds no monitor data.
Subprocessors
These are the providers that receive your data when you use CrawlForge, and what each one receives. Target websites are not subprocessors: they receive our crawler’s request, identified as CrawlForge, and nothing about your account.
| Provider | Purpose | What it receives |
|---|---|---|
| Vercel | Hosting for the website, the dashboard and the REST API | Every request: IP address, headers and request body. |
| Neon | PostgreSQL database | Account data, API key hashes, usage logs, subscriptions and fraud-prevention data. |
| Upstash | Redis | Session verification cache, rate limits, credit reservations, batch results, change-tracking baselines, oversized results, webhook idempotency and email deduplication flags. |
| Render | Browser and language-model worker for five REST tools: stealth_mode, scrape_with_actions, agent, extract_with_llm and list_ollama_models | The target URL, the request parameters and the page content while the call runs. Text that extract_with_llm and agent send to a language model goes to the model endpoint configured on this worker. |
| Stripe | Payments and subscriptions | Email, name, the card details you enter on Stripe’s checkout page, and purchase history. |
| Resend | Transactional email | Email address, name and the content of each email: verification links, password resets, billing and credit alerts, and the newsletter. |
| Sentry | Error monitoring | Stack traces and request metadata. Cookies, authorization headers, API keys, tokens and passwords are removed before an event is sent. |
| Google, GitHub | Sign in with Google or GitHub, only if you choose it | We receive your email, name and avatar from the provider. The provider learns that you signed in to CrawlForge. |
| Google Custom Search, DataForSEO | Search results for search_web and serp_rank | The search query, and for serp_rank the domain being ranked. |
| Arctic Shift, PullPush, reddit.com | Reddit archives for reddit_search | The search query. |
- Provider
- Vercel
- Purpose
- Hosting for the website, the dashboard and the REST API
- What it receives
- Every request: IP address, headers and request body.
- Provider
- Neon
- Purpose
- PostgreSQL database
- What it receives
- Account data, API key hashes, usage logs, subscriptions and fraud-prevention data.
- Provider
- Upstash
- Purpose
- Redis
- What it receives
- Session verification cache, rate limits, credit reservations, batch results, change-tracking baselines, oversized results, webhook idempotency and email deduplication flags.
- Provider
- Render
- Purpose
- Browser and language-model worker for five REST tools: stealth_mode, scrape_with_actions, agent, extract_with_llm and list_ollama_models
- What it receives
- The target URL, the request parameters and the page content while the call runs. Text that extract_with_llm and agent send to a language model goes to the model endpoint configured on this worker.
- Provider
- Stripe
- Purpose
- Payments and subscriptions
- What it receives
- Email, name, the card details you enter on Stripe’s checkout page, and purchase history.
- Provider
- Resend
- Purpose
- Transactional email
- What it receives
- Email address, name and the content of each email: verification links, password resets, billing and credit alerts, and the newsletter.
- Provider
- Sentry
- Purpose
- Error monitoring
- What it receives
- Stack traces and request metadata. Cookies, authorization headers, API keys, tokens and passwords are removed before an event is sent.
- Provider
- Google, GitHub
- Purpose
- Sign in with Google or GitHub, only if you choose it
- What it receives
- We receive your email, name and avatar from the provider. The provider learns that you signed in to CrawlForge.
- Provider
- Google Custom Search, DataForSEO
- Purpose
- Search results for search_web and serp_rank
- What it receives
- The search query, and for serp_rank the domain being ranked.
- Provider
- Arctic Shift, PullPush, reddit.com
- Purpose
- Reddit archives for reddit_search
- What it receives
- The search query.
Our primary infrastructure is in the United States. We add a provider to this list before it receives customer data.
Crawler Identity and Site-Owner Controls
Both surfaces identify themselves the same way and follow the same rules on every fetch. Site owners can verify us, rate-limit us or block us without guessing.
- •Every outbound request carries the User-Agent CrawlForge/<version> (+https://crawlforge.dev). The product token is CrawlForge on both the MCP server and the REST API.
- •Every outbound request is signed with Ed25519 under RFC 9421 (the Web Bot Auth profile). The public key directory is at /.well-known/http-message-signatures-directory, so a site can verify that a request claiming to be CrawlForge really is.
- •robots.txt is honoured by default on both surfaces. A caller may pass respect_robots: false; the override is explicit, returns a warning in the response and is recorded against the API key in the usage log.
- •We do not solve CAPTCHAs or forge challenge tokens. A challenge page is reported as blocked, not bypassed. stealth_mode makes a real browser render like a real browser; it does not defeat access controls.
- •A site owner who opts out is added to a permanent blocklist that applies on both surfaces and cannot be overridden by any customer. Write to simon@crawlforge.dev to opt out.
- •The retired token CrawlForge-Bot is still honoured as a disallow rule, so rules site owners wrote before the rename keep working.
Credentials and Encryption
What we do with the secrets you give us, and what we do not claim.
- •All traffic to crawlforge.dev uses TLS, and the site sends a Strict-Transport-Security header.
- •Passwords are hashed with bcrypt. We never see or store the password itself.
- •API keys are hashed with SHA-256 and a server-side salt before they are stored, together with the key prefix shown in the dashboard. The full key exists in plain text once, in the response that creates it. We cannot recover it, so a lost key is revoked and replaced, not retrieved.
- •Sign-in sessions use a 15-minute JWT and a 7-day refresh token, both in httpOnly, secure, same-site cookies. State-changing dashboard requests carry a CSRF token.
- •Password-reset and email-verification links are single-use tokens stored as hashes with an expiry of 1 hour and 24 hours respectively.
- •Secrets in your requests, such as a password typed by scrape_with_actions, proxy credentials or an authorization header, are masked before a usage row is written and again before it is shown in the dashboard.
- •Neon and Upstash encrypt stored data at rest, and every connection to them is TLS.
- •What we do not claim: CrawlForge holds no SOC 2 or ISO 27001 certification, does not offer multi-factor authentication on accounts yet, and does not commission third-party penetration tests. If your procurement needs one of these, ask us where it stands before you buy.
Account Deletion
Deletion is by request for now. The dashboard does not offer a self-service delete yet.
- •Email simon@crawlforge.dev from the address on your account with the subject "Delete my account". We confirm by reply before anything is removed.
- •Cancel any active subscription first, or ask us to cancel it as part of the deletion. Unused credits are not refunded.
- •Within 30 days we delete the account row, and with it your API keys, usage logs, subscription records, device fingerprints, risk scores and tokens. Entries in Redis expire on the periods in the retention table, 90 days at most.
- •Stripe keeps payment records for as long as tax and accounting law requires. We keep nothing that identifies you after the deletion completes.
- •If you subscribed to the newsletter, say so in the request and the subscription is removed at the same time.
Vulnerability Disclosure
If you find a security issue in the website, the API, the crawlforge-mcp-server package or the SDKs, tell us directly. Details are also published at /.well-known/security.txt.
- •Email simon@crawlforge.dev with the subject "Security". Include the affected URL or tool, steps to reproduce and the impact you observed.
- •We acknowledge reports within five business days and keep you informed until the issue is fixed. We do not run a paid bug bounty.
- •Test only against your own account and API keys. Do not access, modify or delete other customers’ data.
- •No denial of service, no volume scanning, no social engineering of people who work on CrawlForge, and no testing against the target websites our tools fetch.
- •Stop as soon as you can demonstrate the issue and report it. Do not keep the data you reached.
- •We ask for reasonable time to fix an issue before it is published, and we will credit you in the fix if you want that.
Research that follows these rules is authorised. We will not pursue legal action against it and we will not report it to law enforcement.
Related Policies
Privacy Policy
What we collect, why, and your rights under GDPR and CCPA
Acceptable Use Policy
Guidelines for ethical and legal use of our web scraping tools
Verifying the CrawlForge Crawler
User-Agent pattern, Web Bot Auth signatures and robots.txt rules for site owners
Untrusted Content and Prompt Injection
The trust model for page text that reaches your agent, and what we deliberately do not sanitise
Questions About Security?
If you need something this page does not answer, such as a data-processing agreement or a description of a control before you buy, write to us.
simon@crawlforge.dev