Is Web Scraping Legal in 2026? The EDPB's New AI Rules

On 7 July 2026, robots.txt stopped being etiquette.

That was the day the European Data Protection Board adopted Guidelines 03/2026 on web scraping in the context of generative AI — the first comprehensive GDPR framework for large-scale scraping of publicly available data to train generative models. Its balancing-test section carries a position with direct engineering consequences: robots.txt, ai.txt, CAPTCHAs and login walls are indicators of what data subjects reasonably expect. Ignore them at scale and the argument is no longer about terms of service. It is about Article 6(1)(f).

Every page ranking for "is web scraping legal" is a law firm writing for other lawyers. This is the version for the person who writes the crawler.

This is not legal advice. It is an engineering reading of a published regulatory document. Guidelines 03/2026 is in public consultation until the end of October 2026 and the final version is not expected before the end of 2026. Talk to counsel before shipping a pipeline that touches EU personal data.

Table of contents

• Is web scraping legal in 2026?
• What changed on 7 July 2026?
• Does robots.txt have legal force now?
• Which legal basis can you actually rely on?
• Targeted scraping or untargeted crawling?
• What does data minimisation mean in code?
• What if you scrape special category data?
• Do you have to tell people you scraped them?
• Are you the controller or the processor?
• How does the US position differ?
• What should scraper developers do differently?
• Where CrawlForge fits

Is web scraping legal in 2026?

Yes, with conditions. Fetching a publicly reachable page is not itself unlawful in the EU or the US. Legality attaches to what you collect, why, and what happens next — and in the EU, any scrape touching personal data is a processing operation that needs a lawful basis before the first request goes out.

The two regimes ask different questions. The EU asks what your legal basis is and whether you can evidence it; the US has no federal scraping statute, so contract, copyright and state privacy law do that work.

The word doing the most damage here is public. "Publicly available" is not a GDPR category. A name on a conference agenda, a comment under a blog post, a maintainer's email in a commit — all personal data, all in scope. The EDPB's position is that internet content involves personal data "virtually always", and mixed datasets do not escape: the personal-data portion stays subject to the GDPR.

What changed on 7 July 2026?

Before July, EU scraping compliance was assembled by analogy from GDPR guidance. Guidelines 03/2026 replaces that with a framework aimed at generative AI training.

The wording can still move, so do not hard-code it into a compliance program. But the obligations it interprets are existing law, enforceable now — waiting for the final text buys you nothing.

Does robots.txt have legal force now?

Not directly. robots.txt is a convention, not a statute, and skipping it is not an offence in itself. What changed is its evidentiary role: the EDPB treats robots.txt, ai.txt, CAPTCHAs and login walls as indicators of data subjects' reasonable expectations, one of the inputs to the legitimate interest balancing test. Where a site deploys those signals and states its data may not be used for AI training, the EDPB's position is that data subjects have no reasonable expectation of being scraped.

Ignoring a Disallow used to be a terms-of-service and copyright question you could argue about later. It now weighs directly against you in the test your entire legal basis rests on.

Two things get merged here that should not be. Our stealth mode guide covers browser fingerprinting on pages already public to any visitor. A login wall is a different act, and since July 2026 it carries GDPR weight on top of the contract exposure it always had.

Which legal basis can you actually rely on?

Consent under Article 6(1)(a) is "generally not a viable legal basis" for scraping at scale, in the EDPB's words: you cannot obtain valid, specific, freely given consent from millions of people whose pages you crawled. That leaves legitimate interest under Article 6(1)(f) — a three-part cumulative test, so failing one part fails all of it.

Step 2 is the one engineering teams underestimate. "We needed the whole web" is not a necessity argument. Narrower crawl criteria, synthetic data and pseudonymisation at ingest are all named as less intrusive alternatives, and if you never evaluated them you have no record that you tried.

Targeted scraping or untargeted crawling?

The guidelines separate targeted scraping — restrictive criteria, specific domains or topics — from untargeted crawling that follows links without restriction. Untargeted carries higher compliance risk, because a crawler with no boundary cannot show its collection was necessary.

That makes crawl configuration a compliance artifact. Your include patterns, exclusions and page caps are the precisely defined criteria the necessity test asks about, so they belong in version control.

import { Client } from '@modelcontextprotocol/sdk/client/index.js';

const client = new Client({ name: 'corpus-builder', version: '1.0.0' });

interface SourceRecord {
  url: string;
  domain: string;
  collectedAt: string; // ISO timestamp — required for the public source list
}

// Targeted crawl: named domain, explicit include patterns, hard page cap.
const result = await client.callTool({
  name: 'crawl_deep',
  arguments: {
    url: 'https://docs.python.org/3/',
    max_pages: 500,
    respect_robots: true, // default; turning it off is a decision you must justify
    include_patterns: ['/3/library/', '/3/tutorial/'],
    exclude_patterns: ['/genindex', '/search'],
  },
});

const payload = result.content as Array<{ text: string }>;
const crawled = JSON.parse(payload[0].text) as { pages: Array<{ url: string }> };

// Article 14 expects a source list that is as complete as possible, ideally
// with collection dates. That makes provenance a schema requirement.
const provenance: SourceRecord[] = crawled.pages.map((page) => ({
  url: page.url,
  domain: new URL(page.url).hostname,
  collectedAt: new Date().toISOString(),
}));

What does data minimisation mean in code?

The guidelines prescribe minimisation at three points, and each maps to a place in your stack.

Before collection. Precise criteria, structurally sensitive site categories excluded outright, robots.txt and ai.txt respected, synthetic data considered. Seed-list and crawl-config work.

During and after collection. Syntax-based filters for identifiers — social security numbers are the EDPB's own example — then pseudonymisation or anonymisation. This belongs in the ingest path, before anything reaches durable storage.

// Syntax filters run before anything reaches persistent storage. They are
// deliberately coarse: they trade recall for a minimisation record you can show.
const IDENTIFIER_PATTERNS: Array<{ label: string; pattern: RegExp }> = [
  { label: 'us-ssn', pattern: /\b\d{3}-\d{2}-\d{4}\b/g },
  { label: 'email', pattern: /\b[\w.+-]+@[\w-]+\.[\w.]{2,}\b/g },
];

interface RedactionResult {
  text: string;
  hits: Record<string, number>;
}

export function redactIdentifiers(input: string): RedactionResult {
  const hits: Record<string, number> = {};
  let text = input;

  for (const { label, pattern } of IDENTIFIER_PATTERNS) {
    const matches = text.match(pattern);
    if (matches) {
      hits[label] = matches.length;
      text = text.replace(pattern, '[REDACTED:' + label + ']');
    }
  }

  // Keep the counts. They are accountability evidence, not debug output.
  return { text, hits };
}

Regex filters miss plenty: names, addresses and free-text disclosures sail through. Document that rather than claiming the corpus is clean — the point is demonstrable reduction, not perfection.

What if you scrape special category data?

Article 9 data — health, political opinions, ethnicity — turns up incidentally in any broad crawl. The EDPB's position is that incidental collection is not automatically unlawful, but survives only with safeguards across the whole model lifecycle. The guidelines anchor this in the CJEU's ruling in GC and Others (C-136/17).

Those are four different owners. A compliance story that stops at ingest covers a quarter of what is required.

Do you have to tell people you scraped them?

Individually, usually not. Article 14(5)(b) excuses direct notice where it would involve disproportionate effort, and notifying every person represented in a web-scale corpus qualifies. It is not a free pass: you must still publish a public privacy notice covering data categories, purposes, legal basis, and a source list that is as complete as possible — ideally searchable domain names with collection dates. Pre-collection opt-out mechanisms are named as good practice.

That last requirement is a schema decision disguised as a legal one. If the crawler does not record source domain and collection date per record, you cannot produce that page later, and reconstructing provenance across hundreds of millions of documents is not realistic. Two columns at write time, as in SourceRecord above.

Are you the controller or the processor?

The entity running the scraper is not automatically the controller, and the distinction decides who carries everything above.

• Scraping to documented instructions — targets, criteria and purposes set by a client — looks like processing. Get the instructions in writing; that is what makes the role defensible.
• Choosing your own targets and purposes makes you the controller, including when you later sell or publish the corpus.
• Reusing a dataset someone else scraped leaves each party responsible for its own processing, unless you jointly determine purposes and means.

If you sell scraping as a service, the gap between "the customer specified the domains" and "we picked the domains" is the gap between processor and controller obligations.

How does the US position differ?

There is no federal web scraping statute in the US. Computer Fraud and Abuse Act claims aimed at the scraping of publicly accessible data have repeatedly failed, the hiQ v. LinkedIn line of cases being the well-known example. That does not make US scraping unregulated: contract terms, copyright and state privacy statutes carry the exposure.

Copyright is the live front for AI training. Ropes & Gray's May 2026 analysis notes that using scraped data to train models, and to power retrieval-augmented generation, can implicate the reproduction right under US copyright law, referencing the U.S. Copyright Office's 2024 report.

For teams shipping globally the EU analysis is the strict superset: build to satisfy Guidelines 03/2026 and the remaining US questions are about licensing and terms, not architecture.

What should scraper developers do differently?

Reed Smith's breakdown of the guidelines recommends starting with a gap analysis against the three-part test — sensible, because most of the gaps turn out to be logging and documentation rather than code. If you are assembling a training corpus, our guide to web scraping for AI training data covers the pipeline side of the same problem.

Where CrawlForge fits

Honestly: it handles the mechanical parts and none of the judgment.

The crawling tools respect robots.txt by default — crawl_deep takes respect_robots per call, and the server reads RESPECT_ROBOTS_TXT for the global default. Turning compliance off is therefore possible, and since July 2026 that is a decision worth recording with its reason. Rate limiting is polite by default, and the tools fetch publicly accessible pages — no login-wall circumvention. Access controls and SSRF protection are on by default, for reasons covered in our write-up on SSRF in MCP servers.

What no tool does is make an unlawful pipeline lawful. None of the 27 tools chooses your legal basis, writes your privacy notice, documents the alternatives you rejected, or decides whether your corpus needed that domain. Those are operator obligations, and Guidelines 03/2026 puts them on the controller.

Building AI pipelines on live web data? Start free with 1,000 credits — robots.txt respected by default across 27 tools, no card required. See the documentation or the guide to web scraping for AI training data.