Is web scraping legal in the EU? GDPR, database rights, AI Act

Scraping public web pages is not illegal in the EU as such, but “public” does not mean “free to use for any purpose”. Which rules apply depends on what you collect and why, not on the tool you use: GDPR if the page contains personal data, the database right and copyright if you copy protected content, contract if you accepted terms, and the AI Act only if you build a general-purpose AI model. This post is informational, as of 19 October 2026, and not legal advice.

Layer	Applies when	What it asks of you	Primary source
GDPR	The pages contain information about identifiable people	A lawful basis, minimisation, a stated purpose	Regulation 2016/679
Database right	You extract a substantial part of a database, or repeatedly extract small parts	Permission, or an exception	Directive 96/9/EC, Art. 7
Copyright and TDM	You copy protected works for automated analysis	Lawful access, and respect for a rights reservation	Directive 2019/790, Art. 3 and 4
Contract	You agreed to a site’s terms	The terms may bind you	CJEU C-30/14
AI Act	You provide a general-purpose AI model	Copyright policy, training-data summary	Regulation 2024/1689, Art. 53

Yes, whenever the data relates to an identifiable person. GDPR defines personal data as any information relating to an identified or identifiable natural person, and says nothing about whether it was public. Names, job titles, photos and forum posts count.

If you collect personal data you become a controller and need a lawful basis (Art. 6). Consent is rarely workable, because you have no relationship with the people concerned. The usual candidate is legitimate interest, Art. 6(1)(f), which fails if the person’s rights and interests override yours. You must also respect purpose limitation and data minimisation (Art. 5). Art. 14 requires telling people you hold their data, unless that proves impossible or takes disproportionate effort (Art. 14(5)(b)). Special categories such as health data need an extra Art. 9(2) condition.

The EDPB Guidelines 03/2026, adopted on 7 July 2026, are the Board’s first guidelines dedicated to web scraping. They are a draft open for consultation until 30 October 2026, and their scope is scraping to train generative AI, so they are not a rule for every scraper, but they show how regulators reason. In the full text, the Board says publishing data online is not consent to its reuse, and that a missing robots.txt is not consent either. Signals such as robots.txt, ai.txt, CAPTCHAs and login walls shape what people can reasonably expect, so sites that clearly oppose scraping should be excluded. Content behind a login should be left out.

Does the EU database right stop me from scraping?

Sometimes. The sui generis right in Directive 96/9/EC lets the maker of a database that took substantial investment to obtain, verify or present its contents prevent extraction or re-use of the whole or a substantial part (Art. 7(1)). Art. 7(5) also forbids repeated, systematic extraction of insubstantial parts when it conflicts with normal exploitation of the database or unreasonably harms the maker. The right lasts 15 years from completion or first publication (Art. 10), renewed by substantial changes. It protects investment, not creativity, so a directory can qualify even though no copyright subsists in it. Reading a few records is a different matter from mirroring a whole catalogue on a schedule, which is the pattern Art. 7(5) targets.

Terms of use are a separate question. In Ryanair v PR Aviation (C-30/14, 15 January 2015), the Court of Justice held that the directive does not apply to a database protected by neither copyright nor the sui generis right, so its provisions do not stop the owner from setting contractual limits on use, subject to national law. It does not say every clause is enforceable, only that the directive will not rescue you from the contract.

What do the text and data mining exceptions allow?

Directive 2019/790 gives two exceptions for text and data mining, defined in Art. 2(2) as automated analysis of text and data to generate information such as patterns, trends and correlations.

Article 3 covers research organisations and cultural heritage institutions doing scientific research, on works they can lawfully access. Rightholders cannot opt out.
Article 4 covers anyone, for lawfully accessible works, and lets you keep copies as long as necessary for the mining. It applies only if the rightholder has not expressly reserved the use “in an appropriate manner, such as machine-readable means” for content made public online (Art. 4(3)).

Both exceptions are for analysis, not for republishing. A machine-readable reservation is an opt-out you must honour; robots.txt is the best-known signal, not the only one. In Germany, the Hamburg Higher Regional Court decided Kneschke v LAION on 10 December 2025 (5 U 104/24): the reservation on the agency website was not in machine-readable form, so the Art. 4 exception covered the copying. It also accepted the research exception. An appeal to the Federal Court of Justice was admitted, so the ruling is not final.

Italy’s Law 132/2025 (in force 10 October 2025) adds Art. 70-septies to the copyright law: extraction through AI models, including generative ones, is allowed only within the existing TDM articles.

What does the AI Act add for scrapers?

For most developers, nothing directly. The copyright duties in Art. 53 of the AI Act fall on providers of general-purpose AI models: they must have a policy to comply with EU copyright law, including identifying and honouring reservations of rights under Art. 4(3) of the DSM Directive, and publish a sufficiently detailed summary of the training content. The Commission says these obligations applied from 2 August 2025, its enforcement powers started on 2 August 2026, and models placed on the market earlier must comply by 2 August 2027.

If you train or fine-tune a model, check whether you count as a provider. The voluntary General-Purpose AI Code of Practice (10 July 2025) has a copyright chapter: signatories commit to crawlers that follow robots.txt (IETF RFC 9309), not to circumvent paywalls, and to honour other machine-readable reservations. If you only call hosted models, these duties sit with their provider.

What do national regulators say?

Your own authority is where complaints land. What I could verify on each site on 19 October 2026:

Italy, Garante: nota informativa on web scraping (decision no. 329 of 20 May 2024, Gazzetta Ufficiale 7 June 2024). It is written for site operators (reserved areas, anti-scraping clauses, traffic monitoring) and is non-binding.
France, CNIL: focus sheet on legitimate interest and scraping (19 June 2025). Exclude sites that oppose scraping via robots.txt or CAPTCHAs, collect only what is open without an account, offer a prior opt-out, delete irrelevant data fast.
Spain, AEPD: a legal criterion of 25 November 2025 states that scraping which collects personal data is processing. The case was closed because the tool only showed an aggregated chart, so this is narrow guidance.
Germany: I found no scraping-specific text from the DSK or a Land authority, so the EDPB guidelines and the Hamburg ruling above are the usable references.
Brazil (not EU, LGPD): the ANPD’s Technology Radar no. 3 (2024) says content on public sites remains subject to the LGPD, and scraping needs a legal basis under Arts. 7 or 11. It is a technical study, not binding.

A checklist for developers who scrape

Public pages only: no login, no paywall, no fake accounts.
Decide whether you need personal data at all. If not, filter it out.
If you do, write down the purpose, legal basis and balancing test first.
Read robots.txt and any other machine-readable reservation, and honour them.
Keep the request rate low (rate limits).
Keep only the fields you need, only as long as you need them.
Do not republish copied content or mirror whole databases.
Have a privacy notice ready and handle access and deletion requests.

The robots.txt step is a few lines with Python’s standard library. Run on 19 October 2026:

from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser

def allowed(url: str, agent: str = "my-research-bot") -> bool:
    parts = urlparse(url)
    parser = RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt")
    parser.read()
    return parser.can_fetch(agent, url)

print(allowed("https://github.com/about"))     # True
print(allowed("https://github.com/copilot/"))  # False

robots.txt covers crawling etiquette. It does not replace reading the terms or checking for other reservations.

How does FastAF fit into this?

FastAF fetches public pages and returns them rendered; it is not a way around any rule above. Its acceptable use policy forbids content behind logins and paywalls, credentials, personal data without a lawful basis, special-category data, data about children, and infringing copyright or database rights. The scrape and screenshot endpoints refuse private and internal addresses and URLs with login-type path segments such as login, sign-in or oauth. The privacy policy says the content your requests return is streamed back to you and not retained. When it contains personal data, you are the controller and FastAF acts on your instructions under the data processing addendum.

The guard is blunt: it does not judge whether a page carries personal data or a rights reservation, so that stays with you. A login URL, called with the production API on 19 October 2026, returned in 0.15 seconds and cost nothing:

curl -s -H "Authorization: Bearer $FASTAF_API_KEY" \
  "https://api.fastaf.co/v1/scrape?url=https%3A%2F%2Fexample.com%2Flogin"
# HTTP 400
# {"error":{"code":"blocked_target","message":"This target is not allowed."}}

For mechanics, see scraping any website with Python and the web scraping API. LinkedIn has its own case law: Is LinkedIn scraping legal?

The short version

Scraping public pages is workable in the EU when you respect four limits: GDPR if people are identifiable, the database right and copyright if you copy protected content, any machine-readable opt-out, and the terms you accepted. Write down your purpose, build the checklist into your pipeline, and consider the AI Act only if you provide a general-purpose model. The rules are moving: the EDPB text is a draft and the Hamburg case is on appeal. For public pages only, create a free account with 500 credits and read the docs.