Google has published new guidance explaining that robots.txt is designed to control whether automated systems may crawl website content, but it does not by itself govern how content may be used after access. The clarification is relevant for businesses that publish content online and want to manage visibility in search, AI systems and third-party data use.

What happened

According to the reported update, Google’s Gary Illyes said robots.txt covers crawling rather than later uses such as AI training. The guidance also reportedly recommends percent-encoding non-English characters in robots.txt path rules to avoid parsing or matching issues.

This is an important technical distinction. Robots.txt is a long-established web standard used to tell compliant crawlers which areas of a site they may or may not access. It is not, on its own, a licensing or usage-control mechanism for content once obtained through other means.

Why it matters for European businesses

Many businesses have been looking at robots.txt as a way to control how AI systems interact with their websites. Google’s clarification underlines that this file should be treated primarily as a crawling management tool, not a complete answer to concerns about AI training, content reuse or intellectual property.

For European companies, this matters in several practical areas:

• SEO and discoverability: Blocking crawlers in robots.txt can affect how content appears in search and other discovery systems.
• AI visibility: A company trying to limit AI-related use of content may need more than standard crawl directives, depending on the service and the legal terms involved.
• Content protection: Publishers, blogs, product catalogues and knowledge bases may need to review contracts, platform settings and site architecture rather than rely only on robots.txt.
• Technical accuracy: Sites using non-English URLs should check whether robots.txt rules are written in a format that automated systems can correctly interpret.

The update also reflects a broader issue for digital businesses: traditional web controls were created for search crawling, while AI-related content use raises additional technical, contractual and legal questions that may require separate handling.

Who may be affected

• Publishers and content-driven businesses that want to manage how articles, documentation or media are accessed.
• E-commerce companies exposing product data, descriptions and images on public pages.
• Marketing teams and SEO managers responsible for organic visibility and crawl management.
• IT and web teams maintaining robots.txt files, URL structures and multilingual sites.
• Businesses evaluating AI search exposure and how public website content may be surfaced or reused.

What companies should consider

• Review robots.txt expectations: Treat robots.txt as a crawl-control tool, not a full content-usage policy.
• Audit multilingual and non-English URL rules: Check whether path directives need percent-encoding to match correctly.
• Separate SEO decisions from AI governance decisions: Blocking crawlers may reduce visibility in search without fully addressing concerns about downstream use.
• Check provider-specific controls: Where relevant, review whether search engines, AI platforms or hosting providers offer additional controls beyond robots.txt.
• Review legal and commercial protections: For valuable content, consider terms of use, licensing language and other contractual safeguards alongside technical measures.
• Coordinate marketing, legal and IT teams: Decisions about content access increasingly affect traffic, brand visibility, data use and intellectual property strategy at the same time.

The reported guidance does not create a new legal rule, but it does clarify an operational point that many businesses may have misunderstood. For companies publishing valuable digital content, the key takeaway is that crawl control and AI-use control are not the same thing.