OpenAI, Google, Microsoft and Anthropic have said they will adopt a proposed shared standard that would allow publishers and website owners to signal AI scraping preferences in one place. The mechanism, described as a “policy token” and reported by The Verge, is being proposed through the Internet Engineering Task Force (IETF) and would sit inside a site’s robots.txt file.

If implemented broadly, the approach could reduce the need for publishers and businesses to manage separate opt-out settings for different AI companies. For European organisations that rely on web content, brand assets, product data or original editorial material, the announcement is relevant as a practical step in website governance and AI risk management.

What happened

According to the report, several of the largest AI companies said they will support a proposed standard that lets publishers communicate AI training and scraping preferences using a shared signal. The proposal would be added to robots.txt, the long-established file that websites use to tell automated crawlers which parts of a site they may or may not access.

The reported aim is to give publishers a more standardised way to apply preferences across multiple AI platforms rather than configuring rules for each provider individually.

At this stage, the policy token is a proposed technical standard rather than a law or a universally enforced rule. Support from major AI companies is significant, but businesses should not assume every AI crawler will immediately follow it, or that compliance can be independently verified in all cases.

Why it matters for European businesses

For many European companies, website content is a business asset. Product descriptions, pricing structures, help documentation, research, editorial content, images and brand copy may all be of interest to AI systems for training, retrieval or indexing.

A shared opt-out mechanism could matter in several ways:

• Simpler controls: web teams may be able to manage one standard signal instead of maintaining separate policies for multiple AI providers.
• Clearer governance: legal, marketing and IT teams could align internal rules on how company content should be accessed by AI crawlers.
• Operational efficiency: SMEs and lean digital teams may find it easier to implement one website-level policy than negotiate crawler settings platform by platform.
• Commercial protection: publishers, e-commerce businesses and knowledge-based firms may want tighter control over how original content is reused by AI systems.

The development is also relevant in the broader European context of AI governance and data strategy. While robots.txt is a technical convention rather than a legal instrument, a more standardised signal may help businesses demonstrate that they have set explicit machine-readable preferences for the use of their content.

That said, this should not be confused with copyright enforcement, GDPR compliance, or a guaranteed legal right to block all automated collection. Businesses should treat it as a practical control layer, not a complete compliance solution.

Who may be affected

• Publishers and media businesses that want to limit AI training on original articles or archives.
• E-commerce companies concerned about the scraping of product data, catalogues, descriptions or pricing content.
• B2B firms whose websites contain proprietary documentation, research, support materials or specialist knowledge.
• Marketing teams and digital agencies responsible for SEO, content strategy and site governance.
• IT and web development teams that manage robots.txt, crawler behaviour and server-side traffic policies.

What companies should consider

European businesses do not need to wait for the entire market to settle before reviewing their approach to AI crawling. Practical steps may include:

• Audit website assets: identify which content is commercially sensitive, original or strategically important.
• Review current crawler controls: check existing robots.txt settings and any provider-specific AI crawler rules already in place.
• Align internal policy: decide whether the business wants to permit, limit or block certain forms of AI access to public web content.
• Coordinate teams: involve legal, marketing and IT stakeholders, especially if web content has copyright, licensing or contractual value.
• Monitor implementation: watch whether the proposed standard is finalised and how major AI providers document support and enforcement.
• Keep expectations realistic: technical opt-out signals may help with governance, but they may not fully prevent copying, extraction or downstream reuse by all actors.

For SMEs in particular, the main value of the proposal is operational simplicity. If large AI providers genuinely support a shared standard, businesses may gain a more practical way to express content-use preferences without building a separate process for each platform. The real business impact, however, will depend on final standardisation, adoption across the market, and how consistently AI crawlers honour the signal in practice.