OpenAI to honor shared robots.txt training signal

OpenAI has said it will honor a shared robots.txt signal that allows website operators to communicate their preferences about AI training to participating providers. The move gives publishers and site owners a more standardized way to express whether their content may be used for training, without having to negotiate separate agreements with every AI company. According to OpenAI, it will respect the signal alongside its existing controls. The announcement is significant in the debate over consent, crawling and fair use in AI development. For years, websites have used robots.txt to guide search engines and other crawlers. Extending that logic to AI training creates a common channel for preferences, though it depends on providers choosing to participate and on the signal being widely implemented. For publishers, the benefit is procedural clarity. They gain a way to state their position in a format that can be read automatically, rather than relying only on terms of service or individual deals. For AI companies, honoring such a signal may help reduce friction with content owners and demonstrate good faith as scrutiny of training data grows. The approach is not a complete solution to the broader conflict over how AI systems use public web content. Questions remain about enforcement, scope and what happens when sites change their preferences. Still, OpenAI’s commitment marks a notable step toward shared norms for training data. It could encourage other providers to support the same signal and give site owners a more predictable mechanism for managing their content as AI crawlers become more common.