Robots Exclusion Protocol Extension to communicate AI preferences vocabulary
Microsoft Corporation
One Microsoft Way
Redmond
WA
98052
United States
facan@microsoft.com
Microsoft Corporation
One Microsoft Way
Redmond
WA
98052
United States
krmadhav@microsoft.com
Applications and Real-Time Area
Internet Engineering Task Force
This document extends the RFC9309 by facilitating the communication of content usage specifically within the area of Artificial Intelligence (AI).
Although the Robots Exclusion Protocol allows service owners to manage the access of automated clients, known as crawlers, to the URIs on their services as specified by [RFC8288], it does not offer any mechanisms to regulate how the data retrieved by these services can be used in the field of Artificial Intelligence (AI).
Application developers are requested to honor these tags. The tags are not a form of access authorization however.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
This RFC defines solely the communication method for the AI preference vocabulary, without detailing the vocabulary itself. The term AIPreferenceVocabularyTerm is a non-null string that serves as a placeholder for the AI preference vocabulary terms.
DisallowAIPreferenceVocabularyTerm - These lines specify whether a URI and its associated data are prohibited to be used for the scenario outlined by this AI preference vocabulary term.
AllowAIPreferenceVocabularyTerm - These lines specify whether a URI and its associated data are allowed to be used for the scenario outlined by this AI preference vocabulary term.
The values are case insensitive and honor the same matching and prioritization logic as existing Allow and Disallow robots.txt rules.
The same rules can also be set in the Application Layer Response Header X-Robots-Tag:
noAIPreferenceVocabularyTerm - instructs the parser that the data is prohibited to be used for the scenario outlined by this AI preference vocabulary term.
AIPreferenceVocabularyTerm - instructs the parser that the data can be used for the scenario outlined by this AI preference vocabulary term.
The values are case insensitive and honor the same matching and prioritization logic defined for X-Robots-Tag.
Same rules can also be set via an HTML meta tag (Meta Robots):
<meta name="robots" content="noAIPreferenceVocabularyTerm">
<meta name="examplebot" content="AIPreferenceVocabularyTerm">
TODO: https://www.rfc-editor.org/rfc/rfc9110.html#name-field-name-registry