]> Google LLC.

garyillyes@google.com

robotstxt The Robots Exclusion Protocol defined in specifies the user-agent rule for targeting automatic clients either by prefix matching their self-defined product token or by a global rule * that matches all clients. This document extends by defining a new rule for targeting automatic clients based on the clients' purpose for accessing the service. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Source for this draft and an issue tracker can be found at .

Introduction (fill in)

Specification We define user-agent-purpose as the new rule with a predefined set of values. The values are registered with IANA at ... Below is an Augmented Backus-Naur Form (ABNF) description, as described in .

user-agent-purpose The user-agent-purpose rule is semantically equivalent to the user-agent rule defined in Section 2.2.1. of . As the user-agent rule, user-agent-purpose acts as a starter of rule groups.

user-agent-purpose tokens The user-agent-purpose token MUST be a substring of the identification string that the automatic client sends to the service. For example, in the case of HTTP , the purpose token MUST be a substring in the User-Agent header, along with the product token. Here's an example of a User-Agent HTTP request header with the purpose token by the product token: The purpose token MUST be one of the tokens registered with IANA. Unrecognized tokens MAY be discarded by parsers. Crawlers MUST use case-insensitive matching to find the group that matches the purpose token and obey the rules of the group. If there's a group that matches the product token of the automatic client, the client SHOULD obey that group. If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present. If there is more than one group matching the user-agent-purpose, the matching groups' rules MUST be combined into one group and parsed according to Section X.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Security Considerations The security considerations are the same as in the parent .

IANA Considerations The vocabulary used as purpose tokens are registered at IANA-URL.

Examples

References Normative References The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK]

Acknowledgments TODO acknowledge.