]> Google LLC.

Brandschenkestrasse 110 Zürich 8002 Switzerland garyillyes@google.com

robots.txt This document extends RFC9309 by specifying additional URI level controls through application level header and HTML meta tags originally developed in 1996. Additionally it moves the response header out of the experimental header space (i.e. "X-") and defines the combinability of multiple headers, which was previously not possible. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Source for this draft and an issue tracker can be found at .

Introduction While the Robots Exclusion Protocol enables service owners to control how, if at all, automated clients known as crawlers may access the URIs on their services as defined by , the protocol doesn't provide controls on how the data returned by their service may be used upon allowed access. Originally developed in 1996 and widely adopted since, the use-case control is left to URI level controls implemented in the response headers, or in case of HTML in the form of a meta tag. This document specifies these control tags, and in case of the response header field, brings it to standards compliance with . Application developers are requested to honor these tags. The tags are not a form of access authorization however.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This specification uses the following terms from : Dictionary, List, String, Parameter.

Specification

Robots control The URI level crawler controls are a key-value pair that can be specified two ways:

• an application level response header structured field as specified by .
• in case of HTML, one or more meta tags as defined by the HTML specification.

Application Layer Response Header The application level response header field "robots-tag" is a structured field whose value is a dictionary containing list of rules applicable to either all accessors or specifically named ones. For historical reasons, implementors should also support the experimental field name, "x-robots-tag". The value of the robots-tag field is a dictionary containing lists of rules. The rules are specific to a single product token as defined by or a global identifier — "*". The global identifier may be omitted. The product token is the first element of each list. Duplicate product tokens must be merged and the rules deduplicated. For example, the following response header field specifies "noindex" and "nosnippet" rules for all accessors, however specifies no rules for the product token "ExampleBot": abc_123;a=1;b=2;cdef_456, ghi;q=9;r="+w" ~~~~~~~~ Robots-Tag: *;noindex;nosnippet, ExampleBot; ~~~~~~~~ The global product identifier "*" in the value may be omitted; for example, this field is equivalent to the previous example: The structured field in the examples is deserialized into the following objects: ~~~~~~~~ ["*" = [["noindex", true], ["nosnippet", true]]], ["ExampleBot" = []] ~~~~~~~~ Implementors SHOULD impose a parsing limit on the field value to protect their systems. The parsing limit MUST be at least 8 kibibytes .

HTML meta element For historical reasons the robots-tag header may be specified by service owners as an HTML meta tag. In case of the meta tag, the name attribute is used to specify the product token, and the content attribute to specify the comma separated robots-tag rules. As with the header, the product token may be a global token, "robots", which signifies that the rules apply to all requestors, or a specific product token applicable to a single requestor. For example: ]]> Multiple robots meta elements may appear in a single HTML document. Requestors must obey the sum of negative rules specific to their product token and the global product token.

Robots controls rules The possible values of the rules are:

• noindex - instructs the parser to not store the served data in its publicly accessible index.
• nosnippet - instructs the parser to not reproduce any stored data as an excerpt snippet.

The values are case insensitive. Unsupported rules must be ignored. Implementors may support other rules as specified in Section 2.2.4 of .

Caching of values The rules specified for a specific product token must be obeyed until the rules have changed. Implementors MAY use standard cache control as defined in for caching robots-tag rules. Implementors SHOULD refresh their caches within a reasonable time frame.

Security Considerations The robots-tag is not a substitute for valid content security measures. To control access to the URI paths in a robots.txt file, users of the protocol should employ a valid security measure relevant to the application layer on which the robots.txt file is served — for example, in the case of HTTP, HTTP Authentication as defined in . The content of the robots-tag header field is not secure, private or integrity-guaranteed, and due caution should be exercised when using it. Use of Transport Layer Security (TLS) with HTTP ( and ) is currently the only end-to-end way to provide such protection. In case of a robots-tag specified in a HTML meta element, implementors should consider only the meta elements specified in the head element of the HTML document, which is generally only accessible to the service owner. To protect against memory overflow attacks, implementers should enforce a limit on how much data they will parse; see section N for the lower limit.

IANA Considerations TODO(illyes): https://www.rfc-editor.org/rfc/rfc9110.html#name-field-name-registry

References Normative References This memo explains how to use the Upgrade mechanism in HTTP/1.1 to initiate Transport Layer Security (TLS) over an existing TCP connection. [STANDARDS-TRACK] This specification defines a model for the relationships between resources on the Web ("links") and the type of those relationships ("link relation types"). It also defines the serialisation of such links in HTTP headers with the Link header field. This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References

Acknowledgments TODO acknowledge.