Boeing Research & Technology

P.O. Box 3707 Seattle WA 98124 USA fltemplin@acm.org

I-D Internet-Draft IPv6 packets contain a single unit of transport layer protocol data which becomes the retransmission unit in case of loss. Transport layer protocols including the Transmission Control Protocol (TCP) and reliable transport protocol users of the User Datagram Protocol (UDP) prepare data units known as segments which the network layer packages into individual IPv6 packets each containing only a single segment. This specification presents new packet constructs termed IPv6 Parcels and Advanced Jumbos (AJs) with different properties. Parcels permit a single packet to include multiple segments as a "packet-of-packets", while AJs offer essential operational advantages over basic jumbograms for transporting singleton segments of all sizes ranging from very small to very large. Parcels and AJs provide essential building blocks for improved performance, efficiency and integrity while encouraging larger Maximum Transmission Units (MTUs) according to both the classic Internetworking link model and a new Delay Tolerant Network (DTN) link model.

IPv6 packets contain a single unit of transport layer protocol data which becomes the retransmission unit in case of loss. Transport layer protocols such as the Transmission Control Protocol (TCP) and reliable transport protocol users of the User Datagram Protocol (UDP) (including QUIC , LTP and others) prepare data units known as segments which the network layer packages into individual IPv6 packets each containing only a single segment. This document presents a new construct termed the "IPv6 Parcel" which permits a single packet to include multiple segments. The parcel is essentially a "packet-of-packets" with the full {TCP,UDP}/IPv6 headers appearing only once but with possibly multiple segments included. IPv6 parcels represent a network encapsulation for the multi-segment buffers managed by Generic Segment Offload (GSO) and Generic Receive Offload (GRO); these buffers are now known as "parcel buffers" or simply "parcels" which become "IP parcels" following encapsulation in {TCP,UDP}/IP headers. Transport layer protocol entities form parcels by preparing a data buffer (or buffer chain) containing at most 64 consecutive transport layer protocol segments that can be broken out into individual packets or smaller sub-parcels as necessary. All non-final segments must be equal in length while the final segment must not be larger. The transport layer protocol entity then presents the buffer(s), number of segments and non-final segment size to the network layer. The network layer next either performs packetization to forward each segment as an individual IPv6 packet or appends a parcel integrity block preceded by a single {TCP,UDP} header and a single IPv6 header plus extensions that identify this as a parcel and not an ordinary packet. Any included {TCP,UDP} options are associated with all segments, therefore parcels may only include segments that employ compatible options. The network layer then forwards each parcel over consecutive parcel-capable links in a path until they arrive at a node with a next hop link that does not support parcels, a parcel-capable link with a size restriction, or an ingress Overlay Multilink Network (OMNI) Interface connection to an OMNI link that spans intermediate Internetworks. In the first case, the original source or next hop router applies packetization to break the parcel into individual IPv6 packets. In the second case, the node applies network layer parcellation to form smaller sub-parcels. In the final case, the OMNI interface applies adaptation layer parcellation to form still smaller sub-parcels, then applies adaptation layer IPv6 encapsulation and fragmentation if necessary. The node then forwards the resulting packets/parcels/fragments to the next hop. Following adaptation layer IPv6 reassembly if necessary, an egress OMNI interface applies reunification if necessary to merge multiple sub-parcels into a minimum number of larger (sub-)parcels then delivers them to the network layer which either processes them locally or forwards them via the next hop link toward the final destination. The final destination can then apply network layer (parcel-based) reunification or (packet-based) restoration if necessary to deliver a minimum number of larger (sub-)parcels to the transport layer. Reordering, loss or corruption of individual segments within the network is therefore possible, but most importantly the parcels delivered to the final destination's transport layer should be the largest practical size for best performance. Loss or receipt of individual segments (rather than parcel size) therefore determines the retransmission unit. This document further introduces an "Advanced Jumbo (AJ)" service that provides essential extensions beyond the basic IPv6 jumbograms defined in . AJs are parcel variants that provide end and intermediate systems with a robust delivery service when transmission of singleton segments of all sizes ranging from very small to very large is necessary. The following sections discuss rationale for adopting parcels and AJs as core elements of the Internet architecture, as well as the actual protocol constructs and operational procedures involved. Parcels and AJs provide essential data transit for improved performance, efficiency and integrity while encouraging larger Maximum Transmission Units (MTUs). A new Delay Tolerant Networking (DTN) link service model for parcels and AJs further supports delay/disruption tolerance especially suited for air/land/sea/space mobility applications. These services should inspire future innovation in applications, transport protocols, operating systems, network equipment and data links for Internetworking performance maximization.

The Oxford Languages dictionary defines a "parcel" as "a thing or collection of things wrapped in paper in order to be carried or sent by mail". Indeed, there are many examples of parcel delivery services worldwide that provide an essential transit backbone for efficient business and consumer transactions. In this same spirit, an "IPv6 parcel" is simply a collection of at most 64 transport layer protocol segments wrapped in an efficient package for transmission and delivery as a "packet-of-packets", with each segment including its own end-to-end integrity checks. All non-final segments must be equal in length while the final segment must not be larger. IPv6 parcels and AJs are distinguished from ordinary packets and jumbograms through the constructs specified in this document. The term "Advanced Jumbo (AJ)" refers to a parcel variation modeled from the basic IPv6 jumbogram construct defined in . AJs include a 32-bit Jumbo Payload Length field and a single transport layer protocol segment the same as for basic IPv6 jumbograms, but are differentiated from true parcels and other jumbogram types by including an "Advanced Jumbo Type" value plus end-to-end segment integrity checks the same as for parcels. Unlike basic IPv6 jumbograms which are always 64KB or larger, AJs can range in size from as small as the headers plus a minimal or even null payload to as large as 2**32 octets minus headers. The term "link" is defined in as: "a communication facility or medium over which nodes can communicate at the link layer, i.e., the layer immediately below IPv6. Examples are Ethernets (simple or bridged); PPP links; X.25, Frame Relay, or ATM networks; and internet-layer or higher-layer "tunnels", such as tunnels over IPv4 or IPv6 itself". Where the document refers to "IPv6 header length", it means only the length of the base IPv6 header (i.e., 40 octets), while the length of any extension headers is referred to separately as the "IPv6 extension header length". The term "IPv6 header plus extensions" refers generically to an IPv6 header plus all included extension headers. Where the document refers to "{TCP,UDP} header length", it means the length of either the TCP header plus options (20 or more octets) or UDP header plus options (8 or more octets). It is important to note that only a single IPv6 header and a single full {TCP,UDP} header plus options appears in each parcel regardless of the number of segments included. This distinction often provides a measurable overhead savings made possible only by parcels. Where the document refers to checksum calculations, it means the standard Internet checksum unless otherwise specified. The same as for TCP and UDP , the standard Internet checksum is defined as (sic) "the 16-bit one's complement of the one's complement sum of all (pseudo-)headers plus data, padded with zero octets at the end (if necessary) to make a multiple of two octets". A notional Internet checksum algorithm can be found in , while practical implementations require detailed attention to network byte ordering to ensure interoperability between diverse architectures. The term "Cyclic Redundancy Check (CRC)" is used consistently with its application in widely deployed Internetworking services. Parcels that employ end-to-end CRC checks use the CRC32C or CRC64E standards (see: ). AJs that employ end-to-end CRC checks include either a CRC or message digest calculated according to the MD5 , SHA1 or US Secure Hash algorithms. In all cases, the CRC or message digest is included as an integrity header and submitted for transmission in network byte order per standard Internetworking conventions. The terms "application layer (L5 and higher)", "transport layer (L4)", "network layer (L3)", "(data) link layer (L2)" and "physical layer (L1)" are used consistently with common Internetworking terminology, with the understanding that reliable delivery protocol users of UDP are considered as transport layer elements. The OMNI specification further defines an "adaptation layer" logically positioned below the network layer but above the link layer (which may include physical links and Internet- or higher-layer tunnels). The adaptation layer is not associated with a layer number itself and is simply known as "the layer below L3 but above L2". A network interface is a node's attachment to a link (via L2), and an OMNI interface is therefore a node's attachment to an OMNI link (via the adaptation layer). The term "parcel-capable link/path" refers to paths that transit interfaces to adaptation layer and/or link layer media (either physical or virtual) capable of transiting {TCP,UDP}/IPv6 packets that employ the parcel/AJ constructs specified in this document. The source and each router in the path has a "next hop link" that forwards parcels/AJs toward the final destination, while each router and the final destination has a "previous hop link" that accepts en route parcels/AJs. Each next hop link must be capable of forwarding parcels/AJs (after first applying packetization or parcellation if necessary) with segment lengths no larger than can transit the link. The term "5-tuple" refers to a transport layer protocol entity identifier that includes the network layer (source address, destination address, source port, destination port, protocol number). The term "4-tuple" refers to a network layer parcel entity identifier that includes the adaptation layer (source address, destination address, Parcel ID, Identification). The Internetworking term "Maximum Transmission Unit (MTU)" is widely understood to mean the largest packet size that can transit a single link ("link MTU") or an entire path ("path MTU") without requiring network layer fragmentation. The "Parcel Path MTU" value returned during parcel path qualification determines the maximum sized parcel/AJ that can transit the leading portion of the path up to a router that cannot forward the parcel/AJ further, while the "Residual Path MTU" determines the maximum-sized conventional packet that can transit the remainder of the path following packetization. (Note that for paths that include a significant number of routers that do not recognize the parcel construct the Residual Path MTU may be over-estimated.) The terms "packetization" and "restoration" refer to a network layer process in which the original source or a router on the path breaks a parcel/AJ out into individual IPv6 packets that can transit the remainder of the path without loss due to a size restriction. The final destination then restores the combined packet contents into a parcel before delivery to the transport layer. In standard practice, parcel packetization and restoration are functional equivalents of the well-known GSO/GRO services. The terms "parcellation" and "reunification" refer to either network layer or adaptation layer processes in which the original source or a router on the path breaks a parcel into smaller sub-parcels that can transit the path without loss due to a size restriction. These sub-parcels are then reunified into larger (sub-)parcels before delivery to the transport layer. As a network layer process, the sub-parcels resulting from parcellation may only be reunified at the final destination. As an adaptation layer process, the resulting sub-parcels may first be reunified at an adaptation layer egress node then possibly further reunified by the network layer of the final destination. The terms "fragmentation" and "reassembly" follow exactly from their definitions in the IPv6 standard . In particular, OMNI interfaces support IPv6 encapsulation and fragmentation as an adaptation layer process that can transit packet/parcel/AJ sizes that exceed the underlying Internetwork path MTU. OMNI interface fragmentation/reassembly occurs at a lower layer of the protocol stack than packetization/restoration and/or parcellation/reunification and therefore provides a complimentary service. Note that IPv6 parcels and AJs are not eligible for direct fragmentation and reassembly at the network layer but become eligible for adaptation layer fragmentation and reassembly following OMNI IPv6 encapsulation. "Automatic Extended Route Optimization (AERO)" and the "Overlay Multilink Network Interface (OMNI)" provide an adaptation layer framework for transmission of parcels/AJs over one or more concatenated Internetworks. AERO/OMNI will provide an operational environment for parcels/AJs beginning from the earliest deployment phases and extending indefinitely to accommodate continuous future growth. As more and more parcel/AJ-capable links are enabled (e.g., in data centers, wireless edge networks, space-domain optical links, etc.) AERO/OMNI will continue to provide an essential service for Internetworking performance maximization. The terms "(original) source" and "(final) destination" refer to host systems that produce and consume IPv6 packets/parcels/AJs, respectively. The term "router" refers to a system that forwards IPv6 packets/parcels/AJs not addressed to itself while decrementing the Hop Limit. The terms "OAL source", "OAL intermediate system" and "OAL destination" refer to OMNI Adaptation Layer (OAL) nodes that (respectively) produce, forward and consume OAL-encapsulated IPv6 packets/parcels/AJs over an OMNI link. The terms "controlled environment" and "limited domain" follow directly from . All nodes within a controlled environment / limited domain are expected to honor the protocol specifications found in this document, whereas nodes on open Internetworks may exhibit varying levels of conformance. The "Parcel Integrity Block (PIB)" follows the {TCP,UDP}/IPv6 headers of each parcel/AJ. For parcels, the PIB includes a 1-octet field that encodes the number of segments included followed by integrity check fields (and for TCP also a Sequence Number) for each of the segments that follow. For AJs, the PIB includes only the integrity check fields. "Forward Error Correction (FEC)" as discussed in this document refers specifically to the IETF FEC architecture documented in . In the architecture, a source node applies FEC encoding to an original IP packet/parcel/AJ and the corresponding destination(s) in turn apply FEC decoding to retrieve the original data minus any corrected errors. The parcel sizing variables "J", "K", "L" and "M" are cited extensively throughout this document. "J" denotes the number of segments included in the parcel, "K" is the length of the final segment, "L" is the length of each non-final segment and "M" is termed the "Parcel Payload Length".

All IPv6 nodes MUST observe their respective requirements found in the normative references, including . IPv6 parcels and AJs are modeled from the basic jumbogram specification found in , but the specifications in this document take precedence whenever they differ from the basic requirements. Most notably, IPv6 parcels and AJs use an adaptation of the IPv6 Minimum Path MTU Hop-By-Hop Option instead of the basic IPv6 Jumbo Payload Option . IPv6 parcels/AJs are not limited only to segment sizes that exceed 65535 octets; instead, parcels can be as small as the packet and parcel headers plus a singleton segment while AJs can be as small as the headers plus a NULL payload. IPv6 parcels/AJs are not eligible for direct network layer IPv6 fragmentation and reassembly although they may become eligible for adaptation layer fragmentation and reassembly following OMNI IPv6 encapsulation. IPv6 parcels and AJs therefore SHOULD NOT include IPv6 (Extended) Fragment Headers, and implementations MUST silently ignore any IPv6 (Extended) Fragment Headers in IPv6 parcels and AJs. For further Hop-by-Hop Options considerations, see: . For IPv6 extension header limits, see: . For IPv4 parcel and advanced jumbo considerations, see: . The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Studies have shown that applications can improve their performance by sending and receiving larger packets due to reduced numbers of system calls and interrupts as well as larger atomic data copies between kernel and user space. Larger packets also result in reduced numbers of network device interrupts and better network utilization (e.g., due to header overhead reduction) in comparison with smaller packets. However, the most prominent performance increases were observed by increasing the transport layer protocol segment size even if doing so invoked network layer fragmentation. A first study involved performance enhancement of the QUIC protocol using the linux GSO/GRO facility. GSO/GRO provides a robust service that has shown significant performance increases based on a multi-segment transfer capability between the operating system kernel and QUIC applications. GSO/GRO performs packetization and restoration at the transport layer with a transport protocol segment size limited by the path MTU (typically 1500 octets or smaller in current Internetworking practices). A second study showed that GSO/GRO also improves performance for the Licklider Transmission Protocol (LTP) used for the Delay Tolerant Networking (DTN) Bundle Protocol for segments larger than the actual path MTU through the use of IP fragmentation. Historically, the NFS protocol also saw significant performance increases using larger (single-segment) UDP datagrams even when IP fragmentation is invoked, and LTP still follows this profile today. Moreover, LTP shows this (single-segment) performance increase profile extending to the largest possible segment size which suggests that additional performance gains are possible using (multi-segment) parcels or AJs that approach or even exceed 65535 octets in total length. TCP also benefits from larger packet sizes and efforts have investigated TCP performance using jumbograms internally with changes to the linux GSO/GRO facilities . The approach proposed to use the Jumbo Payload Option internally and to allow GSO/GRO to use buffer sizes that exceed 65535 octets, but with the understanding that links that support jumbograms natively are not yet widely deployed and/or enabled. Hence, parcels/AJs provide a packaging that can be considered in the near term under current deployment limitations. A limiting consideration for sending large packets is that they are often lost at links with MTU restrictions, and the resulting Packet Too Big (PTB) messages may be lost somewhere in the return path to the original source. This path MTU "black hole" condition can degrade performance unless robust path probing techniques are used, however the best case performance always occurs when loss of packets due to size restrictions is minimized. These considerations therefore motivate a design where transport protocols can employ segment sizes as large as 65535 octets (minus headers) while parcels that carry multiple segments may themselves be significantly larger. (Transport layer protocols can also use AJs to transit even larger singleton segments.) Parcels allow the receiving transport layer protocol entity to process multiple segments in parallel instead of one at a time per existing practices. Parcels therefore support improvements in performance, integrity and efficiency for the original source, final destination and networked path as a whole. This is true even if the network and lower layers need to apply packetization/restoration, parcellation/reunification and/or fragmentation/reassembly. An analogy: when a consumer orders 50 small items from a major online retailer, the retailer does not ship the order in 50 separate small boxes. Instead, the retailer packs as many of the small items as possible into one or a few larger boxes (i.e., parcels) then places the parcels on a semi-truck or airplane. The parcels may then pass through one or more regional distribution centers where they may be repackaged into different parcel configurations and forwarded further until they are finally delivered to the consumer. But most often, the consumer will only find one or a few parcels at their doorstep and not 50 separate small boxes. This flexible parcel delivery service greatly reduces shipping and handling cost for all including the retailer, regional distribution centers and finally the consumer.

The classic Internetworking link service model requires each link in the path to apply a link-layer integrity check often termed a "Frame Check Sequence (FCS)" over the entire length of the frame. The link near-end calculates and appends an FCS trailer to each packet pending transmission, and the link far-end verifies the FCS upon packet reception. If verification fails, the link far-end unconditionally discards the packet. This process is repeated for each link in the path so that only packets that pass all link-layer checks over their full lengths are delivered to the final destination. (Note that Internet- or higher-layer tunnels may traverse many underlying physical links that each apply their own FCS in series.) While the classic link model has contributed to the unparalleled success of terrestrial Internetworks (including the global public Internet), new uses in which significant delays or disruptions can occur are not as well supported. For example, a path that contains multiple links with higher bit error rates may be unable to pass an acceptable percentage of packets since loss due to link errors can occur at any hop. Moreover, packets that incur errors at an intermediate link but somehow pass the link integrity check will be forwarded by all remaining links in the path leaving only the final destination's integrity checking as a last resort. Advanced error detection and correction services not typically associated with packets are therefore necessary; especially with the advent of space-domain and wireless Internetworking, long delays and significant disruptions are often intolerant of retransmissions. This specification therefore introduces a new Delay Tolerant Networking (DTN) link model. IPv6 parcels/AJs that engage this DTN link model include a limited hop-by-hop integrity check that covers only the headers plus a leading portion of the payload. Each IPv6 parcel/AJ also includes per-segment end-to-end Cyclic Redundancy Checks (CRCs) or message digests plus Internet checksums to be verified by the final destination. For each parcel/AJ admitted under the DTN link model, the original source applies Forward Error Correction (FEC) encoding if necessary. Each link near-end in the path then applies its standard link-layer FCS for only the leading portion upon transmission as indicated by the IPv6 Payload Length then writes the FCS as a trailer following the end of the parcel/AJ payload. The link far-end then verifies the FCS for the leading portion upon reception and discards the parcel/AJ if an error is detected. However, each link in the path passes parcels/AJs with valid headers through to the final destination even if the unchecked portion of the payload accumulates bit errors in transit. The final destination then invokes FEC decoding if necessary, verifies integrity using per segment end-to-end CRCs/digests plus Internet checksums and delivers each segment to the local transport layer which may employ higher-layer integrity checks. The ubiquitous 1500 octet link MTU had its origins in the very earliest deployments of 10Mbps Ethernet technologies, however modern wired-line link data rates of 1Gbps are now typical for end user devices such as laptop computers while much higher rates of 10Gbps, 100Gbps or even more commonly occur for data center servers. At these data rates, the serialization delays range from 1200usec at 10Mbps to only .12usec at 100Gbps (still higher data rates are expected in the near future). This suggests that the legacy 1500 MTU may be too small by multiple orders of magnitude for many well-connected data centers, wide-area wired-line networked paths or even for deep space communications over optical links. For such cases, larger parcels and AJs present performance maximization constructs that support larger transport layer segment sizes. While data centers, Internetworking backbones and deep space networks are often connected through robust fixed link services, the Internet edge is rapidly evolving into a much more mobile environment where 5G (and beyond) cellular services and WiFi radios connect a growing majority of end user systems. Although some wireless edge networks and mobile ad-hoc networks support considerable data rates, more typical rates with wireless signal disruption and link errors suggest that limiting channel contention by configuring more conservative MTU levels is often prudent. Even in such environments, a mixed link model with error-tolerant data sent in DTN parcels/AJs and error-intolerant data sent in classic packet/parcel/AJ constructs may present a more balanced profile. IPv6 parcels and AJs therefore provide a revolutionary advancement for delay/disruption tolerance in air/land/sea/space mobile Internetworking applications. As the Internet continues to evolve from its more stable fixed terrestrial network origins to one where more and more nodes operate in the mobile edge, this new link service model relocates bulk error detection and correction responsibilities from intermediate systems to end systems that are uniquely capable of taking corrective actions. Note: IPv6 parcels and AJs may already be compatible with widely-deployed link types such as 1/10/100-Gbps Ethernet. Each Ethernet frame is identified by a preamble followed by a Start Frame Delimiter (SFD) followed by the frame data itself followed by the FCS and finally an Inter Packet Gap (IPG). Since no length field is included, however, the frame can theoretically extend as long as necessary for transmission of IPv6 parcels and AJs that are much larger than the typical 1500 octet Ethernet MTU as long as the time duration on the link media is properly bounded. Widely-deployed links may therefore already include all of the necessary features to natively support large parcels and AJs with no additional extensions, while operating systems may require extensions to post larger receive buffers.

A transport protocol entity identified by its 5-tuple forms a parcel body by concatenating "J" transport layer protocol segments (for J between 1 and 64) into a contiguous buffer or chain of smaller buffers. All non-final segments MUST be of equal length "L" while the final segment of length "K" MUST NOT be larger and MAY be smaller. The overall parcel length (including headers) is then represented by the value "M". The transport layer protocol entity sets L to a 16-bit non-final segment length that MUST be no smaller than 256 octets and no larger than 65535 octets minus the lengths of the {TCP,UDP} header (plus options) and IPv6 header (plus extensions) (see: ). The transport layer protocol entity then presents the buffer(s) and non-final segment length L to the network layer, noting that the combined buffer length(s) may exceed 65535 octets when there are sufficient segments of a large enough size. If the next hop link is not parcel capable, the network layer performs packetization to package each segment as an individual IPv6 packet as discussed in . If the next hop link is parcel capable, the network layer instead appends a Parcel Integrity Block (PIB) that includes a Number of Segments (Nsegs) value plus CRC code followed by J segment integrity blocks consisting of a 0/4/8-octet CRC followed by a 2-octet Internet Checksum followed by a 4-octet Sequence Number for each TCP segment. Nsegs encodes the number of segments (for J between 1 and 64) and CRC is set to 0 for no CRC, 1 for CRC-32 or 2 for CRC-64 (CRC value 3 is reserved). The PIB is formatted as shown in :

The network layer then appends a single full {TCP,UDP} header (plus options) and a single full IPv6 header including an IPv6 Hop-by-Hop Options extension header with a Parcel Payload Option formatted as shown in :

The network layer includes the Parcel Payload Option as an IPv6 Hop-by-Hop option with Option Type set to '0x30' and Opt Data Len set to 12. The length also distinguishes this format from its use as the IPv6 Minimum Path MTU Hop-by-Hop Option . The network layer then sets the IPv6 Payload Length field to the length of the leading portion of the parcel subject to hop-by-hop integrity checks and sets Parcel Control to L. The network layer next sets Parcel Payload Length to a 24-bit value M that encodes the length of the IPv6 extension headers plus the length of the {TCP,UDP} header (plus options) plus the length of the PIB plus the combined lengths of all concatenated segments. Note that in this arrangement the IPv6 Payload Length determines the leading length of the parcel subject to link layer FCS protection while the Parcel Payload Length determines the end of the parcel payload after which the link layer appends the trailing FCS itself. The network layer next sets Index to an ordinal parcel segment "Index" value between 0 and 63 to identify the initial segment index for the (sub-)parcel and sets the "More (S)egments" flag to 1 for non-final sub-parcels or 0 for the final (sub-)parcel. The network layer next sets the "e(X)treme" flag to 0 unless otherwise specified (see: ). The network layer finally includes a 4-octet Identification, sets Code to 255 and sets Check to the same value that will appear in the IPv6 header Hop Limit field on transmission (see: ). These values provide hop-by-hop assurance that previous hops correctly process parcels without applying legacy IPv6 option processing per . Following this transport and network layer processing, {TCP,UDP}/IPv6 parcels therefore have the structures shown in :

A TCP Parcel is an IPv6 parcel that includes an IPv6 header plus extensions with a Parcel Payload Option formed as shown in with Parcel Payload Length encoding a value no larger than (2**24 - 1) octets. The IPv6 header plus extensions is then followed by a TCP header plus options (20 or more octets) followed by a PIB followed by J consecutive segments. The TCP header Sequence Number is set to 0, each non-final segment is L octets in length and the final segment is K octets in length. The value L is encoded in the Parcel Control field while the overall length of the parcel is determined by the Parcel Payload Length M. The source prepares TCP Parcels in an alternative adaptation of TCP jumbograms . The source calculates a checksum of the TCP header plus IPv6 pseudo-header only (see: ). The source then writes the exact calculated value in the TCP header Checksum field (i.e., without converting calculated 0 values to '0xffff'). For each Segment(i) (for i between 0 and (J-1)), the source next populates the PIB by calculating the Internet checksum beginning with Sequence Number(i) and extending over the length of Segment(i), then writes the value into the Checksum(i) field. When CRC is 1/2, the source then calculates the CRC-32/64 beginning with Checksum(i) then extending over both Sequence Number(i) and the length of Segment(i), then writes the value into CRC(i). See for additional TCP considerations. See for additional integrity considerations. Note: The parcel TCP header Source Port, Destination Port and (per-segment) Sequence Number fields apply to each parcel segment, while the TCP control bits and all other fields apply only to the first segment (i.e., "Segment(0)"). Therefore, only parcel Segment(0) may be associated with control bit settings while all other segment(i)'s must be simple data segments.

A UDP Parcel is an IPv6 Parcel that includes an IPv6 header plus extensions with a Parcel Payload Option formed as shown in with Parcel Payload Length encoding a value no larger than (2**24 - 1) octets. The IPv6 header plus extensions is then followed by an 8-octet UDP header followed by a PIB followed by J transport layer segments followed by any UDP options. Each segment must begin with a transport-specific start delimiter (e.g., a segment identifier, a sequence number, etc.) included by the transport layer user of UDP. The length of the first segment L is encoded in the Parcel Control field while the overall length of the parcel is determined by the Parcel Payload Length M as above. The source prepares UDP Parcels in an alternative adaptation of UDP jumbograms by setting the UDP header Length field to the length of the UDP header plus the lengths of the PIB plus all parcel segments. If this length exceeds 65535 octets, the source instead sets UDP Length to 0 and includes a 2-octet trailing "UDP Option Length" field that encodes the length of the UDP options which immediately precede it, i.e., excluding the length of the UDP Option Length field itself. (If there are no UDP options the UDP Option Length field encodes the value 0.) If UDP checksums are enabled, the source then calculates the checksum of the UDP header plus IPv6 pseudo-header (see: ) while writing calculated 0 values as '0xffff'. If UDP checksums are disabled, the source instead writes the value '0'. For each Segment(i), the source next populates the PIB. If UDP checksums are disabled, the source writes the value 0 into the Checksum(i) field; otherwise, the source calculates the Internet checksum over the length of Segment(i) and writes the value into the Checksum(i) field while rewriting calculated 0 values as '0xffff'. When CRC is 1/2, the source then calculates the CRC-32/64 beginning with Checksum(i) and extending over the length of Segment(i), then writes the value into CRC(i). For the final segment, the source also extends the CRC calculation over the length of the segment to also include the UDP options plus UDP Option Length field when either or both are present. (Note that the length of the UDP Option Length field itself is also included in the Parcel Payload Length.) See: for additional integrity considerations.

The parcel source unambiguously encodes the values J, L and M in parcel header fields as specified above. The value K is not encoded in a header and must therefore be calculated by intermediate and final destination nodes. K is calculated simply as the remainder of the Parcel Payload Length M minus the length of the IPv6 header extensions minus the length of the {TCP,UDP} header (plus options) minus the length of the PIB minus the lengths of all concatenated non-final segments (L * (J-1)).

When the network layer of the source assembles a {TCP,UDP}/IPv6 parcel it fully populates all IPv6 header fields including the source address, destination address and Parcel Payload Option as above. The source sets Hop Limit to the Parcel Limit value discovered through probing (see: ), sets X to 0 for normal or 1 for "e(X)treme path" OMNI link traversal and also sets Parcel Control to L (between 256 and 65535) to distinguish the parcel from other jumbogram types (see: ). The source also maintains a randomly-initialized 4-octet (32-bit) Identification value for each destination. For each packet, parcel or AJ transmission, the source sets the Identification to the current cached value for this destination and increments the cached value by 1 (modulo 2**32) for each successive transmission. (The source can then reset the cached value to a new random number when necessary, e.g., to maintain an unpredictable profile.) For each parcel transmission, the source includes the Identification value in the Parcel Payload Option. The source finally populates {TCP,UDP} header and option fields, includes a populated PIB then presents the parcel to an interface for transmission to the next hop. For ordinary interface attachments to parcel-capable links, the source simply admits each parcel into the interface the same as for any IPv6 packet where it may be forwarded by one or more routers over additional consecutive parcel-capable links possibly even traversing the entire forward path to the final destination. Note that any node in the path that does not recognize the parcel construct may either drop it and return an ICMP Parameter Problem message or attempt to forward it as a (truncated) packet, where the IPv6 Payload Length determines a likely truncation length. Most importantly, each parcel-capable link in the path forwards parcels/AJs with integrity-checked headers even if the remainder of the payload accumulated link errors since each segment is responsible for its own end-to-end integrity. This ensures that the vast majority of coherent data is delivered to the final destination instead of being discarded along with a minor amount of corrupted data at an intermediate hop. When the link far end receives a parcel/AJ it verifies only the leading portion of the parcel/AJ header before forwarding to the next hop while leaving integrity assurance for the remainder as an end-to-end service (see: ). When the next hop link does not support parcels at all, the source breaks the parcel up into individual IPv6 packets. When the next hop link is parcel-capable but configures an MTU that is too small to pass the entire parcel, the source breaks the parcel up into smaller sub-parcels. In the first case, the source can apply packetization (i.e., GSO), and the final destination can apply restoration (i.e., GRO)) to deliver the largest possible parcel buffer(s) to the transport layer. In the second case, the source can apply parcellation to break the parcel into sub-parcels with each containing the same Identification value and with the S flag set appropriately. The final destination can then apply reunification to deliver the largest possible parcel buffer(s) to the transport layer. In all other ways, the source processes of breaking a parcel up into individual IPv6 packets or smaller sub-parcels entail the same considerations as for a router on the path that invokes these processes as discussed in the following subsections. Parcel probes that test the forward path's ability to pass parcels/AJs include "Parcel Path MTU" and "Residual Path MTU" fields as discussed in . Each router in the path may rewrite the fields to progressively smaller values in a similar fashion as for . The fact that the probe transited a previous hop link provides sufficient evidence of forward progress since path MTU determination is unidirectional in the forward path only. Following successful parcel probing, each parcel/AJ transmission may include {TCP,UDP} segment size probes used for packetization layer path MTU discovery per . Such probes may be necessary to refine the Residual Path MTU, for which parcel probes can only provide an estimate. When a router or destination receives a parcel (or parcel probe) it first compares Code with 255 and Check with the IPv6 header Hop Limit; if either value differs, the node drops the parcel and returns a negative Jumbo Report (see: ) subject to rate limiting. For all other intact parcels, each router next compares the value L with the next hop link MTU. If the next hop link is parcel capable but configures an MTU too small to admit a parcel with a single segment of length L the router returns a positive Jumbo Report (subject to rate limiting) with MTU set to the next hop link MTU. If the next hop link is not parcel capable and configures an MTU too small to pass an individual IPv6 packet with a single segment of length L the router instead returns a positive Parcel Report (subject to rate limiting) with MTU set to the next hop link MTU. If the next hop link is parcel capable the router MUST forward the parcel to the next hop while decrementing both Check and the IPv6 header Hop Limit field by 1. If the router recognizes parcels but the next hop link in the path does not, or if the entire parcel would exceed the next hop link MTU, the router instead opens the parcel. The router then forwards each enclosed segment in individual IPv6 packets or in a set of smaller sub-parcels that each contain a subset of the original parcel's segments. If the next hop link is via an OMNI interface, the router instead follows OMNI Adaptation Layer procedures. These considerations are discussed in detail in the following sections.

For transmission of individual packets over links that do not support parcels, the source or router (i.e., the node) invokes packetization the same as for GSO. Routers also invoke packetization if decrementing the parcel Hop Limit would cause it to become 0. Otherwise, the node forwards the intact (sub-)parcel or performs parcellation (see: for discussion of Parcel Limit). To initiate packetization, the node first determines whether an individual packet with segment of length L can fit within the next hop link/path MTU. If an individual packet would be too large the node drops the parcel and returns a positive Parcel Report message (subject to rate limiting) with MTU set to the next hop link/path MTU and with the leading portion of the parcel beginning with the IPv6 header as the "packet in error". If an individual packet can be accommodated, the node next removes the Parcel Payload Option and PIB while retaining the PIB contents for integrity reference. If CRC=1/2, the node then verifies the CRCs of each segment(i) (for i between 0 and (J-1)) and discards any segment(i)'s with incorrect CRCs. The node then copies the {TCP,UDP}/IPv6 headers followed by segment (i) into J individual packets ("packet(i)"). Each such packet(i) will be subject to the independent link-layer CRC verifications of each remaining link in the path. For each packet(i), the node then clears the TCP control bits in all but packet(0), and includes only those {TCP,UDP} options that are permitted to appear in data segments in all but packet(0) which may also include control segment options (see: for further discussion). The node then sets IPv6 Payload Length for each packet(i) based on the length of segment(i) according to . For each packet(i), the node then inserts a Parcel Parameters Option for TCP or UDP . The {TCP,UDP} option is formatted as shown in :

The node includes the Parcel Parameters Option in the TCP header or UDP trailer of each packet(i). The node sets Kind to 253 for TCP or 127 for UDP , then sets ExID to TBD1 (see: IANA Considerations). The node includes the Identification values found in the original parcel header, then sets Index to 'i' and sets S to 1 for non-final packet(i)'s or to 0 for the final packet(i) of the final (sub-)parcel. (For single segment parcels and AJs, the node instead sets Index and S to 0.) For each IPv6 packet, the node then sets Hop Limit to a conservative value that allows for sufficient conventional IPv6 forwarding hops along the residual path from the node performing packetization to the final destination while still providing an adequate termination count to protect against routing loops. For each TCP/IPv6 packet, the node next sets Payload Length according to then calculates/sets the checksum for the packet according to . For each UDP/IPv6 packet, the node instead sets the Payload Length and UDP length fields then calculates/sets the checksum according to . The node reuses the PIB checksum value for each segment in the checksum calculation process. The node first calculates the Internet checksum over the new packet {TCP,UDP}/IPv6 headers (with Sequence Number set to 0 for TCP) then adds the cached segment checksum value. For TCP, the node finally writes the PIB Sequence Number value for each segment into the TCP Sequence Number field. For UDP, if a per-segment Checksum was 0 the node instead writes the value 0 in the Checksum field of the corresponding UDP/IPv6 packet. The node then forwards each IPv6 packet to the next hop. Note: Packets resulting from packetization may be too large to transit the remaining path to the final destination, such that a router may drop the packet(s) and possibly also return an ordinary ICMP PTB message. Since these messages cannot be authenticated or may be lost on the return path, the original source should take care in setting a segment size as large as the Residual Path MTU unless as part of an active probing service.

For transmission of smaller sub-parcels over parcel-capable links, the source or intermediate system (i.e., the node) first determines whether a single segment of length L can fit within the next hop link MTU if packaged as a (singleton) sub-parcel. If a singleton sub-parcel would be too large, the node returns a positive Jumbo Report message (subject to rate limiting) with MTU set to the next hop link MTU and containing the leading portion of the parcel beginning with the IPv6 header then drops the parcel. Otherwise, the node employs network layer parcellation to break the original parcel into smaller groups of segments that can traverse the path as whole (sub-)parcels. The node first determines the number of segments of length L that can fit into each sub-parcel under the size constraints. For example, if the node determines that each sub-parcel can contain 3 segments of length L, it creates sub-parcels with the first containing Segments 0-2, the second containing 3-5, the third containing 6-8, etc., and with the final containing any remaining Segments. The node also includes a PIB in each sub-parcel that contains the corresponding CRC, Checksum and Sequence Number fields for its included segments (where the per-segment fields of the sub-parcel PIB are copied from the PIB of the original parcel). If the original parcel's Parcel Payload Option has S set to 0, the node then sets S to 1 in all resulting sub-parcels except the last (i.e., the one containing the final segment of length K, which may be shorter than L) for which it sets S to 0. If the original parcel has S set to 1, the node instead sets S to 1 in all resulting sub-parcels including the last. The node next sets the Index field to the value 'i' which is the ordinal number of the first segment included in each sub-parcel. (In the above example, the first sub-parcel sets Index to 0, the second sets Index to 3, the third sets Index to 6, etc.). If another router further down the path toward the final destination forwards the sub-parcel(s) over a link that configures a smaller MTU, the router may break it into even smaller sub-parcels each with Index set to the ordinal number of the first segment included. The node next appends identical {TCP,UDP}/IPv6 headers (including the Parcel Payload Option plus any other extensions) to each sub-parcel while resetting Index, S, and Parcel Payload Length in each as above. The node also sets the Hop Limit in each sub-parcel to the same value that occurred in the original (sub-)parcel. For TCP, the node then clears the TCP control bits in all but the first sub-parcel and includes only those {TCP,UDP} options that are permitted to appear in data segments in all but the first sub-parcel (which may also include control segment options). The node then resets the {TCP,UDP} Checksum according to ordinary parcel formation procedures (see above). The node finally sets PMTU to the next hop link MTU then forwards each (sub-)parcel to the parcel-capable next hop.

For transmission of original parcels or sub-parcels over OMNI interfaces, the node admits all parcels into the interface unconditionally since the OMNI interface can accommodate all parcel sizes. The OMNI Adaptation Layer (OAL) of this First Hop Segment (FHS) OAL source node then forwards the parcel to the next OAL hop which may be either an intermediate node or a Last Hop Segment (LHS) OAL destination. OMNI interface parcellation and reunification procedures are specified in detail in the remainder of this section, while parcel encapsulation and fragmentation procedures are specified in . When the OAL source forwards a parcel (whether generated by a local application or forwarded over a network path that transited one or more parcel-capable links), it first assigns a monotonically-incrementing (modulo 64) adaptation layer Parcel ID (note that this value differs from the (Parcel) Index encoded in the Parcel Payload Option). If the parcel is larger than the OAL maximum segment size of 65535 octets, the OAL source next employs parcellation to break the parcel into sub-parcels the same as for the above network layer procedures. This includes re-setting the Index, S and Parcel Payload Length fields in each sub-parcel the same as specified in . The OAL source next assigns a different monotonically-incrementing adaptation layer Identification value for each sub-parcel of the same Parcel ID then performs adaptation layer encapsulation while writing the Parcel ID into the OAL IPv6 Extended Fragment Header. The OAL source then performs OAL fragmentation if necessary and finally forwards each fragment to the next OAL hop toward the OAL destination. (During encapsulation, the OAL source examines the Parcel Payload Option S flag to determine the setting for the adaptation layer fragment header S flag according to the same rules specified in .) When the sub-parcels arrive at the OAL destination, it retains them along with their Parcel IDs and Identifications for a short time to support reunification with peer sub-parcels of the same original (sub-)parcel identified by the 4-tuple information corresponding to the OAL source. This reunification entails the concatenation of PIBs included in sub-parcels with the same Parcel ID and with Identification values within (modulo 64) of one another to create a larger sub-parcel possibly even as large as the entire original parcel. The OAL destination concatenates the segments for each sub-parcel in ascending Identification value order, while ensuring that any sub-parcel with TCP control bits set appears as the first concatenated element in a reunified larger parcel and any sub-parcel with S flag set to 0 appears as the final concatenation. The OAL destination then sets S to 0 in the reunified (sub-)parcel if and only if one of its constituent elements also had S set to 0; otherwise, it sets S to 1. The OAL destination then appends a common {TCP,UDP}/IPv6 header plus extensions to each reunified sub-parcel while resetting Index, S and Parcel Payload Length in the corresponding header fields of each. For TCP, if any sub-parcel has TCP control bits set the OAL destination regards it as sub-parcel(0) and uses its TCP header as the header of the reunified (sub-)parcel with the TCP options including the union of the TCP options of all reunified sub-parcels. The OAL destination then resets the {TCP,UDP}/IPv6 header checksum. If the OAL destination is also the final destination, it then delivers the sub-parcels to the network layer which processes them according to the 5-tuple information supplied by the original source. If the OAL destination is not the final destination, it instead forwards each sub-parcel toward the final destination the same as for an ordinary IPv6 packet. Note: Adaptation layer parcellation over OMNI links occurs only at the OAL source while adaptation layer reunification occurs only at the OAL destination; intermediate OAL nodes do not engage in the parcellation/reunification processes. The OAL destination should retain sub-parcels in the reunification buffer only for a short time (e.g., 1 second) or until all sub-parcels of the original parcel have arrived. The OAL destination then delivers full and/or incomplete reunifications to the network layer (in cases where loss and/or delayed arrival interfere with full reunification). Note: OMNI interface parcellation and reunification is an OAL process based on the adaptation layer 4-tuple and not the network layer 5-tuple. This is true even if the OAL has visibility into network layer information since some sub-parcels of the same original parcel may be forwarded over different network paths. Note: Some implementations may encounter difficulty in applying adaptation layer reunification for sub-parcels that have already incurred lower layer fragmentation and reassembly (e.g., due to network kernel buffer structure limitations). In that case, the adaptation layer can either linearize each sub-parcel before applying reunification or deliver incomplete reunifications or even individual sub-parcels to upper layers. Note: If the original source selects the "e(X)treme path" for OMNI link traversal, the OMNI interface forwards the entire parcel as a (giant) singleton carrier packet using jumbo-in-jumbo encapsulation instead of applying adaptation layer parcellation as discussed in .

When the original source or a router on the path opens a parcel and forwards its contents as individual IPv6 packets, these packets will arrive at the final destination which can hold them in a restoration buffer for a short time before restoring the original parcel the same as for Generic Receive Offload. The 5-tuple information plus the Parcel Parameters Option values included by the source during packetization (see: ) provide unambiguous context for GRO restoration which practical implementations have proven as a robust service at high data rates. When the original source or a router on the path opens a parcel and forwards its contents as smaller sub-parcels, these sub-parcels will arrive at the final destination which can hold them in a reunification buffer for a short time or until all sub-parcels have arrived. The 5-tuple information plus the Index, S and Identification values provide sufficient context for reunification. In both the restoration and reunification cases, the final destination concatenates segments according to ascending Index and/or Identification numbers to preserve segment ordering even if a small degree of reordering and/or loss may have occurred in the networked path. When the final destination performs restoration/reunification on TCP segments, it must include the one with any TCP flag bits set as the first concatenation and with the TCP options including the union of the TCP options of all concatenated packets or sub-parcels. For both TCP and UDP, any packet or sub-parcel containing the final segment must appear as a final concatenation. The final destination can then present the concatenated parcel contents to the transport layer with segments arranged in (nearly) the same order in which they were originally transmitted. Unless a security encapsulation is included, strict ordering is not mandatory since each segment will include a transport layer protocol specific start delimiter with positional coordinates. However, the Index field and/or Identification includes an ordinal value that preserves ordering since each sub-parcel or individual IPv6 packet contains an integral number of whole transport layer protocol segments. Note: Restoration and/or reunification buffer management is based on a hold timer during which singleton packets or sub-parcels are retained until all members of the same original parcel have arrived. Implementations should maintain a short hold timer (e.g., 1 second) and advance any restorations/reunifications to upper layers when the hold timer expires even if incomplete. Note: Since loss and/or reordering may occur in the network, the final destination may receive a packet or sub-parcel with S set to 0 before all other elements of the same original parcel have arrived. This condition does not represent an error, but in some cases may cause the network layer to deliver sub-parcels that are smaller than the original parcel to the transport layer. The transport layer simply accepts any segments received from all such deliveries and will request retransmission of any segments that were lost and/or damaged. Note: Restoration and/or reunification buffer congestion may indicate that the network layer cannot sustain the service(s) at current arrival rates. The network layer should then begin to deliver incomplete restorations/reunifications or even individual segments to upper layers (e.g., via a socket buffer) instead of waiting for all segments to arrive. The network layer can manage restoration/reunification buffers, e.g., by maintaining buffer occupancy high/low watermarks. Note: Some implementations may encounter difficulty in applying network layer restoration/reunification for packets/sub-parcels that have already incurred adaptation layer reassembly/reunification. In that case, the network layer can either linearize each packet/sub-parcel before applying restoration/reunification or deliver incomplete restorations/reunifications or even individual packets/sub-parcels to upper layers.

Unless there is operational assurance that all routers and destinations in the network will recognize parcel/AJ constructs, the original source should send an initial probe to determine whether parcels/AJs can transit at least an initial portion of the forward path toward the final destination. The original source prepares an ordinary IPv6 packet with an alternate encoding of the IPv6 Minimum Path MTU Option that contains Parcel Probe parameters as shown in :

The IPv6 packet can be either a purpose-built probe or part of an existing transport protocol session, but it should cause the destination to return a responsive {TCP,UDP}/IPv6 packet with authenticating credentials and with a Parcel Probe Reply Option - see below. (Note that the probe must appear in an ordinary IPv6 packet and not a parcel/AJ to ensure that it will traverse the entire path to the destination.) The source sets the IPv6 probe Hop Limit to a sufficiently large value to allow the probe to traverse the path. The source then sets Payload Length the same as for an ordinary IPv6 packet. The source next sets "Option Type" to '0x30' the same as for the Parcel Payload Option, sets "Option Data Len" to 14, sets Code to 255 and sets Check to the same value as Hop Limit. Next, the source sets Parcel Path MTU to the 32-bit MTU of the outgoing (parcel-capable) interface for the probe, sets Residual Path MTU to the 16-bit value 'ffff', and sets Identification to a 32-bit identification value for the next packet/parcel/AJ to be sent to this destination. The source then sets the Parcel Limit, Reserved and O fields all to 0, and sets the X flag to 1 if it is probing the "e(X)treme path" for OMNI links (see below). The source finally adds any padding options necessary for 8-octet alignment and sends the packet to the next hop. Each node in the path that observes this specification (including IPv6 routers and the final destination itself) examines the packet and processes the Parcel Probe Option as follows: If Code is 255 and Check contains the same value as the IPv6 header Hop Limit, then set Parcel Path MTU to the minimum of its current value, the previous hop link MTU, and the node's own receive buffer size (but no smaller than the IPv6 minimum MTU ). Next increment Parcel Limit by 1 and, if the previous hop link was an OMNI link, set the O flag to 1. Then (for routers) forward the probe to the next hop while decrementing Hop Limit by 1 and setting Check to the new Hop Limit value. If Code is not 255 or Check contains a different value than the IPv6 header Hop Limit, then set Residual Path MTU to the minimum of its current value, the previous hop link MTU, and the node's own receive buffer size (but no smaller than the IPv6 minimum MTU ). Then, (for routers) forward the probe to the next hop while decrementing Hop Limit by 1 and setting Check to 255. When the destination receives the probe, it performs the above operations and also sets Residual Path MTU to 0 if Code is 255 and Check contains the same value as the IPv6 header Hop Limit. The destination then returns a responsive {TCP,UDP}/IPv6 packet that includes a Parcel Probe Reply Option as a {TCP,UDP} option formatted as shown in .

The destination sets Kind to 253 for TCP or 127 for UDP , then sets Length to 16 and ExID to TBD1 (see: IANA Considerations). The destination then sets Parcel Path MTU, Residual Path MTU, Parcel Limit, Reserved, O, X and Identification to the values included in the probe, i.e., after its own local probe processing as discussed above. The destination then includes any additional identifying parameters (such as authentication codes) in the {TCP,UDP}/IPv6 packet and returns the packet to the source while discarding the probe. The original source can therefore send parcel probes in the same packets used to carry real data. The probes will transit all routers on the forward path possibly extending all the way to the destination. If the source receives a probe reply, it authenticates the message and matches the Identification value with one of its previous probes. If a match is confirmed, then the Parcel Probe Reply Option will contain all information necessary for the source to use in its future parcel/AJ transmissions to this destination. In particular, the Parcel Path MTU determines the largest-size parcel/AJ that can transit the leading portion of the path up to a point that packetization would be necessary. If the O flag is set and X is clear, then the maximum-sized AJ is limited to 65535 octets while parcels as large as the Parcel Path MTU can be accommodated; if both the O and X flags are set, then the maximum-sized AJ is also bounded by the Parcel Path MTU which may exceed 65535 octets. If Residual Path MTU is non-zero, its value determines the maximum-sized packet that can transit the remainder of the path following packetization noting that the maximum packet size may be smaller still if there are routers in the probed path that do not recognize the protocol. (Note that a Residual Path MTU value of 0 instead indicates that the path is parcel-capable in all hops from the source to the destination.) Finally, Parcel Limit contains the value the source must place in the IPv6 Hop Limit field of future parcels/AJ transmissions to this destination. All routers and destinations within a controlled environment / limited domain are expected to forward or accept packets with IPv6 Hop-by-Hop Options extension headers without dropping them, i.e., even if they ignore the option contents. Conversely, for open Internetworks outside of a controlled environment / limited domain some paths may be unable to transit IPv6 packets that contain Hop-by-Hop Options extension headers. Sources that connect to open Internetworks should therefore send "augmented" probes that include a UDP header inserted between the IPv6 header and the Hop-by-Hop Options extension header. The source next rewrites the Hop-by-Hop Options Next Header field per Section 6.4 of with the "Type" component set to OMNI-HBH and with the "Next" component set to the value for the next header that follows (e.g., OMNI-TCP, OMNI-UDP, etc.). Next, the source sets the IPv6 Next Header field to UDP ("17"), sets the UDP port numbers to OMNI ("8060"), calculates and sets the UDP Checksum, then sends the prepared probe to the destination. This implies that all routers that recognize parcels/AJs and all destinations that accept them must be capable of accepting and processing the contents of these OMNI protocol UDP messages as though they arrived as ordinary probes. Such routers and destinations must therefore implement enough of the OMNI interface to be able to recognize and process the messages. When there may be one or more OMNI links in the path, the source can optionally send probes that test and measure the OMNI link "e(X)treme path" which uses jumbo-in-jumbo encapsulation instead of IP fragmentation (see: ). In one approach, the source can first send probes with the X flag set to 0. If the probe reply returns with the O flag set to 1, and if jumbo-in-jumbo encapsulation is needed (e.g., to forward very large AJs and parcels at extreme data rates), the source can next send probes with the X flag set to 1. The source can then remember the MTU and Parcel Limit values for both types of probes, and can subsequently send smaller parcels/AJs using the first set of parameters while sending larger parcels/AJs using the second set of parameters. All parcels/AJs also serve as implicit probes and may cause a router in the path to return an ordinary ICMPv6 error and/or Packet Too Big (PTB) message concerning the parcel if the path changes. If the path changes, a router in the path may also return a Parcel Report (subject to rate limiting per ) as discussed in .

When the destination returns a Parcel/Jumbo Report, it packages the report as a {TCP,UDP} option in a {TCP,UDP}/IPv6 packet to return to the source the same as for a Parcel Probe Reply (see: ). For a positive report, the destination may set Parcel Path MTU and Residual Path MTU to smaller values that reflect its (reduced) receive buffer size. For a negative report, the destination instead sets Parcel Path MTU, Residual MTU and Parcel Limit to 0 as an indication to the source that the path must be re-probed before sending additional parcels/AJs. When a router returns a Parcel/Jumbo Report, it prepares an ICMPv6 PTB message with Code set to either Parcel Report or Jumbo Report (see: IANA considerations) and with MTU set to either the minimum MTU value for a positive report or to 0 for a negative report. The router then writes its own IPv6 address as the Parcel/Jumbo Report source and writes the source address of the packet that invoked the report as the Parcel/Jumbo Report destination. The router next copies as much of the leading portion of the invoking parcel/AJ as possible (beginning with the IPv6 header) into the "packet in error" field without causing the entire Parcel/Jumbo Report (beginning with the IPv6 header) to exceed the IPv6 Minimum MTU. The router then calculates and sets the Checksum field the same as for an ordinary ICMPv6 message then sends the prepared Parcel/Jumbo Report to the original source of the probe. This implies that original sources that send parcels/AJs must be capable of accepting and processing Parcel/Jumbo reports (formatted as above) with coming from either a router or the final destination. Note: For positive Parcel/Jumbo reports, the source can continue sending parcels/AJs into the path with its segment sizes reduced accordingly. For negative Parcel/Jumbo reports, the source should instead re-probe the path before sending additional parcels/AJs.

This specification introduces an IPv6 Advanced Jumbo (AJ) service as a (single-segment) parcel alternative to basic jumbograms. Each AJ begins with a {TCP,UDP}/IPv6 header followed by the additional header encodings specified below. The AJ service employs the IPv6 Hop-by-Hop Parcel Payload Option the same as for IP Parcels; it sets Opt Data Len to 12 but replaces the 3-octet Parcel Payload Length field plus 1-octet preamble by a 4-octet Jumbo Payload Length field as shown in :

The source forms {TCP/UDP}/IPv6 AJs by setting the most significant octet of the Parcel Control field to 0 and treating the least significant octet of the field as an "Advanced Jumbo (AJ) Format" octet as shown in :

]]>

In the AJ Format octet, the source sets "X" to 0 for classic OMNI link traversal or 1 for "e(X)treme path" traversal. The source then sets "FEC" to 0 for "Non-FEC" or 1 for "IANA FEC" (other values may be specified in future documents) and sets "Digest" to one of the CRC/digest types found in . Implementations MUST support the following integrity checking algorithms identified by "Digest":

If FEC is 1, the source includes an "IANA FEC Header" as the first 4 octets of the AJ Payload as shown in :

The source sets FEC Scheme, FEC Instance and FEC Framework according to the appropriate registry values found in and sets Reserved to 0. The source next includes an (N+2)-octet AJ PIB formatted as shown in with the first N octets including the CRC/Digest according to the appropriate length given in and the final 2 octets including the Internet Checksum: "

The source then sets the IPv6 Payload Length to the length of the leading portion of the AJ to be covered by hop-by-hop FCS integrity checks noting that the remainder of the AJ will be covered only by end-to-end integrity checks after FEC is applied if necessary. The source next forms the {TCP/UDP}/IPv6 AJ the same as for parcels as shown in except that the PIB is followed by only a single segment ("Segment 0") and (for TCP) Sequence Number appears in the TCP header the same as for an ordinary TCP/IP packet. Unlike parcels, the AJ PIB CRC/Digest field can also be larger than 8 octets according to the length of the selected Digest. UDP AJs set the UDP Length field the same as specified for UDP parcels, and include a trailing UDP Option Length field if and only if the UDP length field encodes 0. (Note that the length of the UDP Option Length field itself is also included in the Jumbo Payload Length.) The source then includes a CRC/Digest in the AJ PIB for CRC-32, CRC-64, MD5 , SHA1 or the advanced US Secure Hash Algorithms according the to AJ Digest field value. (An AJ Digest value is also reserved by IANA as a non-functional placeholder for a nominal CRC128J algorithm, which may be specified in future documents; see: .) The source then sets the Identification the same as for a parcel, sets Code to 255 and sets Check and Hop Limit to the Parcel Limit for this destination (see: ). The source next calculates the {TCP,UDP} Checksum based on the same pseudo header as for an ordinary parcel (see: ) but with the 3-octet Parcel Payload Length field plus 1-octet preamble replaced with a 4-octet Jumbo Payload Length field and with Parcel Control encoding the AJ Format Octet; see above. The source calculates the header checksum only and writes the value into the {TCP,UDP} header checksum field the same as specified for parcels. For all AJ Digest values other than 0, the source then calculates the checksum of the segment payload, writes the value into the segment Checksum header, then calculates the CRC/digest over the length of the (single) segment beginning with the Checksum field and writes the value into the AJ PIB Digest field. The source then performs FEC encoding if necessary, resets the Jumbo Payload Length to include the additional length introduced by the FEC algorithm, then sends the AJ via the next hop link toward the final destination. At each forwarding hop, the router examines Code and Check then drops the AJ and returns a negative Jumbo Report if either value is incorrect. (Note that the AJ may also have been truncated in length by a previous-hop router that does not recognize the construct.) For all other intact AJs with FEC set to 0, if decrementing would cause the Hop Limit to become 0 the router performs packetization to convert the AJ into a packet the same as specified for parcels (see: ) and forwards the packet to the next hop. Otherwise, the router decrements both Hop Limit and Check by 1 and forwards the intact AJ to the next hop. When the AJ arrives, the destination parses the IPv6 header and Parcel Payload Option then applies FEC decoding for the payload if necessary. The destination then rewrites the Jumbo Payload Length to reflect the payload decrease due to FEC, then verifies the message CRC/Digest and Checksums. If all integrity checks agree, the destination delivers the AJ to upper layers. Note: If the original source selects the "e(X)treme path" for OMNI link traversal, the OMNI interface forwards the intact AJ as a carrier packet using jumbo-in-jumbo encapsulation instead of applying adaptation layer IP fragmentation. These jumbo carrier packets are then subject to best-effort delivery over the (previously-probed) path. The original source may select "e(X)treme" for any parcel/AJ, but must select "e(X)treme" for all AJs larger than 65535 octets - see: . Note: AJs that do not require an Identification can simply set the Identification field to 0.

OMNI interfaces can process parcels of all sizes as well as AJs as large as 65535 octets according to normal OMNI link parcellation, encapsulation and fragmentation procedures. For larger AJs as well as for parcels that may experience better performance by avoiding parcellation and fragmentation, the original source can instead select OMNI link "e(X)treme path" traversal. For probes/parcels/AJs that select the "e(X)treme path", the source sends the probe/parcel/AJ via the first-hop link under standard procedures specified in previous sections with standard IP forwarding providing service for each successive link up to the OMNI link ingress. When the probe/parcel/AJ arrives at the OMNI link ingress, the X flag provides an indication that "e(X)treme path" OMNI link traversal is desired as follows. For parcel/AJ probes, the OMNI link ingress first verifies that all previous hops were jumbo-capable by examining the Code and Check values. If Code or Check are incorrect, the OMNI link ingress clears the X flag and forwards the probe using normal OMNI encapsulation. If Code and Check are both correct, the OMNI link ingress instead inserts the OMNI and L2 encapsulations as specified in then performs "jumbo-in-jumbo" encapsulation by copying the (L3) Parcel Probe Option Hop-by-Hop extension header from the original IPv6 probe packet into the L2 headers as shown in . The OMNI link ingress then calculates the UDP checksum over the entire length of the encapsulated probe (as the UDP payload) and writes the value into the L2 UDP checksum field. Each L2 forwarding hop in the path to the next OAL intermediate node will then process the probe exactly as specified in , where each parcel/AJ capable hop adjusts the Code, Check, Parcel Path MTU and Parcel Limit fields then re-calculates/re-sets the L2 UDP checksum.

When each successive OAL intermediate node receives the parcel probe, it propagates the Parcel Probe Option Hop-by-Hop extension header into the L2 headers for the next OAL hop while updating the probe parameters the same as for an ordinary IP forwarding hop. When the OAL destination receives the parcel probe, it first verifies that all previous hops were jumbo-capable by examining the Code and Check values. If Code or Check are incorrect, the OAL destination drops the probe and returns a negative Jumbo Report to the OAL source, which then returns a negative Jumbo Report to the original source. Otherwise, the OAL destination removes the L2 and OAL headers while copying the L2 probe parameters into the L3 Parcel Probe Option (with the L2 encapsulation header lengths subtracted from the Parcel Path MTU). The OAL destination then forwards the probe to the next hop toward the final destination. If the probe traverses the entire path to the final destination, the Parcel Path MTU will contain the minimum MTU and the Parcel Limit will contain the total number of parcel/AJ-capable L2/L3 hops between the source and destination. (Note that the Residual Path MTU may also indicate that the final portion of the path is not parcel/AJ capable even though the leading portion of the path was.) The destination will then return a probe reply to the source, and if the X flag is set the source can begin sending parcels/AJs with the X flag set to enable the OMNI link "e(X)treme path". If the source receives an intact probe reply with X flag set, it can use the enclosed Parcel Path MTU, Residual Path MTU and Parcel Limit values to prepare future parcels/AJs for transmission via the "e(X)treme path" by setting the X flag. Each L3 forwarding hop in the path from the original source to the OMNI link ingress then forwards the parcel/AJ the same as for the standard procedures specified in previous sections. When the OMNI link ingress receives a parcel/AJ with the X flag set, it performs "jumbo-in-jumbo encapsulation" by leaving the L3 parcel/AJ headers intact, then appending OMNI adaptation layer IPv6 encapsulations plus L2 encapsulations that include a Parcel Payload Option with Advanced Jumbo Type 0 (but without including a segment checksum field as for {TCP,UDP} AJs) in either a full or minimal AJ extension header as an L2 extension. The OMNI link ingress sets the Jumbo Payload Length field to the length of the L2 extension headers (including the L2 UDP header, if present) plus the lengths of the OMNI IPv6 encapsulation header and the L3 packet (including all L3 headers). The OMNI link ingress sets all other OMNI and L2 encapsulation header fields as specified in . The parcel/AJ "jumbo-in-jumbo" encapsulation format is shown in . The OMNI link ingress then calculates the L2 UDP checksum over the L2 UDP/IP pseudo-header and extending to cover the OMNI adaptation layers up to but not including the L3 IP header, then writes the value into the L2 UDP header checksum field. The OMNI link ingress then copies the L3 TTL/Hop Limit into the L2 IP header TTL/Hop Limit and forwards the encapsulated parcel/AJ to the next L2 hop. When the parcel/AJ arrives at an OAL intermediate node, the node discards the L2 headers from the previous hop OMNI segment and inserts L2 headers for the next hop OMNI segment while updating the OMNI encapsulation header fields accordingly (see: ). In the process, the OAL intermediate node decrements the previous L2 hop TTL/Hop Limit and writes this value into the next L2 hop IP header while also transferring the previous hop Advanced Jumbo Type 1 header to the next hop L2 header chain. The node also re-calculates and re-sets the L2 UDP header checksum before forwarding toward the next OMNI hop. When the parcel/AJ arrives at the OAL destination, the OAL destination copies the L2 IP TTL/Hop Limit into the L3 IP TTL/Hop Limit field, then removes the L2 and OMNI encapsulation headers and forwards the packet to the next L3 hop while decrementing the IP TTL/Hop Limit by 1 according to standard IP forwarding rules. The final destination will then receive the intact original parcel/AJ. While a probe/parcel/AJ is traversing an OMNI link "e(X)treme path", it may encounter an L2 link that does not recognize the construct. This may cause a subsequent link to detect a formatting error and return a negative Jumbo Report that will be returned to a previous hop OAL intermediate node or the OAL source. The OAL node that receives the (L2) Jumbo Report must then prepare and generate an (L3) Jumbo Report to return to the original source. The L3 Jumbo Report contains the leading portion of the L3 probe/parcel/AJ with the L2 and OMNI headers removed. This will provide indication to the original source that the OMNI link "e(X)treme path" has failed for this particular transmission. Note: If an OMNI link ingress receives an "e(X)treme path" probe with an incorrect Code or Check, it clears the X flag and forwards the probe as an ordinary IP packet using standard OMNI encapsulation and fragmentation since a previous L3 hop was determined to be jumbo-incapable yet may be able to perform packetization. This is true even if there may be multiple OMNI links in the L3 path, where the X flag applies to all OMNI links in the series and not just the first. Note: The L2 UDP checksum extends over the entire length of each jumbo-in-jumbo encapsulated Parcel Probe, but only over the L2 and OMNI headers for each jumbo-in-jumbo encapsulated parcel/AJ following probing. This is due to the fact that the source must disguise the probe as an ordinary IP packet while probing is in progress; after probing has converged, subsequent parcels/AJs only require an integrity check of the headers.

IPv6 parcel/AJ integrity assurance responsibility is shared between lower layers of the protocol stack and the transport layer where more discrete compensations for lost or corrupted data recovery can be applied. In the classic link model, parcels and AJs are delivered to the final destination only if they pass the integrity checks of all links in the path over their entire length. In the DTN link model, intermediate system lower layers forward parcels/AJs with correct headers to the final destination transport layer even if the upper layer protocol data accumulates link errors at intermediate hops. The destination is then ultimately responsible for its own end-to-end integrity assurance. The {TCP,UDP}/IPv6 header of each parcel/AJ includes an integrity check of the header while the PIB includes integrity checks for each segment. The per-segment Checksums/CRCs are set by the source and verified by the destination. Note that both checks are important (when no other integrity checks are present) since there will be many instances when errors missed by the CRC are detected by the Checksum . IPv6 parcels can range in length from as small as only the {TCP,UDP}/IPv6 headers plus the PIB and a single segment to as large as the headers plus (64 * 65535) octets, while AJs include only a single segment that can be as small as a null segment to as large as 2**32 octets (minus headers). Due to parcellation/packetization in the path, the segment contents of a received parcel may arrive in an incomplete and/or rearranged order with respect to their original packaging. IPv6 parcels and AJs include a separate 2-octet Internet Checksum header for each segment noting that for UDP the per-segment Checksum value 0 indicates that the segment checksum is disabled. The original source calculates the checksum for each segment beginning with the first octet of the per-segment Sequence Number (for TCP non-first segments) then continuing over the entire segment length. IPv6 parcels with CRC=1/2 include a CRC-32/64 integrity check in the PIB. The original source uses either the CRC32C specification or the CRC64E specification and encodes the PIB. AJs that set an Advanced Jumbo Type other than NULL instead include either a 4/8 octet CRC or an N-octet message digest calculated per , or according to the hash algorithm assigned to Type. Under the DTN link model, the link far end discards the parcel/AJ if it detects an FCS error in the leading portion to avoid the possibility of misdelivery and/or corrupted FEC/PIB fields. Otherwise, the link far end unconditionally forwards the parcel/AJ to the next hop even if the upper layer protocol data incurred link errors. Following any FEC repairs, the PIB integrity checks will ensure that good data is delivered to upper layers. To support the parcel/AJ header checksum calculation, the network layer uses a modified version of the {TCP,UDP}/IPv6 pseudo-header found in Section 8.1 of as shown in . This allows for maximum reuse of widely deployed code while ensuring interoperability.

where the following fields appear: Source Address is the 16-octet IPv6 source address of the prepared parcel/AJ. Destination Address is the 16-octet IPv6 destination address of the prepared parcel/AJ. For parcels, the 4-octets of the Parcel Payload Length encode the Index/X/S preamble and 24-bit Parcel Payload Length as they appear in the Parcel Payload Option fields of the same name. For AJs, the Jumbo Payload Length encodes the 4-octet Jumbo Payload Length value found in the Parcel Payload Option. Parcel Control is the value that appears in the Parcel Payload Option header. zero encodes the constant value 0. Next Header is the IP protocol number corresponding to the transport layer protocol, i.e., TCP or UDP. When the transport layer protocol entity of the source delivers a parcel body to the network layer, it presents the values L and J along with the J segments in canonical order as a list of data buffers. (For AJs, the transport layer instead delivers the singleton AJ segment along with the Jumbo Payload Length.) When the network layer of the source accepts the parcel/AJ body from the transport layer protocol entity, it calculates the Internet checksum for each segment and writes the value into the correct PIB field (or writes the value 0 when UDP checksums are disabled). For parcels/AJs that include CRC/digest integrity checks, the network layer then calculates the CRC/digest for each segment beginning with the per-segment Checksum (followed by the Sequence number for TCP) and inserts the result in the correct PIB field in network byte order. The network layer then concatenates all segments then appends the PIB plus all necessary {TCP,UDP}/IPv6 headers and extensions to form a parcel. The network layer next calculates the {TCP,UDP}/IPv6 header checksum over the length of only the {TCP,UDP} headers plus IPv6 pseudo header then forwards the parcel to the next hop without further processing. When the network layer of the destination accepts an AJ or reunifies a parcel from one or more sub-parcels received from the source it first verifies the {TCP,UDP}/IPv6 header checksum then for each segment verifies the CRC/digest (if present) followed by the Checksum (except when UDP checksums are disabled) and marks any segments with incorrect integrity check values as errors. When the network layer of the destination restores a parcel from one or more individual {TCP,UDP}/IPv6 packets received from the source, it verifies the Internet checksum of each individual packet (except when UDP checksums are disabled), restores the parcel, and delivers each parcel/AJ segment along to the transport layer. Note: Classical links often use CRC32 as their hop-by-hop integrity checking service and this specification assumes that future DTN-capable links will also use CRC32. Since the error detection resolution for CRC32 diminishes for frame sizes larger than ~9KB, implementations should select hop-by-hop integrity protection for only the leading portions of parcels/AJs while leaving the remaining payload for end-to-end integrity checks. Hop-by-hop integrity checks should at a minimum extend to cover the {TCP,UDP}/IP headers (plus options/extensions) plus the FEC preamble and PIB. Note: for AJs, the source performs FEC encoding after calculating the CRC/Checksums and the destination performs FEC decoding before verifying the CRC/Checksums. This ensures that the source and destination work from identical copies of the original packet provided any errors incurred in the path were corrected. Note: the source and destination network layers can often engage hardware functions to greatly improve CRC/Checksum calculation performance.

Common widely-deployed implementations include services such as TCP Segmentation Offload (TSO) and Generic Segmentation/Receive Offload (GSO/GRO). These services support a robust service that has been shown to improve performance in many instances. An early prototype of UDP/IPv4 parcels (draft version -15) has been implemented relative to the linux-5.10.67 kernel and ION-DTN ion-open-source-4.1.0 source distributions. Patch distribution found at: "https://github.com/fltemplin/ip-parcels.git". Performance analysis with a single-threaded receiver has shown that including increasing numbers of segments in a single parcel produces measurable performance gains over fewer numbers of segments due to more efficient packaging and reduced system calls/interrupts. For example, sending parcels with 30 2000-octet segments shows a 48% performance increase in comparison with ordinary packets with a single 2000-octet segment. Since performance is strongly bounded by single-segment receiver processing time (with larger segments producing dramatic performance increases), it is expected that parcels with increasing numbers of segments will provide a performance multiplier on multi-threaded receivers in parallel processing environments.

The IANA is instructed to add a reference to this document ([RFCXXXX]) in the "Minimum Path MTU Hop-by-Hop Option" entry in the "Destination Options and Hop-by-Hop Options" table of the 'ipv6-parameters' registry. The IANA is instructed to assign new Code values in the "ICMPv6 Code Fields: Type 2 - Packet Too Big" table in the 'icmpv6-parameters' registry (registration procedure is Standards Action or IESG Approval). The registry entries should appear as follows:

The IANA is requested to assign two new entries in the 'ipv6-parameters' registry "Destination Options and Hop-by-Hop Options" table (registration procedures IESG Approval, IETF Review or Standards Action). The first entry sets "Hex Value" to '0xE2', "acct" to '11', "chg" to '1', "rest" to '00010' and Description to "Minimal Parcel With Errors". The second entry sets "Hex Value" to '0x10', "acct" to '00', "chg" to '0', "rest" to '10000' and Description to "Parcel With Errors". Both entries set "Reference" to this document [RFCXXXX]. The IANA is instructed to assign a new entry in the "TCP Experimental Option Experiment Identifiers (TCP ExIDs)" table of the 'tcp-parameters' registry (registration procedures First Come First Served per ). The table entry should set "Value" to TBD1, "Description" to "Parcel Parameters" and "Reference" to this document [draft-templin-6man-parcels]. The IANA is also instructed to assign the same value TBD1 as an entry in the to-be-created "UDP Experimental Option Experiment Identifiers (UDP ExIDs)" table (registration procedures First Come First served per ). This document places no preferences on the actual TBD1 value assignment. Finally, the IANA is instructed to create and maintain a new registry titled "IPv6 Parcel and Advanced Jumbo Formats and Types" as follows: For IPv6 parcels and Advanced Jumbos, the value in the 'Opt Data Len' field of the IPv6 Minimum Path MTU Option serves as an "Option Format" code that distinguishes the various option formats specified in this document. Initial values are given below:

For IPv6 Advanced Jumbos, when the most significant octet of the Parcel Payload Option Parcel Control field encodes the value 0, the least significant 4 bits of the field encode an "Advanced Jumbo Type" value. The IANA is therefore instructed to establish an "IPv6 Advanced Jumbo Types" registry with the initial values given below:

In the control plane, original sources match the Identification (and/or other identifying information) received in Parcel Reports with their earlier parcel/AJ transmissions. If the identifying information matches, the report is likely authentic. When stronger authentication is needed, nodes that send Parcel Reports can apply the message authentication services specified for AERO/OMNI. For nodes that include {TCP,UDP} Parcel Parameter Options in ordinary data packets, however, the authenticating services that apply to the data packets also authenticate the options. In the data plane, multi-layer security solutions may be needed to ensure confidentiality, integrity and availability. According to , a full IPv6 implementation includes the Authentication Header (AH) and Encapsulating Security Payload (ESP) per the IPsec architecture to support authentication, data integrity and (optional) data confidentiality. These AH/ESP services provide comprehensive integrity checking for parcel/AJ upper layer protocol headers and all upper layer protocol payload that follows. Since the network layer does not manipulate transport layer segments, parcels/AJs do not interfere with transport or higher-layer security services such as (D)TLS/SSL which may provide greater flexibility in some environments. IPv4 fragment reassembly is considered dangerous at high data rates where undetected reassembly buffer corruptions can result from fragment misassociations . IPv6 is less subject to these concerns when the 32-bit Identification field is managed responsibly. IPv6 Parcels and AJs are not subject to fragmentation unless exposed to OMNI interface encapsulation which includes a 64-bit Identification space. For IPv6 parcels and AJs that engage the DTN link model, the destination end system is uniquely positioned to verify and/or correct the integrity of any transport layer segments received. For this reason, transport layer protocols that use parcels/AJs should include higher layer integrity checks and/or forward error correction codes in addition to the per-segment link error integrity checks. The CRC/digest codes included with parcels/AJs that engage the DTN link model provide integrity checks only and must not be considered as authentication codes in the absence of additional security services. Further security considerations related to IPv6 parcels and Advanced Jumbos are found in the AERO/OMNI specifications.

This work was inspired by ongoing AERO/OMNI/DTN investigations. The concepts were further motivated through discussions with colleagues. A considerable body of work over recent years has produced useful segmentation offload facilities available in widely-deployed implementations. With the advent of networked storage, big data, streaming media and other high data rate uses the early days of Internetworking have evolved to accommodate the need for improved performance. The need fostered a concerted effort in the industry to pursue performance optimizations at all layers that continues in the modern era. All who supported and continue to support advances in Internetworking performance are acknowledged. This work has been presented at working group sessions of the Internet Engineering Task Force (IETF). The following individuals are acknowledged for their contributions: Roland Bless, Scott Burleigh, Madhuri Madhava Badgandi, Joel Halpern, Tom Herbert, Andy Malis, Herbie Robinson, Bhargava Raman Sai Prakash. Honoring life, liberty and the pursuit of happiness.

TCP Extensions for High Performance are specified in , which updates earlier work that began in the late 1980's and early 1990's. These efforts determined that the TCP 16-bit Window was too small to sustain transmissions at high data rates, and a TCP Window Scale option allowing window sizes up to 2^30 was specified. The work also defined a Timestamp option used for round-trip time measurements and as a Protection Against Wrapped Sequences (PAWS) at high data rates. TCP users of IPv6 parcels/AJs are strongly encouraged to adopt these mechanisms. Since TCP/IPv6 parcels only include control bits for the first segment ("segment(0)"), nodes must regard all other segments of the same parcel as data segments. When a node breaks a TCP/IPv6 parcel out into individual packets or sub-parcels, only the first packet or sub-parcel contains the original segment(0) and therefore only its TCP header retains the control bit settings from the original parcel TCP header. If the original TCP header included TCP options such as Maximum Segment Size (MSS), Window Scale (WS) and/or Timestamp, the node copies those same options into the options section of the new TCP header. For all other packets/sub-parcels, the note sets all TCP header control bits to 0 as data segment(s). Then, if the original parcel contained a Timestamp option, the node copies the Timestamp option into the options section of the new TCP header. Appendix A of provides implementation guidelines for the Timestamp option layout. Appendix A of also discusses Interactions with the TCP Urgent Pointer as follows: "if the Urgent Pointer points beyond the end of the TCP data in the current segment, then the user will remain in urgent mode until the next TCP segment arrives. That segment will update the Urgent Pointer to a new offset, and the user will never have left urgent mode". In the case of IPv6 parcels, however, it will often be the case that the next TCP segment is included in the same (sub-)parcel as the segment that contained the urgent pointer such that the urgent pointer can be updated immediately. Finally, if a parcel/AJ contains more than 65535 octets of data (i.e., even if spread across multiple segments), then the Urgent Pointer can be regarded in the same manner as for jumbograms as described in Section 5.2 of .

For each parcel, the transport layer can specify any L value between 256 and 65535 octets. Transport protocols that send isolated control and/or data segments smaller than 256 octets should package them as ordinary packets, AJs, singleton parcels or as the final segment of a larger parcel. It is also important to note that segments smaller than 256 octets are likely to include control information for which timely delivery rather than bulk packaging is desired. Transport protocol streams therefore often include a mix of (larger) parcels and (smaller) ordinary packets, AJs or singleton parcels. The transport layer should also specify an L value no larger than can accommodate the maximum-sized transport and network layer headers that the source will include without causing a single segment plus headers to exceed 65535 octets. For example, if the source will include a 28 octet TCP header plus a 40 octet IPv6 header with 24 extension header octets the transport should specify an L value no larger than (65535 - 28 - 40 - 24) = 65443 octets. The transport can specify still larger "extreme" L values up to 65535 octets, but the resulting parcels might be lost along some paths with unpredictable results. For example, a parcel with an extreme L value set as large as 65535 might be able to transit paths that can pass large parcels/AJs natively but might not be able to transit a path that includes conventional links. The transport layer should therefore carefully consider the benefits of constructing parcels with extreme L values larger than the recommended maximum due to high risk of loss compared with only minor potential performance benefits.

This section postulates a 128-bit Cyclic Redundancy Check (CRC) algorithm for AJs termed "CRC128J". An Advanced Jumbo Type value is reserved for CRC128J, but at the time of this writing no algorithm exists. Future specifications may update this document and provide an algorithm for handling Advanced Jumbos with Type CRC128J.

Some modern operating systems include Generic Segment Offload (GSO) and Generic Receive Offload (GRO) services for use by Upper Layer Protocols (ULPs) that engage segmentation. For example, GSO/GRO support has been included in linux beginning with kernel version 4.18. Some network drivers and network hardware also support GSO/GRO at or below the operating system network device driver interface layer to provide benefits of delayed segmentation and/or early reassembly. The following sections discuss the linux GSO and GRO APIs.

GSO allows ULP implementations to present the sendmsg() or sendmmsg() system calls with parcel buffers that include up to 64 ULP segments, where each concatenated segment is distinguished by an ULP segment delimiter. The operating system kernel will in turn prepare each parcel buffer segment for transmission as an individual UDP/IP packet. ULPs enable GSO either on a per-socket basis using the "setsockopt()" system call or on a per-message basis for sendmsg()/sendmmsg() as follows:

cmsg_level = SOL_UDP; cm->cmsg_type = UDP_SEGMENT; cm->cmsg_len = CMSG_LEN(sizeof(uint16_t)); *((uint16_t *) CMSG_DATA(cm)) = gso_size;]]>

ULPs must set SEGSIZE to a value no larger than the path MTU via the underlying network interface, minus header overhead; this ensures that UDP/IP datagrams generated during GSO segmentation will not incur local IP fragmentation prior to transmission (Note: the linux kernel returns EINVAL if SEGSIZE encodes a value that exceeds the Path-MTU.) ULPs should therefore dynamically determine SEGSIZE for paths that traverse multiple links through Packetization Layer Path MTU Discovery for Datagram Transports (DPMTUD). ULPs should set an initial SEGSIZE to either a known minimum MTU for the path or to the protocol-defined minimum path MTU. The ULP may then dynamically increase SEGSIZE without service interruption if the discovered Path-MTU is larger.

GRO allows the kernel to return parcel buffers that contain multiple concatenated received segments to the ULP in recvmsg() or recvmmsg() system calls, where each concatenated segment is distinguished by an ULP segment delimiter. ULPs enable GRO on a per-socket basis using the "setsockopt()" system call, then optionally set up per receive message ancillary data to receive the segment length for each message as follows:

cmsg_len = CMSG_LEN(sizeof(int)); *((int *)CMSG_DATA(cmsg)) = 0; cmsg->cmsg_level = SOL_UDP; cmsg->cmsg_type = UDP_GRO; ... /* Receive per-message GRO segment length */ if ((segmentLength = *((int *)CMSG_DATA(cmsg))) <= 0) segmentLength = messageLength; ]]>

ULPs include a pointer to a "use_gro" boolean indication to the kernel to enable GRO; the only interoperability requirement therefore is that each UDP/IP packet includes a parcel buffer with an integral number of properly-formed segments. The kernel and/or underlying network hardware will first coalesce multiple received segments into a larger single segment whenever possible and/or return multiple coalesced or singular segments to the ULP so as to maximize the amount of data returned in a single system call. ULPs that invoke recvmsg( ) and/or recvmmsg() will therefore receive parcel buffers that include one or more concatenated received ULP segments. The ULP accepts all received segments and identifies any segments that may be missing. The ULP then engages segment ACK/NACK procedures if necessary to request retransmission of any missing segments.

<< RFC Editor - remove prior to publication >> Changes from version -09 to -10: Added Appendix on "GSO/GRO API". Updated text on handling UDP options. Changes from version -09 to -10: Allow UDP options to appear in larger parcels and AJs based on a "UDP Option Length" trailer. Changes from version -08 to -09: Terminology. Changes from version -07 to -08: Add terminology and general cleanup. Changes from version -06 to -07: TCP and UDP options for parcels now apply to all parcel segments and not just the first or final segment. TCP Sequence Numbers for parcels always appear in the PIB and with the TCP header Sequence Number set to 0. Changes from version -05 to -06: Moved all per-segment integrity checks into Parcel Integrity Block header. This allows hop-by-hop integrity checking of the end-to-end integrity check values. Changes from earlier versions: Submit for review.