Problem Statement for High Performance Wide Area Networks
ZTE Corporation
- BDP:
- Bandwidth Delay Product
- DC:
- Data Center
- DCI:
- Data Centers Interconnection
- HPC:
- High Performance Computing
- WAN:
- Wide Area Networks
- PFC:
- Priority Flow Control
- ECN:
- Explicit Congestion Notification
- ECMP:
- Equal-Cost Multipath
- RTT:
- Round-Trip Time
- TCP:
- Transfer Control Protocol
- RDMA:
- Remote Direct Memory Access
- QUIC:
- Quick UDP Internet Connections
Technical Goals for HP-WANs
The services need to be provided in HP-WANs mainly focus on massive
data with timely transmission while multiple services may co-exist over
long-distance WANs as described below.
- Massive data transmission, high-volume data with high-speed transfer,
e.g. the data speed of a flow could be at 2Gbps~1Tbps.
- Requested completion time, the data transmission should be completed
within a requested completion time, e.g. the completion time could be
minutes~milliseconds.
- Scheduled transmission, traffic patterns could be scheduled by the
sender, e.g. data volume, start time, finish time, service type.
- Long-distance transmission over non-dedicated WANs, with multiple hops
and domains, long RTT latency, routing changes, network congestion,
packet loss, and link quality fluctuations, e.g. the distance between
two sites or DCs could be more than 100km or 1000km.
- Multiple services are co-existed with concurrent flows, with different
transport protocols for data transmission, such as QUIC, TCP and RDMA etc.
It is required to achieve high-speed data transmission within
a completion time. Moreover, it is also crucial to maximize bandwidth
utilization while ensuring fairness among multiple services. This
document outlines the technical goals for HP-WANs as described below.
- High throughput: ensuring the high-speed data transmission within
a requested completion time for a flow, which could be impacted by
the bandwidth, convergence speed, start time and RTT.
- Efficient use of capacity: efficiently using available network
capacity with fairness to maximize data transfer rates and minimize
the completion time for multiple flows.
Problem Statement
The specific requirements of HP-WANs may encompass a wide range of
aspects. These include transport-related technologies such as proxy,
flow control, QoS negotiation, congestion control, admission control
and traffic scheduling. Additionally, they also involve routing-related
technologies like traffic engineering, resource scheduling, and load
balancing.
Existing network technologies face numerous challenges and fall short
of meeting performance requirements. This document highlights the key
issues associated with HP-WANs in the following sub-sections.
Poor Convergence Speed
The traditional congestion control mechanisms perform blind transmission
by controlling the size of the congestion window with rate adjusting by
detection of overloaded links. WAN is a black box to provide unpredictable
behaviors for high-speed transmission due to the issues such as multiple
hops and domains, long Round-Trip Time (RTT), routing changes, network
congestion, packet loss, and link quality fluctuations. The BDP (Bandwidth
Delay Product) which represents the maximum amount of data that can be in
transit on the network at any given time is variable over WANs, so the
inflight data is difficult to predict for host-based congestion control
algorithms. It will lead to the poor convergence speed that the host always
takes significantly long time to identify the optimal sending rate comparing
to the requested completion time.
For example, it will use the slow start and blind detection with
unawareness of network capability leading to long convergence time
such as Cubic (e.g.over 50s), BBR (e.g.over 30s) and BBRv2 (e.g.30~50s).
BBR divides the entire process into four stages, Startup, Drain,
ProbeBW and ProbeRTT. The probe cycle of ProbeRTT state is long,
e.g. 10s. The convergence time will be multiple probe cycle which
will impact the completion time at seconds level. There is a significant
transmission capacity gaps between the appropriate sending rate and the
available network capacity. The transport protocols should signal and
collaborate with the network to negotiate the rate for the host to send
traffic.
Unscheduled Traffic
The host sending large unscheduled traffic without collaboration will
lead to the instantaneous congestion in WANs. For multiple high-speed
flows, the random arrival and departure of cross-traffic without scheduling
creates significant fluctuations for available capacity in WANs. The network
infrastructure may struggle to handle high-volume data transfers efficiently
if applications do not proactively schedule the traffic. Without awareness
of the traffic patterns, the network risks unscheduled resource allocation,
leading to low bottleneck bandwidth utilization, reduced overall throughput,
and uncontrolled completion time.
For example, for HPC applications, a large amount of data will be transmitted,
e.g. the data volumes of a single flow may be from 10G to 1TB, the host sends
the unscheduled large traffic leading to the instantaneous congestion, packet
loss, and queuing delay within network devices in WANs, resulting in low throughput.
Considering the multiple services with various types of flows, the optimal
bandwidth and transmission time may be different and the traffic is random to
join and leave without to be scheduled to multiple paths and fine-grained
network resources, which can not achieve the timely transmission. The resource
of WANs should be scheduled at the elements along the path to provide predictable
capability for high-speed transmission.
Long Feedback Loop
The congestion algorithms are implemented by controlling the size
of the congestion window and adjusting the sending rates upon the network
status feedback. It will delay the network feedback due to the long-distance
transmission delays and large RTT, resulting in the inability to adjust
the transmission rate in a timely manner. It will be challenging for congestion
control over WANs for controlling the total amount of data entering the
network to maintain the traffic at an acceptable level, leading to RTT
fluctuation due to long queues and large buffer at network devices
with high-speed transmission upon the long network state feedback loop.
Especially when multiple flows targeting an aggregating node, the maximum
value is exceeding devices buffer capacity.
For example, the loss-based congestion control algorithms, such as
Reno and CUBIC, depends on the congestion notification with packet loss.
Explicit Congestion Notification (ECN) can be used to achieve an
end-to-end congestion notification based on IP and transport layers.
When a congestion occurred, the network may signal congestion
by ECN markings or by dropping packets, and the receiver passes this
information back to the sender in transport-layer acknowledgements,
notifying the source to adjust the transmission rate. It will use the
slow start, requiring large buffer which is impacted by multiple hops
and long RTT latency over WANs.
And the congestion-based congestion control algorithms such as BBR,
depends on the measurement of congestion, it actively measures
bottleneck bandwidth (BtlBw) and round-trip propagation time (RTprop)
based on the model to calculate the BDP and then to adjust the
transmission rate to maximize throughput and minimize latency. But
BBR relies on real-time measurement of the parameters, and will
optimize the buffer overflow, but it is not significant under large
RTT, e.g. retransmission will increase when the buffer size is
less than two BDPs, thereby affecting the control precision of BBR
in long-distance networks.
Multiple Transport Protocols Adaption
Multiple services are coexisted for massive data transmission over
WANs with different transport protocols, such as QUIC, TCP and RDMA etc.
Multiple transport protocols, each handling substantial data transfer
requests, will coexist within the same network. Optimizing these diverse
transport protocols can entail significant overhead. This encompasses
issues such as redesigning congestion control algorithms, mapping parameters,
adapting hardware components, and formulating QoS policies. To improve
such significant overhead, a more flexible deployment strategy, such as
the implementation of a transport protocol proxy, can be enabled for
the adaptation of functionality to suit the requirements of different
transport protocols. The proxy should support high-speed transmission
such as traffic classification, packet processing, buffering, and
implement the collaboration and interaction between proxy and hosts.
Seamless communication between hosts and network infrastructure requires
adaptive coordination across heterogeneous transport protocols
(e.g., TCP, UDP, QUIC, RDMA).
Moreover, in some scenarios, it is difficult to simultaneously ensure
both encrypted data and high-speed transmission. Encryption algorithms
(e.g., AES, RSA) require intensive CPU operations, which reducing available
capacity for data transmission. Edge computing nodes with limited CPU
capabilities struggle to balance encryption and data processing.
The proxy could perform optimizations (e.g., hardware acceleration,
distributed encryption modules) to mitigate the bottlenecks.
Security Considerations
This document covers several of representative applications and
network scenarios that are expected to make use of HP-WAN
technologies. Each of the potential use cases does not raise
any security concerns or issues, but may have security
considerations from both the use-specific perspective and
the technology-specific perspective.
IANA Considerations
This document makes no requests for IANA action.
Acknowledgements
The authors would like to acknowledge Guangping Huang, Yao Liu and
Zheng Zhang for their thorough review and very helpful comments.
References
Normative References