Multicast for Computing and Storage

There are applications in data center with point-to-multipoint communication patterns that would benefit from network multicast service, without which, these applications, when migrating to public clouds, will use server based packet replication techniques. This leads to CPU load inflation and prevents tenants from sustaining high throughputs and low latencies for multicast workloads. In order to better show the requirements for computing and storage, we list 3 typical potential multicast scenarios with P2MP services: Multi-tenant Cloud, Computing and Storage. The multicast requirements could be described with the following 3 aspects: - Network Scale: number of switches, number of links, number of hosts - Multicast Tree Size: number of intermeidate nodes; number of receivers - Multicast Service Number

As illustrated in the following figure, a data-center may contain: a network fabric configured in unicast-only mode, hosts running as virtual machines (VMs) managed by tenants, central replicators (C-R) for providing MSR6 packet delivery service among the hosts of a tenant.

Central-Replicator +-+--+ +-+--+ +-+--+ | | | +-----+--------+--------+-----+ | Spine +Leaf +vSwitch Fabric | | (Unicast Only) | +--+--+--+--+--+--+--+--+--+--+ | | | | | | | | h1 | h3 h4 | h6 | h8 ====>Tenant-1 | | | h2 h5 h7 ====>Tenant-2 ]]> Take tenant-1 for example. The host h1 can send multicast flow using MSR6 packets to C-R1, the MSR6 packets include one or more of the destination hosts h3/h4/h6/h8 encoding in the MSR6 header. An MSR6 packet may be sent to C-R1 where it is replicated and sends to the desired destination hosts. An MSR6 packet may be sent to C-R1 where it is replicated and sends to the part of the destination hosts, and another copy to C-R2 for replication and delivery to the left destination hosts. A Tenant may have a dedicated set of C-Rs for its own use, or a Tenant may use a shared C-Rs for its replication requirement among VMs.

The following figure shows a typical RDMA AI training scenario.

Worker->PS: The gradient of each worker is pushed to PS node PS->Worker: PS will pull the parameters back to all workers after aggregation In this process, the second stage is information distribution, with the same data content. N connections are used to transmit unicast separately. The bandwidth efficiency is 1/N, the larger the scale, the lower the efficiency.

If the source only sends 1 copy to the network and the switches replicate the packet to different distinations. The use of bandwidth is more efficient and the training is faster. The large-scale multicast requirement in this scenario is as the following: - Network Scale: 10-10k GPU - Multicast Tree Size: 10-10k receivers - Multicast Service Number: depends on the scenario

The following is an example of MPI in HPC scenario.

| RoCE |<------------->| RoCE || |+------+| |+------+| |+------+| +--------+ +--------+ +--------+]]> Stage 1: Dispatcher Master senses millions of cores and schedules millions of Rank MPI jobs on demand. Dispatcher Master sends the scheduling results to Dispatcher Agent Stage 2: Dispatcher Agent starts Million Rank MPI on each node The Dispatcher Agent that receives the message broadcast the message to other Dispatcher Agents and do the initialization before starting the MPI application Stage 3: Dispatcher Agent broadcaast the message to start the MPI application. MPI internal initialization Synchronize the RoCE endpoint in allgather way after the MPI application is started The last 2 stages could benefit from multicast and reduce task completion time. The large-scale multicast requirement in this scenario is as the following: - Network Scale: 1000 k CPU/GUP - Multicast Tree Size: 10k~100k receivers - Multicast Service Number: 1~100

Ceph is an open-source distributed software platform. It mainly focuses on scale-out file system including storage distribution and availability, which is widely used in storage. Ceph Object Storage Daemons (OSDs) are reponsible for storing objects on a local file system on behalf of Ceph clients. Also, Ceph OSDs use the CPU, memory, and networking of Ceph cluster nodes for data replication, erasure coding, recovery, monitoring and reporting functions. The following process request P2MP service. - Application initiates "write" operation from a client to a server. - Client finds the server to write in, and 3 copies are sent to 3 services.

The large-scale multicast requirement in this scenario is as the following: - Network Scale: 3k Server (1 Pod) - Multicast Tree Size: 3 receivers - Multicast Service Number: 10k

This document makes no request of IANA.

TBD