In the ever-evolving field of computer networking, protocols play a crucial role in managing data exchange. One cornerstone is the OSI Seven-Layer Model, designed to standardize communication between computers by showcasing its complexity through a layered network model. From the hardware-centric Physical Layer to the application-centric Application Layer, each layer contributes to seamless communication.
Understanding OSI Protocol and the Transition to RDMA in HPC
A protocol is a set of rules, standards, or agreements established for data exchange within computer networks. Legally, the OSI (Open Systems Interconnection) Seven-Layer Model is an international standard designed to meet the requirements of open networks through its seven-layer network model. Each layer has specific functions and responsibilities that facilitate communication and data exchange between computers. It is worth noting that real-world network protocols may deviate from the OSI model based on practical needs and network architecture design and implementation.
TCP/IP is a protocol suite composed of various protocols, roughly divided into four layers: the Application Layer, Transport Layer, Network Layer, and Data Link Layer. TCP/IP is considered an optimized version of the seven-layer model.
Against the backdrop of high-performance computing (HPC) and its demand for high throughput and low latency, TCP/IP has transitioned to RDMA (Remote Direct Memory Access). TCP/IP has some drawbacks, including latency and significant CPU overhead due to multiple context switches and CPU involvement in encapsulation during transmission.
RDMA, as a technology, allows direct access to memory data through the network interface without involving the operating system kernel. It enables high-throughput, low-latency network communication, making it ideal for large-scale parallel computing clusters.
Spine-Leaf Architecture vs. Traditional Three-Layer Networks
Traditional data centers typically employ a three-tier architecture, consisting of the access layer, aggregation layer, and core layer. However, traditional three-tier network architectures have significant drawbacks, which become more apparent with the development of cloud computing: bandwidth waste, large failure domains, and high latency.
The spine-leaf architecture offers significant advantages, including a flat design, low latency, and high bandwidth. In a spine-leaf network, the role of leaf switches is similar to traditional access switches, while spine switches act as core switches. This architecture achieves non-blocking performance. Since each leaf in the structure is connected to every spine, any issue with one spine only results in a slight decrease in throughput performance for the data center.
A Deep Dive into NVIDIA SuperPOD Architecture
SuperPOD refers to a cluster of servers interconnected through multiple computing nodes to provide high-throughput performance. Taking the NVIDIA DGX A100 SuperPOD as an example, the recommended configuration utilizes the QM8790 switch, offering 40 ports, each operating at 200G. The architecture employed follows a fat-tree (non-blocking) structure.
In terms of switch performance, the QM9700 introduced in the DGX H100 SuperPOD recommended configuration incorporates Sharp technology. This technology utilizes an aggregator manager to construct Streaming Aggregated Trees (SATs) within the physical topology. Multiple switches in the tree execute parallel computation, thereby reducing latency and enhancing network performance. The QM8700/8790+CX6 supports up to 2 SATs, while the QM9700/9790+CX7 supports up to 64 SATs. As the number of ports increases, the number of switches decreases.
Switch Choices: Ethernet, InfiniBand, and RoCE Compared
The fundamental difference between Ethernet switches and InfiniBand switches lies in the distinction between the TCP/IP protocol and RDMA (Remote Direct Memory Access). Currently, Ethernet switches are more commonly used in traditional data centers, while InfiniBand switches are more prevalent in storage networks and high-performance computing (HPC) environments.
Modern data centers demand underlying interconnects with maximum bandwidth and extremely low latency. In this scenario, traditional TCP/IP network protocols prove inadequate, resulting in CPU processing overhead and high latency.
For enterprises deciding between RoCE and InfiniBand, careful consideration of unique requirements and cost factors is crucial. Those prioritizing the highest performance network connections may find InfiniBand preferable, while those seeking optimal performance, ease of management, and cost-effectiveness may choose RoCE in their data centers.
Inquiry and Answers on InfiniBand Technology
With the advancement of big data and artificial intelligence technologies, the demand for high-performance computing continues to rise. To meet this demand, the NVIDIA Quantum-2 InfiniBand platform provides users with outstanding distributed computing performance, achieving high-speed, low-latency data transmission, and processing capabilities.
FS’s InfiniBand solutions include AOC/DAC cables and modules with speeds of 800G, 400G, 200G, 100G, and 56/40G, as well as NVIDIA InfiniBand adapters and NVIDIA InfiniBand switches. In IB network cluster solutions, FS’s professional team will provide corresponding hardware connectivity solutions based on the network. Tailored to your needs and network scale, ensuring network stability and high performance.
For more inquiries and answers regarding InfiniBand technology, please read Inquiries and Answers about Infiniband Technology.
How FS Can Help
FS offers a rich array of products supporting RoCE or InfiniBand. Regardless of your choice, it provides lossless network solutions based on these two network connectivity options. These solutions enable users to build high-performance computing capabilities and lossless network environments. Sign up now to improve your connectivity or request a customized consultation for high-speed solution design.
