AI-optimised networking revolutionises data centre scalability and performance

Emerging AI-specific networking technologies, including AI-NICs, new standards, and photonics, are transforming data centres to meet the demands of complex AI workloads, promising ultra-low latency and unprecedented scalability.

As artificial intelligence (AI) workloads become increasingly complex and data-intensive, the traditional networking architectures in data centers are proving inadequate for their unique demands. AI operations can be likened to millions of self-driving trucks transporting valuable cargo, data, at high velocity. This analogy underscores the necessity for a network infrastructure that is not only wide enough to handle massive throughput but also smooth in execution with ultra-low latency and intelligent traffic management to prevent bottlenecks.

While much of the industry’s focus has been on accelerating computation through GPUs and CPUs, recent insights indicate that the efficiency of data movement and synchronization across distributed AI systems is equally critical. AI performance hinges on networks capable of delivering consistent, deterministic speed, delays as low as five to ten microseconds, or even under one microsecond per server, are often the target to maintain training and inference accuracy. In this context, networking emerges as a potential bottleneck, limiting the ability to scale AI models effectively.

A key evolution in this space is the introduction of AI-optimized Network Interface Cards (AI-NICs). Unlike conventional NICs, which primarily shuttle data between systems, AI-NICs incorporate onboard compute capabilities. This allows them to perform in-network processing such as collective operations, preprocessing, and even AI inference acceleration directly within the NIC, alleviating workload from the host CPU, reducing latency, and improving throughput. Programmable and designed to support emergent AI-specific protocols like ultra Ethernet, AI-NICs transform the network from a mere data highway into an integral part of the AI processing ecosystem.

However, the explosive growth in data volume and interconnect traffic for AI workloads poses additional challenges at the data center infrastructure level. Backend networks must now cater to bandwidth requirements four to eight times greater than traditional cloud environments. This scale-up requires significantly more fiber cabling, often five to eight times more per AI server, which complicates deployments and retrofitting in existing facilities. Supply chain constraints, labour shortages, and power and cooling limitations in brownfield data centers exacerbate these difficulties. The dense concentration of GPUs and intricate high-speed networks also elevate risks of network failures, where even minor disruptions can waste precious GPU cycles and delay delivery of insights, inflating operational costs.

In response to these challenges, major technology firms are collaborating to establish new networking standards that better suit AI workloads. A notable initiative is the Ultra Accelerator Link Consortium, which includes Meta, Microsoft, AMD, Broadcom, Google, Cisco, Hewlett Packard Enterprise, and Intel. Their forthcoming open standard aims to enhance communication between AI accelerators and reduce dependence on Nvidia, which currently dominates around 80% of the AI chip market but is absent from this group. This consortium targets more efficient, interoperable AI data center networks, releasing specifications in the third quarter of 2024.

Meanwhile, Nvidia is pursuing its own advanced networking roadmap to maintain its competitive edge. By 2026, Nvidia plans to deploy silicon photonics and co-packaged optics (CPO) technology in AI data centers, significantly boosting bandwidth and efficiency beyond what traditional copper and optical modules offer. This transition will enable throughput at terabit speeds (up to 409.6 Tb/s), drastically reduce power consumption, and improve signal integrity. Nvidia’s upcoming Quantum-X InfiniBand and Spectrum-X Ethernet switches underscore the company’s commitment to supporting generative AI workloads with simplified, power-efficient architectures, positioning CPO as an essential standard rather than an optional upgrade.

Adding to this evolving landscape, Huawei is developing UB-Mesh, a novel interconnect protocol designed to unify AI data center communication by replacing multiple existing protocols such as PCIe, NVLink, CXL, and TCP/IP. Announced at Hot Chips 2025, UB-Mesh offers strikingly high bandwidth (up to 10 Tbps per chip) and ultra-low latency (around 150 nanoseconds per hop). It treats the entire data center as a single mesh supernode, facilitating massive scalability with fault-tolerant mechanisms. Huawei intends to open-source UB-Mesh to encourage widespread adoption, aiming to provide an alternative to Western standards. However, its success remains uncertain amid competition from more established protocols backed by industry heavyweights.

Overall, AI data center networking is transitioning from traditional, latency-tolerant designs to highly specialised, intelligent infrastructures. High-speed interconnect technologies, AI-driven load balancing, congestion control, and automation for dynamic traffic management are becoming fundamental in maintaining consistent performance at scale. The new generation of AI-aware hardware, from AI-NICs to photonics-based switches, represents a critical enabler for the next wave of AI innovation, facilitating seamless synchronization and efficient data exchange across vast, distributed GPU clusters. As AI continues to expand its footprint across industries, the race to optimise and future-proof data center networks will be pivotal in unlocking the technology’s full potential.

📌 Reference Map:

• ^[1] (Data Center Dynamics) – Paragraphs 1, 2, 3, 4, 5, 6
• ^[2] (Data Center Frontier) – Paragraph 4, 5
• ^[3] (Reuters) – Paragraph 6
• ^[4] (Tom’s Hardware) – Paragraph 7
• ^[5] (Juniper Networks) – Paragraph 8
• ^[6] (Tom’s Hardware) – Paragraph 9

Source: Noah Wire Services