Power and cooling limits force AI clusters with 100,000+ accelerators to use multiple data centers

Power and cooling limits force AI clusters with 100,000+ accelerators to use multiple data centers

The transition toward scale-across architectures has already begun. Google introduced its Virgo network architecture earlier this year to support AI workloads that exceed the space and power capacity of individual data centers. The company intends to treat an entire campus of computing facilities as a single system, albeit with specialized networks for communication between accelerators, servers, and data centers. Cisco and Marvell are also developing technologies for scale-across deployments, including high-capacity routers, switching chips, and optical interconnects designed to connect geographically distributed clusters.

Extending AI infrastructure beyond individual facilities introduces substantial technical challenges. While modern scale-up networks offer extremely high bandwidth and low latency over relatively short distances, communication between data centers must contend with the physical limitations of long-distance connections. Even optical networks cannot eliminate propagation delays, whereas congestion, packet loss, and synchronization overhead can further reduce effective performance. Consequently, connecting two large GPU clusters with high-speed links does not necessarily make them behave like one tightly integrated computing system.

These challenges are particularly important for AI training, where accelerators frequently exchange model parameters, gradients, and intermediate results. Large-scale workloads require efficient collective communications, as delays that affect individual accelerators dramatically reduce the performance of the entire cluster. Therefore, the efficiency of distributed computing depends not only on raw network bandwidth, but also on network topology, congestion management, communication protocols, and software implementations.

To address these limitations, network adapters and switches are becoming more sophisticated than ever. Modern network interface controllers can perform Remote Direct Memory Access (RDMA) operations without continuous CPU involvement, whereas technologies such as HPE Slingshot hardware collectives and Nvidia SHARP can offload certain communication operations to the network infrastructure itself. SmartNICs and data processing units further reduce the burden on host processors as they handle selected network and infrastructure functions in dedicated hardware. Such capabilities can improve efficiency of distributed applications without necessarily requiring faster physical interconnects.

Interestingly, software plays an equally important role in the performance of large-scale AI infrastructure, as communication frameworks like libfabric, MPI, or UCX determine how applications exchange data and use the capabilities of modern network adapters. Also, GPU-aware communication and technologies like GPUDirect RDMA can eliminate unnecessary transfers through host memory, while optimized collective operations can reduce communication overhead across large numbers of accelerators, according to the researchers.

The growing complexity of AI infrastructure also changes how operators should evaluate network performance. Although the industry continues to introduce faster interconnect technologies, including versions of GbE with nominal bandwidths of 800 Gb/s and 1.6 Tb/s, peak transfer rates alone do not fully describe the capabilities of a given system because the performance of actual applications depends on multiple factors, including how effectively network hardware and software cooperate to transfer data, prevent congestion, and synchronize computations across thousands of processors.

Ultimately, the emergence of scale-across interconnection architectures reflects a fundamental change in the design of AI infrastructure. As the computational capabilities of individual accelerators increase and AI clusters expand beyond individual facilities, efficient communication becomes just as important as processor performance. As a result, the next generation of AI systems will depend not only on faster AI accelerators or CPUs, but also on sophisticated network adapters, switches, and, perhaps more importantly, software that can unite geographically distributed computational resources without excessive communication overhead.

Anton Shilov Social Links Navigation Contributing Writer Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.

Key considerations

  • Investor positioning can change fast
  • Volatility remains possible near catalysts
  • Macro rates and liquidity can dominate flows

Reference reading

More on this site

Informational only. No financial advice. Do your own research.

Leave a Comment