This isn’t a tweak around the edges being made by tinkerers: some of the biggest GPU firms are doing something similar. Nvidia’s Blackwell power profiles fine-tune everything from GPU compute and memory frequencies, power limits, NVLink states and cache settings to fit the workload. Nvidia believes that can save up to 15% of energy while retaining 97% or more of overall performance, allowing power-constrained facilities to run more GPUs, raising throughput by as much as 13%.
Chetan Visrolia, business development manager for data center infrastructure at SHI, said in an interview with Tom’s Hardware Premium that the most obvious waste he encounters is often legacy equipment supporting old code. “It is always the lowest-hanging fruit for quick savings,” he said. “It can easily be found in any data center, as it is the loudest rack on the floor.”
Cloud instances can also be made a smarter size, and AI prompts that are repeatedly used can be stored in cache rather than recomputed on the fly. There are also efficiencies to be made by getting small models to handle routine requests while holding back the larger ones for harder tasks. At the same time, better batching and compilers can raise accelerator utilisation, and crunching down prompts while throwing limits on outputs can also tamp down the number of tokens processed.
Software can also change when and where electricity is drawn. Sophie Hall, a doctoral student at ETH Zurich’s Automatic Control Laboratory who has studied workload shifting across Google ’s global data-center fleet, argues in an interview with Tom’s Hardware Premium that electricity consumption isn’t necessarily the main problem.
“It’s more like: when do they use it, where do they use it, and how is it interacting with the grid?” she said. If batch jobs aren’t time-sensitive, they could be delayed until local demand falls or routed to a region with spare capacity and lower-carbon electricity. Hall’s research uses day-ahead planning and real-time scheduling to respond to grid signals while preserving performance guarantees.
OpenAI and Anthropic scramble for smaller data centers as massive gigawatt projects lag
US AI data centers projected to become the fifth-largest natural gas consumer in the world by 2035
Taking that kind of action was less necessary when hyperscalers built way more capacity than they actually needed and the grid wasn’t a bottleneck, Hall said. But the massive demand for AI has changed the equation.
There are limits, though: individual countries’ data sovereignty rules can prevent a workload from crossing borders, while moving large datasets across networks is something that inference providers may be wary of in case things go wrong. “The data needs to literally, physically, geographically move to a different location,” Hall said. “That’s not that easy.”
The scale of the data center demand powered by AI means that software isn’t a silver bullet or a substitute for better chips and new electricity infrastructure. But it can act as a control layer that squeezes out every last drop from those increasingly expensive assets, while also helping shift demand to places where there is more capacity.
Jevons ’ paradox is also at play, and needs to be considered when thinking about making systems run better. Efficiency doesn’t reduce a facility’s electricity use if each watt saved gets used up generating even more tokens. “Power is the core bottleneck in AI data centers,” said Chung. “We really want to make the best use of every watt we consume.”
Chris Stokel-Walker Freelance Contributor Chris Stokel-Walker is a Tom's Hardware contributor who focuses on the tech sector and its impact on our daily lives— online and offline. He is the author of How AI Ate the World, published in 2024, as well as TikTok Boom, YouTubers, and The History of the Internet in Byte-Sized Chunks.
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/data-centers/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/data-centers/software-could-be-the-easiest-fix-for-hyperscalers-ai-power-squeeze-researchers-say-data-center-demand-is-expected-to-rival-japans-electricity-usage-by-2030#main
- https://www.tomshardware.com/my-account
- How NVIDIA GPUs Help Accelerate OpenAI’s GPT-6 Astra Ultrafast
- U.S. suspends green card path for H-1B workers at Microsoft and Adobe — labor certification program blocked due to alleged fraud
- How NVIDIA GPUs Help Accelerate OpenAI’s GPT-6 Astra Ultrafast
- Sneak a great deal on the original MSI Claw during Best Buy clearance sale
- U.S. suspends green card path for H-1B workers at Microsoft and Adobe — labor certification program blocked due to alleged fraud
Informational only. No financial advice. Do your own research.