On performance, a tuned vLLM setup serving Moonshot AI’s open-weight, 2.8-trillion-parameter Kimi K3 model reportedly tops 700 tokens per second for a single user on the seventh-generation Ironwood TPU . That speed figure comes from Inferact, the startup founded by vLLM’s creators that helped Google build vLLM-TorchTPU. Its test ran 16 Ironwood chips with speculative decoding, a method of speeding up large language model inference with a small draft model, against vLLM’s published setup on 16 Nvidia GB200s, which reached 452 tokens per second. Its kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the GB200 baseline’s throughput at small batch sizes without speculative decoding.
SGLang-TorchTPU will keep adding models, starting with DeepSeek V4.1 Flash and Reflection Beam. It's also working toward Day 0 TPU support for new open models, meaning support on release day, by the first quarter of 2027, according to RadixArk. Google, Meta and RadixArk plan to share more at SGLang Summit 2026, Nov. 12–13 in San Francisco.
Follow Tom’s Hardware on Google News , or add us as a preferred source , to get our latest news, analysis, & reviews in your feeds.
Shane Downing Social Links Navigation Contributing Writer Shane Downing is a Contributing Writer for Tom’s Hardware, covering consumer storage, PC hardware, and AI.
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/googles-tpus-for-ai-workloads-now-run-in-at-metas-data-centers-tpus-sold-to-external-customers-for-the-first-time-threaten-nvidias-market-dominance#main
- https://www.tomshardware.com/membership
- Iran threatens to attack undersea cables in the Strait of Hormuz, calls them 'legitimate new targets' in future conflict
- Pick up this DDR5 gaming PC for only $899
- China’s tech giants drop $35 billion on export-compliant Nvidia RTX Pro 5500 workstation GPU orders to power AI ambitions, according to reports
- Micron's Taiwan union will launch a one-day strike this week, threatening memory production
- Modder brings full Nvidia DLSS 5 suite to three generations of AMD gaming GPUs
Informational only. No financial advice. Do your own research.