Follow Tom's Hardware on Google News , or add us as a preferred source , to get our latest news, analysis, & reviews in your feeds.
Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals. ","collapsible":{"enabled":true,"maxHeight":250,"readMoreText":"Read more","readLessText":"Read less"}}), "https://slice.vanilla.futurecdn.net/13-4-25/js/authorBio.js"); } else { console.error('%c FTE ','background: #9306F9; color: #ffffff','no lazy slice hydration function available'); } Bruno Ferreira Social Links Navigation Contributor Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals.
usertests It appears to be taking the benefits of a mixture of experts model to an illogical extreme, although maybe it can be made useful with more optimizations. Normally I would just say save up your pennies for Strix/Gorgon/Medusa Halo or DGX Spark, which seem to be good at running big MOE models. But now I'm wondering if Intel has something worthwhile coming soon. Specifically a socketed Nova Lake-S desktop APU with 12 Xe3P cores. Even if it's limited to 128-bit memory, maybe you can min/max it to have better price/performance than Halo boxes. You could even use something like this Colibri project. And then there's Nova Lake-AX. Reply
jp7189 Im a fan boy of local models and this work is really cool. GLM 5.2 is a fine model, but dropping it down to int4 for this project loses a lot. Still, I hope this type of work inspires others towards better optimizations. Reply
timsSOFTWARE usertests said: It appears to be taking the benefits of a mixture of experts model to an illogical extreme, although maybe it can be made useful with more optimizations. Normally I would just say save up your pennies for Strix/Gorgon/Medusa Halo or DGX Spark, which seem to be good at running big MOE models. But now I'm wondering if Intel has something worthwhile coming soon. Specifically a socketed Nova Lake-S desktop APU with 12 Xe3P cores. Even if it's limited to 128-bit memory, maybe you can min/max it to have better price/performance than Halo boxes. You could even use something like this Colibri project. And then there's Nova Lake-AX. I think AMD's Epyc Venice will actually be very interesting on the CPU inference side. With 16 memory channels using MRDIMMs, it's expected to have ~100GB/channel x 16 channels = 1.6 TB/sec of memory bandwidth. While that still trails significantly behind leading edge AI accelerators – that are around 8TB/sec now – it's comparable to something like a 5090 (with 1.8TB/sec). For a large MOE model that has only 30-40B parameters active, that combo will probably actually be pretty usable. Reply
coldspring22 Very clever proof of concept. Maybe it will be useful if civilization collapsed and all you have is an old laptop. But I think old xeon sever with cheap DDR3 server ram/some old GPU cards will be much more feasible approach to run near frontier MOE models (yes still very slow, maybe 1-2 tokens/second) Reply
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/colibri-proof-of-concept-gains-frontier-level-1-5-tb-ai-model-novel-approach-runs-on-only-25gb-of-ram-and-shows-promise-for-local-ai-setups#main
- https://www.tomshardware.com/subscription
- Minecraft shown running on Game Boy Color and Game Boy in 3D with textures — developer coaxed 3D look out of barely-there hardware
- FCC approves orbital space mirrors, first test satellites will launch this year — large spacecraft reflects sunlight to Earth’s surface for construction sites,
- Windows 95 didn’t detect installers, it ‘guessed’ based on the file name, says veteran dev — it simply checked for words like setup, install, inst, or localized
- NVIDIA Nemotron Achieves Benchmark-Leading Performance With LangChain Deep Agents Harness
- Nvidia shows off GeForce Trading Cards Series 1 — collectible cards show off games, GPUs, and tech demos, and will be available for free at upcoming events
Informational only. No financial advice. Do your own research.