Follow Tom's Hardware on Google News , or add us as a preferred source , to get our latest news, analysis, & reviews in your feeds.
Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals. ","collapsible":{"enabled":true,"maxHeight":250,"readMoreText":"Read more","readLessText":"Read less"}}), "https://slice.vanilla.futurecdn.net/13-4-25/js/authorBio.js"); } else { console.error('%c FTE ','background: #9306F9; color: #ffffff','no lazy slice hydration function available'); } Bruno Ferreira Social Links Navigation Contributor Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals.
usertests It appears to be taking the benefits of a mixture of experts model to an illogical extreme, although maybe it can be made useful with more optimizations. Normally I would just say save up your pennies for Strix/Gorgon/Medusa Halo or DGX Spark, which seem to be good at running big MOE models. But now I'm wondering if Intel has something worthwhile coming soon. Specifically a socketed Nova Lake-S desktop APU with 12 Xe3P cores. Even if it's limited to 128-bit memory, maybe you can min/max it to have better price/performance than Halo boxes. You could even use something like this Colibri project. And then there's Nova Lake-AX. Reply
jp7189 Im a fan boy of local models and this work is really cool. GLM 5.2 is a fine model, but dropping it down to int4 for this project loses a lot. Still, I hope this type of work inspires others towards better optimizations. Reply
timsSOFTWARE usertests said: It appears to be taking the benefits of a mixture of experts model to an illogical extreme, although maybe it can be made useful with more optimizations. Normally I would just say save up your pennies for Strix/Gorgon/Medusa Halo or DGX Spark, which seem to be good at running big MOE models. But now I'm wondering if Intel has something worthwhile coming soon. Specifically a socketed Nova Lake-S desktop APU with 12 Xe3P cores. Even if it's limited to 128-bit memory, maybe you can min/max it to have better price/performance than Halo boxes. You could even use something like this Colibri project. And then there's Nova Lake-AX. I think AMD's Epyc Venice will actually be very interesting on the CPU inference side. With 16 memory channels using MRDIMMs, it's expected to have ~100GB/channel x 16 channels = 1.6 TB/sec of memory bandwidth. While that still trails significantly behind leading edge AI accelerators – that are around 8TB/sec now – it's comparable to something like a 5090 (with 1.8TB/sec). For a large MOE model that has only 30-40B parameters active, that combo will probably actually be pretty usable. Reply
coldspring22 Very clever proof of concept. Maybe it will be useful if civilization collapsed and all you have is an old laptop. But I think old xeon sever with cheap DDR3 server ram/some old GPU cards will be much more feasible approach to run near frontier MOE models (yes still very slow, maybe 1-2 tokens/second) Reply
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/colibri-proof-of-concept-gains-frontier-level-1-5-tb-ai-model-novel-approach-runs-on-only-25gb-of-ram-and-shows-promise-for-local-ai-setups#main
- https://www.tomshardware.com/subscription
- Joyride Through July With 12 Games Coming to GeForce NOW
- Retro Nintendo Switch 2 dock looks exactly like the Nintendo 64 and holds six Game Cards in its cartridge slot — the 64BITDOCK is available now starting at $89
- AMD RX 9070 GRE collapses to $499 to save 1440p gaming — RDNA 4 price slips 9% to steal a piece of Nvidia's mid-range pie
- Fake Go DNS scanner spread malware through over 200 GitHub repos — 'Operation Muck and Load' has published 700 malicious modules since January
- This 3D-printed electric motorbike folds into your luggage — creator warns it is 'super fast… way too fast'
Informational only. No financial advice. Do your own research.