Local model running on a sub-$10 microcontroller is impressive despite its obvious limitations.
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works .
If you haven't read any of our previous coverage of this tiny chip, ESP32-S3 boards offer about the best bang for buck in the whole computing world. You can snag one online with a protective case for under $20 here in the States, and bare boards are readily available for under $10 around most of the world. As you'd expect from a chip so cheap, it's not powerful. On this variant, the S3, you get exactly 512KB of SRAM, 8MB of PSRAM, and 16MB of flash memory, which is not very much memory at all. So how exactly do you cram a nearly 30-million parameter model onto a chip with less primary storage than a single raw photo from your smartphone?
Usually, to run an LLM, the entire model has to sit in your system's fast memory because the processor needs to constantly do math against every parameter to generate the next word. If you try to run a 29M parameter model normally on an ESP32, you run out of fast RAM instantly. The previous record for a chip like this was around 260,000 parameters by one Mr. Dave Bennett, as pointed out by Slava himself on X.
Our clever hacker got around this bottleneck by borrowing a brilliant architectural trick from Google 's Gemma called Per-Layer Embeddings. He quantized the model down to 4-bit (making the total file size just 14.9 MB) and changed where the data lives; instead of trying to stuff the whole thing into the tiny 512KB SRAM or the only slightly-less-tiny 8MB PSRAM, he dumped the 25-million-parameter embedding table into the relatively-slow 16MB Flash memory. Because this specific model architecture only needs to pull a few rows from this table per token, the inherent slowness of the Flash memory doesn't choke the processor, and so the 512KB of fast SRAM is kept clear for just the "thinking core", the actual reasoning weights.
Now, let's pump the brakes for a second, because I know someone out there is already wondering if they can replace their server with an $8 chip . The model he used was trained on the TinyStories dataset, and it's really more of a Small Language Model (SLM), or honestly, a "micro LM." Due to the way it was created, it's only capable of writing short, simple, fictional stories. It will not answer questions, it will not follow instructions, it won't write your Python code, and it possesses exactly zero factual knowledge about the real world.
Colibrì proof-of-concept gains frontier-level 1.5-TB AI model
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-developer-runs-28-9-million-parameter-model-on-usd10-esp32-s3-microcontroller-uses-googles-per-layer-embeddings-technique-stores-table-on-16mb-flash-memory#main
- https://www.tomshardware.com/subscription
- Microsoft shipped a native Xbox 360 emulator with its new backwards-compatible releases on PC — modders quickly got 360 games running with minor tweaks
- Get 16GB of DDR5 RAM for just $16 when you buy it with AMD's brand-new 7700X3D — Ryzen 7 with an X870 motherboard, G.Skill Ripjaws, and an AIO for just $588
- AOC U32G4 32-inch 4K Dual-Refresh gaming monitor review: Solid performance and value
- Save $1,000 on Lenovo's over-the-top RTX 5090 gaming laptop — this 18-inch monster packs 64GB of RAM and up to a 440Hz refresh rate
- Gigabyte announces support for Chinese-made CXMT memory — pushes it to 8200 MT/s on Socket AM5
Informational only. No financial advice. Do your own research.