Local model running on a sub-$10 microcontroller is impressive despite its obvious limitations.
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works .
If you haven't read any of our previous coverage of this tiny chip, ESP32-S3 boards offer about the best bang for buck in the whole computing world. You can snag one online with a protective case for under $20 here in the States, and bare boards are readily available for under $10 around most of the world. As you'd expect from a chip so cheap, it's not powerful. On this variant, the S3, you get exactly 512KB of SRAM, 8MB of PSRAM, and 16MB of flash memory, which is not very much memory at all. So how exactly do you cram a nearly 30-million parameter model onto a chip with less primary storage than a single raw photo from your smartphone?
Usually, to run an LLM, the entire model has to sit in your system's fast memory because the processor needs to constantly do math against every parameter to generate the next word. If you try to run a 29M parameter model normally on an ESP32, you run out of fast RAM instantly. The previous record for a chip like this was around 260,000 parameters by one Mr. Dave Bennett, as pointed out by Slava himself on X.
Our clever hacker got around this bottleneck by borrowing a brilliant architectural trick from Google 's Gemma called Per-Layer Embeddings. He quantized the model down to 4-bit (making the total file size just 14.9 MB) and changed where the data lives; instead of trying to stuff the whole thing into the tiny 512KB SRAM or the only slightly-less-tiny 8MB PSRAM, he dumped the 25-million-parameter embedding table into the relatively-slow 16MB Flash memory. Because this specific model architecture only needs to pull a few rows from this table per token, the inherent slowness of the Flash memory doesn't choke the processor, and so the 512KB of fast SRAM is kept clear for just the "thinking core", the actual reasoning weights.
Now, let's pump the brakes for a second, because I know someone out there is already wondering if they can replace their server with an $8 chip . The model he used was trained on the TinyStories dataset, and it's really more of a Small Language Model (SLM), or honestly, a "micro LM." Due to the way it was created, it's only capable of writing short, simple, fictional stories. It will not answer questions, it will not follow instructions, it won't write your Python code, and it possesses exactly zero factual knowledge about the real world.
Colibrì proof-of-concept gains frontier-level 1.5-TB AI model
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-developer-runs-28-9-million-parameter-model-on-usd10-esp32-s3-microcontroller-uses-googles-per-layer-embeddings-technique-stores-table-on-16mb-flash-memory#main
- https://www.tomshardware.com/subscription
- 3D-printed F-14 Tomcat uses an FPGA recreation of the ‘world’s first microprocessor' — CADC’s MP944 chip controls the fighter’s swing-wing system, among other t
- Nvidia weighs $250 billion guarantee so OpenAI can lease SoftBank's 10-gigawatt Ohio campus, report claims — Nvidia also said to be discussing $350 billion deal
- At AI Summit, South Korea Outlines Its AI Future With NVIDIA and Partners
- NVIDIA Harnesses Vera CPU to Speed Up Design of Next-Generation CPUs and GPUs
- Moonshot AI releases weights for Kimi-K3, firing a shot across the bow of OpenAI and Anthropic — open-weight model performs almost as well as frontier models wh
Informational only. No financial advice. Do your own research.