Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per second

Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per second

The speed that came with the GPUs is within the range of what the builder expected, guessing “somewhere between 2 and 4 tokens per second” in a prediction from the original build thread. The GPU install had to wait, in part, for modifications to the case, which required cutting out some of the metal and relocating the PSU to make room for the GPUs and their cooling. Cooling remains a huge issue, with “the improved throughput [not changing] the original thermal warning,” and the two 12W blower fans for the Optane DIMMs are kept in place. Add in dual 350W-class GPUs , and the changes are understandable.

Things won’t stop here, as more optimizations are possible, including “customized quantization,” which may be similar to the techniques used by lawyer Nick Mermiges . Quantization selected per tensor or layer, or by where the data lives, might reduce bandwidth and capacity pressures for the slower tiers; the setup “might be faster yet.” Speed, or tokens per second alone, is not enough, and output quality must be taken into consideration in any future plan.

Follow Tom’s Hardware on Google News , or add us as a preferred source , to get our latest news, analysis, & reviews in your feeds.

Shane Downing Social Links Navigation Contributing Writer Shane Downing is a Contributing Writer for Tom’s Hardware, covering consumer storage, PC hardware, and AI.

Key considerations

  • Investor positioning can change fast
  • Volatility remains possible near catalysts
  • Macro rates and liquidity can dominate flows

Reference reading

More on this site

Informational only. No financial advice. Do your own research.

Leave a Comment