Additionally, more local capacity could also reduce communication between accelerators. Conventional expert parallelism distributes experts across GPUs and requires all-to-all communication at every layer. OXMIQ claims that inexpensive HBF capacity could allow considerably more experts to reside locally, reduce the number of expert-parallel shards, and reduce network traffic. In this case, HBF effectively trades memory capacity for interconnect bandwidth and power consumption.
Long-context inference is another potential use case. Sparse-attention models access only a small portion of their large KV cache during each decoding step, which allows the rest to remain in slower HBF memory. OXMIQ believes that HBF could store this large KV cache while the accelerator fetches only the data needed for each step from HBF to HBM.
The HBM-as-cache idea has a serious limitation. Intuitively, one would put popular experts in HBM and cold experts in HBF. Such a strategy works mainly at low batch sizes or when similar queries can be deliberately batched. As batch size rises and queries become more heterogeneous, however, expert popularity flattens, and the workload accesses a broader range of experts, according to OXMIQ. The working set can then outgrow the relatively small HBM cache, which results in more frequent expert transfers from HBF and reduces the performance benefit provided by HBM caching.
OXMIQ does not expect HBF to work simply as slower GPU memory. Instead, it proposes using HBF in place of host DRAM to store large amounts of less frequently accessed data, such as MoE experts and KV cache. Frequently used data would remain in HBM, while even colder data could still be kept in remote memory or SSDs. This certainly contradicts SanDisk's original vision for HBF: sitting next to AI accelerators. Furthermore, adding HBF support will be complicated on many levels.
The software side of HBF is particularly complicated. To achieve maximum bandwidth, HBF requires large transfers — 64 KB reads and 1 MB writes — and data is moved through DMA rather than the CPU/GPU cache hierarchy. When HBF and HBM are used together, software must also decide which data goes into each memory type and manage HBF's limited write endurance.
Meanwhile, current inference software is not ready for such a configuration. OXMIQ says vLLM would need dedicated HBF support to manage memory allocation and data placement, prefetch data before it is needed, and monitor flash endurance, which requires a major software overhaul. An effort like this has to be a joint effort between the HBF hardware vendors, AI accelerator vendors, and inference-framework developers. At the lowest level, AMD, Nvidia, and other accelerator vendors would need to provide the hardware/driver/runtime mechanisms for efficiently moving data between HBF and HBM. Then vLLM developers, who work with vendors, would implement the higher-level memory allocator and policies that decide which experts/KV blocks live in HBM and which reside in HBF, when they should move, and how to hide HBF latency.
On the one hand, if AMD or Nvidia adopt HBF, they will provide its partners with everything needed to use it, and while this would take time before everything works as intended, this is a straightforward way to add HBF support to AI platforms. On the other hand, the biggest question is whether hardware vendors like AMD or Nvidia need HBF. As per OXMIQ, HBF's advantage is limited to select use cases, so it may not make sense for AMD or Nvidia to support it universally, especially keeping in mind that managing multi-tier memory hierarchy is hard.
SambaNova is perhaps the most obvious candidate to support HBF. Its SN40L already uses a three-tier hierarchy: SRAM => HBM => DDR, with up to 520MB of SRAM, 64GB HBM, and 1.5 TB of DDR. Conceptually, HBF could become another tier or replace some of that DDR capacity. Then again, this is merely speculation.
While we still have a lot to learn about how HBF works, OXMIQ's model suggests that HBF has a much weaker general-purpose value proposition than the original claim made in early 2025 suggested. It is not useless: it is a specialized solution whose strongest applications depend on particular workload characteristics.
The fundamental problem is that HBF solves memory capacity, while modern AI accelerators are frequently constrained by memory bandwidth. OXMIQ's simulation makes this rather obvious: HBF provides about 14X more memory capacity but only 0.6X the aggregate bandwidth of HBM. Once the workload becomes sufficiently bandwidth-intensive, the enormous capacity stops offsetting the bandwidth deficit.
While the hybrid HBM+HBF solution makes sense for some use cases, it is not a magic fix. When HBM is used as an expert cache, heterogeneous requests at larger batch sizes flatten expert popularity, cause the workload to touch more experts, and reduce cache efficiency dramatically.
For now, HBF has three particularly compelling use cases: reduce the number of GPUs required simply to fit huge models, store massive but infrequently accessed MoE expert pools, and keep large KV caches for sparse long-context inference. For MoE models, its large local capacity could also reduce expert parallelism and expensive all-to-all communication between GPUs. In all three cases, HBF makes sense because capacity requirements are enormous while bandwidth demand remains relatively low.
(Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) (Image credit: OXMIQ) Image 1 of 22 View Original
TOPICS SanDisk SK Hynix HBM See all comments (0) Anton Shilov Social Links Navigation Contributing Writer Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/pc-components/ssds/SPONSORED_LINK_URL
- https://www.tomshardware.com/pc-components/ssds/hot-chips-2026-high-bandwidth-flash-promises-massive-bandwidth-and-capacity-but-its-usability-is-extremely-limited-new-memory-format-strikes-a-balance-between-hbm-and-nand-flash#main
- https://www.tomshardware.com/my-account
- Rockstar releases statement after a week of GTA VI leaks, avoids mentioning leaker's demands — says that gameplay leaks have been ‘heartbreaking for our team’
- Crypto bro faces 280 years in prison for defrauding investors with promises of an 'AI supercomputer' for mining — jury convicts businessman of running $24-milli
- Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents
- OpenAI bans Russian ChatGPT accounts posing as a fake Israeli think tank — used VPNs to push pro-Kremlin narratives and steal academic papers for its website
- Hot Chips 2026: Nvidia presents Groq 3 LPX architecture and unveils its first third-party inference benchmark — LP30-based rack already in production, company s
Informational only. No financial advice. Do your own research.