What is HBM memory, and why does AI need so much of it?

Every AI chip you've heard of, from Nvidia's GPUs to Google's TPUs, has a few small towers of memory sitting right beside it. That memory is HBM, short for high-bandwidth memory. It's one of the most important, and most constrained, parts of the AI build-out, and it's a big reason your laptop's RAM costs more this year.

The problem HBM solves

An AI accelerator can do trillions of calculations a second, but only if data reaches it fast enough. A large model's weights have to be streamed from memory into the chip over and over. If the memory can't keep up, the chip sits idle, waiting. Engineers call this the "memory wall," and for AI workloads it's often the real limit on performance, not the chip's raw compute.

Normal computer memory (the DDR DRAM in a PC or server) sits on separate sticks, several centimeters away, connected by a relatively narrow channel. That's fine for everyday computing. For AI, it's like feeding a factory through a garden hose.

How HBM works

HBM fixes this with three design choices:

• It's stacked. Instead of chips laid side by side, HBM stacks 8, 12 or 16 thin DRAM dies on top of each other, like floors in a building.
• It's wired vertically. Thousands of tiny vertical connections, called through-silicon vias (TSVs), run straight through the stack, so every layer can send data at once.
• It sits right next to the processor. The stacks are mounted on the same package as the GPU, millimeters away, connected through a very wide interface. HBM3E uses a 1,024-bit connection per stack; the new HBM4 standard doubles that to 2,048 bits. A typical DDR5 channel is 64 bits wide.

The result is enormous bandwidth. A single modern AI accelerator can move several terabytes per second between its memory and its compute, many times what a standard server can manage.

Why it's so hard to make

HBM is still DRAM, but manufacturing it is far more demanding than making ordinary memory chips:

• It eats wafer capacity. The dies are larger and need extra circuitry, and stacking losses add up. By common industry estimates, producing a bit of HBM takes roughly three times the wafer capacity of a bit of standard DRAM.
• Yield compounds. A 12-high stack only works if all 12 dies and every connection between them work. One bad layer can scrap the whole stack.
• Packaging is a second bottleneck. HBM has to be bonded onto the accelerator package using advanced packaging, which has its own limited capacity.

This is why HBM matters for more than AI. When memory makers shift production lines to HBM, there's less capacity left for the DRAM that goes into phones, PCs and ordinary servers. That squeeze is a big part of why conventional DRAM prices have surged in 2026.

Who makes it

Only three companies produce HBM at scale, and the market is highly concentrated.

The trend worth noticing: SK hynix is still dominant, but its share is slipping as Micron and Samsung catch up. More competition could eventually pressure HBM pricing, even while demand stays strong.

What to watch as an investor

1. HBM4 qualification. The next generation ships with Nvidia's next platform. Whichever suppliers qualify early and in volume will shape market share for the next two years.
2. Contract pricing. HBM is mostly sold on long-term agreements. Watch whether next year's contracts are signed at higher or lower prices.
3. Capacity additions. All three companies are building new fabs. If capacity arrives just as demand growth slows, the classic memory boom-bust cycle comes back.
4. Standard DRAM prices. As long as HBM keeps absorbing capacity, ordinary DRAM stays tight. That's the link between the AI build-out and memory stocks' broader earnings.

For the bigger picture on how memory fits into the AI spending cycle, see The AI supercycle is moving down the stack.

Sources

• SK hynix F-1 filing: HBM market share (IDC), 2025 and Q1 2026
• HBM supplier positions and the shift to HBM4
• HBM capacity sold out through 2026

Not investment advice. I may hold positions in companies mentioned.