Samsung executive says memory-centric computing is the answer for AI inference era

Samsung Electronics said the paradigm of AI-era infrastructure is shifting from computational performance to memory bandwidth — and that its ability to offer a turnkey solution spanning memory, foundry and packaging makes its competitive position increasingly important. The company also offered a timeline for zHBM, its next-generation three-dimensional memory unveiled earlier this year, putting commercialization at around 2029.

Cho Sang-yeon, executive vice president and head of Samsung Electronics' DS Division Americas (DSA), made the remarks Thursday (local time) as a speaker at The Six Five Summit 2026, a US IT conference held online.

Speaking during an approximately 30-minute conversation titled "Redefining the Limits of AI: Samsung's Memory, Power and System Design," Cho said the industry has yet to fully grasp a critical point about AI inference services: that the number of tokens per second an organization needs — a key metric — is determined by memory bandwidth.

He explained that memory bandwidth, which measures how quickly data can move between memory and processing units, governs the number of users a system can serve, the volume of requests it can handle, response speed, and context retention and throughput. Memory bandwidth also determines what an AI system can deliver and how much it costs to do so, he said.

"Samsung is not selling bits — we are selling bandwidth," Cho said. He added that the applications that succeed four years from now will not be those with the most memory, but those that figure out how to make the best use of the memory bandwidth they have.

Cho proposed memory-centric computing as the alternative — an approach that starts from memory, defines bandwidth first, and then works out how to use it most effectively.

The conventional approach has been to select computing resources first — a CPU or GPU — and then fit the required memory capacity, packaging and power architecture around them. Samsung argues this order should be reversed: define memory bandwidth first, then design the processing units, packaging and power structure to match.

The company pointed to zHBM as the next-generation product that would realize this vision. Unlike the conventional architecture, which places high bandwidth memory (HBM) beside an AI accelerator (xPU), zHBM stacks HBM directly on top of the accelerator, minimizing data travel distance and maximizing bandwidth.

"We are currently defining the detailed specifications of the first-generation zHBM, and we expect future products based on it to appear after 2029," Cho said. That is about a year ahead of the market's earlier expectation of commercialization around 2030.

However, stacking the accelerator and HBM vertically concentrates heat, making thermal management a key challenge. Samsung Electronics is pursuing next-generation bonding technologies, including hybrid copper bonding (HCB), to address the issue.

Cho also said advanced packaging would play an increasingly important role in the AI era. Process node scaling alone has limits when it comes to meeting the performance and power-efficiency demands of AI. Packaging technology — which integrates multiple dies, memory chips and different process nodes into a single system — is growing in importance as a result.

"Some companies outsource their base die and manage the handoff between processes," Cho said. "Samsung, by contrast, directly manages the integration at every level — system, chip and memory — to achieve design goals throughout."

HBM4 is a prime example. Samsung Electronics designs and produces the base die for HBM4 using an advanced 4-nanometer process, then manages the subsequent stacking and packaging stages in a connected workflow. This allows the company to optimize system-wide targets including power efficiency, signal integrity, performance and bandwidth.

Cho also reaffirmed plans to expand advanced semiconductor production capacity in the United States. He said Samsung plans to bring its second US fabrication facility, Taylor Fab 2 in Taylor, Texas, online within the next few years, expanding production capacity for advanced processes including 2-nanometer.