"In conventional computer architecture, system performance is generally understood to be determined by processors — CPUs, GPUs, NPUs," said Joo Young-pyo, executive vice president for system architecture at SK hynix. "But I see it differently. In AI systems, how much bandwidth memory can supply determines the performance of the entire system."
Joo made the remarks Tuesday in a keynote address at the Dell Technologies Forum 2026 at COEX in Gangnam-gu, Seoul. He argued that the launch of ChatGPT in 2022 fundamentally transformed the memory chip market as the AI era arrived in earnest.
"I truly love large language models," Joo said, adding that AI systems today move data to GPUs at a rate equivalent to the full contents of six or seven 650-megabyte CDs every second — a staggering volume of memory traffic.
He said the rise of agentic AI, which now handles tasks such as vibe coding, has driven token consumption to an entirely different scale. "Traditional compiler and runtime software was built long ago and is hard to speed up," he said. "To cut total processing time and improve cost efficiency, you need not tens of tokens per second but hundreds or thousands."
SK hynix is working to resolve AI infrastructure bottlenecks through improvements in memory performance. Joo said the company has kept memory bandwidth advancing in step with GPU performance gains over the years.
"From HBM1 through HBM4, we have run hard for just over a decade and achieved strong results, and we intend to keep that innovation going," he said. "But as we enter the agentic AI era, HBM alone is no longer enough to deliver the performance required."
SK hynix is researching ways to stack memory directly on top of GPUs to boost bandwidth, framing the challenge as a geometry problem. "Memory bandwidth faces physical limits because of geography," Joo said. "Placing memory directly on top of a GPU — as with HBM — can deliver bandwidth comparable to the SRAM-based accelerators drawing so much attention lately, but the heat generated by an already-hot GPU makes that extremely difficult to manage." He added that memory companies, logic semiconductor firms and NPU makers are working together to solve the problem.
Energy efficiency is another challenge. AI systems consume far more energy moving data than actually computing with it, so minimizing data movement is essential.
"We are developing processing-in-memory, or PIM, technology that performs computation right next to DRAM data to cut the commute time for data," Joo said. "Energy consumption is very low and efficiency is high, but as securing data storage capacity has become critically important, the idea of minimizing energy spent on data movement is gaining traction."
Memory utilization is also a key area of development. As Joo noted at the outset, the context and key-value cache generated through user interactions — beyond the AI algorithm itself — are emerging as factors that determine AI intelligence.
"After Google's TurboQuant algorithm was released earlier this year, the entire industry turned its attention to memory storage capacity," he said. "Jevons paradox — the idea that higher resource efficiency leads to greater demand, not less — still holds."
He said SK hynix has incorporated memory pooling technology to address the problem of expensive GPUs sitting idle while waiting for data to be transferred. "We are using CXL — Compute Express Link — technology to improve overall system performance and cost efficiency," he said.
jeongwan@heraldcorp.com
