KAIST's AI pianist improvises in real time with jazz musicians

When a jazz pianist pauses mid-performance, an AI seamlessly takes over the keys. It then generates bass and drum accompaniment to match the human musician's improvisation.

KAIST announced Friday that a research team led by Professor Nam Juhan of the Graduate School of Culture Technology has developed a real-time music collaboration technology between humans and AI, showcased at the "Jazz AI Lounge" stage of the 23rd Jarasum International Jazz Festival, held from Saturday to Monday.

Twelve jazz artists participated in the performances, collaborating with the AI across 12 sets and 26 songs.

Jazz is defined by its spontaneity — tempo, rhythm and melody shift constantly according to the performer's emotions and split-second decisions. Because no two performances of the same piece are alike, musicians must listen closely to one another and stay in sync.

To replicate these qualities in AI, the research team developed an "AI pianist" capable of taking over a human's performance and an "AI accompanist" that generates improvised backing in real time.

The AI pianist, named Scott, collaborated with jazz artists on nine pieces.

Scott compares a performer's live playing against the sheet music in real time to track where the piece stands. Even when a musician slows down or alters the timing, Scott picks up the performance according to the actual flow of the music rather than a preset schedule.

In some sets, Scott went further, generating new solo passages that reflected the preceding musician's melody and rhythm — a musical exchange that resembled a conversation between human and machine.

The AI's output was transmitted to Yamaha's Disklavier, a self-playing piano, which rendered it through actual key and pedal movements.

The team also demonstrated technology that enables the AI to generate accompaniment in response to a human's improvisation.

The team's AI accompaniment system, Deepocket, analyzes the notes, rhythm and tempo of a piano performance and generates bass and drum accompaniment in real time.

Across eight pieces performed alongside a jazz pianist, Deepocket kept pace with the human musician from start to finish without any pre-prepared backing tracks.

Nam's team previously demonstrated related technology in 2023 during a performance with soprano Jo Su-mi, where the AI adjusted the timing of piano accompaniment to match the singer's live phrasing.

This time, the team went a step further, having the AI generate its own solos and accompaniment on the fly in a jazz setting where improvisation is central.

The team expects the technology to find applications beyond live performance and music creation — including as an AI ensemble partner for musicians practicing alone and as a tool for personalized music education.

"This performance showed on a real stage that AI can go beyond playing fixed music — it can listen to a human's playing and create music together in the moment," Nam said. "We will continue our research so that AI can evolve from a simple performance tool into a collaborative partner that exchanges musical inspiration with people."