'My tastes are gone — only the AI's remain': the paradox of ChatGPT recommendations

The movies, books and restaurants that generative AI systems such as ChatGPT recommend may be shaped less by a user's actual preferences than by what the AI itself recommended in the past, according to new research.

The Ulsan National Institute of Science and Technology (UNIST) announced Sunday that a research team led by Professor Lee Yeon-chang of its AI Graduate School has developed EchoTrace, a diagnostic technology that tracks errors and bias in large language model (LLM)-based recommendation systems.

Personalized recommendation services powered by generative AI have been spreading rapidly across online shopping platforms and video streaming services. These systems analyze user preferences to suggest products, or generate synthetic user profiles to improve recommendation accuracy.

Problems can arise, however, when AI-generated recommendations are fed back into the system as training data. The products an AI previously recommended carry more weight in shaping the next round of suggestions than a user's actual preferences do, creating a self-reinforcing distortion.

To examine this problem, the research team analyzed recommendation systems for movies, books and restaurants.

When AI directly generated recommendation lists, the share of items presented that did not exist in the eligible pool reached 4.07 percent for movies, 7.40 percent for books and 13.14 percent for restaurants. This is the so-called "hallucination" phenomenon, in which AI presents nonexistent or ineligible items as if they were normal recommendations.

Serious errors also emerged when the AI tried to infer user information. In 93.16 percent of cases, the occupation the AI estimated based on a user's movie-viewing history did not match the actual job — and in some instances the system invented occupations entirely, such as "film critic," a category absent from standard job classifications.

A concentration of recommendations on certain popular products was also observed. When AI-generated training data was used, the system tended to over-recommend popular films, specific genres or particular business categories compared with recommendations derived from actual user records.

The deeper problem is that these errors do not disappear even after repeated retraining.

When the research team ran the retraining process five times using AI-generated recommendations, the initial biases and errors persisted. In some systems, the skew toward popular products grew worse.

Particularly in systems where AI directly recommends films, the AI's own previous recommendations were the single most influential factor in generating the next round of suggestions for between 86 and 92 out of every 100 users.

A feedback loop forms in which AI-generated recommendations are recycled as training data — treated as records of actual user choices — and then used to produce new recommendations. The more this cycle repeats, the less likely it becomes that a user's genuine preferences are reflected.

EchoTrace is designed to analyze these risks systematically. While conventional evaluations of recommendation systems focus on accuracy at a single point in time, EchoTrace tracks how AI-generated information and recommendations change as they pass through repeated rounds of retraining.

The team analyzed 116 papers published at international academic conferences to classify the roles LLMs play in recommendation systems into five categories, then verified the errors and biases arising at the stages of information generation, final recommendation and iterative retraining for three of those roles.

"This research has laid the groundwork for systematically diagnosing how bias and hallucination in LLM-based recommendation systems accumulate and amplify through repeated recommendation cycles," Lee said. "It can be used going forward to design and evaluate recommendation systems that are safer and more trustworthy."