ETRI develops robot AI benchmark that accounts for individual user preferences

South Korean researchers have developed an AI technology that evaluates how accurately robots understand and respond to the distinct characteristics and needs of individual users.

The Electronics and Telecommunications Research Institute (ETRI) announced Friday that it has developed Trinity, a benchmark designed to assess a robot's task-planning capabilities by comprehensively accounting for the user, the AI agent, and the surrounding environment.

Conventional robot AI evaluations have focused on whether a robot can accurately perceive its environment and successfully complete assigned tasks. Trinity goes further, assessing whether the actions a robot chooses are appropriate for the person receiving the service.

Robot AI systems powered by large language models have advanced to the point where they can understand human commands and independently plan multi-step tasks. However, they have not adequately accounted for the fact that the same command may call for different actions depending on a user's dietary habits, health condition, or personal preferences.

To address this gap, ETRI researchers introduced a new evaluation concept called "personalized human-aware task planning," or PHA.

Trinity is built around 151 user conditions — combinations of dietary habits, allergies, religious dietary requirements, and hobbies — and a simulated home environment closely modeled on real households. Using 95 types of objects, the benchmark evaluates a robot's ability to select the option best suited to a given user from among multiple choices.

Even if a robot successfully identifies and moves an object to the correct location, it is judged to have failed the task if it selected an item inappropriate for the user.

In experiments, the AI selected food unsuitable for a vegetarian even when it correctly understood the surrounding environment and the method required to complete the task. The finding suggests that strong environmental perception alone may not be enough — a robot may still fall short when it comes to translating a user's personal characteristics into concrete action.

Trinity also distinguishes between the sources of such errors, analyzing whether a failure stemmed from a misunderstanding of the user's characteristics, an inadequate scan of the environment, or a plan that called for actions the robot could not physically carry out.

ETRI expects the technology to be used in verifying the performance of physical AI systems that share daily life with people, including home robots, care robots, service robots and humanoid robots.

The research team has released Trinity's code and data, and plans to expand the benchmark's scope beyond food and beverages to cover other lifestyle domains such as cleaning, organizing and safety. The team also intends to refine the technology so it can reflect users' needs as they change over time.

"While previous robot AI research focused on understanding the environment and accurately executing given tasks, Trinity is significant in that it also evaluates whether a robot's actions are appropriate for the person being served," said Kim Do-hyeong, head of ETRI's Social Robotics Research Laboratory. "It will serve as a research foundation for advancing robots toward an intelligence that understands the distinct characteristics and needs of each individual and can truly share life with people."