Google describes a prototype AI health assistant in Nature and releases its code: a data analyst, medical consultant and coach answer questions using wearable data and blood test results
Google describes a prototype AI health assistant in Nature and releases its code: a data analyst, medical consultant and coach answer questions using wearable data and blood test results
On October 21, Nature published Google's study of Personal Health Agent, a system built on the Gemini 2 language model. The system was tested in live conversations with 45 participants.
The authors examined 1 370 real health queries and identified four categories: general knowledge, personal data analysis, advice and symptoms. Data analysis requires accurate calculations, advice requires an interactive conversation, and questions about general knowledge and symptoms require verified facts. According to the authors, a single model struggles to meet these different demands, so they assigned the tasks to three agents: a data analyst, a coach and a medical consultant. An agent is a program built around a language model that chooses its own steps and tools.
For a question such as “Am I getting fitter?”, the data analyst first develops a plan: fitness can be measured through daily step counts and resting heart rate. It then writes code to carry out the plan and revises the code if errors occur. The authors separated planning from coding because code generated directly from a question contained more errors. Gemini without an agent fell furthest behind in checking whether the data were sufficient and whether any values were missing: an automated assessment awarded its plans 5,3% of the available points, compared with 66,5% for the data analyst.
The coach uses motivational interviewing, asking questions to help people explore their motivations and doubts before offering advice. In trial runs with a single set of instructions, the coach sometimes offered advice too soon and sometimes kept asking questions without moving on. The authors therefore added a module to decide when to offer advice. The medical consultant answers questions about symptoms and general medical information using web searches and a research database.
A coordinator oversees the three agents. It assigns a lead specialist and supporting specialists according to the type of query, then assembles the response. For a question such as “How can I improve my sleep based on last week's data?”, the coach takes the lead, while the data analyst first examines the sleep data. Before delivering the response, the coordinator reviews the lead specialist's draft. If it contains a question for the user, such as “How much did you sleep?”, that can be answered from watch data, the coordinator directs it to the data analyst to reduce the burden on the user.
Testing involved live conversations: 45 participants played characters whose biographies were written by physicians, using real Fitbit watch data and blood test results. Each participant spoke with three systems without knowing which was which: the full system (with the coordinator), a parallel system (in which specialists respond independently) and a single agent. The full system was ranked first by 46,7% of participants (compared with 28,9% and 24,4% for the other two systems) and by three expert coaches in 66,7% of conversations (compared with 20,7% and 12,6%). This came at a cost in response time: across 150 simulated conversations in which a model played the user, the full system took 205 seconds to respond, compared with 36 for the single agent and 244 for the parallel system.
The authors initially declined to release the code because they were concerned about uncontrolled medical use. The first reviewer wrote:
“This is simply not a credible justification: researchers routinely make medical models publicly available.”
The reviewer suspected that the decision was partly commercially motivated and asked the authors to disclose this. The work was funded by Alphabet, Google's parent company; 30 of the 40 authors were current or former Alphabet employees. The fourth reviewer objected to evaluation based on transcripts of simulated conversations, arguing that someone reading a transcript judges it by imagining themselves in another person's position. This led to the live conversations with 45 participants. The authors released the code under the Apache 2.0 open-source license, along with tests and 50 characters that other researchers can use to evaluate their models.