AI in-home multi-modal mental health system
Older adults who live alone often lack family attention and may slowly develop unnoticed mental health symptoms
My Role
Researcher UI/UX Designer
Skill
Design System, Research, Brand Identity
Timeline
Nov - Dec 2024
Teammates
Nuo Chen Ming Li
According to the World Health Organization, the world’s population is rapidly aging, with the number of people aged 60 and over expected to double between 2015 and 2025. During the pandemic, adults aged 50-80 experienced mental health symptoms. Of these, 19% were depressed or sad, 19% had sleep problems, and 28% were worried and anxious.
The increasing prevalence of mental health issues among older adults highlights the urgent need for innovative and accessible solutions to provide timely care and support.
We aim to focus on assessing the mental health condition of older adults, particularly depression.
Final Design
My teammate and I designed our prototype in 3 views: smart watch, web app/Vision Pro (augmented reality (AR) and virtual reality (VR)), and mobile app
We introduce an AI-infused in-home multi-modal mental health diagnosis system.
-
Designed to make traditional mental health services, such as mental health diagnosis with professional psychiatrists, more accessible for older adults.
-
By leveraging advances in artificial intelligence and integrating data from wearable devices and smart home technologies such as Alexa and Google Home Assistant
-
The system enables continuous monitoring and proactive intervention for mental health management by automatically responding with mentally relieving activities
-
Sending frequent summaries and health reports to the older adults' caregivers or healthcare professionals
Solution
Competitive Analysis
We researched apps with similar features to our idea. Here is what we learn:
-
Emoji, data, graph, message, and cross-platform interfaces are helpful to attract people to use the system/app
-
When people use technologies, they want it to "talk" to them like a real person is talking to them
-
Telling the problems is good, but providing a solution would be much more helpful for people to solve the problem
Literature Review
Voice Assistant for Mental Health
Voice Assistant for Older Adults
Persona
The primary users of this AI-assisted in-home healthcare system are older adults who may struggle with mental health issues, often unnoticed or untreated.
Storyboard
Story:
Mr. Smith, an elderly man, begins showing signs of depression, losing interest in life, and spending his days quietly staring out the window. He occasionally talks to his Alexa for companionship, but it offers little comfort. Unaware of his mental state, he dismisses his sadness as grief from losing his wife.
Meanwhile, an AI system passively monitors his behavior through wearable devices and smart assistants. It detects emotional decline and alerts his doctor, but Mr. Smith declines help, feeling alone and unsupported by his busy son.
As his condition worsens, the AI picks up alarming language and elevated stress signals, prompting an urgent alert to his family and doctor. They intervene just in time, ensuring Mr. Smith gets the help he needs. The AI system proves vital, offering continuous support and early intervention for mental health in older adults.
Workflow
We shaped the user flow around enabling stronger family connections for older adults
-
Voice Assistant have conversation with older adult
-
Generate report with reliving activities, score, and detail information
-
Score less than 5 is consider Normal
-
Between 5-10 is Moderate
-
Bigger than 10 is Severe
-
-
Family caregivers or user can share it to healthcare professionals to see the report generated based on the conversations and AI. If the report is in severe status, it will automatically sent to the user's healthcare professionals for professional care
Metrics
Our goal is for the assistant to respond with warmth and empathy, making users feel supported and understood as it guides them through the questions
Based on our literature review, we have developed an approach for integrating screening questions into the conversation in a way that feels friendly, natural, and humanized, specifically designed for a voice assistant supporting older adults.
Here are the points we will consider when coming up with the questions:
-
The Geriatric Depression Scale (GDS) – A Depression Screening Tool for Older Adults (Yesavage et al.)
-
Used extensively with older adults
-
The form is a 15-questionnaire in which participants are asked to respond yes or no to how they have felt over the past week. For the GDS-15, a score of 0-5 is normal. A score greater than 5 suggests depression.
-
-
Natural Language Conversation
-
Make screening questions sound natural/human-like
-
Prompt structure: friendly, supportive, but not monitoring
-
-
Create variances
-
-
Fit to Scenarios
Voice Assistant for Older Adults
Goal:
-
Enable tailored AI responses for various relevant user needs
-
Fit psychological screening questions into a proper conversational context
-
Ensure natural, friendly, and relieving conversational flows
Conversational Structure Diagram:
The conversations were structured as five user scenarios, and we incorporated GDS screenings into these various use cases.
If conversations are related to part of the GDS screening, we use the user's answer to calculate a mental health score. Otherwise, users
interact and get responses from the AI model.
Examples of Conversation Types:
All conversations with voice assistant can be divided into two categories:
-
Freeform AI Interaction for users to seek advice or have a casual conversation with voice assistant.
-
We insert Behavioral Activation events into the conversation where voice assistant will raise GDS Screening questions related to the activation scenario and do a behavioral check-in that requires users to reflect on their past activities or feelings.
Final Prototype in detail (Web App)
Web App version of the Health Report in which users, caregivers, and healthcare professionals can view it on desktop or with vision pro
Based on our research, we designed interfaces that allow users, caregivers, and healthcare professionals to view and monitor the health report. So the users can get better care from families and professionals. While the voice assistant interacts and analyzes the users' behavior, we also want them to be able to see their data visually.
A report summary after the user's conversation with NEURA(our voice assistant)
User's health insurance information
Notification of new reports and alerts
User or caregivers can send the report to healthcare professionals for more solutions
Brief summary of what is the user's depression level based on the AI's analysis from the conversation with NEURA
Keywords analysis from the conversation
Body measurements recorded from a smart watch
Provided relieving plans for the user's depression level
Transcript for caregivers and healthcare professional to see the conversations in details
Report History for caregivers and healthcare professionals to see the pass report so they can better understand the user's feeling and depression level
Voice Assistant
-
Voice input: engage in daily conversation
-
More accessible for guiding interaction
AI Interpretation
Depression Assessment
-
Clear communication
-
Correct input and enhance accuracy
-
Build user trust
-
Encourage users to open their minds
-
Provide the depression score and report
-
Give the user the option to send the report
Final Prototype in detail (Mobile App)
Mobile app version of the Health Report in which users, caregivers, and healthcare professionals can view it on smart phone at anywhere
Notification of new reports and alerts
User's mental health metrics from our NEURA (AI)'s analysis
User can also chat with our NEURA when they are outside
Body measurements recorded from a smart watch
Provided relieving plans for the user's depression level
Final Prototype in detail (Smart Watch Interface)
Smart Watch interface shows the user what their mood is at the moment and has a message to give them advice and help.
After NEURA analyzes the user’s mood based on the conversation, it displays an emoji that reflects how the user might be feeling in the moment.
Have a warm, human-centered version of message that sounds like a real person from NEURA is reaching out to support the user
We designed four levels of mood emojis, each paired with a supportive message that feels like it’s coming from a real person. Depending on the mood, the message offers empathy, encouragement, or help—creating a more human and comforting experience for the user.
What makes NEURA different?
