Meta Smart Glasses
Multimodal AI for Blind and Low Vision Users
/ Background
At Meta Reality Labs, I led product design work across our Privacy, Security, Safety, Accessibility, and Equity teams.
As part of our Accessibility and Equity initiatives, I helped develop multi-modal AI features on the Meta Ray Ban Smart Glasses to enable blind and low-vision users to use the smart glasses as assistive devices.
/ Role
Product Design Manager, Meta Reality Labs Trust
/ Year
2022 - 2023
Video: Meta Ray Ban Smart Glasses Multi-Modal AI
Product Goals (MVP Scope)
Read text from a physical source
Read a body of text aloud to the user from a sign, a piece of mail, etc
Describe an item in the environment
Describe an individual physical item to the user
Describe what is happening
Describe an entire scene to the user, including descriptions of people and items in the scene, hazards, and motion
Video, audio, and voice assistant features on the smart glasses unlocked new opportunities to integrate multi-modal AI and computer vision.
Image: Prototype of computer vision image recognition
Once we had a prototype, I ran product red teaming sessions to identify and mitigate risks. One particular area of focus was Responsible AI and mitigating risks from our AI agent interactions.
Product Responsibility Considerations
Incorrect interpretation
Low lighting, off-angle capture, or lack of representation in training data could cause Meta AI to incorrectly identify text or an object.
Biased descriptions
When Meta AI is asked to describe a person, the description could include offensive or inappropriate descriptions.
Failing to alert user of hazards
When describing scenes, Meta AI could fail to recognize hazards leading the user to trip and fall.
Privacy
Meta AI could voiceover a piece of sensitive information (ex. medical test results) causing people nearby to overhear.
Since there is no visual UI input, we couldn't use warnings or content flags to mitigate some of these risks.
Instead, I helped develop agentic conversation design principles in order to mitigate harms.
Scale of confidence
For example, on the issue of incorrect interpretation the solution was in how our Conversation Designers wrote voice interactions based on confidence scores.
We didn't have a visual UI that we could put a tooltip or a disclaimer in, so we needed to use natural language to convey a sense or certainty or uncertainty.
By thinking through how to use content, voice, tone, and inflection we could communicate to blind users how much or little they could rely on our interpretation.
We also encouraged users to use supplements, like asking other people around them, rather than relying 100% on our interpretation.
Future Accessibility Features
Live captions for deaf users
Future models of the smart glasses will include an Augmented Reality (AR) display, and could help deaf users see live captions of conversations.
Alternative computer inputs for people who are limited in hand/arm mobility
The upcoming electromyography (EMG) wrist band could allow users with limited hand/arm mobility to have an alternative computer input method.
Spoken communication for people who are nonverbal
A combination of the EMG wrist band and the AI voice feature could help people who are nonverbal speak to other people.
AR reminders or labels for people with degenerative memory disorders
AR display features could allow people to pin reminders and labels to items in their home, allowing people with memory disorders, like Dementia or Alzheimer's, to have an easier time remembering contexts.
