/ Case study
Meta Smart Glasses
Multimodal AI for Blind and Low Vision Users
At Meta Reality Labs, I led product design work across our Privacy, Security, Safety, Accessibility, and Equity teams.

/ Background
As part of our Accessibility and Equity initiatives, I helped develop multi-modal AI features on the Meta Ray Ban Smart Glasses to enable blind and low-vision users to use the smart glasses as assistive devices.
My primary roles were to develop the product accessibility roadmap for the 2023 release of the Meta Ray Ban Smart Glasses, work with the Artificial Intelligence (AI) engineers and the Optical Character Recognition (OCR) engineers on prototyping, and to run Responsible AI red teaming to mitigate AI risks.
Role
Product Design Manager, Meta Reality Labs Trust
Timeline
2022 - 2023
Related content
Video
Meta Ray Ban Smart Glasses Multi-Modal AI
/ Product goals (MVP scope)
—
Read text from a physical source
Read a body of text aloud to the user from a sign, a piece of mail, etc
—
Describe an item in the environment
Describe an individual physical item to the user
—
Describe what is happening
Describe an entire scene to the user, including descriptions of people and items in the scene, hazards, and motion
01
Multi-modal AI and computer vision
Video, audio, and voice assistant features on the smart glasses unlocked new opportunities to integrate multi-modal AI and computer vision.
To design the blind and low vision features, I worked with the OCR engineers developing the technology. Together we built out the conversation design flows, onboarding, and settings that enable the glasses to read aloud descriptions of text and scenes in the real world.



Image: Prototype of computer vision image recognition
02
Responsible AI red teaming
Once we had a prototype, I ran product red teaming sessions to identify and mitigate risks. One particular area of focus was Responsible AI and mitigating risks from our AI agent interactions.
Product Responsibility Considerations
Incorrect interpretation
Low lighting, off-angle capture, or lack of representation in training data could cause Meta AI to incorrectly identify text or an object.
Biased descriptions
When Meta AI is asked to describe a person, the description could include offensive or inappropriate descriptions.
Failing to alert user of hazards
When describing scenes, Meta AI could fail to recognize hazards leading the user to trip and fall.
Privacy
Meta AI could voiceover a piece of sensitive information (ex. medical test results) causing people nearby to overhear.
03
Product safety mitigation design
Since there is no visual UI input, we couldn’t use warnings or content flags to mitigate some of these risks. Instead, I helped develop agentic conversation design principles in order to mitigate harms.

Scale of confidence
For example, on the issue of incorrect interpretation the solution was in how our Conversation Designers wrote voice interactions based on confidence scores.
We didn’t have a visual UI that we could put a tooltip or a disclaimer in, so we needed to use natural language to convey a sense or certainty or uncertainty.
By thinking through how to use content, voice, tone, and inflection we could communicate to blind users how much or little they could rely on our interpretation.
We also encouraged users to use supplements, like asking other people around them, rather than relying 100% on our interpretation.
/ Outcome
This generation of Meta Ray Ban Smart Glasses with Multimodal AI was released at Meta Connect 2023 along with these accessibility-focused AI features.

04
Future accessibility features
Looking ahead, I developed a longer-term smart glasses accessibility roadmap to plot out how future hardware and sensor capabilities could benefit people with other kinds of disabilities.
Live captions for deaf users
Future models of the smart glasses will include an Augmented Reality (AR) display, and could help deaf users see live captions of conversations.
Alternative computer inputs for people who are limited in hand/arm mobility
The upcoming electromyography (EMG) wrist band could allow users with limited hand/arm mobility to have an alternative computer input method.
Spoken communication for people who are nonverbal
A combination of the EMG wrist band and the AI voice feature could help people who are nonverbal speak to other people.
AR reminders or labels for people with degenerative memory disorders
AR display features could allow people to pin reminders and labels to items in their home, allowing people with memory disorders, like Dementia or Alzheimer’s, to have an easier time remembering contexts.
05
Featured on The Vergecast
Victoria Song from The Verge talks about how the accessibility community uses Meta Rayban Smart Glasses.
Video
How the low-vision community embraced AI smart glasses | The Vergecast
Other work

Cash App Trust & Safety
Privacy, Safety, Security, Equity

Privacy Protections for Meta's VR Products
Privacy, Integrity

LGBTQ Safety on Facebook Dating
Privacy, Safety, Integrity

Meta Responsible AI
AI Safety