/ Case study

Meta Smart Glasses

Multimodal AI for Blind and Low Vision Users

At Meta Reality Labs, I led product design work across our Privacy, Security, Safety, Accessibility, and Equity teams.

A man wearing Ray-Ban Meta smart glasses, sitting outdoors and looking at his phone

"Hey Meta, what am I looking at?"

Copied ✓

/ Background

As part of our Accessibility and Equity initiatives, I helped develop multi-modal AI features on the Meta Ray Ban Smart Glasses to enable blind and low-vision users to use the smart glasses as assistive devices.

My primary roles were to develop the product accessibility roadmap for the 2023 release of the Meta Ray Ban Smart Glasses, work with the Artificial Intelligence (AI) engineers and the Optical Character Recognition (OCR) engineers on prototyping, and to run Responsible AI red teaming to mitigate AI risks.

Role

Product Design Manager, Meta Reality Labs Trust

Timeline

2022 - 2023

Video

Meta Ray Ban Smart Glasses Multi-Modal AI

/ Product goals (MVP scope)

—

Read text from a physical source

Read a body of text aloud to the user from a sign, a piece of mail, etc

—

Describe an item in the environment

Describe an individual physical item to the user

—

Describe what is happening

Describe an entire scene to the user, including descriptions of people and items in the scene, hazards, and motion

01

Multi-modal AI and computer vision

Video, audio, and voice assistant features on the smart glasses unlocked new opportunities to integrate multi-modal AI and computer vision.

To design the blind and low vision features, I worked with the OCR engineers developing the technology. Together we built out the conversation design flows, onboarding, and settings that enable the glasses to read aloud descriptions of text and scenes in the real world.

Side view of Ray-Ban Meta smart glasses showing the touch-control area on the arm with an audio waveform
Close-up of Ray-Ban Meta smart glasses showing the built-in camera on the frame
Prototype illustration of computer vision: a person wearing a headset looks at a desk while the system identifies and announces 'Laptop'

Image: Prototype of computer vision image recognition

02

Responsible AI red teaming

Once we had a prototype, I ran product red teaming sessions to identify and mitigate risks. One particular area of focus was Responsible AI and mitigating risks from our AI agent interactions.

Product Responsibility Considerations

Incorrect interpretation

Low lighting, off-angle capture, or lack of representation in training data could cause Meta AI to incorrectly identify text or an object.

Biased descriptions

When Meta AI is asked to describe a person, the description could include offensive or inappropriate descriptions.

Failing to alert user of hazards

When describing scenes, Meta AI could fail to recognize hazards leading the user to trip and fall.

Privacy

Meta AI could voiceover a piece of sensitive information (ex. medical test results) causing people nearby to overhear.

03

Product safety mitigation design

Since there is no visual UI input, we couldn’t use warnings or content flags to mitigate some of these risks. Instead, I helped develop agentic conversation design principles in order to mitigate harms.

Ray-Ban Meta smart glasses with a 'Hey Meta' voice prompt indicator

Scale of confidence

For example, on the issue of incorrect interpretation the solution was in how our Conversation Designers wrote voice interactions based on confidence scores.

We didn’t have a visual UI that we could put a tooltip or a disclaimer in, so we needed to use natural language to convey a sense or certainty or uncertainty.

By thinking through how to use content, voice, tone, and inflection we could communicate to blind users how much or little they could rely on our interpretation.

We also encouraged users to use supplements, like asking other people around them, rather than relying 100% on our interpretation.

↑ Lower confidence

Not confident

<30%

“Hmm, this might say ‘caution, wet floor’ but you should ask someone nearby to confirm.”

Somewhat confident

~30–80%

“I think this says ‘tenant meeting in the lobby on Tuesday at 6pm’.”

Very confident

>80%

“This says ‘organic red apples’.”

Not confident

<30%

“Hmm, this might say ‘caution, wet floor’ but you should ask someone nearby to confirm.”

Somewhat confident

~30–80%

“I think this says ‘tenant meeting in the lobby on Tuesday at 6pm’.”

Very confident

>80%

“This says ‘organic red apples’.”

↓ Higher confidence

Lower confidence

Higher confidence

/ Outcome

This generation of Meta Ray Ban Smart Glasses with Multimodal AI was released at Meta Connect 2023 along with these accessibility-focused AI features.

Ray-Ban Meta smart glasses on display at Meta Connect 2023 with a 'Pre-order today, $299' sign

04

Future accessibility features

Looking ahead, I developed a longer-term smart glasses accessibility roadmap to plot out how future hardware and sensor capabilities could benefit people with other kinds of disabilities.

Live captions for deaf users

Future models of the smart glasses will include an Augmented Reality (AR) display, and could help deaf users see live captions of conversations.

Alternative computer inputs for people who are limited in hand/arm mobility

The upcoming electromyography (EMG) wrist band could allow users with limited hand/arm mobility to have an alternative computer input method.

Spoken communication for people who are nonverbal

A combination of the EMG wrist band and the AI voice feature could help people who are nonverbal speak to other people.

AR reminders or labels for people with degenerative memory disorders

AR display features could allow people to pin reminders and labels to items in their home, allowing people with memory disorders, like Dementia or Alzheimer’s, to have an easier time remembering contexts.

05

Featured on The Vergecast

Victoria Song from The Verge talks about how the accessibility community uses Meta Rayban Smart Glasses.

Video

How the low-vision community embraced AI smart glasses | The Vergecast

Other work

A 3D illustration of a phone showing Cash App, surrounded by a protective bubble that blocks icons representing scams and vulnerabilities

Cash App Trust & Safety

Privacy, Safety, Security, Equity

A man wearing a Meta Quest Pro VR headset

Privacy Protections for Meta's VR Products

Privacy, Integrity

A queer couple on a green couch, one sitting behind the other with their arms around their partner’s shoulders

LGBTQ Safety on Facebook Dating

Privacy, Safety, Integrity

The Meta AI logo with a search box on top that reads “Ask Meta AI or Search”

Meta Responsible AI

AI Safety

©2026