Genius Captions
Making captions flexible for everybody.
We decided to design an app that would let users access caption and audio description information in various ways as needed - in other words, everything I have been doing for years.
My husband has low vision, and we spend a lot of time relaxing and watching movies together. Over the years, I have learned how to be his assistant and read subtitles or text on the screen and describe what is happening when it is dark.
He can see the majority well, and often read the captions or subtitles, but sometimes they are too small, or there are too many and it is tiring, or he is sitting farther away.
I have almost ten years of experience in closed captioning videos, and my husband has more than seventeen years of experience as an accessibility consultant.
More and more people are getting their information from videos, and a big part of our culture is and conversation is about movies and tv shows. I realized there is a big opportunity here to develop an app to help all people have access to and enjoy movies, tv, educational videos, news and more.
Limitations of Currently Available Closed Captions, Subtitles and Audio Description
Currently, there are a limited number of ways to make videos accessible, mostly closed (or open) captions or audio description. However, there are many different types of users, and they have unique needs – and they may have different needs depending on the type of video, the environment (noise, distractions or interruptions), lighting of scenes, whether there are foreign language subtitles, a lot of text or small details on the screen that may be difficult to see, the accent of the speakers or if they are mumbling or whispering, and the other sounds that may interfere with understanding of the dialog.
Samantha
Watching a movie with subtitles for some scenes and important plot points that happen in the dark.
Narrate subtitles
Turn audio description on and off
Nikki
Characters have strong Australian accents
Read Mode
“What did they say”
Scrolling backwards and then synching with current point and playing movie
Toby
User who is blind is watching a movie with friends
Audio description with headphones
Sketches to explore the best format of the design
I initially thought of designing the app for tablets. This would give maximum flexibility in types of information the user could see, as well as more screen real estate to customize the size and format of the captions. However, currently the types of information in captions are not separated into different streams or encoded in any different way. In addition, a phone app could be used on a tablet for users who needed a larger font. And since we anticipate a lot of users will just be listening without viewing the screen most of the time, the phone seemed most practical and easiest to hold for long periods.
Designing for the phone in the vertical position provides little flexibility for users to change the size of the captions. I decided to focus on designing for phones in the horizontal position, and then making sure the user could optionally use in the vertical position in line with current WCAG 2.1 accessibility standards.
Initial Medium-Fidelity Mocks
Read and Listen Mode
Listen Only Mode
Next Steps
This project is in the very early stages. We are planning to make a short video with many different types of issues and create subtitle and audio description tracks, and then create a prototype to experiment with each of the features and options to see which are best for each situation, which work with technology available now, and which we can design for in anticipation of technology that may be available in the near future.
Other Projects
Other Projects
