Lookout: an app to help blind and visually impaired people learn about their surroundings

There are over 253 million blind or visually impaired people in the world. To make the world more accessible to them, we need to build tools that can work with the ever-changing environment around us. Our new Android app Lookout, coming to the Play Store in the U.S soon, helps people who are blind or visually impaired become more independent by giving auditory cues as they encounter objects, text and people around them.

We recommend wearing your Pixel device in a lanyard around your neck, or in your shirt pocket, with the camera pointing away from your body. After opening the app, and selecting a mode, Lookout processes items of importance in your environment and shares information it believes to be relevant—text from a recipe book, or the location of a bathroom, an exit sign, a chair or a person nearby. Lookout delivers spoken notifications, designed to be used with minimal interaction allowing people to stay engaged with their activity.

There are four modes to choose from within the app: Home, Work & Play, Scan or Experimental (this allows you to test out features we’re working on). When you select a specific mode, Lookout will deliver information that’s relevant to the selected activity. If you’re getting ready to do your daily chores you’d select “Home” and you’ll hear notifications that tell you where the couch, table or dishwasher is. It gives you an idea of where those objects are in relation to you, for example “couch 3 o’clock” means the couch is on your right. If you select “Work & Play” when heading into the office, it may tell you when you’re next to an elevator, or stairwell.  As more people use the app, Lookout will use machine learning to learn what people are interested in hearing about, and will deliver these results more often.

The core experience is processed on the device, which means the app can be used without an internet connection. Accessibility will be an ongoing priority for us, and Lookout is one step in helping blind or visually impaired people gain more independence by understanding their physical surroundings.