Google is Developing an Android App to Help Guide Visually Impaired Users

“Lavender at 12 o’clock,” a virtual assistant calls out.

Meet Lookout, an Android app being developed by Google that can identify objects around you—from a pair of scissors to an exit sign to a lavender bush—and tell you exactly where they are located. The goal of the app, which will be made available on the Google Play store later in 2018, is to help guide people who are blind or visually impaired.

Lookout is specifically built for Pixel phones. Users can place their phone in a lanyard worn around their neck, with the camera facing out. Once the app is open, users can select a mode that best describes the environment they’re currently in, whether that's being home, at work, in a shopping mall, or in a situation where they need to have text read aloud to them ("scan mode"). The “work and play” mode, for instance, will likely alert users when they’re next to an elevator or stairwell, while the “home” mode will identify your TV, washing machine, and kitchen table.

After selecting a mode, the app begins to detect objects, text, and people using the phone’s camera. It uses machine learning to determine what information is most critical to the individual user based on their usage history, allowing it to improve over time.

It’s also designed to be mostly hands-free, allowing users to navigate their surroundings without having to constantly tap on the app. Users can cover the camera to pause detection, knock twice on their phone to resume detection, and use the fingerprint sensor to switch to a different mode. The app can also be controlled via bluetooth or work offline.

Lookout follows the 2017 release of Microsoft’s Seeing AI app for iOS, which acts as a “talking camera” by describing objects surrounding the user, according to The Next Web. As the tech news site points out, there’s one key difference, though:

“Lookout seems like it could be more useful as its various modes can help highlight only the important objects in one’s surroundings based on what they’re doing—and therefore cut out a whole lot of noise from the app.”

To see how the Lookout app works, check out this video from Google: