What is Microsoft’s Seeing AI App and What you need to know

In celebration of the International Day of Persons with Disabilities (IDPD), Microsoft introduced the revamped Seeing AI app. Powered by advanced Microsoft AI, it’s now available on Android devices through the Google Play Store. Speaking in 18 languages* already, to reach 36 by 2024, this app is a breakthrough for accessibility. It’s a game-changer, aiding people globally by providing assistance and support and making daily tasks more accessible and inclusive. This updated version promises a brighter future for individuals using the Seeing AI app, showcasing Microsoft’s commitment to accessibility for all.

What does the Seeing AI App do?

The Seeing AI app, free to use, describes the world for blind and low-vision users via mobile devices. It helps with daily tasks like reading mail, recognizing products, and explaining photos. This app goes beyond just identifying things; it’s like having a helpful guide always with you, enhancing accessibility and easing everyday routines for those with visual impairments.

The latest generative AI features recently launched on iOS have been incorporated into the new Android version:

• Enhanced Depictions of Images: Apart from giving a concise overview of the pictures on the Scene channel, you have the option to select ‘more info’ and access a comprehensive description that provides extensive information about the contents of the image.
• Interact with your documents through chat: In addition to having the document read aloud, you can engage in a conversation with Seeing AI after scanning it. This allows you to inquire about various aspects, such as menu items, the cost of a specific item on a receipt, or even request a summary of an article.

How Seeing AI Empowers the Visually Impaired

The Seeing AI app, launched in 2017 as a research project at Microsoft, stems from its dedication to innovation for people with disabilities. Their team collaborated closely with the blind community, aiming to identify how technology could boost independence and joy. Initially for iOS, this app evolved through insights gathered directly from users. Engineers worked hand-in-hand with the blind community, learning about their needs and aspirations. Together, they strived to develop a tool that enhances independence and enriches daily experiences for individuals with visual impairments. Seeing AI continues to grow, driven by the commitment to empowering lives through accessible technology.

With the assistance of Seeing AI, one can effortlessly direct the camera or capture an image to receive an auditory description. It is also possible to switch between channels to obtain specific and targeted information: 

Short Text

The Seeing AI app swiftly converts text into spoken words as it appears within the camera’s view. This feature ensures instant, audible feedback for any text detected, enabling real-time accessibility by vocalizing the content observed. Through the app’s camera functionality, text seamlessly transforms into speech, offering immediate auditory information to users for enhanced accessibility.

Documents

Seeing AI provides audio guidance for capturing and reading printed pages, maintaining their original layout. Interact with the app for quick information retrieval through conversation, enabling swift access to your desired content.

Products

Audio beeps guide barcode scanning, offering continual assistance. You’ll hear product names and details whenever available, aiding seamless scanning.

Scenes

Explore further details by tapping ‘more info’ for an in-depth scene description. Glide your finger across the screen to hear the locations of different objects in the photo, enabling a detailed auditory exploration of the captured scene.

People

Uncover nearby companions or friends within your surrounding area. Explore and connect with individuals or groups present in your immediate vicinity, fostering opportunities for social interaction or forming new bonds based on proximity and shared interests.

Currency

This feature recognizes and distinguishes various types of currency bills. Using visual recognition, it can differentiate between different monetary notes, providing information about their denomination or value. This capability aids users, especially those with visual impairments, by allowing them to identify and differentiate currency bills easily through an app or device equipped with this functionality.

Colors

This feature refers to identifying the primary color or shade perceived by an observer or a system. It involves analyzing the dominant wavelength of light emitted or reflected by an object to categorize it within the color spectrum. This determination helps recognize and characterize the specific hue perceived visually, aiding in color identification or analysis processes.

Handwriting

The app recognizes and reads handwritten text commonly found in greeting cards, enhancing accessibility. However, this feature’s language availability is limited, offering recognition and interpretation in only a select number of languages.

Light

This feature emits varying tones based on ambient light levels. Higher brightness triggers a higher-pitched sound, while lower light levels result in lower tones. It translates environmental brightness into auditory cues, aiding visually impaired users by providing real-time feedback about light intensity through distinctive sound levels.

Images in other apps

Activate the Seeing AI App’s image identification feature by uploading a photo. The app processes the image using advanced recognition technology, providing detailed descriptions or information about objects, scenes, or text within the photo. This capability allows seamless accessibility and understanding of visual content for users who are visually impaired or benefit from additional context.

The Seeing AI app is heading to Android, reaching 3 billion users. This move empowers more blind and low-vision individuals, making everyday life easier. Microsoft is eager for customer feedback, collaborating closely with the community to enhance the app. Microsoft’s motto, “nothing about us, without us,” guides us. User input shapes future versions, ensuring the app suits their needs. Customer feedback remains crucial as the app rolls out new AI-powered updates, continually improving the Seeing AI app for everyone’s benefit.

You may also like: Microsoft Copilot AI Chatbot is now generally available, OpenAI Investor Envisions Free AI-Enabled Doctors and Lawyers in 10 Years