Google for India 2022: Using AI to make Google more helpful, secure, and private for India's changing digital needs

Today we hosted our 8th annual Google for India event where we shared our progress on building AI-powered and India-first innovations that reflect the country's unique opportunities. From breaking down language barriers, to helping a billion people access information in more natural ways, to keeping people safe at every step, these innovations underscore our commitment to helping make a digital India a reality.

And these advancements are more important than ever. In the wake of the pandemic, India has led the way in defining and embracing digital practices. From farmers in the agricultural heartlands to young urban entrepreneurs, as people everywhere benefit from the convenience of being connected, we at Google India have been privileged to be a part of people’s embrace of the power of technology.

Introducing more natural and intuitive ways for people to search

Thanks to advancements in artificial intelligence, we're now able to create search experiences that reflect how we as people make sense of the world – across visual, voice and language experiences. Search has moved well beyond the typing of a text query – across India, people are increasingly seeking out information on the web in their own language, using their voice to make the most of the internet, and their phone cameras to learn more about the world around them.

Information needs are inherently complex, and to help people navigate these, we’ve focused on building search experiences that empower anyone to interact with technology and explore information just like they would in the real world.

Making it easier for people to search and explore information visually

Sometimes what you're looking for can be difficult to describe in words. With Google Lens, you can search the world around you with your camera or an image. In fact, Indians already use it every month more than any other country worldwide! Now, we're making visual search even more natural with the introduction of multisearch, a major milestone in how you can search for information using images and text simultaneously.

Simply open the camera in the Google app to take a picture or use a screenshot, then add text to your question – just as you would naturally point at something and ask about it. From finding out whether shirts are available with that interesting pattern you saw, or knowing the recipe of a new dish you ate.

Multisearch is available in English in India, and will be coming to many Indian languages in the next year, starting with Hindi.

Besides images, videos are also a rich source of visual information but it’s often hard to find something buried inside a long clip. For example, perhaps you’re watching a long video about Agra, and you’re curious if it also covers Fatehpur Sikri. With a new feature we’re piloting, you’re now able to search for anything that’s mentioned in a video, right from Search. Simply enter a search term using the ‘Search in video’ feature and quickly find what you’re looking for.

Building a more intuitive search experience in Indian languages

Part of seeking information more naturally is being able to use the language, or languages, you’re most comfortable with. Language needs in India are becoming more dynamic, and multilingual, and we’re also seeing this reflected in people’s search experiences. Notably, the majority of Google users in India consume more than one language.

To make it easier for people who use more than one language to seek and explore information, we’re now making search results pages bilingual, for people who prefer it that way. Using advanced machine learning-based translation models and a cross-language search technology, we’ll serve you high-quality and relevant content in your local language alongside English results, if that's how you choose to view them. And we’re excited to share that this is another innovation we’re bringing to India first.

This functionality has already started rolling out in Hindi, and will be expanding to other Indian languages, including Tamil, Telugu, Marathi, and Bengali, in the coming year.

Improving speech recognition technology for Hinglish speakers

We are also addressing India’s dynamic language needs when it comes to voice. Voice-based technologies have had a transformational impact in India, helping people – especially those new to the internet – find information and get things done online by simply speaking in their own language. In fact, the percentage of Indians using voice search queries each day is nearly twice the global average.

We want to help more people, and specifically Indians, ask questions naturally and intuitively with their voice. Today we’re announcing a new speech recognition model that can more effectively understand people who speak in Hinglish. We’re doing this through the use of a new, neural-network inspired speech recognition model that takes into account the individual’s accents, surrounding sounds, context, and speaking styles.

Piloting a new way to help serve people with non-standard speech

Most people take for granted that when they speak, they will be heard and understood. But there are 250 million people worldwide who have non-standard speech, who find it difficult to communicate with others and be easily understood.

Speech recognition innovations can play an important role in such cases, where standard speech models haven’t worked well to date.

Years of research has led to us creating Project Relate – an app that is trained to unique speech patterns of people with non-standard speech. We have been piloting the app with English users in India, and will be expanding this pilot to Hindi users in early 2023. Through this project, we endeavour to make products that rely on speech recognition technology, like Google Assistant, more accessible to everyone.

Easy and organized access to verified official documents from Digilocker on your Android phone

It's not uncommon for many of us to store multiple documents and images of Aadhaar, PAN cards, and driver’s licenses on our phone. People open these files regularly, many especially for availing crucial government services – like filling an online passport form or signing up for citizen benefit schemes.

And so we are announcing our partnership with the National eGovernance Division (NeGD) to enable people with easy access to their authentic digital documents, directly from the Files by Google app on Android.

We are also announcing a feature that does away with hunting through your phone for these files, or struggling with copying important information within them. This functionality is built on a solid foundation of privacy and security – these documents stored in Files by Google will be in an isolated environment on your device, and can only be accessed using a unique lockscreen authentication.

We use a custom AI model to scan for and identify key documents from across your phone, which then uses our proven OCR technology and machine learning models to recognize relevant text and collate this important information into a quick and simple view. And all of these models run offline, completely on device, and cannot be accessed by anyone, including Google.

You’ll see this functionality first in the Files by Google App on Android smartphones in India – tap the new tab, and you can access all of these important documents and important information in a single organized location. If you have documents of other family members on your phone, it organizes their information as well – simply tap on the name to conveniently pull up the relevant details.

Using deep learning to massively improve fraud detection in Google Pay

As Google Pay completes five years in India, it’s hard to overstate the pervasive impact of UPI and digital payments for millions of people for whom transacting online is now second nature. As we continue to find ways to simplify paying online, safety continues to underpin everything we do.

Handling money has now moved from cash in a purse to taps on a phone, and this can sometimes be prone to human error. When a user is presented with a suspect payment request, it isn’t always easy to determine whether it is legitimate.

At Google Pay, we use advanced methods in our safety engine that analyze aggregate transaction patterns of millions of users, relating this to a graph of a billion nodes. This system helps identify possible flags on fraudulent behaviour, then uses its intelligent language engine to flash a warning and notify the user in their most preferred language. In some cases, we also generate a quick vibration to really get your attention!

We are also introducing a useful new feature in Google Pay that makes it easier than ever to access your transaction history. You can now query Google Pay out loud using natural language questions like, “Show me how much I spent on coffee last week”.

Through it all, security is the cornerstone of our product strategy. We focus on the fundamentals of security and software development to design products that are secure by default, private by design and put users in control. While people navigate the internet, it is important that they get safe access to quality information.

As we head into the next year, we remain committed to putting our technology to work to help even more people across the country. India’s unique approach to embracing digital technology has had transformative effects across society. At Google in India, we are equally honored to build solutions to the challenges and to find ways to help more people share the benefits of using the internet in their daily lives.