Reducing gender-based harms in AI with Sunipa Dev

Natural language processing (NLP) is a form of artificial intelligence that teaches computer programs how to take in, interpret, and produce language from large data sets. For example, grammar checkers use NLP to come up with grammar suggestions that help people write grammatically correct phrases. But as Google’s AI Principles note, it’s sometimes necessary to have human intervention to identify risks of unfair bias.

Sunipa Dev is a research scientist at Google who focuses on Responsible AI. Some of her work focuses specifically on ways to evaluate unfair bias in NLP outcomes, reducing harms for people with queer and non-binary identities. Sunipa’s work was recently featured at a workshop at the ACM Fairness, Accountability, and Transparency (FAcct) conference in Seoul, Korea.

In our interview, she emphasizes that her work is achievable only through forging collaborative partnerships between researchers, engineers, and AI practitioners with everyday users and communities.

What inspired you to take on this career path?

While working on my PhD at the University of Utah, I explored research questions such as, “How do we evaluate NLP tech if they contain biases?” As language models evolved, our questions about potential harms did, too. During my postdoc work at UCLA, we ran a study to evaluate challenges in various language models by surveying respondents who identified as non-binary and had some experience with AI. With a focus on gender bias, our respondents helped us understand that experiences with language technologies cannot be understood in isolation. Rather, we must consider how these technologies intersect with systemic discrimination, erasure, and marginalization. For example, the harm of misgendering by a language technology can be compounded for trans, non-binary, and gender-diverse individuals who are already fighting against society to defend their identities. And when it’s in your personal space, like on your devices while emailing or texting, these small jabs can build up to larger psychological damage.

What is your current role at Google?

I am currently a Research Scientist at the Responsible AI - Human Centered Technology team. In my current role, I am working to build a better understanding of how to avoid unfair bias in AI language models across different cultures and geographies, aligned with Google’s AI Principles.

This is a challenge because language changes, and so do cultures and regional laws as we move from one place to another. This can all impact how people express themselves, what identities they choose and how they experience discrimination on a daily basis. Gender bias can manifest in entirely different ways in different parts of the world. In some of my ongoing work that focuses on a non-Western point of view, we are working with social scientists and NGOs in India while engaging with local communities. We are using the voices of many people who are living in a specific region and asking, “What are the biases prevalent in their society?”

What is gender bias in NLP?

Written text and training data for language technologies can lack representation or misrepresent different gender identities; this can reflect social biases. As a result, some NLP technologies can reinforce gender stereotypes and slurs, erase people’s gender identities, or have reduced quality of service for marginalized communities. What drives me in my work is my goal to make language technologies more inclusive and usable.

Why does this matter for AI?

Gender can be such an integral part of someone's identity, and having that wrongly assumed by an AI system can be triggering, unfair, and harmful. We need to work towards systems and societies that do not encode unfair biases and harmful stereotypes in order to break out of the cycle of perpetuating harms of stereotyping, misgendering, and erasure.

How can people who are not researchers, engineers or AI practitioners engage in this work?

A very direct way is for people to report potential harms as bugs within products they use. People can also participate in open discussions in workshops, panels and town halls. These are all helpful ways to build inclusive AI.

I want to emphasize, however, that the onus can’t only be on the user. It’s also on the side of the researcher, engineer and AI practitioner. The goal is to create a continuous feedback loop between humans and machines, with real people stepping in to ensure the creation of more responsible AI. As AI practitioners, we need to work with the people we’re trying to serve and have users collaborate with us to tell us what we need to do better.