Using AI to study 12 years of representation in TV

The show is a popular medical drama, and two police officers have just rushed a victim to the hospital. Orderlies lift the patient onto a gurney and dash into the ER. Nurses hook up IVs. A doctor takes command, shining a light into the victim’s eyes, barking orders.

Now hit pause. Look closer: Who plays the doctor and who plays the orderlies? Are the cops men or women? Is the victim light or dark-skinned? Who gets the most screen time? Who gets to speak, and who doesn’t?

See It, Be It: What Families are Seeing on TV is a new study that analyzes trends in the screen and speaking time of the visually presenting attributes of the characters — gender, skin tone and age — in scripted television over the last 12 years. The report — led by the Geena Davis Institute on Gender in Media (GDI), in partnership with Google Research as the technology provider, and the Signal Analysis and Interpretation Laboratory (SAIL) at the University of Southern California (USC) as the academic advisor — examines representation in Nielsen’s top 10 scripted U.S. TV shows for each season from 2010 to 2021.

Some of the findings from the report

The report analyzed 440 hours of programming, from hit comedies and political dramas to romances and supernatural sci-fi, to reveal insights like:

• The screen time of female characters is rising, but the male-female gap remains. In 2021, male characters still received about 16 percent more screen time than females.
• The screen time gap between characters with light and dark skin tones narrowed from 81% in 2010 to 55% in 2021. But a large gap still remains. Characters with medium and dark skin tones saw screen time increases of 8 and 9 percentage points, respectively, from 2010 to 2021.
• Older men but younger women get more screen time. The most common age group on screen for male characters appears to be 33 to 60; for female characters, it appears to be 18 to 33. Women over the apparent age of 60 still receive less than 1 percent of all available screen time.
• Speaking time for female characters of dark skin tone rose the most. But they are still the group least likely to speak when shown on screen. The speaking time increased at an average rate of 1.2 % per year, but female characters with dark skin tone are speaking only 16 % of the time.

A first-of-its-kind study using advanced technology

The study, a unique partnership between the GDI, USC and Google Research, is the first collaboration of its kind using advanced AI models at this scale. And the partnership was forged from these organizations recognizing how their missions could complement each other.

GDI was founded in 2004 by Academy Award-winning actor Geena Davis, and received a Governor’s Award Emmy in 2022 in recognition of its work to promote gender balance and foster inclusion, and reduce negative stereotyping, in family entertainment. Madeline Di Nonno, President & CEO of GDI notes: “Although our findings demonstrate progress over the past decade for gender and screen time, there are still significant gaps in gender, age and skin tone and much more work to do.”

Google Research’s MUSE (Media Understanding for Social Exploration) project builds AI-enabled technology designed to understand patterns in how people are portrayed in mainstream media, and to inspire more equitable content. That makes the ongoing partnership with the GDI a natural fit.

To reveal the insights in this study, 440 hours of footage were processed in less than 24 hours. More than 100 frames were processed each second, for a total of over 12 million second-by-second face appearances analyzed using machine learning models. This kind of analysis would be impractical to do manually.

Building off our earlier study of representation in media

Our earlier joint study of gender equity in Hollywood movies was the first use of AI to effectively study representation in media. For this TV study, Dr. Shri Narayanan and his team at SAIL once again provided its deep expertise to co-analyze the data with GDI. As Dr. Shri tells us, “Computational media intelligence not only allows us to gain nuanced insights about the stories we tell using media, at scale and in detail never possible before, but offers new tools for creating inclusive and equitable experiences.”

As the technology evolves, the MUSE project continues to add and improve signals. Google has open-sourced a more inclusive skin tone scale, based on the Monk Skin Tone (MST) Scale developed by Dr. Ellis Monk of Harvard University. Our latest study involves one of the first applications of a skin tone classifier based on MST Scale to study representation patterns in a socio-technical system.¹

We plan to collaborate further with GDI and USC on studies that look at cross-culture and mixed-media content in other areas of the world. We intend to evolve the work from what is “presence” signals to “portrayal signals” — meaning examining details like who is or who isn’t shown as a leader in social interactions, or whether certain communities are projected in stereotypical ways. This requires multidisciplinary thinking across computational and social sciences – foundational research, multimodal content understanding, a responsible AI lens, and more. This is an exciting area of work that must be done collectively and responsibly.

The ultimate goal of this work is to understand, with the help of technology, how people oriented stories are portrayed in the media — and to equip all of us with information on how those stories influence our perception of the world. We believe that's a critical step in informing the creation of equitable media that reflects the richness and diversity of all humanity.