ShadowPlay: Using our hands to have some fun with AI

Editor’s note: TensorFlow, our open source machine learning platform, is just that—open to anyone. Companies, nonprofits, researchers and developers have used TensorFlow in some pretty cool ways and at Google, we're always looking to do the same. Here's one of those stories.

Chinese shadow puppetry—which uses silhouette figures and music to tell a story—is an ancient Chinese art form that’s been used by generations to charm communities and pass along cultural history. At Google, we’re always experimenting with how we can connect culture with AI and make it fun, which got us thinking: can AI help put on a shadow puppet show?

So we created ShadowPlay, an interactive installation that celebrates the shadow puppetry art form. The installation, built using TensorFlow and TPUs, uses AI to recognize a person’s hand gestures and then magically transform the shadow figure into digital animations representing the 12 animals of the Chinese zodiac and in an interactive show.

We debuted ShadowPlay at the World AI Conference and Google Developers Day in Shanghai in September. To build the experience, we developed a custom machine learning model that was trained on a dataset made up of lots of examples of people’s hand shadows, which could eventually recognize the shadow and match it to the corresponding animal. “In order to bring this project to life, we asked Googlers to help us train the model by making a lot of fun hand gestures. Once we saw the reaction of users seeing their hand shadows morph into characters, it was impossible not to smile!”, says Miguel de Andres-Clavera, Project Lead at Google. To make sure the experience could guess what animal people were making with high accuracy, we trained the model using TPUs, our custom machine learning hardware accelerators.

We had so much fun building ShadowPlay (almost as much fun as practicing our shadow puppets … ), that we’ll be bringing it to more events around the world soon!