Turning graphs into music with Science Journal
We try to make products that work for users of all abilities and backgrounds. Making tools accessible does more than just make them better for users with disabilities; it makes them better for everyone. Our Science Journal app — designed for kids (and adults!) to observe and experiment the world around them through their Android devices — is an example of this.
Together with the Exploratorium, we just released a new activity in Science Journal called Light Instruments: an activity that enables you to design and build a musical instrument played by changing the light that reaches it. The ability to turn graphs into sound was actually originally a feature we designed for accessibility.
Science Journal enables people to observe, record, experiment and make predictions with real-world data. The app makes use of the many built-in sensors our phones contain, as well as other external sensors connected over Bluetooth. It graphs that data over time to help analyze, compare and reflect trends in a way that exploring individual sensor values would never allow us to do. The issue here though, is that not all users are able to see these graphs. And we wanted to do something about that.
Consulting with Google’s Accessibility Engineering team, we started to brainstorm ways to communicate data trends in real-time for users with visual impairments. We decided to try to graph sonification, and use non-speech audio to convey data trends. By implementing a simple sonification that generated tones with frequencies that changed depending on the value of the current data point within the graph’s minimum and maximum, we allowed users to listen to live data and then “play back” graph sonifications of already-recorded data (source code). In short, we translated points on the graph to musical notes!
When we shared this feature with our partners at the Exploratorium, they saw more than an accessibility tool for users with low vision. As believers in “low floor / high ceiling” learning experiences (where users can enter at any background level and continue to learn and grow within the experience), they saw graph sonification as a way for kids with little-to-no experience analyzing graphs to start developing graph literacy and understanding data trends, lowering the “floor” for kids to do scientific exploration.
So together, we created the playful Light Instruments activity where musical “instruments” play based on changes in light levels. These instruments are made from light sensors connected to external Bluetooth devices which, when connected to the Science Journal app, will show the sensor’s value and change as the levels of light do, and produce different “notes” through the graph sonification.
As an example, here’s a “flute” that uses light and a photoresistor connected to the Science Journal app:
Visualizing data is cool, but listening to it adds a whole new way to help us understand trends and outliers in new ways. We can’t wait to see what sounds you’ll generate!
