The beat goes on: Using AI to build a drumming prosthetic
Since he was young, Jason Barnes loved drumming. But when he lost his arm in an accident at the age of 22, he wasn’t sure if he’d ever drum again. Since typical prosthetics weren’t built with drumming in mind, he couldn’t quite keep rhythm in the way he wanted to. His goal was to build a robotic drumming prosthetic that could.
Jason worked with a team of researchers and developers to use artificial intelligence (AI), specifically Google’s open source machine learning platform TensorFlow, to build a prosthetic arm that would let him keep creating music as naturally as possible. Ultimately, his drive to keep playing music helped push an entire area of research forward to help build better products for the more than a billion people around the world with a disability.
We chatted with Jason to learn more about building a prosthetic that helped him keep making music and the importance of building accessible technology. If you’re interested in learning more about our recent accessibility work or tips for designing more accessible products, you can watch on-demand sessions from I/O, our annual developer conference.
How did you find music and what was your journey to becoming a musician?
I found my love of music through my dad and random jam sessions in the garage. It was one of the first things in life that actually interested me. My love for music grew and led me down a bunch of other paths, playing in multiple bands and different genres of music.
What inspired this project?
I was in my first private music lesson with Eric Sanders, a drum instructor at the Atlanta Institute of Music. I explained this idea I had of making an AI-powered drumming prosthesis. He thought it was a super cool idea and connected me to Gil Weinberg, a professor at Georgia Tech Center for Music Technology, and his students. One thing led to another and within a week Gil reached out and was interested in the project.
What was it like to work with this team to build technology that fit your needs as a musician?
A lot of the people working on this project came from a more technological background. Even if they were musicians, they weren’t necessarily drummers. It was challenging to fuse those different worlds together to figure out how to make this applicable for a drummer and also meet the standards of building an advanced technological robot.
Where do you think this technology can take us next?
If we can build a fairly niche prosthetic specifically for drumming, then the options are endless — the sky's the limit. For example, based on the technology we developed you could help amputees who can’t use a computer mouse at all. They could essentially put on this prosthetic device and use a motion sensor to control the location of where the cursor is and be able to click and navigate.
Why is it important to build products that make something like music accessible to everyone?
Music is such a healing thing. I’ve seen a kid with cerebral palsy get a hold of drumsticks and play a beat almost perfectly and stay in rhythm. He’s completely oblivious to everything else that’s outside of that, he’s locked into what he’s doing and killing it — that’s amazing to me.
Personally, I don't know how I would have gotten through my accident without having music to turn to. It's something that has also helped me deal with losing my dad — it keeps my connection to him alive.
