Better A/B Testing with Firebase

Earlier this year, the Google Optimize and Firebase teams worked together to bring A/B testing functionality to Firebase. Last week, at the Firebase Dev Summit, we announced that A/B testing is now available in beta to all app developers.

This post originally appeared on The Firebase Blog.

Announcing Better A/B Testing with Firebase 

If you're like most app developers, you know that small changes can often make a big difference in the long term success of your app. Whether it's the wording that goes into your "Purchase" button, the order in which dialogs appear in your sign-up flow, or how difficult you've made a particular level of a game, that attention to detail can often make the difference between an app that hits the top charts, or one that languishes. 

But how do you know you've made the right changes? You can certainly make some educated guesses, ask friends, or run focus groups. But often, the best way to find out how your users will react to changes within your app is to simply try out those changes and see for yourself. And that's the idea behind A/B testing; it lets you release two (or more!) versions of your app simultaneously among randomly selected users to find out which version truly is more successful at getting the results you want. 

And while Firebase Remote Config did allow you to perform some simple A/B testing through it's "random percentile" condition, we've gone ahead and added an entirely new experiment layer in Firebase that works with Remote Config and notifications to make it quick and easy to set up and measure sophisticated A/B tests. Let's take a quick tour of how it works! 

Getting to Know the New A/B Testing Feature 

With the new A/B testing feature, you can create an A/B test that will allow you to play with any combination of values that you can control through Remote Config. Setting up an A/B test allows you to define how the experiment will behave in a number of different ways, including determining how many of your users are involved with the experiment at first…

…how many variants you want to run, and how your app might behave differently for each variant…

...and what the goal of the experiment is.

Different experiments might have different desired goals, and A/B testing supports a number of common outcomes, like increasing overall revenue or retention in your app, reducing the number of crashes, or increasing the occurrence of any event you're measuring in Google Analytics for Firebase, such as finishing your in-app tutorial.

Once you've defined your A/B test, Firebase takes over by delivering these different variations of your app to randomly-selected members of your audience. Firebase will then measure your users' behavior over time, and let you know when an experiment appears to be performing better, based on those goals you've defined earlier. Firebase A/B testing measures these results for you with the same Bayesian statistical models that power Google Optimize, Google's free testing and personalization product for websites.

Using A/B Tests for Better Onboarding: A Case Study 

Fabulous, a motivational app for building better habits, recently made improvements to their app's onboarding by using Firebase A/B testing. When the user first starts an app, Fabulous shows them how to complete a habit, presents them with a letter about forming better habits, and then asks them to commit to a simple routine. The team suspected that if they removed a few steps from this onboarding process, more people might complete it.

So they ran an A/B test where some users didn't see the letter, others didn't see the request to commit to a simple routine, and others skipped both of those steps. The Fabulous team found that by removing both of these steps from the onboarding process, there was a 7% improvement in the rate of users completing the onboarding flow. More importantly, they confirmed that this shorter onboarding experience didn't have any impact on their app's retention.

Test Your Notifications, Too! 

You also have the ability to A/B test your app notification messages through the Firebase Notifications console. You can try out different versions of your notification message and see which ones lead to more users opening up your app from that notification, or which messages lead to users performing some intended goal within your app, like making a purchase.

Getting Started 

A/B testing is available in beta to all Firebase developers starting today. If you're excited to get started, you should make sure that your app is wired up to use Remote Config and/or Firebase Cloud Messaging, and that you've updated these libraries to the latest and greatest versions. You can always find out more about A/B testing in our documentation, or check out the A/B Test Like a Pro video series we've been building.

Then, head on over to the Firebase Console and start making your app better — one experiment at a time!