Mobile App A/B Testing

What Is Mobile App A/B Testing?

Mobile app A/B testing is the practice of using A/B testing to test different experiences within mobile apps. App users are randomly bucketed into different segments, and each is shown a different experience. After enough users go through the experiment, it is possible to tell with statistical confidence which version of the app leads to higher conversions.

For example, an app developer might have a mobile game that includes in-app purchases for extra levels. The app developer might want to test out different banners for the levels, to see which one is clicked on the most.

By running an A/B test, they will be able to determine, based on their actual users, which banner performs the best. Once the test reaches statistical significance, they can roll-out the winning variation across all their users and improve their overall conversion rate.

Benefits Of Mobile App A/B Testing

The benefits to running A/B tests on mobile apps is similar to those of running A/B tests on the web. A/B testing allows you to test out different hypotheses for experiences within your app and make changes to your app based on actual data instead of gut instinct. A/B testing allows you to determine with statistical confidence what the impact of changes you make to your app will have, and measure exactly how much that impact will be.

By continually running tests on your mobile app, you will constantly be able to improve the experience of your app and improve your conversion metrics. It's the reason why top companies such as Facebook, Amazon, and Google constantly run A/B tests in their mobile apps. A/B testing is indispensable for experience optimization and conversion rate optimization on mobile devices.

Another benefit of using a mobile A/B testing platform such as Optimizely is that it allows you to make changes directly to an app that has been published to Apple's app store, without having to go through the app review process. This allows you to iterate faster on changes and make fixes and updates to live apps.

Mobile App A/B Testing In Action

The power of mobile app A/B testing is captured in this case study of Secret Escapes, a travel app for iOS. In the experiment, Secret Escapes used Optimizely to test the impact of removing the option to skip signing into their app. What they found was that making the sign-in mandatory resulted in a higher average lifetime value for the user, and improved the ratio of user LTV to customer acquisition costs.

In another case study, the photography app French Girls was able to increase user activation rate by optimizing their testing the order in which they showed specific screens during their onboarding flow. Because they used A/B testing, they were easily able to see which version performed better and quantify the impact of their change.

Mobile App A/B Testing Ideas

Every app is different, and there are countless changes that can be tested on mobile apps. Keys areas to focus on when conducting A/B testing are user onboarding, user retention, user engagement rate, and for apps that feature transactions, every step along the sales funnel to maximize conversions.

Examples of things to test include the number of forms required to sign-up for an account, the placement of buttons and other elements on the screen, the text used in calls-to-action (CTA), images, and ad placement. For more ideas, see our blog post on 50 Mobile App Development Tips.

Mobile App A/B Testing With Optimizely

Optimizely is the leading platform for A/B testing on the desktop and on mobile devices. Optimizely is simple to install, easy to use, and provides enterprise level statistics that gives you confidence in your test results.

If you've used Optimizely before on the web, then you will find the interface for mobile to be very familiar. Even if you've never used Optimizely before, Optimizely features an easy-to-use visual editor that allows you to make changes to your app with a WYSIWYG editor. No programming experience needed!

If you want to make to improve the experience of your iOS or Android app, improve conversions, and make data-driven product changes, start testing with Optimizely today!