A/B Testing at MXL

Testing plays a large role in our decision making process at MobileX Labs. Dynamic metrics in an always-evolving market mean that marketing strategies and elements of those strategies are constantly changing. As Benjamin Graham, father of value investing, explained to his readers, past performance is no guarantee of future outcomes. Therefore it is necessary to test ideas often and adjust accordingly.

For example, if the cost-per-install (CPI) for one of our apps increases significantly in an ad campaign, I test new images, copy, and segmentation to keep the ads fresh and test out how new audiences will respond. A chi-square test or regression analysis of the data can determine if the new ad performs significantly better (or worse) than it's predecessor. I know it may seem difficult to implement, but in this series of blog posts I will lay out the process of setting up, implementing, and analyzing an effective A/B test.

In this first post I will outline how I set up an A/B test to ensure the reliability of the data.

Testing is not as hard as you think.

Scared testing

I have to admit that I’m a former statistics major, and A/B and multivariate testing is actually quite rewarding to me. I know from teaching that there are many who say that testing is either too difficult to learn or do effectively, so in this series of posts I will shut the naysayers up and show that A/B testing is easy, practical, and extremely beneficial. Let’s get started.

Part I: Setting Up an A/B Test

Traffic cop

Earlier this week I had the opportunity to teach a group of aspiring technical marketers at the Startup Institute how to optimize with A/B testing. During our discussion, I outlined the most important steps of implementing an effective A/B test. Note that the following steps must be taken prior to calculating the actual test statistics; they are to ensure that you collect reliable data.

1. Identify the response variable

Defining the one metric that best represents success or failure is vital to implementing an effective A/B test. Generally, your success metric should be as far down your conversion funnel as possible. Think about it - you could have an advertisement with a lower click-through rate (CTR) generating more revenue, so an A/B test to maximize the CTR could have a negative effect on your revenue!

2. Decide what element to test

There are so many factors that may have an influence on your success metric that it becomes difficult to decide which one to test first. Further amplifying this problem is the fact that different factors, such as images and calls-to-action, can have interaction effects that can produce different results based on the version of the other factor (we can save this discussion for a later post on multivariate testing). Test what you believe will have the largest impact on your success metric. Factors that have big impacts, such as layout or images, take less time to test and can help you move forward to test minor changes. Remember to only test one at a time - don’t try to test the call to action and heading simultaneously!

3. Run an A/A test

Unreliable data will render your A/B test worthless. It is imperative that all other factors besides the one you are testing are held constant, so that the effect of your factor is not confounded with other effects. Splitting your traffic and sending them to identical versions of your site or ad should help you verify that you do not have extraneous factors acting upon your response variable.

4. Set up the control group

When you run your A/B test, you need a baseline to compare your results with - to do this you need to keep the original version of the element you are testing. When you run your test there should be 2 variants, the control and the treatment, between which there will only be one changed element.

5. Make sure your traffic is split randomly

In order to effectively attribute an effect to an element, you must avoid sampling error and ensure that your traffic is been split at random between the two variants. Data from the two variants must be collected simultaneously, so that extraneous factors such as time of day or day of the week are nullified.


Congratulations! You have your A/B test set up! Don’t grab a celebratory beer yet though, the fun is just beginning. Now that you have your test set up, begin directing traffic to your variants and collecting data. You will soon have a large enough sample size to gather insights in a relatively short amount of time. If you don’t yet know how to collect or analyze the data, have no fear! I will be writing about collection and analysis soon!

If you have any questions or comments, feel free to email me at julian@mobilexlabs.com or tweet at @julheimer!