Using A/B testing in mediation

Use A/B tests to make informed decisions about how to set up your ad sources in mediation, which can help you to increase your earnings. 

How A/B tests works

A/B tests let you evaluate two variations of a mediation group and compare the results. Use an existing mediation group and create a variation of it to see which performs better. The test splits your app’s users between your existing mediation group setup and the variation, so that their performance can be measured side by side.

You can change your mediation group set up in Variant B by:

• Adding or removing ad sources to see their impact on your mediation group
• Using a different waterfall order by changing the manual eCPM value of ad sources
• Enabling or disabling optimisation on waterfall ad sources
• Changing the number of times an ad source is called

Click to view an example test

The test runs for 90 days, but you can stop at any time. Based on the results of the test, you can decide to use the original mediation group setup (Variant A) or the variation (Variant B). 

Set up an A/B test

Complete the following steps:

1. Sign in to your AdMob account at https://apps.admob.com.
2. Click Mediation in the sidebar.
3. Next to the mediation group that you want to create an A/B test for, click Set up A/B test. 
4. Enter a name for your test. 
5. Use the slider to set the user allocation. This determines the percentage of app users that are allocated to variant B during the A/B test. You can assign from 1% up to 50% of your app user traffic to variant B. The remainder will remain in variant A (original). 
   Note: Once you set the users' allocation and begin the test, you can’t change it.
6. Change the ad source configuration in variant B. You can:
   • Add or remove ad sources to see their impact on your mediation group
   • Change the order of your waterfall by changing the manual eCPM value
   • Enable or disable optimisation on waterfall ad sources
   • Change the number of times an ad source is called
7. To ensure accurate reporting, we recommend using unique ad unit mappings when using the same ad source in Variant A and B. For example, use a mapping that isn't already used in variant A.
   • Click Edit in the Ad unit mapping column. In most cases, the mapping details are available in your account with the ad network. 
   • Learn more about adding waterfall ad sources and bidding ad sources. 
8. Click Save.
9. Click Start test. 
   Once you start the A/B test, you won’t be able to make changes to the mediation group, including Variant A or Variant B, until the test is stopped or has completed. Any change while the experiment is running may impact the results. This means that you can’t edit ad units, location targeting, etc. 
10. Click Start in the confirmation box, and the test will begin in a few minutes.  

By default, the test will run for 90 days but you can choose to take action and end the test based on the data available. Learn how to analyse your A/B test & take action.

Best practices

• Choose ad units that have enough traffic for the test to determine a result. A test must receive at least 10,000 ad requests before AdMob can determine a result. 
• Change only one setting at a time so you can more accurately attribute any performance difference to a particular change.
• Run the test for at least two weeks. More data helps us more confidently estimate which variant will perform best.
• Consider using different mappings in Variant A and B. This helps keep test data and reporting accurate for your waterfall ad sources.