Setting up an A/B Test
What is A/B Testing
A/B Testing, also known as split testing, is a method of optimization that lets you run two (or more) versions of recycling strategies to compare which is more effective at achieving your retention goals.
Metricsco allows you to set up and run A/B tests. The different versions of the recycling strategy that a transaction might be processed through see are known as Variations.
Subscription transactions are randomly bucketed into one version (“A”) or the other (“B”). By measuring how each version performs — you can determine which version is most effective at achieving your retention goals.
Creating an A/B Test
You create A/B Tests using a simple wizard that guides your through the steps. To get to the wizard, just go to the the A/B Tests tab of the user profile and click ‘Add a new Test’.
Tip: You can also access the A/B Tests tab through the sidebar menu:
The Five Key Steps in Effective Tests
There are five essential steps that you’ll use in every test. Let’s walk through the steps, one-by-one.
1. Select the Type of Experiment
Metricsco allows you to set up and run A/B tests, as well as other test types, called “multivariate” tests.
Tip: multivariate tests coming soon.
2. Select Variations
You can select two (or more) Recycling Strategies as your variations for the test.
3. Traffic Allocation: Dividing Subscriptions among “A” and “B”
Now that you’ve selected variations for your experiment, the subscription transactions volume will be randomly allocated into different variations. You can control how much volume goes to each variation. You can even decide how much of your total volume should be participating in the test.
4. Schedule your Test
Tip: Experiment Scheduling is a premium feature available to our Professional and Enterprise customers. It allows you to schedule your tests to start and stop at specific times.
Choose a start and stop time and date. Use the calendar picker to select a date, then the dropdown menus to select a time. You can also select only a start time or stop time, rather than both.
Your experiment will be scheduled to run at the dates and times you specified.
5. Save your Test
Simply give your Test a name and associate it with a profile using the dropdown menu and you are all set.
What’s next ?
Now that you’ve launched your test you are ready to collect the results! You’ll do this on Metricsco’s A/B Tests Results Page accessible through the sidebar menu.
The performance summary shows you the number of unique subscriptions that were processed through this test, how many days the experiment has been running and when it was started, and the variations and results: Each variation’s subscriptions and how each variation performed compared to the baseline.
Each goal module will display conversions, subscriptions, conversion rates, and improvement for each variation.
Variations in green can be considered winners compared to the baseline, while variations in red can be considered losers.
The Charts show you conversion rates over time.