A/B Tests: Do’s And Don’ts
Performing A/B testing with Metricsco for your Decline Recycling Strategies is super-simple, but it’s worth keeping some practical things in mind.
- When doing A/B testing for your decline recycling strategies, never wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong. It’s possible that version B was actually worse but you just happened to have better subscriptions while testing it. Always split real time traffic between two versions.
- Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant (that is, whether you should take the results seriously). It prevents you from reading too much into the results if you have only tested a few subscriptions or transactions for each variation.
- Don’t let your gut feeling overrule test results. The winners in A/B tests may be surprising or unintuitive. Your goal with the test is a better recovery rate so don’t reject the results because of your arbitrary judgment.
- Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer. Giving up too late isn’t good either, because poorly performing variations could cost you subscription retention and sales.
- Route repeat payment declines for the same subscription to the same variation. Luckily, Metricsco handle this for you. We have a mechanism for remembering which variation a subscription was tested through.
- o multiple A/B tests. An A/B test can only three outcomes: no result, a negative result or a positive result. The key to optimizing recovery rates is to do multiple A/B tests, so that all positive results add up to a huge boost to your recovery rate, customer retention, and revenue.