A/B Test Overview
- The goal of experiment is to measure the impact of placing Frankie's recommendations on Collections under strict A/B test conditions.
- Collections are a grouping of products on a Shopifyplus store
- A/B test: 50% of Collections traffic viewed Frankie's recommendations, 50% of Collections traffic did not view the recommendations
- A/B test: 27 days; 50,000 visitors; 2,630 orders; 4,152 items $126,000 sales (USD)
- Shopifyplus Fashion and Apparel store
- Customer base: 46% Nth America, 32% Europe, 22% Other
- Store turnover: ∼$4 million USD p.a.
- 30% lift in average visitor value
- 27% lift in total sales ($)
- 19% lift in conversion rate
- 9% lift in AOV (average order value)
- results statistically significant at 99% confidence level
What is A/B Testing
A/B testing lets you test and determine which of your 'improve performance' ideas actually produce the results you're after. The standard A/B test involves testing two variations, A and B, to determine which gives you the best results. Put simply, A/B testing splits your traffic into two equal halves, and sends half of your traffic to variation A and the other half to variation B. By comparing the results from each variation, you can conclude which version produces the best results.
"Every aspect of marketing is entirely useless unless it produces conversions"
A/B Testing - the Gold Standard
When evaluating new ideas and their impact on Ecommerce, you need to control for variability in multiple factors that affect outcomes. If you do not do this, you cannot reliably conclude what caused an observed outcome, and therefore understand with confidence, what actually improves your results.
Comparing ideas under the same conditions, at the same time, with the same audience controls for the factors you're not measuring or potentially aware of. This is the essence of A/B testing. It's the gold standard in understanding which changes produce repeatable improvements.
Frankie is a personalized recommendation platform for Ecommerce. Currently available on Shopifyplus and Shopify, Frankie provides personalized recommendation options for each stage of the shopping journey and the most popular store pages - ensuring your shoppers are always served the most relevant recommendations possible. Frankie utilizes advanced Enterprise grade artificial intelligence (A.I.) to generate personalized recommendations that understand both the shoppers intent and their personal tastes. Frankie's A.I. is developed by Hiplee.ai and incorporates the latest techniques in visual A.I. optimized for Ecommerce.
A/B Test Design - what we did
- Activated Frankie on a Shopifyplus Store,variation:
- A: Frankie visible on Collections
- B: Frankie not visible on Collections
- Fashion & Apparel store
- Test over: 27 days; 50,000 visitors
- Transactions: $126,000 sales (USD); 2,630 orders; 4,152 items
A/B Test Design
Total - Mobile & Desktop
All key metrics were positively impacted. Average visitor value was observed to be 30% better. Sales were 27% higher and conversion rates were 19% improved.
Mobile accounted for 87% of total traffic. Visitor value on mobile improved by 30%. The increase in AOV on mobile outperformed Desktop (11% increase vs 4% increase).
Desktop accounted for 13% of total traffic. Frankie on desktop outperformed Frankie on mobile in conversion rates 25% higher which flows through to improvements in Purchasers and Orders.
Confidence levels in A/B tests indicate whether you can be confident that the results did not occur due to chance. The results in this A/B test are statistically significant with a confidence level greater than 99%. This means that the results can be relied upon as being real and repeatable. Statistically speaking, the null hypothesis that the two distributions are the same was rejected in all tests, with p = 0.000 (3 decimal places).