- A/B test: With Frankie vs No Frankie
- Shopifyplus Fashion and Apparel store
- A/B test ran for more than 30 days
- 32,000 visitors; 1,300 orders; 2,300 items; $61,000 sales USD
- 12.4% lift in revenue
- 13.8% increase in items ordered
- 14.2% increase in total orders
- Statistically significant outcome
This case study details a study of the benefits of Frankie under strict A/B test conditions on a Shopifyplus store. Shopifyplus stores are representative of the premium performing stores on Shopify. Frankie is found to significantly boost sales with statistically significant results.
What is A/B Testing
A/B testing lets you test and determine which of your 'improve performance' ideas actually produce the results you're after. The standard A/B test involves testing two variations, A and B, to determine which gives you the best results. Put simply, A/B testing splits your traffic into two equal halves, and sends half of your traffic to variation A and the other half to variation B. By comparing the results from each variation, you can conclude which version produces the best results.
Good, better, best. Never let it rest. 'Til your good is better and your better is best.
A/B Testing - the Gold Standard
When evaluating new ideas and their impact on Ecommerce, you need to control for variability in multiple factors that affect outcomes. If you do not do this, you cannot reliably conclude what caused an observed outcome, and therefore understand with confidence, what actually improves your results.
Comparing ideas under the same conditions, at the same time, with the same audience controls for the factors you're not measuring or potentially aware of. This is the essence of A/B testing. It's the gold standard in understanding which changes produce repeatable improvements.
Frankie is a personalized recommendation platform for Ecommerce. Currently available on Shopifyplus and Shopify, Frankie provides multiple recommendation options for each stage of the shopping journey and the most popular store pages - ensuring your shoppers are always served the most relevant recommendations possible. Frankie utilizes advanced Enterprise grade artificial intelligence (A.I.) to generate personalized recommendations that understand both the shoppers intent and their personal tastes. Frankie's A.I. is developed in house and incorporates the latest techniques in visual A.I. optimized for Ecommerce.
A/B Test Design
What we did
- Activated Frankie on a Shopify Store
- variation A: with Frankie
- variation B: no Frankie
- Frankie modules on the following store pages: Home, Collections, Product, Cart and 404
- Test covered 32,000 visitors
- Transactions: 1,300 orders; 2,300 items; $61,000 sales (USD)
A/B Test Design
Sales attributable to recommendations were well distributed across store pages. This result indicates the value in a recommendation platform that provides different recommendation options for each store page and stage of the shopper journey.
Frankie increased the total number of orders by 14.2%, (refer graph below). Indicating that Frankie's personalized recommendations assist shoppers to reach the final stage of the buying process and successfully checkout after finding what they're looking for.
With Frankie led to a 13.8% increase in the number of items ordered during the A/B trial, (refer graph below). This result indicates Frankie's positive impact for Retailers - they end up selling an increased numbers of items to the same number of shoppers who visit their store.
With Frankie led to a 12.4% increase in total sales during the A/B trial, (refer graph below). This result indicates Frankie's positive impact on store revenue.