A/B Test Overview
- The goal of experiment is to measure the impact of placing Frankie's recommendations on Collections under strict A/B test conditions.
- Collections are a grouping of products on a Shopifyplus store
- A/B test: 50% of Collections traffic viewed Frankie's recommendations, 50% of Collections traffic did not view the recommendations
- A/B test: 27 days; 50,000 visitors; 2,630 orders; 4,152 items $126,000 sales (USD)
- Shopifyplus Fashion and Apparel store
- Customer base: 46% Nth America, 32% Europe, 22% Other
- Store turnover: ∼$4 million USD p.a.
- 30% lift in average visitor value
- 27% lift in total sales ($)
- 19% lift in conversion rate
- 9% lift in AOV (average order value)
- results statistically significant at 99% confidence level
"Every aspect of marketing is entirely useless unless it produces conversions"
Frankie is a personalized recommendation platform for Ecommerce. Currently available on Shopifyplus and Shopify, Frankie provides personalized recommendation options for each stage of the shopping journey and the most popular store pages - ensuring your shoppers are always served the most relevant recommendations possible. Frankie utilizes advanced Enterprise grade artificial intelligence (A.I.) to generate personalized recommendations that understand both the shoppers intent and their personal tastes. Frankie's A.I. is developed by Hiplee.ai and incorporates the latest techniques in visual A.I. optimized for Ecommerce.
A/B Test Design - what we did
- Activated Frankie on a Shopifyplus Store,variation:
- A: Frankie visible on Collections
- B: Frankie not visible on Collections
- Fashion & Apparel store
- Test over: 27 days; 50,000 visitors
- Transactions: $126,000 sales (USD); 2,630 orders; 4,152 items
A/B Test Design
Total - Mobile & Desktop
All key metrics were positively impacted. Average visitor value was observed to be 30% better. Sales were 27% higher and conversion rates were 19% improved.
Mobile accounted for 87% of total traffic. Visitor value on mobile improved by 30%. The increase in AOV on mobile outperformed Desktop (11% increase vs 4% increase).
Desktop accounted for 13% of total traffic. Frankie on desktop outperformed Frankie on mobile in conversion rates 25% higher which flows through to improvements in Purchasers and Orders.
Confidence levels in A/B tests indicate whether you can be confident that the results did not occur due to chance. The results in this A/B test are statistically significant with a confidence level greater than 99%. This means that the results can be relied upon as being real and repeatable. Statistically speaking, the null hypothesis that the two distributions are the same was rejected in all tests, with p = 0.000 (3 decimal places).
Impact on Revenue
Revenue ($) increased by 27%
Total sales was 27% higher with Frankie's recommendations on Collections when compared with No Frankie recommendations on Collections. This A/B test clearly indicates that Frankie's personalized recommendations significantly improve the return on Collections.
Summary: Frankie Improves key Ecommerce metrics
A/B testing is the gold standard in Ecommerce to measure the impact of change.
A/B testing was used to determine the impact of traffic going to Collections on a Shopifyplus store.
The store specialises in Fashion and Apparel and has a turn over ∼$4 million USD pa.
The test split traffic going to Collections so that 50% saw recommendations and 50% did not see recommendations
Strong benefits were observed With Frankie vs No Frankie:
- Visitor value increased by 30%
- Total sales increased by 27%
- Conversion rate increased by 19%
- Average order value increased by 9%
30% increase in average visitor value 'With Frankie' vs 'No Frankie' on Collections