A/B Test Overview

  • The goal of experiment is to measure the impact of placing Frankie's recommendations on Collections under strict A/B test conditions.
  • Collections are a grouping of products on a Shopifyplus store
  • A/B test: 50% of Collections traffic viewed Frankie's recommendations, 50% of Collections traffic did not view the recommendations
  • A/B test: 27 days; 50,000 visitors; 2,630 orders; 4,152 items $126,000 sales (USD)
  • Shopifyplus Fashion and Apparel store
  • Customer base: 46% Nth America, 32% Europe, 22% Other
  • Store turnover: ∼$4 million USD p.a.

 

Results

  • 30% lift in average visitor value
  • 27% lift in total sales ($)
  • 19% lift in conversion rate
  • 9% lift in AOV (average order value)
  • results statistically significant at 99% confidence level

 

 

What is A/B Testing

A/B testing lets you test and determine which of your 'improve performance' ideas actually produce the results you're after. The standard A/B test involves testing two variations, A and B, to determine which gives you the best results. Put simply, A/B testing splits your traffic into two equal halves, and sends half of your traffic to variation A and the other half to variation B. By comparing the results from each variation, you can conclude which version produces the best results.

Test_A_B (1)

 

"Every aspect of marketing is entirely useless unless it produces conversions"

Jeremy Smith


 

 

Image result for gold bullion

A/B Testing - the Gold Standard

When evaluating new ideas and their impact on Ecommerce, you need to control for variability in multiple factors that affect outcomes. If you do not do this, you cannot reliably conclude what caused an observed outcome, and therefore understand with confidence, what actually improves your results.

Comparing ideas under the same conditions, at the same time, with the same audience controls for the factors you're not measuring or potentially aware of. This is the essence of A/B testing. It's the gold standard in understanding which changes produce repeatable improvements.

 


 

frankie_full_logo

Frankie is a personalized recommendation platform for Ecommerce. Currently available on Shopifyplus and Shopify, Frankie provides personalized recommendation options for each stage of the shopping journey and the most popular store pages - ensuring your shoppers are always served the most relevant recommendations possible. Frankie utilizes advanced Enterprise grade artificial intelligence (A.I.) to generate personalized recommendations that understand both the shoppers intent and their personal tastes. Frankie's A.I. is developed by Hiplee.ai and incorporates the latest techniques in visual A.I. optimized for Ecommerce.


 


A/B Test Design - what we did

  • Activated Frankie on a Shopifyplus Store,variation:
    • A: Frankie visible on Collections
    • B: Frankie not visible on Collections
  • Fashion & Apparel store
  • Test over: 27 days; 50,000 visitors
  • Transactions: $126,000 sales (USD); 2,630 orders; 4,152 items

A-B testing collections

A/B Test Design


 

Results

Total - Mobile & Desktop

All key metrics were positively impacted. Average visitor value was observed to be 30% better. Sales were 27% higher and conversion rates were 19% improved.

 

Mobile results

Mobile accounted for 87% of total traffic. Visitor value on mobile improved by 30%. The increase in AOV on mobile outperformed Desktop (11% increase vs 4% increase). 

 

Desktop results

Desktop accounted for 13% of total traffic. Frankie on desktop outperformed Frankie on mobile in conversion rates 25% higher which flows through to improvements in Purchasers and Orders.

 

Confidence Level

Confidence levels in A/B tests indicate whether you can be confident that the results did not occur due to chance. The results in this A/B test are statistically significant with a confidence level greater than 99%. This means that the results can be relied upon as being real and repeatable. Statistically speaking, the null hypothesis that the two distributions are the same was rejected in all tests, with p = 0.000 (3 decimal places).

 


 

Untitled

Impact on Revenue

Revenue ($) increased by 27%

Total sales was 27% higher with Frankie's recommendations on Collections when compared with No Frankie recommendations on Collections.  This A/B test clearly indicates that Frankie's personalized recommendations significantly improve the return on Collections.


 

Summary: Frankie Improves key Ecommerce metrics

A/B testing is the gold standard in Ecommerce to measure the impact of change.

A/B testing was used to determine the impact of traffic going to Collections on a Shopifyplus store.

The store specialises in Fashion and Apparel and has a turn over ∼$4 million USD pa.

The test split traffic going to Collections so that 50% saw recommendations and 50% did not see recommendations

Strong benefits were observed With Frankie vs No Frankie:

  • Visitor value increased by 30%
  • Total sales increased by 27%
  • Conversion rate increased by 19%
  • Average order value increased by 9%

 

30% increase in average visitor value 'With Frankie' vs 'No Frankie' on Collections

 


Boost your store's results with Frankie

Get Frankie on Shopify app store

- free 15-day trial -

Give your shoppers a personalized service today.

or, book your own Frankie demo