Summary

• A/B test: With Frankie vs No Frankie
• Shopifyplus Fashion and Apparel store
• Customer base: 46% Nth America, 32% Europe, 22% Other
• A/B test ran for more than 30 days
• 32,000 visitors; 1,300 orders; 2,300 items; $61,000 sales USD

• 12.4% lift in revenue
• 13.8% increase in items ordered
• 14.2% increase in total orders
• Statistically significant outcome

This case study details a study of the benefits of Frankie under strict A/B test conditions on a Shopifyplus store. Shopifyplus stores are representative of the premium performing stores on Shopify. Frankie is found to significantly boost sales with statistically significant results.

What is A/B Testing

A/B testing lets you test and determine which of your 'improve performance' ideas actually produce the results you're after. The standard A/B test involves testing two variations, A and B, to determine which gives you the best results. Put simply, A/B testing splits your traffic into two equal halves, and sends half of your traffic to variation A and the other half to variation B. By comparing the results from each variation, you can conclude which version produces the best results.

Good, better, best. Never let it rest. 'Til your good is better and your better is best.

St Jerome 

A/B Testing - the Gold Standard

When evaluating new ideas and their impact on Ecommerce, you need to control for variability in multiple factors that affect outcomes. If you do not do this, you cannot reliably conclude what caused an observed outcome, and therefore understand with confidence, what actually improves your results.

Comparing ideas under the same conditions, at the same time, with the same audience controls for the factors you're not measuring or potentially aware of. This is the essence of A/B testing. It's the gold standard in understanding which changes produce repeatable improvements.

Frankie is a personalized recommendation platform for Ecommerce. Currently available on Shopifyplus and Shopify, Frankie provides personalized recommendation options for each stage of the shopping journey and the most popular store pages - ensuring your shoppers are always served the most relevant recommendations possible. Frankie utilizes advanced Enterprise grade artificial intelligence (A.I.) to generate personalized recommendations that understand both the shoppers intent and their personal tastes. Frankie's A.I. is developed by Hiplee.ai and incorporates the latest techniques in visual A.I. optimized for Ecommerce.

A/B Test Design

What we did

• Activated Frankie on a Shopify Store
  • variation A: with Frankie
  • variation B: no Frankie
• Frankie modules on the following store pages: Home, Collections, Product, Cart and 404
• Test covered 32,000 visitors
• Transactions: 1,300 orders; 2,300 items; $61,000 sales (USD)

A/B Test Design

Results

Sales Distribution

Sales attributable to recommendations were well distributed across store pages. This result indicates the value in a recommendation platform that provides different recommendation options for each store page and stage of the shopper journey.

Total Orders

Frankie increased the total number of orders by 14.2%, (refer graph below). Indicating that Frankie's personalized recommendations assist shoppers to reach the final stage of the buying process and successfully checkout after finding what they're looking for.

Total Items

With Frankie led to a 13.8% increase in the number of items ordered during the A/B trial, (refer graph below). This result indicates Frankie's positive impact for Retailers - they end up selling an increased numbers of items to the same number of shoppers who visit their store.

Total Sales

With Frankie led to a 12.4% increase in total sales during the A/B trial, (refer graph below). This result indicates Frankie's positive impact on store revenue.

Impact on Revenue

12.4% Increase in Sales

With equal traffic, total sales 'With Frankie' was 12.4% higher than with the 'No Frankie' option.  This result under strict A/B test conditions, clearly indicates that Frankie's A.I. personalized recommendations add a significant increase to store revenue.

Return on Investment (ROI)

As part of the case study, we examined the return on investment (ROI). The 12.4% lift in sales from Frankie produces a return on investment (ROI) of 3,620% - using a 40% cost of goods sold (COGS). This is a phenomenal return - even Warren Buffet doesn't achieve that! Based on the same COGS, we've calculated the expected ROI from Frankie generating a 10% lift in sales is 2,900% ROI. A 5% lift in sales from Frankie produces a 1,400% ROI. These returns clearly indicate a strong business case for implementing Frankie.

Conclusion

A/B testing is the gold standard in Ecommerce to measure the impact of change.

With Frankie vs No Frankie was A/B tested on a Shopifyplus store selling Fashion and Apparel

The test covered 32,000 visitors; 1,300 orders; 2,300 items; $61,000 sales USD

Shopifyplus stores are representative of the premium performing stores on Shopify.

A range of recommendation modules from the Frankie platform were activated on the store to conduct the A/B test.

Revenue attributed to Frankie's A.I. personalized recommendations was distributed throughout the store from different recommendation modules operating on different store pages.

Strong benefits were observed With Frankie vs No Frankie:

• 12.4% lift in revenue
• 13.8% increase in items ordered
• 14.2% increase in total orders

12.4% lift in revenue 'with Frankie' vs the 'No Frankie' option with equal traffic.

Boost your store's results with Frankie

- free 15-day trial -

Give your shoppers a personalized service today.