Simply, a global leader in online learning

Simply is a pioneering company behind the popular apps Simply Piano and Simply Guitar, with millions of users in over 180 countries. Simply makes it easy for families and individuals to pursue creative hobbies that spark joy in every home. Known for its creative and data-driven approach to online learning, Simply’s high testing velocity and ambition necessitated relying on trustworthy data and running complex tests.

Client

Simply

Challenge

Ensure robustness and accuracy in testing methodologies, particularly in multi-variant tests and CAC calculations

Solution

Review and correction of statistical formulas, development of an A/B testing cookbook, and introduction of SRM and sequential testing

Result

Reduced test durations by up to 65%, uncovered critical bugs, and improved testing efficiency and accuracy

Ensuring Robustness and Accuracy

Despite their innovative spirit and “impact velocity” approach, Simply faced challenges in ensuring the robustness and accuracy of their testing methodologies. They particularly struggled with executing multi-variant tests and calculating confidence intervals for Customer Acquisition Cost (CAC).

Comprehensive Support to Enhance Testing Processes

Review and Correction of Statistical Formulas

Ensuring all formulas were accurate and reliable.

A/B Testing Cookbook

Developing a standardized guide for testing methodologies across the company.

Training Sessions

Equipping analysts and relevant staff with the skills needed for effective testing.

Introduction of SRM and A/A Tests

Monitoring and identifying data problems to enhance reliability.

Sequential Testing

Enabling daily monitoring of results and significantly reducing test durations.

Improved A/B Testing Efficiency and Accuracy

Bug Identification

SRM tests uncovered critical bugs and technical issues.

Reduced Tests Durations

Sequential testing reduced test durations by up to 65%, allowing for quicker decision-making and iteration.

Improved Efficiency and Accuracy

Standardized testing methodologies and correct statistical formulas provided more valuable insights.

"One of the problems with testing is the ‘curse of knowledge’ – you know what you know, and you don’t know what you don’t know. Allon and the team really helped by unlocking the testing black box, helping us to find issues with our AB Tests flow, resolve them and ultimately gain much more out of our testing."

Tali Fulman, BA Guild Head, Simply