Menu
Bell Statistics (and Allon) reviewed all statistical formulas and fixed what needed fixing. They supplied the right formulas and wrote (with the team) their A/B testing cookbook to align the methodology across the whole company. They also trained all of the analysts and other people who touch A/B testing in the company. Regarding reliability, Bell introduced Simply to SRM (sample ratio mismatch) and A/A tests to help monitor problems in the data. For the sample size, Bell helped implement sequential testing that lets the team monitor results on a daily basis and made the tests shorter by 65%!
The team found important bugs and technical issues using SRM.
Some of the tests were shortened by up to 65%!
”One of the problems with testing is the ‘curse of knowledge’ – you know what you know, and you don’t know what you don’t know. Allon {and Bell} really helped by unlocking the testing black box, helping us to find issues with our AB Tests flow, resolve them and ultimately gain much more out of our testing.”
– Tali Fulman,
BA Guild Head, Simply
(formerly Joytunes)
Roundforest didn’t have the required infrastructure & methodology to perform A/B testing. This led to uncertainty about which changes or investments in the product and marketing would yield the best results, resulting in missed opportunities, ineffective decision-making, and potentially costly mistakes.
With the help of Allon and the team, Roundforest started laying the groundwork to be able to make informed and evidence-based decisions using our data.
In the recent quarter, Roundforest was able to run A/B tests regularly (some positively impacted company KPIs and were rolled out to all users). This included: user-friendly dashboards that allow the team to review results and understand if there was a significant increase/decrease/no change; ability to run multiple tests simultaneously; ability to run tests and analyze different buckets within the population.
“We are extremely proud of the progress we made with the help of Allon and the team that allowed us to form our data analysis team, and provide them with the tools needed to effectively evaluate and optimize our strategies and decisions. One of the company’s strengths is the amount of data that we hold, but for a long time, we have been struggling to use this data in a way that allows us to test and learn fast. We made a huge leap in this domain and feel we’re set up for success.”
– Amit Drenger,
Director of Product, Roundforest