How to set up and evaluate an AB test

So you need to run an A/B test and you need to figure out how many users you need in order to have valid results. What does “valid” results even mean? How do you decide what is the proper test to use and what does a p-value mean?

In the following jupyter notebook, which you can access here, I go into how to run an AB Test from start to finish. I’ve highlighted which cells you need/might want to edit so that you can just step through the notebook. Think of it as a step by step guide or a template that you can plug in your own data.

Feedback

I wanted to share some of the kind words that my colleagues used to describe the notebook.

• “I used a bunch of your code from the AB testing notebook for proportions in a HP testing notebook. It worked pretty smoothly!”
• “This is a fantastic notebook. It is thorough and comprehensive. You’ve left no stone unturned.”
• “Isn’t it great how fast I can asnwer all these questions?! (love this notebook)”
• “Starting to see some positive results… Thanks again for all your help getting the analysis notebooks together”