Most of the a/b testing we do is on our email notifications such as invitations, and friend requests. We built some in-house instrumentation to measure how many emails are sent (stored in the database), how many emails are opened (measured using an embedded image pixel and stored in the database), and how many users come back to the site (measured using various Google Analytics campaigns).
The link above has been a useful tool for analyzing the results of our tests. It automatically performs all the calculations needed to determine statistical significance. I’ve been looking for a tool like this for a while because I’ve unfortunately forgotten most of the statistics I’ve learned in high school. I hope my old stats teacher isn’t reading this!
