One of the first things I learned about stats is that the accuracy of your result is prorportional to the size of our sample size. The more people you test on, the more confident you can be that your result is equal to the true value that you would get if you were able to test the entire population.
A practical implication of this is that when you run tests on websites, there’s going to be a tradeoff between how many segements you can divide your population into and how long it’ll take for you to collect statistically significant results. The link above is to a tool that calculates how large your segments should be which will help you decide how many segements you should have and how long you should run your test for.
I’m not a stats major, so feel free to critique my explanation. =)
1 year ago