Hacker News new | past | comments | ask | show | jobs | submit login

In my experience the most helpful and generalizable resources have been resources on “experimental design” in biology, and textbooks on linear regression in the social sciences. (Why these fields is actually an interesting question but I don’t feel like getting into it.)

A/B tests are just a narrow special case of these.




A/B testing misses the point of statistical design of experiments, which is that your variables can interact. One factor at a time experiments are pretty much guaranteed to stick you in a local maximum


There's that, and also all of the common pitfalls highlighted in this famous paper: https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

I do believe "doing A/B testing" is probably better than "not doing A/B testing", more often than not, but I think non-statisticians are usually way too comfortable with their knowledge (or lack thereof). And I have very little faith in the vast majority of A/B experiments run by people who don't know much about stats.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: