You can probably come up with a proper formula, but you do actually need to put some effort into it to get it right.
Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often.
Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.
Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often.
Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.