Thompson sampling does not need to assume stability. You can inject time features into the model if you want to model seasonality (or, more accurately, ignorances of seasonality) and you can also have a hidden random-walk variable.
Yes, if you assume stability and things vary, you will not have good results. That seems like any statistics.
However, with an A/B test you don't need to change the math or eliminate the stability assumption from it. You just need to choose a good test duration.
As I pointed out to pauldraper in the other thread, when you start fixing bandits by only changing the split every season, suddenly bandits start to look a lot like A/B testing.
Actually, Chris, I think you misunderstand my comment.
Thompson sampling (and Bayesian Bandits in general) can be applied with a model for estimating conversion that is more complex than P(conversion|A). It can include parameters for time of day, day of week and even be non-parametric.
If you do this, you the standard Thompson sampling framework with nochanges* whatsoever will still kill losers quickly (if the loss is big enough that seasonality cannot save it) and will also wait until it has enough data seasonally to make more refined decisions. This is very different from simply waiting for an even season point to make decisions.
You do need more data to understand a more complex world, but having an optimal inference engine to help is better than having a severely sub-optimal engine.
I understand that. The blog post I linked to describes doing exactly that.
But the point I'm making is different. This is a lot of stuff to get right and most people aren't that sophisticated. Getting A/B tests right is a lot easier, mainly because they are significantly more robust to model error.
Yes, if you assume stability and things vary, you will not have good results. That seems like any statistics.