I have great admiration for the amount of time and precision put into this article--as well as Amazon's relentless, cold, uncool devotion to optimization.
These meditations on A/B optimization reminded me of a genetic algorithm-based A/B testing framework mentioned not too long ago: http://ejohn.org/blog/genetic-ab-testing-with-javascript/ (I'd link to Genetify itself, but there's no homepage, just the demo)
An interesting thought: Why hasn't this caught on? Machines are good at exactly the things A/B testing aims at: measuring performance, permuting a test-matrix, and reporting on the results. This is not to say Genetify is the answer, but rather that I rarely hear of conversion optimization from developers.
Part of the reason why developers don't really focus in conversion optimization is that it requires continuous creative effort. It is not something like you do once and then be happy with it. Coming up with different variations which can perform well is hard and doing it consistently is even harder.
Another reason is that proper optimization requires a good amount of traffic, without which your A/B tests would never achieve statistical significance.
That said, optimization is pretty popular with online retailers and big web companies from whom even a 0.1% change in conversions/signups/etc can result into addition of tens of thousands, or even millions of dollars of revenues. For small-websites, the benefit of 0.1% may not be obvious. Hence, not much motivation for investing in optimization.
These meditations on A/B optimization reminded me of a genetic algorithm-based A/B testing framework mentioned not too long ago: http://ejohn.org/blog/genetic-ab-testing-with-javascript/ (I'd link to Genetify itself, but there's no homepage, just the demo)
An interesting thought: Why hasn't this caught on? Machines are good at exactly the things A/B testing aims at: measuring performance, permuting a test-matrix, and reporting on the results. This is not to say Genetify is the answer, but rather that I rarely hear of conversion optimization from developers.