I have great admiration for the amount of time and precision put into this article--as well as Amazon's relentless, cold, uncool devotion to optimization.
These meditations on A/B optimization reminded me of a genetic algorithm-based A/B testing framework mentioned not too long ago: http://ejohn.org/blog/genetic-ab-testing-with-javascript/ (I'd link to Genetify itself, but there's no homepage, just the demo)
An interesting thought: Why hasn't this caught on? Machines are good at exactly the things A/B testing aims at: measuring performance, permuting a test-matrix, and reporting on the results. This is not to say Genetify is the answer, but rather that I rarely hear of conversion optimization from developers.
Part of the reason why developers don't really focus in conversion optimization is that it requires continuous creative effort. It is not something like you do once and then be happy with it. Coming up with different variations which can perform well is hard and doing it consistently is even harder.
Another reason is that proper optimization requires a good amount of traffic, without which your A/B tests would never achieve statistical significance.
That said, optimization is pretty popular with online retailers and big web companies from whom even a 0.1% change in conversions/signups/etc can result into addition of tens of thousands, or even millions of dollars of revenues. For small-websites, the benefit of 0.1% may not be obvious. Hence, not much motivation for investing in optimization.
Amazon supposedly has an amazing experiments system. Someone can launch an experimental feature, and if it succeeds, it gradually becomes the default.
Having your users vote is definitely the best way to make decisions. Lately, when my cofounder and I disagree, I say "let's build both versions and ask our users which one they like best." (in most cases, the differences are trivial enough to make this feasible).
I expect this is why the quantity is more up-front now. I bet it is pretty unusual for anybody to buy multiple copies of a given book at once, so defaulting to one and giving the opportunity to adjust later made sense. Now that they sell many things that are probably bought in multiple quantities (say, socks), customers are probably having to adjust the quantities more often.
All very well, but sometimes I wish they would spend a bit more time on their wishlists - not much has changed since the beginning and some more effort in this area is overdue by now.
These meditations on A/B optimization reminded me of a genetic algorithm-based A/B testing framework mentioned not too long ago: http://ejohn.org/blog/genetic-ab-testing-with-javascript/ (I'd link to Genetify itself, but there's no homepage, just the demo)
An interesting thought: Why hasn't this caught on? Machines are good at exactly the things A/B testing aims at: measuring performance, permuting a test-matrix, and reporting on the results. This is not to say Genetify is the answer, but rather that I rarely hear of conversion optimization from developers.