I think that many people that post about AB tests fail to mention that one's expectations of AB testing have to align with the types of results one can get from AB testing.
AB testing isn't meant to used through the entire conversion funnel. That is, you can't measure an optimization on your homepage by sales. Sales is an indication of your entire conversion funnel. Each page of every site should have a single goal and you should AB test to that goal. In your case I would say that you're AB test was successful; however, you didn't go far enough. Once you were getting increased traffic to your signup page, it's time to AB test the signup page and improve that conversion.
The statistics behind AB testing are often lost by the hype. The tests used to calculated statistical significance require a random sampling and isolated variables. The moment you try to attribute success or failure of your AB test beyond the primary measurable goal, you're introducing outside variables that will skew your results.
All of that being said, I would argue that true AB testing is not cost effective for most startups. It takes a lot of timer and energy to test every part of your conversion funnel.
In your case I would say that you're AB test was successful; however, you didn't go far enough. Once you were getting increased traffic to your signup page, it's time to AB test the signup page and improve that conversion.
This was my thought at first, but thinking about it now, their AB test actually caused a decrease in conversion on the signup page. And that's the point they're making: they drove traffic to the signup page more effectively, but that traffic was no more interested in actually signing up. You can do a lot of things to massage the stats if you're only optimizing for a single step. You can make your homepage promise people a golden pony if they'll only click on the link to your pricing page, but they're not going to get there and suddenly decide they don't care about the pony and want to give you money instead.
> "That is, you can't measure an optimization on your homepage by sales."
Sure you can. If you are holding all other things constant and randomizing people to A or B on the homepage, there's nothing to stop you from using "sales" as your outcome variable in an AB test. It's probably what you should be measuring, in fact.
AB testing isn't meant to used through the entire conversion funnel.
Wrong.
Not using A/B testing through the entire conversion funnel is a shortcut to get results faster. You sometimes get the wrong result, but you get there faster. However if you have the traffic, you ALWAYS want to measure through the entire conversion funnel.
That's my point, is that in most cases the amount of traffic isn't there to support such an AB test.
It is also very difficult to isolate all other variables in the conversion funnel. For the case of an online software startup with an inside sales team like we have at Central Desktop, it's nearly impossible to say that every signup gets the same sales attention and is pitched in the same way.
Of course every case is different, but I'd argue that most startups don't have the traffic or the ability to isolate ALL potential variables in the conversion funnel.
If you don't have traffic, you don't have traffic.
You don't have to isolate other variables. Just make them randomized, and they become just another factor you don't know that is handled by the statistics.
AB testing isn't meant to used through the entire conversion funnel. That is, you can't measure an optimization on your homepage by sales. Sales is an indication of your entire conversion funnel. Each page of every site should have a single goal and you should AB test to that goal. In your case I would say that you're AB test was successful; however, you didn't go far enough. Once you were getting increased traffic to your signup page, it's time to AB test the signup page and improve that conversion.
The statistics behind AB testing are often lost by the hype. The tests used to calculated statistical significance require a random sampling and isolated variables. The moment you try to attribute success or failure of your AB test beyond the primary measurable goal, you're introducing outside variables that will skew your results.
All of that being said, I would argue that true AB testing is not cost effective for most startups. It takes a lot of timer and energy to test every part of your conversion funnel.