First of all, great work. It looks like you boosted your conversion rate from 0.19% to 0.43%. Which is a 125% improvement, or with confidence intervals, 55% - 179% improvement.
However, before everybody goes out and puts puppies on their homepages, they need to realize that there are a bunch of things being tested.
Image vs. no image: Is it possible that having any image at all improves the conversion. You should test with other pictures: perhaps some animals, people, nature, and see if the puppy is what makes it work.
Call to action: The 'puppy' version also features a more succinct call to action in "Sign up now" rather than "Start your 30 days free trial." Perhaps this also contributes some of the difference.
Button size: The button size in the 'puppy' version is smaller. Perhaps this has some effect as well.
Length of text: The 'puppy' version has more description of what is involved in the free trial. It says "Pick a plan & sign up in 60 seconds. Upgrade, downgrade cancel at any time." vs. the no puppy version that says "Start you 30 days free trial."
Vertical vs. Horizontal Layout: The 'puppy' version has a vertical layout of the text and button, where they are stacked on top of each other rather than left or right.
So there are at least five different changes made between these two designs. Clearly the second design wins on conversions, but it's not entirely clear to me why it wins.
Not sure how you can get much of value from an A/B test with multiple changes, especially if one is claiming that only 1 of those changes is what is responsible for all the improvement.
If nothing else, they have a hypothesis to test in the next experiment.
Even when possible to isolate and remove ancillary changes to improve split test purity, it's often not beneficial. If there's a significant number of changes, achieving statistical significance across the full matrix probably isn't even possible.
But that's ok, because limiting changes to a single test queue restricts your ability to move fast and try lots of stuff, which is beneficial. So when you test, try cheap multivariate methods (there's a bunch!) to quickly understand how interactions between multiple changes affect results.
You can iterate on the other tests over time. Many A/B tests start with a larger change that may include multiple variables but with that baseline increase, now they can go ahead and test Dog vs Cat vs Human as the image. Or can test a variety of different text sizes and lengths. This seems like a fantastic start, with plenty of room for further iteration and improvement.
Emotions do sell things. I like to think of myself as a rational person, but it certainly works on me.
For example, when I was living in Japan around 1996, we needed a kerosene heater for our apartment. We went to the big department store, and they had a lot of different ones. Each of them had a sticker affixed describing the various features and advantages, but they were mind-numbingly similar. But one of the stickers had a photo of Mai-chan, the cheerful and engaging child star of a cooking show that we had enjoyed on Japanese TV a couple years earlier.
My wife pointed it out to me, and we ended up getting that heater. Not because it was any better, but because we liked Mai-chan, and her photo was on it. Of course Mai-chan has absolutely nothing to do with heaters, and we knew it. But we didn't care.
We always called it the "Mai-chan" heater, and even though we left it behind when we returned to the states, I guarantee if I mention the Mai-chan heater to my wife today, nearly 20 years later, she'll smile and know exactly what I'm talking about.
The puppy screenshot in the original post has a similar effect on me. A part of me wants to reach out and pat the cute dog on the head. But I can't. But I can click the button, which is what he obviously wants me to do. Sure it's not at all rational. But it motivates people. I suggest keeping the dog and adding it all your advertising. Make it your mascot. You can't go very wrong with a puppy!
I suggest keeping the dog and adding it all your advertising. Make it your mascot.
That's exactly what I imagined when I saw the title; a startup testing/comparing the conversion rates of a dog mascot/logo. It goes hand-in-hand with establishing a memorable brand. I'd be really curious to see how the conversion rates are affected by a complete website/brand overhaul containing a dog mascot.
Not a dog, but my startup (tabuleapp.com) added a koala as our mascot, and our conversion rates more than doubled. We help college students organize homework, so the fun koala face helped differentiate us from most homework planners that just use a book as their logo.
If you ever plan opening in Poland, add a panda instead.
There's a known few years old meme now, featuring a panda and a number "3". It's a rebus that should be read as "panda 3", and understood as "pan da 3", which is a kind of impolite way of saying "sir, please give me a 3 (D grade)".
The article title gives the puppy image credit but I think the additional text had as much if not more of an impact on the conversion lift than the image.
"Start your 30 days free trial" text
vs
"*Sign up now* 30 days free rial. Pick a plan & sign up in 60 seconds. Upgrade, downgrade, cancel at any time." with a puppy image.
The second version is so much better because it tells the viewer what to expect... the 60 second signup, flexibility in changing plans, no long-term commitments. The dog image does help since he is looking at the call-to-action, which also draws the viewer's eyes there too.
It is a great post and just goes to show how testing can really help boost conversion. It's often a lot cheaper to grow your business by improving conversion rates than by increasing traffic.
"But we can still be fairly confident that the image was the main factor since this test had a much higher change in click through than what we typically see when we just test text vs text."
How can you be fairly confident? They broke the cardinal rule of the scientific method.
Well, it's worth noting that while we're interested in the general science of puppies and conversions, our end goal with this test was not to determine the exact value of the puppy picture, but just generally finding the best way to get people to click our call to action button...
Except you have 2 variables - puppy/no puppy; and copy A/copy B, and therefore 4 combinations. You only tested two combinations, so you have no way of knowing if the puppy or the alternate copy was the reason for the increase.
Why a few months? You can rack up visits by just doing blog posts. I can take most blogs and turn it into a 20K visitors/week blog in no time. Not a great deal of traffic, but enough to start making good tests.
November 2012. The headline was something along the lines of "If X does not fix Y, our Alpaca leave you a voicemail." It improved conversion by about 20% (image vs no image). Same headline.
At one point in my career I had two job offers on the table - my wife was helping me decide, and when she saw that one of the companies had a dog on their site, she said I had to work for them. In the end, I did accept that job offer for my own reasons - but it's interesting how the emotional response is to the cuteness factor.
Coincidently I now work at a company with currently 4 dogs in the office.
"Coincidently I now work at a company with currently 4 dogs in the office"
Jesus, and I complain about how hard it is to concentrate in a noisy open plan office. A bunch of non-human animals roaming about might just tip me over the edge.
I was concerned about this at first - but overall it's really great. There have obviously been a few altercations between them, but those are rare. One of the dogs is owned by a member of my team so he sits right by me all day - occasionally he'll come by just to be pet, and when I am stuck on something work related we play fetch for a few minutes to clear my head. Our daily standups are pretty awesome too since we generally have a dog or two there with us, for example, yesterdays standup: http://i.imgur.com/3DABSti.jpg
At least where I worked, the dog just kept us company and wasn't obnoxious at all. It was very calming to just play with her when you're stuck somewhere, most of the time petting the dog yielded beautiful technical solutions. She should have been on the payroll (well, she did get snacks, but was worth way more than that IMHO).
It's interesting you mention that. I remember just a month ago, seeing pictures of dogs in the offices at Palantir (on their website), and wondered if seeing that became the deciding factor for prospective employees.
We all know that puppies can sell CMSs. That's a given. What they should A/B test is the effectiveness between a black lab and golden retriever.
Joking aside, the text version has no picture at all (could be just that screenshot), one thing I haven't forgotten from my marketing 101 is that pictures grab attention 10X better than text.
(to elaborate for the downvote-debbies: there is an entire industry designed around getting users to use your product based on ideas such as this, and this link is one place to learn more about that)
In my experience "image beats text" is a really common result of an AB test. For an encore, I'd suggest testing an image of a (smiling) person in that same space.
Worth testing, but in my experience stock photos of people make websites look cheesy and fake. Untrustworthy, even. Like one of those fake sites to gather extra domain traffic.
This reminds me of Windows Bob but I don't think it worked very well for Microsoft putting that dog there. Initially people (women and children) were impressed with how cute it was but five minutes later they were trying to kill that annoying yellow dog.
But was it really the puppy? There was a lot more text explaining what was involved (okay, not all that much, because there wasn't a lot on either, but significantly more) on the puppy version.
It could just have been that you described what signing up entailed. Plus you included 'cancel at any time', which might help some people decide.
More people clicked through to the sign-up page, but did the actual number of sign-ups increase too? After all, in the end, that's what matters. If people click your link just because they want to see more pictures of puppies, it might not be such a great improvement.
People used to game Google Adsense by putting their own images next to the adsense blocks in order to get more clicks. MFA sites did this so much, Google banned the practice of using images next to their ads. This is the same result.
I'd like to see tests between that dog, and a person, or a group of people. Having images generally will help with conversion rates, but what kind of images would produce a better result? And for what kind of services?
It says "the number of clicks on the signup button" was measured. That does not seem like the metric they are interested in which would be "number of actual signups".
i wonder if it simply has to do with the fact that there was AN image there ? Maybe verify by putting something not as cute - say an ORC or something like that? and something neutral like a fruit?
However, before everybody goes out and puts puppies on their homepages, they need to realize that there are a bunch of things being tested.
Image vs. no image: Is it possible that having any image at all improves the conversion. You should test with other pictures: perhaps some animals, people, nature, and see if the puppy is what makes it work.
Call to action: The 'puppy' version also features a more succinct call to action in "Sign up now" rather than "Start your 30 days free trial." Perhaps this also contributes some of the difference.
Button size: The button size in the 'puppy' version is smaller. Perhaps this has some effect as well.
Length of text: The 'puppy' version has more description of what is involved in the free trial. It says "Pick a plan & sign up in 60 seconds. Upgrade, downgrade cancel at any time." vs. the no puppy version that says "Start you 30 days free trial."
Vertical vs. Horizontal Layout: The 'puppy' version has a vertical layout of the text and button, where they are stacked on top of each other rather than left or right.
So there are at least five different changes made between these two designs. Clearly the second design wins on conversions, but it's not entirely clear to me why it wins.