"But we can still be fairly confident that the image was the main factor since this test had a much higher change in click through than what we typically see when we just test text vs text."
How can you be fairly confident? They broke the cardinal rule of the scientific method.
Well, it's worth noting that while we're interested in the general science of puppies and conversions, our end goal with this test was not to determine the exact value of the puppy picture, but just generally finding the best way to get people to click our call to action button...
Except you have 2 variables - puppy/no puppy; and copy A/copy B, and therefore 4 combinations. You only tested two combinations, so you have no way of knowing if the puppy or the alternate copy was the reason for the increase.
"But we can still be fairly confident that the image was the main factor since this test had a much higher change in click through than what we typically see when we just test text vs text."
How can you be fairly confident? They broke the cardinal rule of the scientific method.