I can see how the ugly ad could get more clicks. It looks so different from everything else that you're naturally curious to see what it leads to.
I will also add that the "professionally designed" ad is kind of busy. It has so many things going on (some text in all caps, different colors, some quoted, illustration doesn't add much value, arrows that don't mean anything) that it doesn't really catch your attention or call you to action. The "ugly" ad, however, gets to the point.
I don't know, I've seen that style on Facebook ads a lot and I'm kind of numb by now to the style of "oh, look, this was hastily scrawled together, isn't that so edgy and real?"
Personally I'm completely numb to the "every word in a different font and some random ones on banners" style that was so inescapable in the 2010s, that it ended up on those horrible t-shirts for, like, Deranged Girlfriends Who Are Proud Gun Nuts
there is a bot that poorly creates phrases for targeted subcultures and digitally affixes them onto T-shirt designs and makes targeted ads for them. per usual with T-shirt sites, when people purchase, the design is created on a real t-shirt at that point in time.
I am humble, so humble! My humility is even written on my shirt! Also, there are some badass mock angel wings drawn on my back, in case you had been forgotten of my magnanimity.
Agreed. Along with the LinkedIn messages with my name or my companies name on some piece of paper some random person is holding at whatever company bought a list with me on it.
Ignoring the other points about bounce rate, brand identity, and statistical significance for a second, I think an aspect that's missed a lot is that good design accomplishes a goal. The hand-drawn ad, by this definition, stands out more than the "well-designed" one just by virtue of being visually distinct. Asking a firm to help you design an ad capaign should involve iteration, testing, and refinement as different ideas are tested on your audience. If you ask a firm to take some words and make them look good you're missing out on most of the design process.
I'm not sure the author really understands the math (from his comments) but he's right that it's significant, where significant means "I am at least 95% confident that the observed improvement is caused by this change". The test is fine (assuming he didn't just check the results every day until he found one that would make for a fun blog post).
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
> The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
I think the objection is about the phrasing that can be interpreted as "I will only be wrong in at most 5% of significant results", while what we mean is "if there is no effect, we will only get so large differences in at most 5% of cases".
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
Well, with low probability events you can go a bit further with your back of the envelop calculations, because that means they're more or less Poisson distributed. The average is somewhere around 16 so that gives a standard deviation of 4.
So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.
Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.
So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.
I assume that one takes the uneven sampling into account? That should boost the significance a bit, though perhaps a bit more than I'd originally assumed. It remains a back of the envelop calculation after all.
Their argument consists of linking an optimizely screenshot.
12 vs 24 clicks is not significant, it could’ve gone either way. Also given this minuscule sample, it’s easy to conduct p-hacking to get the desired outcome
Could you explain the calculations that lead to the claim that the result is not significant? From what I can tell, if we assume that clicking the ad is a weighted binary variable, eg. what is modelled as a "proportional distribution" there's a statistically significant difference between the two results. It's even pretty strong at P=0.006 (per https://epitools.ausvet.com.au/ztesttwo), but I might be missing something?
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
The author is using a tool designed for landing pages. The quality of the samples are going to be wildly different and that needs to be taken into account.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
As long as there's not a bias favoring one group in the collection of samples, statistically significant is statistically significant. There's no such statistical thing as a "tool designed for landing pages", but instead tools that compare occurrences in different population.
It’s statistically safest to live on Mars to avoid bear attacks.... the data matters. There’s limited real world application, and that’s what matters most. Semantics and arguing definitions isn’t useful.
You're handwaving about why there's some special definition of "significance" here, and when called on it, it comes down to "I don't feel like this is true".
Valid arguments, better formed, are: A) you're not sure it's representative of a real campaign, B) you're not sure it predicts the end-outcomes we really care about instead of some intermediate measure. Neither of these improve with more n, and so they're not significance related issues--- 4000 clicks doesn't improve either A or B.
I'm not off in the semantic weeds. And I think it's kind of embarrassing you're trying for a witty rejoinder instead of giving -some- kind of cogent argument.
Sibling comments are right on the technical side, but to help build intuition: Statistical significance isn't solely about 12 vs 24, you have to take into account the total population as well (5563 vs 4403). In particular, the smaller population had more than double the results - this is a hint that you have to do the math and can't just say "it's not significant", because it really well could be.
You can have a sample size of 5 and have it be statistically significant. Or maybe you are using some non-statistical measure of statistical significance?
The fact that it has a significant p-value is interesting, but the lack of information about how the author decided when to stop is suggestive of p-hacking (i.e. we don't know how many screenshots were taken, but we understand that the author posted only the most favourable one)
Also when running campaigns, I’ve seen an ad suddenly get way more clicks than usual in a short timespan. Depending on how clicks are counted, just one user who gets click-happy could skew the result.
But in this case I suspect people were just curious why the ad looked like that and clicked to find out. Those may not be the people likely to convert.
In the real world, we often have to work on a "preponderance of evidence" standard to actually get things done.
Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.
I think there can be prior that a professional looking ad can generate more clicks. Your argument shows a lack of statistical understanding - conditional on this data, the Bayesian approach would be to update the prior (whether A is better, or they’re equally as good) with the data collected. With such a small dataset, you might end up with a belief that there’s a 60% probability that B is better than A, but that’s not significant enough to conclude that B is in fact better than A, as you still have a lot of uncertainty.
With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.
I addressed in my second sentence that I disagree with that prior. I understand the statistics perfectly well.
And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.
P=.95 still is an arbitrary threshold, even if it's a commonly used one.
> An ad that took me 15 seconds to create had a 2.5x better clickthrough rate than one done by a paid designer. If this were an actual campaign it would mean 2.5x more sales leads or user signups at a lower acquisition cost, on top of shipping faster with less overhead.
Are you sure about the latter? How do you know that the people that click the ugly ad weren’t just curious about what it was because of how it looked rather than what it was about?
I find the paid-for design to be uninteresting and unattractive. If you were to ask me what I'd find when I clicked through, I would say that it's likely I'd be asked to sign up for a low-quality mailing list to receive a bland white paper.
The scribble version is, at least, novel. Still probably wouldn't click through, but at least it would grab my attention for a moment.
What was the bounce rate, avg visit time and conversion rate (email signup) between people who came from the non-designed vs. designed ad though?
A higher clickthrough rate isn't necessarily better if more of the people who came from the non-designed ad didn't do much or anything on the page and mostly clicked out of curiosity.
Making people click is not the problem. People follow garbage click-bait links all the time. Design is not just about "making things look nice". It's just as much about creating identity. What does it matter if you bring users to your site if it's the wrong kind that does not convert to real customers? Seems like there's a particular type of business leaders that thinks marketing is just a numbers game.
Yeah exactly, another thing he could've done is put a girl in a bikini or a gross pic of toenails, a la Outbrain. That'll get people to click more than a "normal" ad but only the hackiest of hacks thinks that's the ultimate goal.
Exactly, which is why the most garbage click bait ads you will find on the internet are for websites that make their money on advertising rather than by selling products. Because those sites are just trying to get eye balls to increase impressions and thus make more money.
They don't have to worry about actually selling you a product.
Funny story about that effect gone wrong. I was helping a university with digital ad buying in collaboration with their ad agency. I pitched the idea of using online display split tests to measure the effectiveness of messages, then use that data to help inform what messages would get printed for outdoor billboards. In online tests, an ad variation with a poorly-drawn smiley face did exceptionally well relative to the more traditional designs. The “ugly ad” that won the online tests didn’t translate well to billboards. Data is harder to obtain in that sphere, but the ad got plenty of feedback and not much of it was good.
I got to see the lesson I had to learn with our clients everyday on my commute for 3 months. I won’t soon forget it.
There's a lot of problems with the conclusion of the results on <$30 of total spend especially without actually selling a product when the author extrapolates to a sold product (vs actually giving a free one).
But, if the goal is the download the yes this works. In large part because of why 'one weird trick' took off. It's simply different. Different causes our brain to pay more attention. The professional ad, just like all ads today are relatively ignorable because we've "seen it all" and are mostly annoyed by them.
This tends to be why trends are cyclical. There are only so many things you can do with a specific medium. So, it's relatively expected that once a medium hits a certain tenure, it'll recycle on itself.
Read "Oglivy on Advertising" (1983). Oglivy is one of the all-time greats in advertising. He's noted for making the original Volkswagen Beetle cool. He points out the problems of ad overdesign.
You don't get to talk to the prefrontal cortex without first getting past the hind-brain.
It's the guardian at the gates.
People are just more willing to give their attention to things that are novel. Which is of course based on context.
It's also one of the reasons why I've taken to using drawings within my own blog posts [1]. I've seen time on site that is MUCH higher than industry benchmarks.
Probably worse than most textbook corporate ads: the 'free guide' and 'read more' are too small to catch the eye, and the title is boring and generic even so even the sort of person that reads ads and is in the target audience might pass. Unsurprising the novelty ad performs better, but a better conventional one might too.
Help me understand this a bit -- why would a subreddit buy advertising? Is there a feedback loop or revenue share with mods?
Or is this purely a ponzi thing where you buy bitcoin worth 100 bucks, spend 100 bucks on advertising and end up with the same bitcoin now worth 1000 bucks.
In 1991, Bob Kaufman switched from slick ad campaigns featuring local (central Connecticut) radio celebrities to advertise his two-location furniture store chain to shooting commercials himself with whatever grotty camcorder he could get his hands on, and appearing in them himself with his face for radio and voice for print.
The self-starring commercials were an enormous success, and became a meme among Connecticut viewers through the 90s, but they also got people into Bob's locations. Today, Bob's Discount Furniture has opened stores up and down the Eastern seaboard and even on the west coast, He's retired from starring as himself in his own commercials, being replaced with a puppet likeness and becoming a sort of founder-turned-mascot like Col. Sanders. But the chain's enormous success wouldn't have happened without those initial, cheaply made, cheesy ads.
I like the premise of the post but you have to be transparent about your downstream funnel otherwise how can anyone be sure of the quality of the clicks?
Sure Option B had around half the CPC by the looks of it but what happened next? That’s the most interesting thing of all and it’s missing from your post.
The design of this site is unconventional by today's design standards, but it has pricing clearly presented and is 100x more memorable, just like the OP's example. (As a side note, sites like this make me miss the old days of the Internet when everything was unique.)
People are indifferent to car dealerships (They're all typically the same. Yes, luxo-brands will have free cookies and nicer waiting areas...), just like with insurance-- they are "boring" verticals so having something so out there works great. When you look at insurance companies, which companies have memorable marketing? Companies like Liberty Mutual with their vanilla "Save $200/year" commercials, or the Geico Gecko, "We are Farmers....", All State's Mayhem, etc.. The cheesier the better, since auto insurance is a boring, fungible product to consumers. Compare to another boring vertical like Men's Razors. They're essentially all the same. How are any of the marketing campaigns from P&G/Unilever actually differentiated enough to be memorable (ignoring the duopoly considerations)?
Having unfortunately worked in marketing, everyone seems to shoot for the middle-- and they end up being bland and forgettable. Chances are, you're not going to have a super memorable marketing campaign by doing something slightly different than everyone else. The companies I've worked with where their marketing is intentionally cheesy have always had better reception than the vanilla "professional" brands.
This reminds me somewhat of my time as a journalist in the late 1980s. Laserprinters were just becoming common and companies started sending out nicely layed out press releases, using nice typefaces.
It became apparent that journalists tended to ignore them because they didn't look like new - quickly produced and hof the press. So many PR agencies went back to using double-spaced Courier.
> An ad that took me 15 seconds to create had a 2.5x better clickthrough rate than one done by a paid designer. If this were an actual campaign it would mean 2.5x more sales leads or user signups at a lower acquisition cost, on top of shipping faster with less overhead.
It's irrelevant whether one ad gets a better result than another ad, what matters is does any ad drive traffic at a rate cheap enough to build a sustainable business. The "ugly" ad cost $11 to drive a webpage visit. Is this enough to result in a profitable conversion? I'm guessing almost certainly not.
So what does it matter if the "ugly" ad performs better? All that means is you go bankrupt slower. Is there room to improve the performance of the ugly ad so it wanders into the realm of profitability? I'm guessing there's far less room than for the professional ad and, probably, there doesn't exist a design that will drive profitability with the current offering and more focus should be made in customer development to discover and communicate a clear value prop rather than mess around with ad copy.
I'd wonder about click through vs. conversion and if there's a difference there.
Might just be because of novelty factor, but also may be tricking less sophisticated users because it looks different and those users may be less likely to convert.
It'd be interesting to see conversion results too.
I've seen variations of this article before. It's the novelty. "That's different... What would someone pay real money to advertise with something that looks so cheap and crappy? Click."
In the direct response world, a rule of thumb is that if an ad doesn’t look like an ad, it would perform well. This is the reason all ads on Fox News blend in look like they are articles. Ads on reddit start with TIL, amateur looking ads taken with iPhone with a talking head often beats professional ones on fb. This is also the reason drawing works better here. But, This is at the expense of eroding the brand image. over using this tactic overtime will win you the battle and lost the war.
I have an ad blocker in my mind that filters out any commercially looking or sounding message. not only it works on websites but in the streets for billboards, during voice phone calls and face to face conversations.
the ugly ad that violates most of the ad traditions is more likely to be followed by me. (especially if it lacks the call to action part. I hate calls to action. especially when it's me who's going to pay)
I've read somewhere that porn ads and political ads perform better when they don't look professionally made. If I were a publisher, I would be very selective about the quality of the ads being displayed. I would also be in the minority since even CNN has ugly ads at the bottom of every article ("You won't believe what she looks like today!")
I'm reminded of enterprise-created videos that explain their business/products to generic "happy" music. Something different would immediately get my attention too.
As other comments have pointed out, I think it's a matter of standing out, and in this case lowering the standard of quality works, but soon the novelty will be something else.
CTR is high because version B is hard to read. So more people have to visit the website to read more. It is bad because we pay for each click in CPC model. Failing to weed out uninterested people leads to increased cost of aquisition.
So measuring CTR does not tell anything about if version B is better. It may be worse.
To compare we'd need to check CPA with A/B testing.
The first thing I read on the hand drawn version was "free guide". In the "professional" version, my mind glossed over those words completely. I don't know why, other than maybe the pretty pictures at the bottom pulled my eye away.
It's old news that in copywriting, copy with some misspellings typically performs better. While you may lose some who see the ad as less trustful, people respond to ads that make them feel like they've discovered something secret, new or underground.
Digital ads used to be noticeable (because flash and gifs). Google and fb ads are bland and don't stick out, and you don't even have an option. I wonder how advertisers missed this bit, novelty used to be the desired goal in old media advertising.
I think the biggest red flag for me is the large disparity in sample sizes between the two arms of the test. Unless the author purposely set up a 55-45 test for some reason, it feels like a test that was not set up correctly.
If this article gets a lot of love in the right circles, I expect we'll see hand-drawn ads for a few years--until they're not novel or rare anymore, and then the pretty ones will come back.
Web advertising will always be an enigma to me. I would assume that the kind of person who clicks internet ads is not an average person so it must be difficult to create a good ad.
let's compare apps with orange.
Option A is awful, no wonder if got so few clicks. It is too hard to busy and more difficult to read. B is simple and easy to read.
I would have assumed B was done by the professional, because it is clear and concise while A may have been drawn by a professional artist, it wasn't done by a marketer.
It seems to me, that this experiment is evidence that an easy to read ad is better than a noisy one.
I don't know as I don't have a marketing background. Maybe it would. My point is that the article assumes that a design that costs $148 is automatically "professional" (and "good").
The professional "design" is too busy and noisy, distracting me from title. The call to action is tiny. The "ugly" design increases the lettering to a larger % of the total graphic. A design with these changes, and graphics that are a better abstraction for the message, would likely perform much better. (tl;dr pretty design != good design)
I will also add that the "professionally designed" ad is kind of busy. It has so many things going on (some text in all caps, different colors, some quoted, illustration doesn't add much value, arrows that don't mean anything) that it doesn't really catch your attention or call you to action. The "ugly" ad, however, gets to the point.