Could you explain the calculations that lead to the claim that the result is not significant? From what I can tell, if we assume that clicking the ad is a weighted binary variable, eg. what is modelled as a "proportional distribution" there's a statistically significant difference between the two results. It's even pretty strong at P=0.006 (per https://epitools.ausvet.com.au/ztesttwo), but I might be missing something?
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
The author is using a tool designed for landing pages. The quality of the samples are going to be wildly different and that needs to be taken into account.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
As long as there's not a bias favoring one group in the collection of samples, statistically significant is statistically significant. There's no such statistical thing as a "tool designed for landing pages", but instead tools that compare occurrences in different population.
It’s statistically safest to live on Mars to avoid bear attacks.... the data matters. There’s limited real world application, and that’s what matters most. Semantics and arguing definitions isn’t useful.
You're handwaving about why there's some special definition of "significance" here, and when called on it, it comes down to "I don't feel like this is true".
Valid arguments, better formed, are: A) you're not sure it's representative of a real campaign, B) you're not sure it predicts the end-outcomes we really care about instead of some intermediate measure. Neither of these improve with more n, and so they're not significance related issues--- 4000 clicks doesn't improve either A or B.
I'm not off in the semantic weeds. And I think it's kind of embarrassing you're trying for a witty rejoinder instead of giving -some- kind of cogent argument.
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.