On: The price of solar power just fell 50% in 16 month...

[bemmu]

Maybe people like fresh things?

Here's an A/B test I recently ran on some notification messages. There were two messages, only difference is the word "new" in there.

    A: "You received a message from the admins about your Facebook page."
    B: "You received a new message from the admins about your Facebook page."

A clickthrough 15.64%, B clickthrough 15.95%.

About 1000 samples per group. Not sure if noise.

[pjc50 → bemmu]

https://www.mccallum-layton.co.uk/tools/statistic-calculator...

  Sample Size: 1000
  Observed Proportion: 15.64%
  Confidence Level: 95%
  Confidence Interval: ±2.25
  Range for the true population proportion: 13.39% to 17.89%

It's noise.

[bemmu → pjc50]

Hmm, I tried playing around with this tool.

When I enter 250000 in sample size I get ranges that don't overlap. Does that mean I would need that many samples to answer this?

[cscheid → bemmu]

Yes. The bigger your sample size, the bigger your ability to see small effects. A test with 1000 elements can only tell apart big effects; a test with 250000 can tell apart much smaller ones. This is the idea of https://en.wikipedia.org/wiki/Statistical_power

[nothrabannosir → bemmu]

.3% on 1k? That's three people.. Yes, noise.

[nkrisc → nothrabannosir]

That's a great way to get a gut check on the validity of the result. Put it in real terms.

[msellout → nothrabannosir]

I'm often annoyed when reading reports that use percentages instead of the actual ratios. For example, "2 out of 3" is much better than "66.7%".