Thanks Maxime!
I was looking for a new way to discuss art. There are a couple of nice subreddits to discover new art but I found the discussions that went with them a bit disappointing.
So with Peerdiem, I'd like people to take their time (a day) to observe, reflect and share their thoughts about the artwork.
Congrats with the Birdy.so launch! Hopefully the free trials convert in a week :)
> How do you make sure the same user who was on A variant yesterday doesn't get the B variant today?
You can't. You make the point that the stats generated by Birdy are not 100% accurate, but they're the best we've got given the limitations of the Twitter API :D
I make the point that the data is good enough, and statistically significant enough (especially if you let the test run enough) to help you find the better profile version.
I agree that this subset of visitors that may come twice or more can skew the data.
Birdy's stats are not to be taken as the absolute truth, but as solid clues as to which profile version it is alternating between is providing the best results.
I'm looking forward to improving Birdy as Twitter adds more capabilities to their API!
I suppose that the issue is that a single user could see both versions, which could skew the data.
I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version.
But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.
> There is no “testing” going on here, unfortunately. You’re just taking turns placing users in the treatment or control group, arbitrarily, depending on when they visit the page. How can you measure any treatment effect when everyone is part of either group?
This is not a perfect solution, but I have found that it is good enough to be able to identify clear winners (if there is actually a winning version). A lot of followers won't visit your profile multiple times anyway. They visit it, and they either follow it or they don't - and they will of course be influenced by the currently displayed version :). They won't just come back to your profile over and over again for no reason, but if they do, they will also convert better on the version they prefer. So a version might nudge the user into following you while another one might not. I would disagree that this is not testing. I think it is for the majority of the profile clicks you receive.
> I guess you could try something like switchback testing, but I’m not convinced visits to the average Twitter profile will yield enough samples.
I don't think that would be possible with the current Twitter API capabilities anyway.
> I think it’s a well-executed idea, but I don’t think it’s fair to sell results under the guise of statistical validity when they don’t appear to have that. (although it’s just Twitter profiles, not eg medical treatment, so no real harm done)
While the results might not be perfectly accurate, I think they are accurate enough to provide value, especially if you let the test run long enough to get a big sample size. I personally use Birdy (obviously :D) and I have noticed much better conversion, which is why I'm confident.
I'm looking forward to seeing new capabilities appear on the Twitter API to always make the process more accurate though.
Thank you! Yes, custom built with Tailwind. First I wanted to use a landing page template, but then I decided it would be easier to just build it myself hehe
Oh! That's a good idea. I didn't think about that.
I think at some point I'll have a free plan for everyone (with limited features). I just need to finish implementing all the profile components and then enable only some of them for the free plan.
