What's the reason behind the sudden, violent drop off from the front page for a few posts? They seem to be on a fairly linear trajectory and then go down by 30+ positions within a 10 min window.
Post that have more comments than points, for example, get massively deranked rather quickly.
The reason this is done is to remove from the front page hot-button topics (i.e. controversial political news), but often this also includes friendly, niche posts with vibrant conversation, which is very sad to see. These tend to have < 1 upvote/comment ratio than most "Generic tech-adjacent news" like "Apple releases new iPhone".
For example, the Ask HN I posted a few hours ago suffered the same fate: https://hnrankings.info/39788649/: from #13 on the frontpage, to 3rd page in minutes.
Last time I saw it happen on a lovely post about Forth and its various implementations over the decades — not really at risk of flame wars, but alas.
Lately, my gut feel (not based on data analysis) is that there are more people commenting without upvoting than there used to be. I wonder whether that's true.
HN has worked this way for over a decade. It’s a combination of user flags, flamewar detector, and manual moderator classification. The goal is to have a front page that consistently surprises or delights anyone bookish. So the stories that get bumped tend to be ones that do the opposite.
I think you underestimate the number of flags from users and overestimate the amount of direct moderator intervention.
There have been countless times where I've been the flag that tips a comment or a post over into the dead (or for posts, deranked) territory. I flag comments that are rude, abusive, or deliberately filled with misinformation. I flag posts that are likely to start flame wars, that are highly off topic, or that are clearly low-effort blogspam marketing with no valuable content.
If any of that sounds subjective to you, that's because it is. Most of the favoritism and bias you observe can be attributed to users like me who have an idea of what the HN culture should be and subjectively flag things that don't fit.
I imagine also, that plenty of HN readers use “flag” as a mega-downvote, for articles they don’t like or are offended by or whatever. Since there is no downvote for articles (only for comments), the “flag” is the only way for readers to try to bury one.
It’s the other way around, I think. We probably underestimate the amount of direct moderator intervention, both for stories and comments. It’s their full time job.
Ask yourself: what do you think Dan’s team does all day, every day, if not make decisions that influence the site? His comments are all that are publicly visible, so most people assume he just writes comments — an assumption as far from the truth as can be. And most people have no idea that there are even other members of the team, let alone know what they do.
This is by design. HN works best when people are focused on the content, not the site. It’s one of the most influential newspapers in the world, precisely because it focuses on the content. And you’d be kidding yourself if you feel they don’t decide which stories should stay on the front page, every day, a dozen times. Again, it’s their whole job.
Note that them seeing a story and allowing it to stay is also a choice, even though it’s implicit. So them doing nothing because they chose to let a story stay is not the same thing as them doing nothing because the community chose to put it there. Frankly, you wouldn’t want to visit HN if it was community run, because the community is terrible about choosing which stories should be on the front page. This has been proven true for over a decade, and the main reason Dan doesn’t come out and say so is because it’s one of those things that shouldn’t be said out loud. There’s no reason to call attention to how utterly awful the majority of users’ decisions are about what should be on the front page.
Remember, millions of people visit the site now. You wouldn’t want that many people to try to decide. The voting system is an indicator to the mods of potential community interest, but it’s ultimately up to the editors what stays and what goes.
People call this favoritism or censorship or other frankly silly terms, but there’s nothing sinister about it. The way onto the front page is simple: make something that someone bookish would find interesting or surprising. The "surprising" criteria is the trickiest part, because it covers what most people mean by "newsworthy" (e.g. https://news.ycombinator.com/item?id=39778999). But HN has a particular kind of newsworthiness that filters out the majority of stories that the majority of users want to put on the front page.
It’s a tricky subject, and it’s long past time someone should write an essay to explain it all. I’m not quite sure how to do that without angering the community xor mods, but it would be interesting if it were possible. One reason this idea appeals to me is because I’d like to at least thank all of the dozens of people who help run HN behind the scenes; they rarely get any kind of recognition, because they don’t need it, let alone want it. But that makes them worth thanking all the more.
I recognise the huge amounts of effort involved in this and I applaud the moderators for keeping HN an interesting place to be.
That said, I think it's reasonable for us to have visibility on their manual interventions, and this could be easily surfaced via the Hacker News API (https://github.com/HackerNews/API), if the Story JSON included the values of "contro," "bury," and "gag" fields, which are currently opaque to users of the API
One other positive thing to mention about HN: the cost of running HN is non-zero, yet it remains ad-free. I'm grateful for this. So many websites have become unusable due to monetisation dark patterns.
This sort of system just encourages more meta debates about what is and isn't on the front page and why. Most of them are repetitive and boring. This is analogous to the 'receipts for downvotes' perennial - if you think about what it would mean in practice, it's not hard to see the result would be a crappier site.
That doesn't fix the boringness problem and people have meta-discussions about HN on other forums all the time so there's that for those who are into that sort of thing.
What makes you believe Dan has a team? And what do you mean by "team"? My presumption is that there are a number of unpaid volunteers who have been delegated small pieces of the moderation effort, but that there are no employees except Dan. Am I wrong?
Personally, I'd feel better about the future of the site if I thought there was a strong moderation team making numerous decisions behind the scenes. Instead, I think most stays/goes content decisions are made by one vastly overworked Dan and by a bunch of flagging users who frequently have ulterior motives.
I reverse-engineered the HN ranking algorithm a few years ago [1]. To answer your question, posts have a sudden drop due to several types of penalties. Many posts are penalized from the start due to their title or domain. A "controversy" penalty kicks in if an article has more comments than upvotes and at least 40 comments; this penalty is usually the reason why an article catastrophically drops. Other penalties are applied by the moderators. If the voting looks suspicious, a "voting ring" penalty is applied. I found that penalties were very common: 20% of front page articles were penalized and 38% of second page articles. There have been changes to the ranking algorithm since I looked at it, but it is generally the same.
+1, and thanks for taking the time to work on this! I've always wondered how HN ranking works until this moment where every question I had regarding this is being answered in the article.
Also, the article is more than a decade ago, and I'm wondering if there have been any considerable updates to the HN ranking algorithm since then?
I've also tried reverse engineering the algorithm recently to plot a scattergraph of story ranks vs age-adjusted score. I'm looking for outliers in order to work out where HN moderators have intervened.
It turned out to be very difficult as there are so many arbitrary factors, and the HN API doesn't expose those factors.
It's good to see the OP's graphs over time, showing the large jumps which clearly indicate manual intervention. That's a much more effective approach than my own analysis.
My motivation is to surface any patterns in those interventions. Annecdotally I notice that positive news about Elon Musk and his companies has been suppressed more than once.
Obviously, the moderators are responding to things like user flags, but in my opinion there should be more transparency around this, in order to prevent users brigading certain topics, using flags inappropriately, to manipulate the narrative.
