HN Uptime Visualized: Lots of mini-outages lately?

[chaosmachine]

http://hnstatus.net/

[pg]

I forget exactly why, but attempts to connect get ignored when MzScheme is in the middle of GCing. I'm going to start tossing old stuff back out of memory, which should mitigate the problem, but I've been too busy with YC lately to work on HN.

[pmjordan]

It's weird, when HN goes down, it seems to accept the HTTP connection, but closes it again once the request is sent. I'm not sure on the exact timing, but I'm definitely never getting "connection refused" or an actual HTTP error code. No other site behaves like that.

[derefr → pmjordan]

I'm guessing there's a proxy or load-balancer involved—the backend goes down, but the proxy stays up, so when you send your request it gets forwarded first, and then fails.

[wmf → derefr]

A working proxy would send a 500 error. The closed connections are a telltale sign of an amateur Web server; probably written in Arc in this case.

[Zak → wmf]

An outdated version of the HN source code, including its web server is distributed along with Arc.

[chaosmachine]

Almost seems to be doing a hockey stick, if you look at the last 6 months.

[Groxx]

I've been noticing it too, was wondering if it was just me ( downforeveryoneorjustme.com said it was just me, though that could be because it came up in the meantime).

[swombat]

I think I heard somewhere that HN is auto-rebooted every half hour or so.

[pg → swombat]

No, I manually restart it when it gets slow.

[fossguy]

what tool do you use to generate these stats?

[caller9 → fossguy]

You could combine Nagios, Lilac, and PNP4Nagios to get some pretty good monitoring with graphs(PNP4Nagios), notification(Nagios), and the ability to configure new servers from templates(Lilac). Not for the feint of heart, but not outright tedious. With some slight hackery you can even graph and alert on windows drive freespace.

[ronnier → fossguy]

The graphs look like MRTG. Probably running a request every few minutes and looking at the response, if there is one, and plotting it.

http://oss.oetiker.ch/mrtg/

[rogermugs]

if you manage to hack HN you're going to lose some karma... but lets be honest... you're also gonna get some mad props.