On: App Engine down

[davis_m]

I think this is larger than just GAE.

http://internettrafficreport.com/namerica.htm

It seems like large portions of the internet are down.

[EwanToo → davis_m]

Internet Traffic Report, while a nice concept, is unfortunately very misleading.

Their sample size is extremely small, and most of those are permanently down.

Have a look through their list of north american routers and find one of them where packet loss has gotten worse as their main overall graph for packet loss would suggest - I've just been through them all and couldn't find one.

[calinet6 → EwanToo]

Their baseline values are very misleading.

But their relative metrics can still be useful.

For example, it's very inaccurate to say that 51% of the internet is down.

But it's precise to say that packet loss among the working nodes has increased about 30% in the last 24 hours, and sharply.

[EwanToo → calinet6]

But that's likely 2 or 3 nodes, not a meaningful sample

[obituary_latte → EwanToo]

Agreed. I was surprised to see only one[1] of the "down" routers with any packet loss over the last 24 hours.

That said, it's still interesting how the overall traffic trends so sharply downwards. I wonder if they have more data than they are showing in the graphs.

[1]http://internettrafficreport.com/history/190.htm

[jre → EwanToo]

It seems like the individual router graphs most recent update is 10/26 00:00 whereas the overall graph has been updated at 10/26 08:30.

[untog → davis_m]

Yes, Tumblr is experiencing difficulties and AFAIK they don't run anything on App Engine.

[neanderdog → davis_m]

May not be anything or may be..

I noticed a couple of days ago that some of our dns entries were mysteriously removed from level 3 servers which out of old habit are used for resolution (some of the ip's go back to uu-net/worldcom/mci)

Now the interesting bit is they were for private subnet ip's. They're working fine everywhere else.

Today the last of their dns servers removed the entries so I had users go to google (8.8.8.8) and all's well with our apps.

Level 3's entries for our external stuff is there, just the private subnet stuff is removed.

If others do this too and resolve with level 3....

edit: just found this: http://tracker.outages.org/reports/view/59

[X-Istence → davis_m]

And here we have another website that does something similar:

http://www.internetpulse.net

And according to them all routes are up and running just fine,not only that bit ping times aren't elevated.

[j_baker → davis_m]

My money is on Hurricane Sandy: http://www.nhc.noaa.gov/refresh/graphics_at3+shtml/084338.sh...

[sparkinson → davis_m]

Now those are some interesting graphs.

[FiloSottile → davis_m]

The internet is burning :O Well, seriously what could be a root cause that affects so many nodes?

[bsaul → davis_m]

only 51% of north american internet is working ?? Am i reading this correctly ?

[EwanToo → bsaul]

You're reading it correctly, it's just telling you the wrong thing.

51% of a very small self-selecting set of machines are down, mostly small businesses or home users which have obviously been shutdown or renamed.

If internettrafficreport automatically removed all devices which returned no results for 7 days, they'd be a slightly more useful resource.

[seldo → bsaul]

As Ewantoo mentioned, ITR is pretty unreliable. We have found it works as a relative measure -- when their packet loss rises, we do indeed see consumer traffic fall, though usually not by the amount their graphs would indicate.

[aidos → davis_m]

Add Dropbox to that list.

[cfontes → davis_m]

Great find, I did not know this site.