"SBG: ERDF is trying to find out
the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. "
"An incident is ongoing impacting our network. We are all on the problem. Sorry for the inconvenience."
"SBG: 1 gen restarted."
"RBX: all optical links 100G from RBX to TH2, GSW, LDN, BRU, FRA, AMS are down."
It started with all our SBG servers going down simultaneously. Approximately 1h later all our RBX servers went down as well including the OVH status page and all other OVH web applications. Either their SBG and RBX data centers are somehow connected or those are indeed two independent incidents.
SBG was a power failure, followed by a generator failure. Not sure if they set up their network infrastructure in a way that this could spread but I find it hard to imagine that the outage in SBG triggered RBX going completely black. Unless of course they store their configuration files in SBG.
I moved away from OVH after I paid 3 months advance (~$300) for a server which burned down after 1 1/2 months. They did not issue any refunds (data, blood, sweat and tears were lost that day). I have been an OVH customer for 12 years.
Today, I'm glad to have moved away all my production environments as well.
I'm still at OVH (support is reasonable and prices cheap) but would never trust one provider with all my infrastructure. In the end it always turns out that there is a single point of failure and if it's just the billing department. Using two providers protects you from that and if you chose some with good peering and free traffic, keeping both in sync is relatively easy.
That's just the problem with services as AWS, traffic is too costly to have production live with another provider as well.
Mirror your data onto another provider continuously (log shipping/rsync), Switch DNS.
Ansible works for this stuff as it allows you to can the task of “quick get me a production environment up on Linode!”
If you can afford some downtime you don’t need a hot standby just roll out everything into new provider and you’re done.
I’ve done this on very large scale environments and small ones and it’s achievable for even small organisations. The killer is avoid anything you can’t run on bare metal servers.
I also do this approach. Get a decent, from your infrastructure independent DNS provider and take care of your Ansible scripts. This way in emergency you do a one liner, have a new production server running and change DNS settings.
In addition I'd suggest getting a second domain in a TLD operated by another company in another country than the primary TLD is, and teaching your customers/users that both are valid. This protects you from three things:
2) legal issues, when one of your domains gets seized or the provider gets pressured into cancelling, just as has happened with Pirate bay, SciHub and friends, gambling or sites with user generated content that may be illegal or frowned upon in some countries or for the latter to the Nazi site Daily Stormer (although I'm glad for it being down, it's a perfect example what can happen in a very short time frame)
3) (edit, after suggestion below) the entire TLD going down because the TLD DNS provider has issues, which also happens from time to time.
My experience with OVH VPSes is that they are cheap and reasonably reliable. But when one starts failing support never helped me to get it fixed.
Just fire up another one, install everything on there, switch off the old one and forget about it. This means never paying for more than one month in advance.
Just an good advice: Never ever trust a single point/server without backups. Do backups, do snapshots (check if and how you can do that), store them somewhere else. Try to do automated reinstall with e.g. Ansible so that you can move easily. It's an investment at the beginning and during running the server but if the stuff is critical you should always do that.
It happens to all of them. Whenever I read comments like yours I wonder where you moved your servers to, and if you'll move them again when that goes down too.
Damn, every emergency power supply I have encountered (the big ones with fuel and hundreds of batteries) always fail to start when they have to... Why is that ?
I've seen places that /did/ test their backup power - but they got failures anyway, because of faults the test didn't reveal.
For example you switch off the data centre circuit breakers and everything fails over to generators just fine. Test successful, right?
Then when there's a real outage you have problems because the operations team's computers have all gone off, so they can't migrate load to a different data centre. It didn't happen in testing, because they aren't in the data centre so their kit isn't connected to the breakers you turned off.
Or it turns out the wireless APs aren't on UPSes. Or it turns out there's a switch in a closet somewhere that isn't on a UPS. Or they tested for a single loss of power, but when the mains power toggles on and off every 30 seconds the UPS batteries get run down. Or they need to top up the generator and they discover you can't get fuel delivered at 9pm on a Friday. Or the generator doesn't recharge the UPS, but you have to turn off the generators to refuel them. Or a guy had a standalone UPS for his desktop, but his monitor wasn't connected as the UPS only came with IEC C13 power cables and his monitor needed IEC C5...
One place I worked at found this during the big storm in the UK UPS worked fine and all Telecoms Golds Machines stayed - but they had forgotten to put the Modems on the UPS :-)
While at $bigco we halted testing of generation equipment because it was sending DCs offline more often than it kept them up. Lawyers were involved, things got ugly
I'm completely unfamiliar with electrical generators/power generation, so take this question in the spirit of ignorance:
Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face?
I don't know if it's feasible to dump all that power to ground or whatever, but that way you could test the generator under full load at will and identify issues without impacting the (operationally) live datacenter itself.
This equates a bit in my head with the 'verifying the backup' bit you might get in software, whereas actually using the generator to power the live operational datacenter equipment would be more like 'restore from backup'.
> Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face?
That would exercise transfer switches in addition to generators. Transfer switches are always energized, except for the time an equivalent of a big red mechanical switch is flipped into "OFF" position. When it is in the off position, neither main or generator are going to provide power to the customer. The biggest power consumer is actually a cooling system. If a HVAC system stops functioning in a typical data center, the temperature would quickly rise to the level of Arizona desert, destroying metric tons of equipment. Transfer switches have certain properties where after a flip they may not go back to the correct state on power restore ( say 1% ). Normally it is not a big deal because you really need to have lose power under full load often to get bit by it and if you are losing power that much you should probably address it with the utility company. However, if you are doing your full test once a week, in one year you would introduce 52 power failures.
Say your transfer switch is stuck in a wrong position. Now you need to shutdown all heat generating equipment to when you drop power to fix/replace the failed component you don't melt your data center... Congratulations, your data center now has to go offline.
You can if you have to. But then you're really only doing a fancy simulation.
I accompanied my dad (power engineer) to a water purification plant where they were testing new equipment for the back up generator. There their weekly tests involved moving the entire plant to the diesel generator and running it of back up power for a couple of hours (once you start a big generator you have to let it run or it wont last long).
Potential problems for your generator that a resistor bank wont capture include, power factor (phase shift from a motor or switching supply), harmonics (from switching power), startup transients (from every power supplies' capacitors).
All these things can trip the generator, or worse, burn it out.
So if you can't test with the real load, supersize it!
P.S. Every test is a simulation of reality. At Fukushima the diesel generators flooded. Lesson - the unknown reason that'll knock out your grid can knock out your backup
P.P.S If you can, gently turning the load back on is very beneficial. Don't flip the master switch that controls all your load - flip a part of your load, wait a while for the system to stabilize, and flip part of it back.
P.S. Every test is a simulation of reality. At Fukushima the diesel generators flooded. Lesson - the unknown reason that'll knock out your grid can knock out your backup
well the lesson there was more like, that it's stupid to put your diesel generators deep in the ground when they should sustain burst sea level raises. (well I think it's never a good idea to do that, I've seen special places to put them even deep inside germany, just because some panicful people that might think that it still could overflow with ground water, etc)
The generators were placed low to ensure they would not be disabled by a major earthquake, as shaking intensity increases with height above ground.
The basement was, according to plan, protected by a seawall.
The height of the seawall failed to take into account the fact that on a subduction tectonic plate, as pressure builds, the land-side plate rises, and as the earthquake relieving that pressure strikes, the land falls -- by as much as several meters.
The seawall's height failed to account for this.
That among other elements, but it proved sufficient to kick off the Fukushima disaster, given other aspects.
In hindsight, placing the generators at ground level in an elevated location might have been a better bet. Or locating the entire generating plant further upslope.
> In hindsight, placing the generators at ground level in an elevated location might have been a better bet. Or locating the entire generating plant further upslope.
yeah, well thats what I meant. I mean they were buried deep...
There were even studies that this was dumb:
and basically tepco knew that. they were just too lazy (it was prolly too cost intensive) to do something against it (i.e. placing them higher or creating water bunkers (u-boot style)..
besides the generators there was also the human failure part. (most of the time human failure happens. I mean If I do not automate something, I might do it right 3 times and the fourth time I most often fail hard...)
You know all that cooling equipment data centers tend to have?
Dumping power to ground is also known as an electric furnace.
Now you have twice as much heat to move, but you only have the usual cooling system. Toasty servers are sad servers. Toasty engineers are dead engineers.
Ok your reply makes no sense to me (see first line of my original comment).
I don't understand why you'd be putting a generator test load through the servers in addition to the normal power supply??? Why would you do that???
I would have thought that you have normal operation going on in the datacentre, normal cooling infrastructure, normal power coming in. Then the generators turned on with their in built cooling, then them delivering power to the test area and not the datacenter - which is then the only additional area you'd need to cool.
Seems like dealing with the heat would be not that hard, but perhaps I have too much faith in engineers? :-)
Even just a massive heating element in a tank of water, giant kettle style would work wouldn't it? Big kettle mind you, but big tank too. Seems like a cheap way to test the generator at full load for an extended test?
Your test would only test the generators. It would not test the transfer switches, or whether everything you think is connected to the backup generator is actually connected, and any equipment in between.
Interesting. I've worked at a few hospitals in the UK (not in IT) and all of them do a generator test one a week. However they do it, only one of them lost all power (for a couple of seconds) every week. It was quite disruptive, but I guess cutting the main power and checking the auxillary power kicks in is a better test than simply powering up the generators.
We had a generator that ran a two hour test every Thursday. It ran fine one Thursday, the next day we had a power outage and it failed to start because a capacitor went bad.
Lack of testing; ideally you throw a switch once a month or so to make sure the UPSes work and the generators power on, but I can imagine it's something you'd rather not do, if it doesn't work you've just intentionally borked things for your customers. Still, it is something that datacenters should do, do it a few times before you have any customers, keep doing it. Maybe offer a discount (if it's a new datacenter for an existing provider, like AWS or whatever) for the first couple months, indicating that emergency power has not yet been tested properly.
There's tooling available and running at some of the larger companies (like Netflix) called Chaos Monkey, which triggers random outages constantly to determine if the system is resilient and self-healing.
We always had our generators cycle on once a week for 20 minutes or so without assuming load. You don't want the (in our case propane) engines sitting too long without being run.
UPDATE: not all datacenters are down, it seems like that in europe because ovh routing hasn't been updated so from our point of view everythign is down but really it is not :)
I think they don't know themselves what works. The CEO said GRA is down while I can access it without issues but that could be depending on where you try to connect from.
Looks like he deleted the tweet but he wrote earlier that GRA was down with the others while BHS was up. I'd guess he is in RBX and couldn't reach GRA so assumed it was down until they realised that the RBX-GRA line is down.
EDIT: He did indeed delete the tweet, [1] is the url in case anyone knows a website archiving them quickly enough.
wow, yesterday I was playing with their public cloud because considering choosing them. I had some connection problem with my private networking there (deleted it more than once) and opened a ticket. If it was me... sorry, haha. Not good advertisement but it can happen to everyone.
Huh, I was just looking at them too. Contrary to popular opinion, I kinda prefer it when these things happen before I sign up so that in the post-mortem usually whatever architectural failure lead to the outage is corrected and you get a stronger service.
OVH are very open about their outages and root causes. Even routine maintenance tasks are cataloged with updates at http://status.ovh.net
As a more technical user it's nice to have providers that give this information rather than the boilerplate "Issue with an upstream provider" over and over.
Not saying this is the case here but that's also sometimes where you spot amateurism and should run away. I remember a host provider a long time ago (15y) who was storing its backups on the same machine as the main data. Guess how I figured out!
Oh man that sounds bad! You're right, sometimes these events do expose the inability to handle failure and you're right that that means walking away.
Thankfully many times we're reminded that there are good people out there working hard against difficult constraints and they finally get their chance to do things 'correctly' in the wake of the SHTF.
Wouldn't this require that there be a very limited amount of things that can go wrong?
Unless by stronger you mean some kind of a change in mentality over care for the service, but that would probably have a limited lifespan until it's back to normal.
Hey I would suggest not completely disregarding OVH. Their prices are good and their network is usually very reliable and fast. You simply won't find a North American hosting provider with those prices and reliability. You won't find one at double OVH prices, either (and I have tried).
no, I'm not completely disregarding OVH, I now about their network and reliability in the past. Will rethink only about my disaster strategy in the future.
9 by their count (7xRBX+2xSBG). When this was posted, the CEO wrote that everything except BHS (Canadian DC) was offline (tweet now deleted). Presumably he was in RBX and noticed that they couldn't reach any of the other DCs. So it looked like all DCs were down for a while.
We are experiencing an electrical outage on Strasbourg site.
We are investigating.
Comments (2)
Comment by OVH - Thursday, 09 November 2017, 10:55AM
SBG: ERDF repared 1 line 20KV. the second is still down. All Gens are UP. 2 routing rooms coming UP. SBG2 will be UP in 15-20min (boot time). SBG1/SBG4: 1h-2h
Comment by OVH - Thursday, 09 November 2017, 12:04PM
Traffic is getting back up. About 30% of the IP are now UP and running.
-------------
VPSes are still marked as read in the dashboard. I can't access mine.
Comment by OVH - Thursday, 09 November 2017, 12:44PM
Everything is back up electrically.
We are checking that everything is OK and we are identifying still impacted services/customers.
Comment by OVH - Thursday, 09 November 2017, 13:25PM
Hello,
Two pieces of information,
This morning we had 2 separate incidents that have nothing to do with each other. The first incident impacted our Strasbourg site (SBG) and the 2nd Roubaix (RBX). In SBG we have 3 datacentres in operation and 1 under construction. In RBX, we have 7 datacentres in operation.
SBG:
In SBG we had an electrical problem. Power has been restored and services are being restarted. Some customers are UP and others not yet. If your service is not UP yet, the recovery time is between 5 minutes and 3-4 hours. Our monitoring system allows us to know which customers are still impacted and we are working to fix it.
RBX:
We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be cut from our site in RBX. We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident.
We are in the process of retrieving the details to provide you with information on the SBG recovery time for all services/customers. Also, we will give all the technical details on the origin of these 2 incidents.
We are sincerely sorry. We have just experienced 2 simultaneous and independent events that impacted all RBX customers between 8:15 am abd 10:37 am and all SBG customers between 7:15 am and 11:15 am. We are still working on customers who are not UP yet in SBG.
Best, Octave
They now posted their explanation [1] but I don't buy it. I find it hard to believe that the RBX incident happened shortly after the SGB incident without any connection between these two. They should have redundant networking (at least that's what they say) so one corrupted DB in RBX shouldn't have brought down the whole DC (or 7 DCs according to their system). Maybe they pulled corrupt data from SGB because it was down but I don't believe that at the same time of a power failure, two redundant network nodes got corrupted without any notice. Otherwise wouldn't that mean that one hardware issue can also bring down a whole region?
Some servers in GRA still appear to work if that's of any help. All data centres offline at once sounds more like an attack than a power failure in one location. According to them, there was a power failure in SBG but I don't see how that should affect routing in data centres several hundred miles away.
It seems more likely that their data centers aren't quite as isolated as they thought they'd be. The outage also appears to be limited to their locations in Europe.
I always wondered why they promote having 6 datacentres in Roubaix when google maps shows that they're all within 50m. Can't be too much redundancy there.
Then they could call it one DC. A DC can have more than one building. But by saying you have several data centres you imply redundancy, similar to AZs at AWS. And (at least to Amazon), two AZs are far enough away from each other so that one building could blow up without affecting the other one.
The network equipment failure was due to a software error[1]. I would not rule out some kind of cascading failure triggered by the initial outage until the cause of the bug is known, though OVH seems comfortable with such a statement.
How is that even a thing? Don't you have separate routing for every switch? Otherwise you don't have redundancy? I'd even expect data centres to use different hardware for networks to avoid having a single point of failure.
> "Diagnosis: All the transponder cards we use, ncs2k-400g-lk9, ncs2k-200g-cklc, are in "standby" state. One of the possible origins of such a state is the loss of configuration. So we recovered the backup and put back the configuration, which allowed the system to reconfigure all the transponder cards."
Their interfaces lost their configuration, and they re-applied configuration, and state came back. This does not equal critical software bug.
> "One of the solutions is to create 2 optical node systems instead of one. 2 systems, that means 2 databases and so in case of loss of configuration, only one system is down. If 50% of the links go through one of the systems, today we would have lost 50% of the capacity but not 100% of links."
This is a crap mitigation. They're still depending on the same hardware and process that led to the first outage, only now there's more of it, so there's more chances to fail.
If they had continuous configuration automation they would have detected when the router's state changed, identified the missing bits, and applied configuration.
"New" routers (as in, since 2011) have APIs and can even run code directly on the router in order to fulfill these requirements. Cisco has multiple white papers, and even provides complete products to manage and certify configuration is applied as desired, even in cloud-agnostic multi-tier networks. Even on old routers, practically all config management solutions out there have plugins to manage Cisco routers.
It's also ridiculous that they had no access to remote hands. This is IT 101.
Maybe their main DCs, or their largest, but not all of them. I have virtual servers in thier Quebec DC (BHS) and it hasn't gone down since the last time I rebooted it.
It's in the Gravelines datacenter, which is not actually down contrary to what the initial reports said (only Strasbourg and Roubaix are, and for two different reasons).
Huh. I have services active on two dedicated machines from OVH in Canada, and I was logged into both via SSH all night, and didn't have any interruption at all.
OVH is a giant hosting company in Europe, providing dedicated servers to quite a lot of companies. Half the internet here was borked. A translation service, a few developer tools, etc...
Sure, it's less dramatic than AWS going down, but it still hits, and hard.
The Canadian data center is also a very low cost way to serve the US. I'm an OVH fan. They have their quirks, but the pricing is great. You just make sure you compensate for their quirks with backups and DR plans.
My data is with me in Europe and the company that has my data is in Europe with me too. If I was using DO or AWS, then my data may be in Europe but the control over the data is in the US, free for all to the three letter agencies and lacking privacy laws there.
"guys we have a single point of failure in our architecture with SBG, maybe we should...
- naaah it's fine, we do not have time nor resources"
Then shit happens.
edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me (or backup systems, like generators, not being correctly tested... whatever). I am really curious about the future explanation with what happened exactly
edit2: Why all the downvotes? Even the status page of OVH is down, do not tell me it is good design. We are not here to be charitable, but realist.
its OVH:
The Hardware is good
DDoS protection is good
The Prices are high
but
support/administration does not work well, i have a lot of really weird story's with them, from them plugging in a keyboard in our server to reboot it (without any reason) to taking down a server for a requested maintenance only to notice after 4 hours of downtime that they did not ask their bosses if they were allowed to even perform the maintenance requested (and then not getting permission to do so after another 2 hours ..)
For me it feels like there are some really deep issues somewhere in the whole administration that make incidents like this no real surprise
Problem is most other providers dont work any better, so ...
Everyone makes mistakes, let's just hope they learn from it.
Their prices are pretty amazing for some of the configurations you can put together, especially since you can pay a large setup fee in exchange for a lower monthly rate.
Just the bandwidth alone would cost us 3x what we pay at OVH if we were with one of the big cloud providers.
I've tried Hetzner but the Network Peering to Telekom is subpar compared to OVH. On Hetz I got about 40Mbps up/down to my local computer while on OVH I can easily load my DSL 100% without issues.
OVH is one of the few with direct peering to Telekom, most don't want to pay for that. Otherwise Hetzner is good, their own network is limited but peering works reasonably well. But apart from them I'm not aware of anyone with cheaper dedicated prices than soyoustart
> - naaah it's fine, we do not have time nor resources"
Yup, been there multiple times in smaller hosting companies.
It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.
The person you're replying to was relating their experiences at hosting companies that are smaller than OVH. How does that imply that OVH is a smaller hosting company?
This happens all the time. Every single thing that you see in software development happens in network engineering and data center engineering, except that where in development in general senior people who write software are capable of at least guestimating complexities to provide a marginally unified front against the unreasonable expectations of execs, it is pretty much never the case in neteng or dcops as those that develop software cannot write their heads around the complexities of working with physical hardware.
It is rather counter-intuitive. In neteng and dcops, "I don't know and I cannot find out. I can only attempt to mitigate what i think might have caused it for next time" is a very reasonable answer to 99% of the "why this happened?" questions because in order to replicate the situation to test the theory one needs to recreate the same problem again on the same scale.
This also means that certain things cannot be tested. Most of generator tests are garbage - turning on generator and running it without production load delivered over the transfer switch does not test anything other than that one can turn on a generator and run it. The problem typically happens not because the generator ( also is there the generator or the first and the second generator? Why is there no generator bank for a non monkey-sized company? ) does not start - the problem is because over time transfer switch develops a problem and unlike generators it is not possible to test a transfer switch where in the event of a test failure the customers won't lose power unless the data center is designed from the beginning to deliver A and B powers over separate circuits to every single customer and every single customer has per system ( not per rack ) transfer switches.
Of course it costs a lot more money, something that companies are reluctant to spend.
I think I'm a pretty decent senior network engineer but I've been hit by firmware bugs more than once. No matter how much redundancy you build in, there's always shit that can go wrong and there will always be things that you just can't foresee (like the bug in this case). The software on these network devices and optical gear is written by humans, like all software, and is not perfect, like all software.
(In one case I experienced, the vendor reassured me that what I reported was not even technically possible -- until one of their engineers flew out and witnessed it firsthand.)
If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that this was too unlikely to happen or that there are other failovers but never tested them properly.
No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.
In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.
So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect.
They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.
When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"
I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.
So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.
Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.
What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.
To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.
Generators in critical situations like this are usually provided with crankcase heaters which keeps the oil at temperature and makes them easy to start. Also, any emergency power system should be (and usually is) tested monthly if only to prevent the fuel lines going bad.
Data centers can go down while testing the generators too.
There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.
Their generators are almost certainly tested frequently. But there could be any number of causes underlying the failure, and unfortunately sometimes failure does happen.
The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes.
While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature
It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that.
Improbable. Even in a mostly-normal office building, the monthly generator test consisted essentially of "cut the mains power and see that no impact is perceived as UPS, batteries, and autostarted generators bear the load, in their turn for ~2 hrs total."
It's supposed to be two separate incidents: power going down in Strasbourg, and fiber network equipment going down in Roubaix (the main center of OVH's network) due to a "software bug".
We're nerds, so how about checking the facts (?). Please explain me if there has been some kind of time anomaly lately or?
My submit timestamp: 07:21:25
Your submit timestamp: 07:28:23
About the title. I did consider the title for a while, because I wasn't sure how bad the situation was. But from my own independent monitoring system I did see that RBX and SBG servers were unavailable. Of course I also did some basic trouble shooting and confirmation work before posting.
Btw. Right now, there's some network traffic present on SBG network. Let's hope that the systems are soon up'n'running.
I'm waiting for the postmortem. I'm very curious to see, what was the root cause of all this mess. As usual, there probably were several overlapping causes.
Yes, very important. I have two dedicated machines in different racks in Canada that were serving requests all night. I was also logged into both machines via SSH all night with no interruption.
We selected their three data center EU region precisely because they were three separate data centers, so not happy. This is clearly bad design.
I think we're now going to have to look into multi-provider options. The only way to be solidly up is to be hosted by more than one company at more than one data center.
I've also heard stories of billing nightmares where you get locked out of a cloud provider account, so that's another thing.
I guess this is already a reason by its own. It, among other problems, is what happens when we go from small "local" providers you can actually call to automated global providers that cannot provide immediate support even if they tried.
That's the problem with this trade off. The small providers tend to have good support and someone you can reach if anything goes wrong but they won't have experts on site 24/7. The large ones have dozens of them on call at all times but lack in support (unless you pay a lot).
and on https://twitter.com/ovh_support_en
"SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. "
"An incident is ongoing impacting our network. We are all on the problem. Sorry for the inconvenience."
"SBG: 1 gen restarted."
"RBX: all optical links 100G from RBX to TH2, GSW, LDN, BRU, FRA, AMS are down."