That's pretty crazy, though with the hack publicly disclosed, it was bound to be exploited. What I don't understand is why this report incident comes so late. The hack has been public for a large amount of time. Could anyone explain?
The linked issue isn't "we need to patch Shellshock". It's "we have 800 hacked machines on the network, they are DDoSing like crazy, and it's saturating the internal network". Their problem isn't that servers have been hacked. It's that it's overloading the network.
OVH isn't responsible for what people do with the servers. They provide the initial installation, networking, hardware monitoring and some management tools, but that's it. If they detect that a server is sending too much traffic, they can guess that it has been hacked, so what they usually do is disable it and notify the owner to fix it. But they don't do more, that's not their business. If you want a host that does more for you, it's managed hosting you want, not a dedicated server provider.
Actually if they detect a server is compromised, they will put your server in a FTP only data recovery mode and usually the only way to get your server back is to do a fresh install of the OS. Luckily OS reloads are fast and provisioning dedicated servers takes minutes.
Yeah their anti-DDOS is actually to null route everything to your server. It's been a long time since I've seen that but that how they did it in the past (which is actually making any ddos against you really effective but your neighbors won't be as affected).
If it's actually your server that actually attack another server, they will shutdown your server and give you a warning. They will let you boot in their recovery os that let you access your file system but if your server does it again, they terminate your account.
Like I said, maybe it's different from the last time I saw an attack on an OVH server, however when I saw it, it was literally impossible to reach the server even though it was still up. Using their ip failover system was the only way.
Their anti-DDOS system is mostly designed to protect against external attacks. It works at the network level, probably at the connection between their network and the outside world. Because that's the most efficient way: detect them and block them where you have the most bandwidth available.
This is an internal attack, which requires different mitigation measures, and is seen less often in the wild (compromising 500 servers from a specific provider is more difficult than 500 random servers on the internet, and you're pretty much guaranteed that the provider will deactivate most of them after the first attack), so I guess their protection systems aren't as developped against it.
Its on network/hardware level. The issues with internal DDOS is that they are software related. OVH cannot access your server (or it shouldn't) so the only choice they have is to shut the server and wait till you will fix it.
But if you receive DDOS attack to your server from outside, they can defend you using network resources.
I manage 2 VPSs. I'm responsible for security patches and everything. If shit hits the fan, it's because of me not because of my VPS provider :-) That's what VPS is all about.
True. However, since most clients (especially those that don't maintain their servers well) use standard images provided by the hosting provider, I don't understand why those images don't come with automatic security updates configured by default.
I know that many of us that run systems for which availability criteria mean they want full manual control over all updates, but those are the people that either use custom images or know how to turn those automatic updates off.
Security updates are finally pushed hard on desktop users for obvious reasons, with everyone and their mother running cheap VPS's these days the same logic should apply to virtual servers.
Automatic updates for server software can be tricky.
Ubuntu, for example, can be configured to run apt-get update && apt-get upgrade automatically every day (using unattended-upgrades), but AFAIK the default configuration only checks for updates, notifies the admin, and waits for approval before actually changing anything.
Even though Ubuntu LTS, Debian Stable, CentOS, etc. are supposed to be stable distributions, it is not unheard of for routine updates to break something mission-critical. For example, the package maintainer might have made some changes to the default configuration. (C'mon, do you really need to make minor changes to nginx.conf all the time?) A daemon might go down and fail to come up again after an update. (apt is particularly annoying in this regard, since it forcibly restarts daemons during an upgrade, and not always in a sensible order. Result: HTTP 502 Bad Gateway.) So unless a human is present to spot any issues immediately after an update, you might be left with a broken system at an inconvenient hour. Which is why the documentation doesn't recommend fully automated updates.
Unfortunately, the set of VPS owners who don't apply updates regularly probably has a large intersection with the set of VPS owners who won't be able to troubleshoot a broken update anyway, so maybe this concern need not apply...
We're working on the auto-update model for servers at CoreOS. If interested, you can read more about our update philosophy here: https://coreos.com/using-coreos/updates/. We think that this is one of the best ways to secure the backend internet, similar to how auto-updating browsers moved the front-end of the internet forward.
This can be difficult as some updates can break applications. PHP is problematic that way for our customers on servers that we manage for them. Its not as bad as it used to be (e.g. register_globals), but updates can break customer apps. So if you automatically apply them and their web server breaks, who does the clean up ? The web developer who got paid to create the site but not maintain it ? The VPS provider who gets paid to just host the image ? The customer from their POV (and these are not technical people) will often say, "but I didnt do anything to break it. You broke it! You fix it!" Also, as there are (typically) no regression testing, breakage can be silent and undetected for some time.
As I understand it, server updates run counter to the idea of servers as static, externally configured fixtures ala cloud formation or its competitors. To merge the concepts you'd need to have aws or whoever manage your cloud formation templates and automatically re-deploy whenever they make a change to them for security reasons... I think maybe our tooling isn't quite there yet, but I'm sure its coming. Disclaimer: I don't have a very deep knowledge of cloudformation, so I might be talking out of my ass.
I have some vps with digitalocean, I haven't logged in to them in a long time, even after shellshock surfaced publicly. about aftter a month later I checked to see if I am vulnerable through shellshocker.net test and it was negative. I am guessing my provider did something to fix it.
I know they dont, I didn't say they do. But my non up-to-date Ubuntu server wasn't vulnerable when I checked about a month after shellshock vulnerable went public.
Have you considered you may be using a software stack that is not (obviously) vulnerable? Not every software is vulnerable by default on all URLs, just the ones that wind up calling out to bash in a specific way.
If I am not wrong the vulnerability test on shellshocker.net tests bash directly. Either way I could very well be wrong and I might have missed something.
Yes that's the one I was talking about. Ran the test before and after updating. It was the same (negative). I ran the same test on the relative up todate home computer (Ubuntu) and it was vulnerable on some of the tests. This IS weird.
So somebody used shellshock to break in, then plugged the hole to prevent others from accessing the server, and is now using your server for... whatever?
Ubuntu, for instance, has unattended-upgrade (that it defaults to on), the premise being that it will forcefully install and use critical security patches. Ideally you'd have done this yourself ASAP, but if not I have to think that unattended-upgrade came into play at some point.
Default means "the default when you install Ubuntu", not the unconfigured state of that process. For kernel and in-use things it would still require a restart, however maybe at some point DO forced a restart, even if unrelated.
Probably because they are a virtual and colo hoster so aren't generally responsible for the security of the machines they host. Also, a big chunk of people who buy these machines are less responsible than they should be.
At best this is monitoring and playing whack-a-mole on their part which they've done a pretty good job of by the looks.
The involved machines are dedicated servers, according to the posts.
Also, although I have many things to blame OVH for, they at least take a somewhat proactive stance regarding security, even for the very cheap dedicated servers. They usually contact you when they notice a traffic spike on unusual ports for example.
But yeah... judging by the people I know in real life who own cheap OVH dedicated servers, I'm not surprised at all by these news.
that's why I find crazy all those people wanting to have a dedicated server, I don't want to spend my life on call 24/7.
I'm pretty sure 90% of the customers are not professionally trained in server maintenance. I am, and I don't want to do it anyways unless I have a good reason to do it.
I use a VPS for email (configured opensmtpd + IMAP over SSL), VPN server, HTTP Proxy and host 3-4 personal applications (nginx + mysql + ruby apps).
The hard part was to set-up all these services, keeping them up-to-date, since no other party depends on them it is considerably easy and almost risk-free. Other than that is not time-consuming at all. Only 2 times in 3 years I had to re-configure a service. Most of the times I just run an upgrade-script on a tmux session and everything runs smoothly (I rarely use pre-compiled packages).
Now, if I had a production-level web application, I would probably set-up as carefully as I could a new VPS and roll my application there, up to the point where I needed to scale.
If it would be a one-man-show I'd probably go with Heroku (since I write sinatra/ruby/rails applications) or similar (possible cheaper) service to avoid spending time on sys-admin. But these service IMHO are still, expensive for projects with no income while a VPS can do all that and mode at once at a considerably lower price.
ps. As a side note. Before I join HN, I though that all programmers were capable of sys-admin (UNIX servers). Then I realized that tasks that seem trivial to me, are considered somewhat difficult for others and vice-versa of course.
Agree entirely. That's why I go half-way and use Windows Azure's PaaS platform. Much less painful.
The company I currently work for have to full racks in two data centres each, most of which is near idle and it's a full time job for 2 people. Could do away with all the machines and all the associated administrative staff, just use a devops team and shave 40% of capex and opex as well by moving to something else.
But then again, they might not be getting nice lunches with HP and the DC vendor.
Depending upon how much they're spending, I kind of doubt HP would really pay a whole lot for trying to woo that customer. The usual method is to just cut pricing if someone gets antsy in TSG and over on the Procurve / networking side. This is a company that nickel and dimed people on trips to force them to exclude alcohol from reimbursement and had probably the lowest per diem I've ever had while traveling.
To easily maintain ratio (of uploaded to downloaded amounts of data) on a private torrent tracker. Due to restricted access (members only), it's pretty safe.
If you're asking about the hosting company, they largely don't care what you do unless someone complains.
Would you think most of torrents are private? So you need to be introduced to be part of them? So if you're introduced, a police investigator could be introduced too, right?
Because you're trained you realize the actual costs, most people don't factor in maintenance as cost and see VPS's being as no-brainer cheap to get stuff running on.
I have had a bad experience with the OVH abuse department. I sent them email twice (once about spam, once about network attacks) and never received anything, not even an automated email delivery confirmation.
OVH/Kimsufi has non-existent support. They claim to offer support (yes, every kind - from listing your passport verification request to tax exemption request) via a publicly accessible forum. Even that support is non existent. All you get there is few useless answers from few uselessly enthusiastic (sometimes just smartass) fellow customers.
They've been having capacity issues for almost 2 weeks now at peak, you can see it at http://weathermap.ovh.net/usa usually. Capacity inside their network between Montreal and Newark was at 100% until an upgrade was done last week (edit: the upgrade isn't finished yet, but so far it's helped):
It is sad that so many servers are still vulnerable on an issue that has been reported through all major mainstream news networks. We've witnessed several attacks many days after Shellshock was fixed, and chasing down the botnet scripts we saw hundreds of servers compromised.
The story and scripts were published on a blog post in case anyone wants to check out a standard botnet attack:
I feel it's my clue to ask a question I've been wondering lately: how good is OVH's dedicate server service? (Regarding network, uptime, system outages etc).
I used one of their dedicate servers for six months and never had any problem with it, however I plan to buy (rent) dozens of big-data servers for my next project, and while my experience so far have been good, I'd like to hear from someone else.
Despite of that, I had several downtime issues with VPSs hosted at OVH. I guess the difference is you can't oversell dedicated servers.
I had a few expensive top end storage servers (36 drives), they worked fine for years, only 1-2 drive failures, no power failures or major network outages I can remember.
Last year i switched to collocation instead, but still have 1 server with them, things have improved alot lately, now the servers come with ipmi and new ovh control panel is nice.
Problem with OVH are noobs buying their servers trying to save every buck only to realise that the support will not fix anything but hardware issues (loose cable, broken drive etc). Obviously alot of people do not understand that unmanaged means unmanaged
I have been using OVH for a few years now, and had no issues to report. This is only a few low-mid level dedicated boxes and a half dozen VPS's though, so others may have differing results.
Completely forgot I had a server running there, I think it is actually a dedicated server, not VPS. It was vulnerable but seems to be uncompromised. Fixed it now.
Just having a vulnerable bash isn't enough to break in. You also need something that uses it. A CGI script available via webserver, or some other service (I hear OpenVPN had issues...)
I run a server on OVH, but I figured as I don't run any dynamic web app on it, I was unaffected by shellshock - the only way in is through the web server (which basically hosts a bunch of static files), or SSH.
From what I understood, shellshock was caused by poor sanitation of bash variable extension, and so only posed a threat to people who used bash from their public-facing interfaces, hosted bash CGI scripts, ...
So even static servers could have been vulnerable. I'm not sure if your server was vulnerable or not, but I think you were definitely too soon with assuming you were safe.
This exploit only works if bash is called at all during the handling of the request. This is typically only the case with CGI scripts, or when the requested application does a system() call or directly calls out to bash
Though it should certainly not happen, it's not uncommon for various applications to 'shell-out' certain tasks. The point I was trying to make is that you're not necessarily secure just because you don't use CGI or only host static content, and that exploitation via the web server could certainly have been possible.
The big problem with shellshock is that the attack surface is found pretty much everywhere. The HTTP server attacks using the CGI environment (HTTP_USER_AGENT, etc) was only the most obvious way to exploit the bug. There is no way to predict all the ways a vulnerable bash could be exploited; we are going to see these attack for decades. Old firmware and poorly-maintained servers are just waiting for someone to find some clever trick that lets an environment variable be set remotely. I would guess that we haven't even begun to see the subtle ways shellshock can be exploited.
So yes, patch everything, or you are taking a bet that there will not be any sufficiently clever attackers in the entire future lifetime the server.
Hard to say, without knowing what you are running.
I've seen people try to exploit this by email, hoping that users have ".forward" or ".procmail" files which are processed with bash, or which use it to execute commands, for example.
The title is misleading -- it was customer servers on OVH that were exploited en masse and then proceeded to flood the network with DDOS and bot traffic, not OVH itself (apparently).
This has likely happened to loads of VPS and dedicated server providers. One consequence of hosting such a service is that you typically end up with a lot of poorly administered crack houses on your network.
It sucks that so many servers are still vulnerable to the shellshock issue. I am impressed with OVH's openness in regards to reporting this issue. I am amused and impressed by their final comment/conclusion; "Obviously it is time to go to 2x100G on the private network between Europe and Canada."
This seems like the tip of the iceberg. Shellshock didn't generate nearly as much attention as heartbleed, but is far more devastating and easier to exploit. How many home routers have been taken over by now?
That's a pretty serious limitation of their automation if they are unable to just whack the ACLs on the private network to the affected customer nodes.
VPS providers should take a little responsibility to protect their customers. I'm not talking about full on sys admin or managed services, but even a scan and report or taking off badly infected hosts offline till the owner upgrades it.
This will save them much time in the future, save their own some time, and save other sites (that will be attached through the compromised hosts some time).
Being a good citizen of the Internet boils down to taking some level of responsibility. I wont put grandma on the internet without malware bytes or avg, etc on her computer. So why should I give her a VPS and tell her she's on her own?
> OVH do not have the right and the access to the system of thoses servers.
You should check your TOS. They reserve the right to access your server and if you use any of the default install images it comes pre-configured with their SSH key.
One of the most command ways a virtual machine is compromised is by brute-forcing a root SSH password.
Do you suggest that every hosting company run non-stop dictionary attacks against their clients?
Or even nmapping open ports, non-stop, looking for backdoor services?
95% of the time virtual machines are running unmanaged, and the hosting company has no insight into what is running inside the machines, nor how up to date they are.
