I'm thinking Rackspace might well have been in the right on this one. If the customer was in fact phishing, Rackspace was well within their rights to shut down the account. It's really up to the application creator to prevent that abuse.
That said, it's good to have a reminder of the risks of outsourcing your hosting. I still think the tradeoff is worth it for experimental products where you don't want to invest too much upfront.
May be technology can help a bit here? I think the issue in this case was more about the granularity of "takedown". If there was some kind of an API contract between the infrastructure provider and the application service provider - an API that lets you shutdown a single subdomain, an email service and so on, the situation would be more tolerable.
Or even something simpler - call the account holder's phone and play a simple recorded message that says "hey, its Rackspace, we need you to deal with an urgent issue, please check your email ASAP". Cheap for Rackspace, gives the customer fair warning, and if there's no response within some reasonable time frame then shutdown the machines in question.
Well I didn't agree or disagree with one hour, I said "some reasonable time frame" explicitly because the expected response time needs to consider what the real risks are.
I keep my phone permanently on vibrate (so I don't have to remember to silence it when in class, etc.), and besides, it'd take a hell of a lot more to wake me up than a ringing phone.
I agree. Hosting providers usually reserve the right to shutdown your server if it has been hacked, and phishing is often more directly harmful to people.
The problem here is Rackspace as a infrastructure provider judging on behalf of service provider. They give no explanation of the complaint details or why it is justified.
Many comments here take the phishing as "a fact", their terms might grant them the power to shut any server down but this is a threat I think every startup should learn if they use Rackspace Cloud or consider to. And we learnt that.
24 hrs is not just for respond to remove the content, it is also for the server providers to verify the complaint and react responsibly.
Taking the contents of the article on faith, I think the point was that their customers were abusing the startup's service for phishing.
This would be analogous to, for example, AWS taking reddit offline because a user posted a phising link.
Nearly every web business which lets people put information on the web that others can see will face abuse issues at some point or the other - and cutting off a business and its legitimate customers because of one client misusing the service does not inspire confidence.
Everyone hate spam. I don't object Rackspace to shut down an account that is obviously phishing/spam, but not take down as soon as they think there is an abuse. Grace period must be given, so the the site holder can respond.
I don't think it is possible for few-man startup can responds in 1 hours for 24x7. I would choose to use an alternative hosting that give a longer gracing period.
> Grace period must be given, so the the site holder can respond.
Unfortunately, during that grace period, numerous people may be receiving spam emails directing them to the site, and some of those people may be naively entering their information ...
I really dislike the way most service providers and the like handle spam, but unfortunately, I too must side with Rackspace on this one. They simply can not afford to "wait and see" until the site owner responds, or provide a grace period while the site owner tries to figure things out.
Phishing attempts must be handled by site owners as though their server has just been compromised and someone is currently downloading the entire password database: the server must be shut down immediately, the problem fixed offline, and the server only brought back online once the issue is fixed.
The real issue to me is their apparent zero tolerance policy. Unless I'm misreading something, if there are two incidents where your site is used for phishing, you will lose your Rackspace account. I understand that Rackspace doesn't want to go chasing these things left and right, but it seems that's a little extreme, especially when they're supposed to be infrastructure providers, and should recognize that their clients have clients, and their clients shouldn't be held entirely responsible for the actions of their clients' clients.
For your argument, I just created an wufoo form which should take down immediately once discovered. http://rickmak.wufoo.com/forms/phishing/. IN that case, I am sure only my account will be taken down, not the whole wufoo.
Actually, it depends on size. If someone created a phishing site on Heroku's, Amazon probably won't shutdown all Heroku sites. But to let Heroku to investigate. For small startup like pandaform, no luck. Rackspace just regards you as one site.
Pandaform can handle things better, like banned "password" field like wufoo do.
It seems like it would be an improvement to either:
1. Keep the very short notification period but also try to reach the site owner via phone or IM
2. Lengthen the notification period if using email only
(Note that I have no problem with short notice and email only if the customer was given the option of providing an emergency contact method but chose not to, and that I otherwise generally agree with the response.)
It seems like the real flaw here is the combination of lack of communication and lack of warning.
The reality is that each minute the phishing site remains up, another account may get its information stolen. Imagine if you are the person that had your bank account information stolen and drained during the "grace period" for the company to respond to the takedown notice.
This is the kind of thing where a customer who gets their information stolen while Rackspace is waiting for the grace period to expire might have a legal cause of action against Rackspace.
Ultimately, I think Rackspace did exactly the right thing here. If you are operating a service that would potentially allow fishing, then you are bearing the risk of policing your users. Asking Rackspace and affected users to give you a grace period is asking them to bear the risk instead. I 100% agree with the decision to immediately shut the site down.
No, and that's a strawman argument. That's like asking if it were reasonable for Level 3 to pull the plug on Rackspace if Level 3 got a phishing complaint.
If Amazon got complains about Heroku then I'd certainly expect them to be investigated, and in Heroku's case I'd expect Heroku would take over and shutdown the phishing site.
exactly, same in this case. I expect Rackspace should ask pandaform to investigate the case and shutdown the phishing site. I won't expect the whole pandaform would be taken down.
Also pandaform doesn't allow use to put any script or password field in the form, which the quality of the "phishing" form is not as serious as what we thought as a normal phishing site do.
In the case of Heroku, I'd expect them to be able to shut the phishing site down within the 40 minute period Rackspace apparently gave Pandaforms before shutting the whole service down themselves.
I'm sure if Pandaforms had done this (which is difficult when you're a much smaller startup than Heroku) then their server would have been left untouched.
You can argue that Heroku would have most likely got a phone call and that Pandaforms deserve the same treatment, but I don't think that they'd have been allowed to leave phishing sites up for any period of time without their servers being placed in jeopardy either.
I think everyone agreed on that the service provide have to investigate and take action on any abuse claim. But what is questioning now is that is it reasonable to shutdown a suspect case of abuse without giving time for the service provider to investigate and respond to this case?
Once a phishing form is “in the wild,” every minute counts.
The burden is on the service (your site) to prevent or quickly act to rectify a situation, but if your provider determines that it must intervene, then it is well within it's right to.
If heroku got enough complaints (relative to it's size) they would get shut down or asked to leave. Now, heroku has a lot more than two servers, so it's going to take more than one or two complaints to take them out, and they are probably going to get more than an hour of notice, but if you provide a hosting service, you need to make sure that your users and customers are not using your service to host phishing sites.
They were in their right.
And I'm in the right of choosing a provider that won't disrupt my business.
I'm not talking about bulletproof hosting but 1 hour notice before takedown? Can't upvote this story enough.
What is missing from this article (and comments so far) is a more comprehensive analysis of available options.
If I lease a server from linode or AWS or theplanet or serverbeach or ${your favorite hosting provider}, would the situation be any different? I understand the article's author frustration with Rackspace, but it's a single data point hence hardly enough to be a basis for an intelligent choice of hosting provider.
I'm not even sure if I sympathise with him. You can argue whether 1 hour notice before disabling a server is enough or not but there is an obvious conflict of interest.
The interest of the person hosting server, who can potentially be a phisher himself, is for the site to stay up as long as possible.
The interest of the public is served by terminating the server as quickly as possible.
Everywhere, the more you pay, the more effort they will put in to helping you clean up your messes rather than shutting you down right away, which makes sense, because helping you clean up your mess is an expensive business to be in.
This may sound contrived and sarcastic, but I assure you it is not.
Try your best to get Robert Scoble's attention on this. Rackspace has tasked him with being an evangelist primarily focused on startups and bringing them into the Rackspace services. It's my understanding that things like this should be his primary concern.
I am really not a fan of all the defending RackSpace here. They pulled the plug unreasonably without proper notification.
If you're going to pull the plug or even thinking about it... email simply isn't going to cut it. You need someone to call the owner and make contact to explain what's going on or how to resolve it. I've had my servers compromised, I've had phishing content setup before, I have never had the plug pulled. I've had hosts contact me, give me appropriate amounts of time to handle it and some of them even offered to help secure my box or look into how it got compromised in the first place.
All this talk about how Rackspace should be treating startups make me think, "Why?"
I understand startups may not have the personel to react as fast as a larger company with dedicated personel, and I understand that there might be a very large percentage of startups on Rackspace which might account for a nice chunk of revenue.
But what makes startups different from my personal website or a larger corporation? If Rackspace receives a complaint about a phishing site hosted on their servers, they should do what they can to correct that, regardless of which client is using the server that has the phishing attack. The startup should get the same treatment from Rackspace as the large corporation and the guy with some simple homepage.
It is like asking why we need special facilities in building for the disables. Why didn't we treat everybody the same and don't provide any special facilities for disable?
I do think that site owner of any scale IS responsible to abuse complains, but the scale of the company do make a different. The influence and harmfulness of a fake shop in Amazon is not same as a fake shop in a very-small-online-shop. Thus people expect Amazon to take action instantly.
A start-up is not possibile to react as fast as large company, so why shouldn't we give them a reasonable time for them to do their job?
Rackspace shutting down your site might have done you a favor.
If they hadn't it's possible law enforcement could have gotten involved, you could have been arrested, all your business records and source code confiscated and have been offline for months. If you'd been using your own hardware that might have been taken away, too.
There are plenty of cases where that kind of thing has happened before.
So you're comparing your startup to a disabled person?
Unlike the hypothetical disabled person, however, your startup, when used for phishing, is allowing real harm to happen to other people.
Your attitude suggests that a) you care less about your clients than about what's convenient for you as a startup founder, because b) you're small. But that's a very bad excuse, and a dumb one, at that. Clients don't care about your size. All they care about is the results your products give them. And that leads to another question: why should anyone buy into your services if you don't/can't act promptly and responsibly in response to problems?
And yet you seem to be getting angry about Rackspace for caring about the clients affected by such criminal activity further downstream from them.
I'd think twice before buying into your products, with an attitude like that. Rackspace appears to care. You do not.
Welcome to the joys and sorrows of hosting user-generated content. If this makes you feel better,rackspace is just your first problem. You'll also find soon enough that abuses can get your domain blacklisted in chrome, firefox, and for opendns users.
When a server is 'shut down' for phishing or spam, a firewall blocks all incoming/outgoing except for traffic to/from a pre-determined IP address along with the notice that the server is being quarantined.
Site owner/admin can then access the server, perform any investigations or deletions necessary, notify data center, then data center opens traffic again.
Alternatively, something like a web-based shell allowing access, but all other traffic denied, would be acceptable.
I've had servers shut down for 'abuse' which was one complaint from someone at 1am local time for me. I supposedly got a call from xxxxxx at 2 am, notifying me that action would be taken, and my server was taken offline at 3am. I wasn't awake until 7am, and couldn't get things resolved until about 10am. I was told I needed to 'rectify the situation', but how can I do that when access to the server is blocked? It's a ridiculous execution of policy, and only serves to heighten everyone's frustration. A private web-shell or single IP in the firewall to allow access would resolve most of the ill-feelings site owners caught in this situation have had.
My policy is that I shut down the server, I create a fresh server for the user and then I attach the old disk read-only to that server, in case the user needs to retrieve data.
The problem is that there is no way to 'clean up' after a break in without booting from trusted media. Otherwise, there is no way to know if you have closed all the backdoors the attacker left.
We had a similar problem with another vps provider. Some php script was remotely exploited and sending spam. They disabled the VPS. Our website and email was down for 24 hours. They enabled the VPS the next day for us to look at it. What's worse I was asked to reinstall the VPS and it took me some convincing for them to enable it so I get my data from the server.
Lesson learned? I think you should never have any operations where you don't have full control. This still holds true but it was a bit ironic because to have full control I ran everything on the VPS, effectively transferring full control to the VPS provider! It was the single point of failure.
Now I will transfer each different service to a different provider. Email to rackspace, websites to a host, git services to github and so on.
Post is lacking information on whether the site was actually a phishing site and who (or which entity) submitted a complaint.
I think verification of the legitimacy of a complaint should be a critical step before disabling a site, otherwise you're prone to DoS.
It would also be good to know what steps the complainant took. Did s/he try to contact Pandaform, or immediately go to Rackspace as the owner of the IP?
Without knowing whether the complaint was legitimate, and what steps Rackspace took to verify this (or not) its tough to say whether their actions were appropriate.
The majority of commentors here assume guilty until proven innocent.
In fact, the author of the article has still been unable to identify the "phishing" form. Was there even really a phishing form?
With all this anti-phishing technology built into modern browsers, why is it Rackspace's responsibility to "protect" users from "alleged" phishing sites?
For copyright, we have the DMCA (for better or worse). Perhaps we need some similar sort of due process for hosting providers or cloud providers.
Most appalling is Rackspace's lack of transparency in handling this.
Pandaform was never contacted by the complainant. The complainant only contacted Rackspace. Rackspace assumed, without investigation, the complaint was legitimate and gave the guy notice via email.
Also, the real rub, is the fact the Rackspace would terminate his account if he got a second complaint.
So, it is not necessary to have an actual phishing form on his site to have his Rackspace hosting terminated. Someone simply needs to allege this to Rackspace, and his account and data will be gone forever.
That is fanatical? Again, I cannot recommend Rackspace.
this will happen at any responsible web host. If you are hosting phishing sites, expect to get taken down. This trickles up. if you run a hosting company, and you get enough complaints that you don't deal with, then yeah, you will get shut down or asked to leave.
That said, I think especially for higher-priced services, a phone call would be nice. (Note: I don't call my customers, though this is a policy I've considered implementing.) I'd be interested in what other people think about other notification systems.
I don't think the issue is that he was hosting an active phishing site. The main issue here is the amount of time he was given to fix the problem was too small. You think Rackspace's upstreams would shut the pipes down if there were a bunch of phishing sites that set up shop? Doubtful. Usually they only get involved when there is a MASSIVE DDoS.
Well, from my experience in industry, big providers of bandwidth do apply pressure (up to and including the threat of disconnect) to large hosting companies in an effort to get them to clean up their act, even when it comes to things like spamming and phishing that are less outright destructive to the network than DDoS activity. You are right that a million dollar customer will get a lot more slack than a twenty dollar user... but, uh, to me that should be expected. Dealing with abuse is very expensive. Some places I've worked that has been the majority of support costs.
What do you think about granularity of filtering: do not take all your customers' servers down, just filter the offending pages/services at network level until they sort it out.
again it's a matter of cost. If you only have one IP, and say, many domains on that one IP, only one of which is a phishing site, it is a /whole lot/ of work for me (who really can only mess with things at the network level) to selectively take out one domain or one url. I mean, it can be done, but the equipment and expertise required to pull it off would drive up my costs.
Now, if I was a 'fully managed' provider that had a login on all my customer boxes, in that case, assuming the problem was a customer of a customer phishing rather than a compromised server, it would be fairly easy to log in and fix it. But that's not how I'm set up.
yes. these sorts of things need to be examined... this is a lot of why running an abuse desk is so difficult, because you /will/ get bogus claims. Most claims are fairly obviously legitimate. A few are obviously bogus, and a few really are questionable. It takes skill to know the difference, and mistakes are obviously quite bad for your reputation.
Rackspace (and others) need to call their clients on the phone in these situations. They feel it is so important that it must be taken care of in one hour, yet they use email. Not exactly fair to the client....
Isn't the rackspace cloud their cut-rate service? would they have called if it was a slicehost (premium price) customer?
Still, I think it is a good idea, and one I ought to implement in my own service. The problem is that we all hate dealing with the phone, but that sounds to me like a lazy answer.
On the other hand, really, if you are an abuse problem, honestly, I don't want you as a customer.
imo the biggest mistake here was in PandaForm trusting all of its user-generated content.
It seems like they should have been doing some sort of verification of the UGC. Alternately, if they really wanted to go with a fully laissez-faire no-security-checks approach, they shouldn't have picked a hosting company (in this case, Rackspace) that requires customers to sign an agreement that has strict anti-phishing rules.
Did the customer simply set up a form that asks for a user's email address? because if so that describes tons of other services out there... e.g. Wufoo, Google Documents (but I guess they are not hosted on Rackspace!)
Good to know I wasn't wrong when I switched to Linode (From Rackspace). While phishing and spam is damaging, owners need more than 1-2 hours sometimes, especially when they are in a different part of the world.
@matrix, I think overall direction is correct, Rackspace should try their best to notify their customer if they received complaints. However the ridiculous point of this case is totally shutting down all servers of a startup in just 1 hour, which is really horrible. Instead I believe they should at least give 12 - 24 hours for a startup to react and investigate before taking down all their services?
Is that really on the same direction that they want to serve startups?
I wonder if this abuse monitoring service can be outsourced somehow. Pay a team of people around the world to monitor the abuse email address and hand them the kill switch for so they can respond immediately to take down one account only instead of the hosting provider taking down the whole server.
First of all, what is up with Linode's London datacenter? I barely press enter on commands and they've already been carried out. Seriously, it installs packages before the text from the server reaches me, and I get 15 ms latency on it (the SSH echo feels faster than my local PC).
Is it just me, or are their other datacenters slower than their London one? It might be the latency, as I was actually in London, but the one I have in Georgia doesn't seem to have that fast disk accesses... Bottom line, though, the London datacenter is just time-traveling fast.
That said, it's good to have a reminder of the risks of outsourcing your hosting. I still think the tradeoff is worth it for experimental products where you don't want to invest too much upfront.