This doesn't help my impression of Digital Ocean at all (even if I am a paying customer currently). A few years ago you could impersonate Digital Ocean staff on their support pages with no effort. They grabbed the username from your email, so whatever you put in front of the @ becamse your username on the forums, visible to everyone. And the avatar came from one of those email->avatar services where you can sign up and set it to anything. So when I signed up with a username like digitalocean@mydomain.com, I ended up being called "digitalocean" on the support forums, and if I had wanted I could just change the avatar to the Digital Ocean logo and impersonate DO or anyone else.
I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.
Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.
And the ban reminds me of the recent case, where DO invalidated the credits of many people within of 2 weeks with a simple TOS change.
I had to pay 5€ to even be able to add the 100$ credit from the GitHub students pack to my account (for "verification purposes"), and then they – illegally – delete it just like that? (I never got to use any of it)
On one hand, the policy change was made a year prior. On the other hand, we didn't communicate it as well as we should have. I apologize for that. I consider myself as much responsible as anyone else on that. If you ever want that account credit back, please let me know. I'm an easy find on Google, or you can open a support ticket anytime.
I was affected by this - I lost some credit that I had remaining from the Github Student Pack due to it expiring with relatively short notice. If I open a support ticket, will I be able to ask to get that credit back?
The change was made a year prior. It was not communicated via email. That email you received was a reminder. We only put up a notification in the control panel. That was the communication failure, and I apologize for it.
At no point should anyone be advising you to take legal action to retain your credit, and I would be grateful for the opportunity to review the ticket in question.
> Why did I not receive any notification? A change in contract of that magnitude should have led to me being directly notified per email or other communication methods, so that I’d actually have a chance at still using the credit.
I agree with you completely. That is precisely why I refer to it as a communication failure. I had a year to ask "Why didn't we email about this?" I didn't ask that question. I failed. I'm sorry.
I understand that you're not interested in helping, but my offer stands, and I appreciate you and your feedback. We're all just people here trying to do what we think is right. I know I fall short, I make mistakes, but all I want is to own them and make them right.
Why did I not receive any notification? A change in contract of that magnitude should have led to me being directly notified per email or other communication methods, so that I’d actually have a chance at still using the credit.
This way, you notified me right when the credit ran out, which was extremely shady, and I’ve met hundreds of others with the same issue in IRC when you invalidated it.
I can’t and won’t be able to assist you with an investigation to improve your public image, though, as I’m quite busy at the moment, and my time is better spent contributing to open source projects than discussing the wrongs of a hoster that won’t change anyway on HN.
>I had had the credit on my account for over 2 years by that time, but hadn’t used it.
>A change in contract of that magnitude should have led to me being directly notified per email
It seems to me the credit is a form of charity for students, and you had not bothered to redeem it for 2 years. Now, you seem to be implying that DO is acting in an evil manner for only alerting you to the change via the dashboard (which the DO rep in this thread admitted was poor execution on their part). I don't think you are being charitable, and I think the underlying assumption you are making almost sounds conspiratorial.
I'm not a DO customer, and don't have any affiliation.
Yes, I had not redeemed it, because, as a student, I was trying to save it up knowing I’d want to do a project where I’d require servers during summer break between 2nd and 3rd year of my undergrad degree.
The credit was invalidated 3 months before that.
With the legal 3 year validity period of promotional credit and coupons that German law guarantees, I had expected being able to do that.
I can think of at least Cloudflare (somewhat), Linode, and Hurricane Electric off the top of my head. Anybody who operates a well-known ns1 type of resolver. It's more a problem with zone hygiene than hosts, honestly.
Is that true? The nameserver I was given by CloudFlare is "nelly.ns.cloudflare.com". From some cursory searches, it seems like a large number of domains have that same nameserver. AWS nameserver hostnames have all kinds of numbers in them that seem a lot more like they're generated per user.
While you can generate unique hostnames of name servers per user you can't assign all of them unique IP addresses (at least in IPv4) as dns resolving doesn't have anything like vhosts (why should it?). So there will be overlap. Which is not a problem as long as the set of name servers for one users doesn't overlap the set of name servers for another user for a single domain.
So you confirm that a set of name server belongs to a user by verifying that its domain record has those name servers configured. You then don't allow another user to control the same name servers for that domain as long as the domain record still points to them. New users can then freely create new name servers but the domain still uses the servers assigned to by the domain owner.
What DO does is that it allows any user to control the name server for a domain if the previous owner gives up control of them. It should only do that, once the domain records start pointing anywhere else.
This same thing happens with CloudFlare & is being actively exploited. We reported it to them within the last two weeks and we were told that it's expected behaviour and that they weren't going to do anything about it.
I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme.
They told us that they wouldn't be making any changes.
FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time).
Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account.
Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account.
In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc.
Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!
According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN.
It seems that this is getting downvoted, and I want to explain why I agree with the downvotes:
The policy on that web page has two egregious issues.
1) It does not have any provisions for SPAM. I regularly get e-mail SPAM that has CloudFlare-protected links. That abuse page does not even apply. Effectively, CloudFlare offers spammers a 'pink contract'.
2) CloudFlare has a reputation for ignoring abuse, and that page effectively says nothing about whether abuse will be stopped; that page only offers to transmit the personal information of someone who reports abuse off to an abuser. This is not a theoretical; this has happened in the past, and the personal information of someone who reported a site that contained child pornography was then posted all over the net for its users to begin harassment with.
So if you find something abusive or malicious, your best bet really is not to report it. CloudFlare won't act, other than to put you at risk.
For what it's worth, in the support ticket exchange I did indeed offer to submit details of the attack vector along with proof that it's currently being actively exploited, for financial gain & at the direct expense of your customers.
I was basically told "you will be wasting your time"...
I don't care whether it's reviewed by a human or not, the fact remains that CloudFlare does not act on abuse reports because they claim "reverse proxy = not responsible" in every abuse report I've sent them even on matters that do not pertain to their reverse proxy functionality at all.
They just don't want to be responsible, and unlike any other large network service provider that I sent abuse reports to, they just try and deflect and keep servicing spamming and phishing operations instead of booting them from their systems.
I work at CloudFlare, not in DNS or on this code, and have mentioned this incident in the all-company chat. There is a healthy conversation happening there and it turns out a fix was already in the works for the underlying issue.
adanto6840 has supplied the support ticket number (thank you), and this specific incident is also being reviewed.
I will never stop being infuriated by responses like this from companies - how many more megaleaks have to happen before they realize that they need to embrace white hats, not ban their accounts, not sue them, not swat them / have them arrested, not silence them.
Why, when the author realised this was likely to be possible, didn't they get in touch with DO? Or try with a domain they knew was OK to use? Or at least just try with one domain.
They took almost twenty thousand, sent all the requests to their own server and logged them.
Domains can only be added if they are not in another account, so he added only domains which were previously deleted but still pointed to DO nameservers (hence almost certainly not being used).
And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
Well they were still being accessed by real people.
> And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
Both seem reasonable. And it does sound like the security team are actually doing something about it.
> I reached out to DigitalOcean’s team to see if they can assist in deleting the domains from my account (sadly leaving them vulnerable again) or sinkholing the DNS to 127.0.0.1. I received a very helpful response from someone on the security team and it appears they will look into it.
I assume these things happened from different departments. Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?
> Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?
If I was a domain reseller, adding my 20k domains to digital ocean just to get banned without warning, explaination or option for reconsideration I would be rightfully upset. If they didn't want people adding large numbers of domains they could just have limited the feature instead of banning people who reach some arbitrary threshold.
If on the other hand the department that executed the ban knew that the registrations weren't made by the domain owners, they should be discussing such a huge incident with the security team. That discussion would naturally lead to them knowing about the specifics of this case, unless this case wasn't widely shared in the security team.
So the options are:
1. Digital Ocean bans legitimate customers without warning or option for reconsideration; for no obvious reason
2. Big security incidents don't get reported to the security team
3. The responsible people thought that this was not a big security incident
4. This incident wasn't discussed in the security team
5. They knowingly banned a white hat hacker (who may or may not have gone too far)
Of all those options, the last one is by far the one that looks best for Digital Ocean.
Or 6, they spoke to the security team in the interim period of several hours between the addition of all the domains and when the author reported the issue to the security team.
Neither of us really know what's happened here, but the author at least thinks DOs actions were justified so I'm reasonably happy to leave it at that.
Not that I don't believe you, but you already lost white hat status in this case the moment you log traffic on 20k domains. Just do an attack on another domain you own is white hat, 20k domains is not.
You're probably right about the logging being a bit too far, it was mainly my curiosity getting the best of me. One of my big assumptions was that all of these domains were just owned by one domain broker and this wasn't actually a systemic problem with the implemented importation methodology. I also thought it would be mild because if they had been deleted from an account they were likely no longer used (or so I had wrongfully assumed).
I certainly don't attribute malice on your part, and I'm sorry if my comment came across this way. My point was (at least intended to be) that I'm not really surprised at DO banning you.
I would be careful about doing something like this on a large scale, I would actually be surprised if this doesn't technically violate some laws. Please be careful, you don't want to end up trying to explain nameservers to a judge. [edit - and the rest of us don't want that either, even just selfishly I would like as many security researchers practicing their craft as possible, but also I don't want people being punished for trying to do the right thing]
I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here.
I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well.
I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here.
I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward.
Hey Jarland thanks for the thoughtful response. I don't think your actions were rude or inappropriate (what host would see this behavior and not act similarly?). I apologize that the discussion on the topic became so negative towards DigitalOcean (this might be my fault, I was merely trying to point out why I hadn't been able to delete the domains - not say that DigitalOcean was a bad company/etc). The disclaimer I added early on I thought would mitigate some of this negativity but apparently not quite.
At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.
I think most of the "outrage" is with the complete lack of validation and the "This is a known workflow within our platform" response more than the ban. Any plans to address the issue other than reactive bans?
I think it's absolutely appropriate to sit down and talk about ways that we can prevent this, both in the short term and long term. I don't think it's as simple of a problem as some might suggest, because you've got to balance expected behavior, a reasonable expectation of convenience, and added security measures.
I don't think you can make a proper decision where you're only looking out for protection, or only looking out for convenience, or only looking out for expected behavior. I think you have to mesh all of these items together and make a change that addresses each item, and I don't think that's necessarily a one day discussion. Certainly security does come at a cost of convenience, and that is okay, but it is important not to toss convenience aside as something not worthy of consideration.
So yeah not trying to be vague, my position is not with engineering or security but with the support team. I do think we need to talk about this, and conversations are taking place, but I don't honestly have the ability to say "Yes we're going to implement _____ within ____ days" or something like that. At least not today.
Linode also does not verify domain ownership before you add a domain to their DNS. And they also use the same nameservers for all accounts. So I think they're in the same boat as DO as far as this "vulnerability" is concerned.
I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
Then you've been through their back end being compromised and customer details leaked several times. I'd been using them since about 2009ish and used them until earlier this year.
Something similar happened to me a few months ago with Cloudflare. I set up a new domain to use Cloudflare's nameservers but did not immediately get around to setting it up on the admin panel. By the time I wanted to add the domain, someone else had already grabbed it and set up some sort of spam page.
Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it.
Wonder what else might be vulnerable to this... CloudFlare seems like it may, they only have a handful of nameservers in any case.
Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.
I have an account with DigitalOcean (and several competitors) and I'm not going anywhere or moving any sites around because of this. Sure, they could have handled things better and the security researcher could have too. I don't see any malice or incompetence here, nor do I see a reason to make the effort to switch to another provider.
Where are you going to run off to? How is their security better over there? How many hours of work does that involve?
It's pretty understandable actually. Most DNS providers do similar, it's really just the admins fault for not switching their DNS, likely because they were abandoned and unused.
I'm sorry, "We aware that we make it easy for about 20k domains to be directed to a malicious host, but we're not going to do anything about it" is reasonable?
But of course, it's because Matt "was messing around with things he shouldn't be". It's all solved - we just need everyone to stop doing things that DO "don't want people to do".
If the domain is added to the account there is no PoC, it's only for domains that have been removed from accounts but still have the nameserver values(meaning the domain is not being used at this point, there's no zone file if it isn't added to an account).
So this is mostly only going to affect currently derelict domains. I'm not saying it isn't something to worry about, but I do think it's a reasonable solution.
If I buy a domain from a registrar, I can point the registrar at Digital Ocean's nameservers (or AWS, CloudFlare, etc) for my domain (by adding NS records at the registrar). Then I need to go to Digital Ocean and add my domain and records to their nameservers (via their control panel or api).
If I remove my domain from Digital Ocean, it's my responsibility to then go to the registrar and point the registrar away from DigitalOcean's nameservers. (I own the domain, so I'm the only one the registrar allows to do this. Digital Ocean cannot do this.)
Now, your suggestion is that Digital Ocean goes and verifies that I'm the one who owns that domain. But how would they do this (legitimate question)? I imagine manual verification of ownership of every domain upon creation isn't feasible for their scale. Digital Ocean could query DNS, and see NS records pointing to Digital Ocean, but this only tells them someone configured the nameservers for that domain - it doesn't imply ownership. Digital Ocean can check Whois for the owner of the domain. Checking Whois might work for many cases, but at least some registrars have the option of obscuring Whois data.
It seems simpler to put the onus of security on the owner of the domain. I should cleanup my registrar's NS records before removing my domain from Digital Ocean to ensure nobody hijacks it. I would be satisfied as long as Digital Ocean maintained a simple eviction policy (I don't know if they do) as a way for legitimate owners to add their domains to Digital Ocean's nameservers.
Huh? 99% of domains point to some nameserver they aren't contracting service? That is, 99% have invalid NS?
(Obviously when I say somebody else's NS I mean a NS they have zero reason to think would respond with correct records. Obviously not talking about outsourcing DNS hosting.)
Except this isn't even a vulnerability, this is expected behavior given the design I think, to encounter an issue with it, you'd need to leave your domain with DO's nameservers, but delete it from your account. This would only happen if it's something you're not using anymore, or if you're a severely terrible admin.
> this isn't even a vulnerability, this is expected behavior given the design
Those aren't mutually exclusive. It's certainly possible to be broken by design.
> This would only happen if it's something you're not using anymore
From the writeup it seems like it would also happen if you registered a new domain and assigned the DO name-servers but didn't immediately point it to something.
Typically when registering a new domain you add it on your panel first and then change the NS as instructed. If you didn't follow that order, well, that's on you. It's the same design just about everyone providing DNS uses (CloudFlare, XName, FreeDNS, probably others).
That's a community tutorial, however it does appear their panel is very sparse in instructions, when you add the domain you see a nameservers list. They don't even tell you that it's not pointed at them and that you should now configure your nameservers as other providers (like CloudFlare) do. I think you're right, some improved documentation would definitely be good here.
However it's the way I've always done it, because otherwise somebody else could add the domain first... it just made sense to me I guess.
I believe Vultr offers FreeBSD (and custom ISO) hosting at about the same price and offers storage servers too. Their documentation and remote console tools leave a lot to be desired though (I wanted to install openSUSE and ended up resorting to manually entering the iPXE commands at a console to get the damn thing installed).
Their support also leaves a lot to be desired. Had a persistent network problem due to a misconfiguration of their system and it took two weeks of back and forth before they really looked into it.
Are Vuktr and DigitalOcean related in any way? their websites look really similar down to the animation that shows you how to create a new droplet/server.
Looking at their DNS setup workflow[1] and API functions[2] I don't see any step where you would have to verify domain ownership - which is this whole thing is about, isn't it?
I was providing an example of a host that provides FreeBSD support. To be clear, I don't think Vultr is a good host so I'm not really sure why I mentioned them to be honest...
"Supports" is a strong phrase. Colin Percival creates FreeBSD AMIs. They work mmmmmostly as you might expect, except there's no cloud-init (instead it uses a gizmo that cperciva wrote), but AWS doesn't support it--that's a community thing.
TransIP is similar to DO (except they've had large storage for years) and they support FreeBSD. I've been a happy customer for a couple of years now, never had any problems.
Meh, it was just mild incompetence. I expect the only providers (hell, large companies in general) who never responded this way are the ones who haven't been around long enough.
Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept?
Why not stop at 10 or 20, and then alert DO to the findings?
Fair point, my relucatance to stop was mainly due to companies usually disreguarding reports unless I have strong proof. Stopping short of the full scope would've left it up to speculation as to the full amount of vulnerable domains.
It was my plan to delete the domains (or at least null route them so others couldn't take them over with more malicious intent). However my account was banned before I could do so.
Have you had any contact with DO's support or security team prior to this? You are correct that companies frequently ignore or downplay reports but it's nice to at least make that first attempt for each new find (per company). This is only ever done as a courtesy, I'm not saying you must or even that you should have here.
Remember, it's just another nerd out there somewhere who receives your report. Sometimes they are super stoked to have it and will be very responsive. This can be more satisfying than a public disclosure without first contacting the company.
That turned out to indeed be the case after their first response :/
How would it have ended if you hadn't reached out to one of their security people after being banned? They'd have given you the traffic, locked your account, and congratulated themselves on a job well done?
How does making a domain that does not resolve suddenly resolve negatively impact a user, again? The domain could not have possibly been operational before the circumstances that brought about this scenario, and there is no legitimate traffic they could possibly be receiving. DigitalOcean could certainly improve authentication here but there are dozens of authoritative services that do not, and this is not a new problem (but is a valuable reminder). I can think of four free providers off the top of my head that this trick would work against.
Put another way, the domains were entirely unresolved and offline. Then they weren't. If anything, this is a nice lesson about keeping your zone and delegations clean, and I'm glad I read it. Nobody got hacked, nobody lost traffic, nobody was impacted. If there was sensitive traffic going to a non-resolving domain, I have more questions than answers. I agree adding the full set was probably a bit much, but you can make that point without misplaced concern for alleged harm.
I'm not impressed with signing up for HN to hit someone like this and your far worse and flagged followup, particularly since it really looks like astroturfing. I guess take solace that the OP pretty much guaranteed he won't get the zones again, since they're for research.
> How does making a domain that does not resolve suddenly resolve negatively impact a user, again?
Adding 20k domains to your nameservers at once might cause issues in resolution while it catches up with itself which will affect other users using the DNS service. It shouldn't on DO's scale, but it might. This is why the whole thing isn't a "white hat" situation, the author tested without thinking of potential consequences.
dang would ask me not to import the content to the discussion, and in this case I agree. Suffice to say it was a snarky comment directed at you, deleted after being flagged.
Why run a loop 20 times when you can run it 20000 times. I guess the thoughts of the developer were 'check domain, redirect if not used, repeat till end', not 'stop after 20'.
Amazon S3 has similar problems. To host static website you need use your domain name as the S3 bucket name. Amazon does not verify ownership of your domain, and bucket names use global namespace.
Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.
Good point. It's actually generally cheaper to serve content through Cloudfront instead of S3 directly as bandwidth is less expensive with Cloudfront.
However you are going to pay for Cloudfront requests in addition to S3's. In the vast majority of cases, it's insignificant compared to the savings you can achieve on the bandwidth side. (And at a certain level you can ask AWS to waive CF request fees entirely).
EDIT: Just tested it and looks like I'm wrong. Proxying with CloudFlare doesn't help either... Looks like I may have done this with CloudFront instead?
That's not correct. The S3 bucket name is always prefixed. The format is: bucketname.region.amazonaws.com.
To clarify, you're going to have to add a DNS record either way. Doesn't matter what you call your bucket. And no, you don't need to put CloudFlare in front of it for this.
Bye bye digitalocean - account deletion request submitted 1178917. When you have reckless people like Cashan Stine (trust & safety specialist - WTF is that title? sounds like a road safety officer?) that close accounts due to a security report then it won't win any business from me or my clients.
That is exploiting the bug. That is literally exploiting the bug. In the same paragraph where you say he did not exploit the bug, you describe the peripherals of his exploit of the bug.
If he was operating responsibly, he would have applied it to a domain he controlled and provided that as a proof of concept. Instead, he ganked twenty thousand domains. That is at best irresponsible and at worst malicious and DigitalOcean (who I am no fan of, for what it's worth) has no obligation to figure out, or even care, which is which before showing him the door.
If you'd just straight up cancel an account that fast I don't believe you had any service or clients hosted on DigitalOcean. Idle threats belong on Facebook and Twitter, not Hacker News.
You don't need to make a deletion request, you can deactivate your account from your account settings, and it offers to delete everything for you. That's what I did when I wanted to close my account recently (I wasn't using the droplets I had, and liked my other VPS better anyway.)
Ramnode. Their prices are good and their support is phenomenal. I've always gotten replies to my tickets within minutes, often from Nick Adams himself (Ramnode's CEO.)
A few examples:
- Whenever a new release of OpenBSD comes out, I send a request for them to add the new install media, and they make it available ASAP.
- I switched from OpenVZ to KVM after a couple months and having paid for a year of the OpenVZ service, and they put my remaining credit toward the new service.
- I completely hosed my machine one time and had them restore from backup. Took a while because they wanted to be double-extra-sure of what I wanted, but it was overall a quick and painless process.
I don't know how things will change as they grow, but right now they just seem like a good company, run by people who care about the quality of their product and the satisfaction of their customers.
I've reported multiple vulnerabilities to DigitalOcean before and they've fixed them rapidly, credited me for the effort, and gave me free time on their services.
The difference is I didn't exploit 20 thousand domains to make flashy headlines and prove a point about something that isn't even a serious bug.
You're coming across as a shill either to make DO seem like an infalliable company or to blatantly astroturf enough to get people to hate DO (See jsmthrowaway's sibling comment).
I'm less inclined to believe the latter. I'm looking forward to transferring my domains on DO to elsewhere tonight.
More throwaway astroturfing? You say "20 thousand" the same way as your other likely throwaway account, V8OaSsoA (that is to say: somewhat identifiably) and complained about someone ripping off DigitalOcean's Web design on this account.
I'm doing math on the throwaways that are oddly attracted to this thread. You are making it very obvious that you are almost certainly a DigitalOcean employee across the two throwaways you've created so far, and that's giving me a whole lot of pause on DigitalOcean that I didn't have from reading the incident itself. If you're an employee or, less likely, a superfan, are you sure this is the type of sustained attack you want to levy? It's not making anybody look good.
I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check the authority of DNS due to the chicken-and-egg problem. The AWS way to handle this issue is definitely awesome but the infrastructure required is not worth for those companies who are providing "free DNS service" as an add-on to their existing customers. Anyway, IMO, it is your fault if you point to a nameserver but not utilizing it.
The random nameservers are only accidentally a defense against this attack. They're avoiding SPOFs, including TLDs -- you never receive nameservers in the same TLD for example. It's a reliability and scaling consideration with this accidental benefit.
Most admins don't think about a complete TLD failure. Amazon did.
Banning his account was totally unjustified since he approached them first with the issue. A less ethical person could have tried to make money or sold this off on the back market. People like him should be rewarded not have their accounts banned. For all we know he just saved DO a lot of headache in sorting this issue had it gone wrong. I really wish the response from DO on this was different.
Adding 20k domains to your account is probably enough to flag as abuse even if you own the domains. Next time the author should probably try just the one or two. Bonus points if they're their own domains.
If the service doesn't understand the issue at all, then when you explain that they're your domains, then they'll probably just tell you it's working as intended and that users should be able to add their own domains.
> The main reason I did not reach out with the theory instead of the proof-of-concept was because I believed that it would be ignored due to lack of evidence (as is my experience with past disclosures)
The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation.
Feel free to reach out to security@digitalocean.com and we will be happy to help.
"I was walking down the street and I noticed your house wasn't locked very well. So I stole all your stuff and put it in my own house.
Now I'm in prison because of this so it's really hard for me to put it back."
The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible.
If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....
Comparisons of events like this to violent crime often seem inaccurate.
A better comparison is removing 10 people's lunch money from their school lockers, maybe due to a careless sequential scheme of creating combinations, and giving the money back. And doing this before talking to the principal or any teacher.
Either way, he still should have contacted them before.
> Theft is defined as the physical removal of an object that is capable of being stolen without the consent of the owner and with the intention of depriving the owner of it permanently.
This is a really bad comparison. He didn't deny anyone access to anything or take anything at all. The real-world equivalent would be if he found a bunch of empty community/neighborhood whiteboards and drew a (trivially erased) line on them. The signs weren't there for him to use but they were empty (he didn't erase them) and the change he made was absolutely minimal.
He just configured a bunch of deleted/unconfigured domains to point them to a blank web page running on his own server. The point is that he could have done all sorts of nefarious things with that redirection but he didn't. He made a harmless change to demonstrate the vulnerability. That's what white hats do. It's what they're supposed to do. We should thank him for his efforts not lambaste him for actually doing something about the problem.
It's not very re-assuring for users of DigitalOcean to know that issues like this can put our domains at risk to "idiots". I don't care whether the guy was malicious or a nice guy or what. I care that a system I may be trusting my domain name to is trivially exploitable like this.
I believe I was clear about it. However sometimes my writing can be unclear so perhaps it wasn't properly understood (I assume you're talking about their security team's response and not Trust & Safety?). Kind of sad about the massive amount of hate for DigitalOcean in this thread as their security team really seemed quite nice. Their support was just acting on an anomaly they had seen so shrugs.
It was the guy who said "Thank you for sending this in. This is a known workflow within our platform. We are committed to always improving our customer’s experience and have been examining ways of minimizing the type of behavior you are describing."
It sounds to me like he read your email about the vulnerability, dismissed it as "on the backlog" and skipped the bit about the fact you had sinkholed lots of traffic.
We only have one side of the story though, so who knows. Lessons can be learnt.
Not quite, only one person can, so if you add first, it's yours. Typically they have you add the domain and then you switch the DNS and in that configuration you wouldn't be vulnerable at all.
It sounds like this only impacts people that weren't using their domain, eh? If I leave my NS pointing to some random provider and delete my account, I'm sorta handing off control...
At first I thought it was an issue where DO was using their own DNS servers but letting you add domains. So if Microsoft.com wasn't registered, you'd register it then all DO customer traffic would get your DNS records.
Some telcos have this. Sign up, say you're porting a number. Provide number of local bank and invalid port info. Account gets set up, telco routes their own calls to you. Port is rejected and customer service is asking for info. All while you are getting the bank's calls and forwarding them. Perfect MITM.
Yes, security of number porting is too lax. My number was once ported by a telco (accidentally in this case). They did call me and I had to confirm that I would get a new SIM card and which size I needed, but since I was expecting a new SIM card from the very same telco I didn't understand that they were in fact porting my number. They forgot to say that.
Luckily I was able to get my number back, after many calls to customer support (they managed to undo the porting when I wanted to check how it was going) and a couple of weeks of waiting.
An additional vector for this kind of attack is to create a zonefile for a subdomain off of a working, live domain administered by the same DNS server.
EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server.
This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.
I don't think that setting up custom DNS for everyone as suggested by the author is quite as simple as it sounds.
It's not enough to just come up with the custom nameservers. In order to use them in most TLDs they also need to be "registered" with the registry that operates the TLD.
So let's say you have myDNSdomain.com. You get a new customer who owns NewCustomer.com and wants to you your DNS, so you create these nameservers for them:
ns237.myDNSdomain.com
ns2323.myDNSdomain.com
In order for your new customer to be able to use those on their NewCustomer.com domain, you will need to go to your registrar and set up these nameservers. The registrar will then create the corresponding nameserver records with Verisign, the registry. Only then, the customer will be able to use the nameservers on his domain.
On the topic of having to pay for traffic to the sinkhole server: how about just closing ports 80 and 443? Then you only get a SYN, and answer with a NACK, that's far less traffic than processing a complete HTTP(s) request.
I find myself asking WW$D where $ is any large tech company with a "good" reputation. What would Google have done? Lyft? Spotify? Blizzard? Use some imagination to apply a similarly dangerous security breach to these companies.
I feel like this question yields better context to ethical arguments because it makes us aware of the cognitive biases and view things from a more abstract perspective..
EDIT: Is there a way to include plain asterisks in HN posts?
He'd still have to pay. By neutering it he means that nobody with bad intensions should be able to use the domains. It is neutered now, but if he was allowed to change DNS entry to point to localhost, he wouldn't have to pay Amazon.
I made the mistake of applying for a job there once. I was discriminated against. Despite being in a protected class, I was so surprised to be so obviously discriminated against by them. (I've interviewed a lot, I don't always get a follow up, this isn't sour grapes, this was very different.) But of course the HR people are careful to not say things that are overtly discriminatory.
But when a company insists on a VIDEO call rather than a phone call (despite asking them to do the first one by phone since I was not in a location with good bandwidth at the time they wanted the call)... and then visibly reacts to your image when they first see it, and then pretty much blows you off, despite being well qualified for the position... yeah, it's not what they say.
I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.
Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.