I had my 2FA at Singlehop bypassed by social engineering attack. They helpfully changed the entire account contact info without any notice to me, presumably from a phone call. The attacker didn't even have any information to go off other than the IP address. I only found out when I saw the server rebooting into rescue mode and luckily I still had an active management portal cookie (changing the password doesn't log you out of the portal, another big problem) so I immediately knew what was happening.
I wish there was a way to disable the "customer support backdoor" in all these kinds of services. I've started to deploy full disk encryption to all my servers now so if the attacker does manage to get into the management account the server itself is still protected from single user / rescue mode / etc.
It's tricky because a lot of customers really DO lock themselves out of a service, and forget their password reset code.
Fun story time. I use to play MTGO, the online Magic the Gathering game. Played it from beta for a few years say 2002-2004. Wanted to check it out in 2014 to see how it changed. Failed password reset online, had to call in to support.
The support guy was like chortle what was your security passcode? I had no idea. He tried giving some hints. I said it has literally been 10 years, I am never going to remember. So he went ahead and rest my password. He told me for the record my security code was "I am the nacho king", but I would be prompted to change it at next login.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
One option is to look at when the user last logged in. I would be a lot less pissed if an account that I've never touched in 10 years got compromised... I'm probably going to remember my info for recent accounts and want it to be difficult to social engineering those
People using a password manager might not ever know their password. Funny things happen with password managers where history is missing, changes don't save, keystrokes break things. We can't penalize users who use them.
That's why it should only raise a flag rather than totally stop. Perhaps the customer service rep can ask a few more questions.
It's a similar situation to someone who only ever uses their credit card to buy small amounts from their local supermarket. Then suddenly they use it to buy a flight in another country. It might be legit, but it's often not, and should suggest that customer service need to do more investigation before approving.
This is more about preventing the social engineering attacks. The example you're replying to is where the actual user logged in 20 minutes ago, while the attacker is trying to claim to customer service that they forgot the password. If customer service were looking at login attempts, they would see that it doesn't make sense for the user to not know their password, when clearly they provided it to the site just 20 minutes ago.
> I would be a lot less pissed if an account that I've never touched in 10 years got compromised...
You don't need to log into your VPS provider's account or domain name provider's account very often, compared to how often you use the machine or domain. But you don't want those getting reset more easily just because you haven't logged into them in a while.
AT&T has a security code which is "What is your favorite restaurant?" that we set a decade ago when signing up for internet service.
My wife and I have made, I don't know, 10 guesses over the years and have never been able to figure out what our response was back then.
Questions with fact-based answers are much better. But...I once had a site ask me for my best man's first name (Good! This probably won't change over time!). I filled in "Dave" ... and the site gave me an error "Your answer must be at least 6 characters."
I'm always amazed at how little thought seems to go into these questions.
My wife filled one out a few weeks ago where both the questions and answers were selected from popup menus. One of the questions was "What's your favorite summer activity?" Her answer was, "Swimming." Yeah, that's going to add about one bit of entropy to most people's accounts, you idiots.
Another favorite is "middle name of your youngest child." That answer can change over time!
I can't reply to the sister comment for some reason, so I'll piggyback on the parent.
I always fill these with awkward or absurd questions/anwers that would be amusing if a human operator ever needs to verify them. E.g.
Would you like to go on a date with me?
What color pants am I wearing?
What is the square root of insanity?
Obviously you need to store these in a password database in order to remember them, which kind of defeats the purpose. If I have to choose from predefined questions, it goes along these lines.
I've just started filling them with randomly generated strings that my password manager helpfully creates for me. Though, apparently my bank uses those answers for phone verification also, which makes answering questions like "What's your Significant Other's nickname?" awkward when the answer is "F9-#g7a2<qj"
I imagine that may make make your significant other who was previously friendly, markedly less so, if they see your "friendly value". But that may have been your point, as it may be quite a bit easier to remember. :)
Best way to handle these are to use a random string for all the answers if you can, and if they let you create your own questions use more random strings; same goes for login names.
What city was my dad born in? xGU,wT&Yvcn6vr?]#,mE of course.
Until someone says "I know my dad was born in Minneapolis, what does it say??" and the customer service representative replies "Huh, it looks like the answer is just gibberish...", "Ah! I must have just mashed on my keyboard when I made the account, sorry about that!!", "No problem, your password is now reset to foobar".
I think all account credentials related things should be handled by special support people, who were trained to understand the situation. Shouldn't be that big percentage of all requests when people need to change/remember password and can't use regular ways.
I did that to my payroll account to try and prevent this very issue.
Little did I know that it's one of those services you need the password (I had written that down at the time as it was temporary) AND these questions that are usually used for password resets.
I don't think that will ever get fixed until I change jobs again.
Security questions should be treated as secondary password fields, since they are that. Use Diceware for a good tradeoff between entropy and memorability/pronounceability or more complex random passwords and store them in a safe place.
This works well until you get to the "Our site is so secure that we need you to answer three security questions from our canned list, and they can't all be the same string" geniuses. Such an antipattern.
I think that the drop-downs are trying to prevent people from mistyping things and locking themselves out because "Accordien" doesn't match "accordion".
If you have a password manager that successfully tracks the questions, then there's no reason to need to recover the password, as you'll just track the password in the same system.
The catch-22 of these systems is that recovery questions need to be obvious, memorable and unchanging enough to the user that they are useful for recovery, while also being hard for a third party to guess/research. I feel like for the most part those are more often than not mutually exclusive.
I had something along those lines tryin to log in to mojang on a new computer. "We've not seen you log into this pc before (although I had on that IP), please answer these three security questions. Of course I don't remember so I just reset them. I imagine the new answers and the old answers had a lot in common - they were composed primarily of expletives.
Favorite restaurant! What percent of the full business name did you use? Did you capitalize all the letters the same way or in a consistent predictable way? Did you add a word to meet the minimum character/word count? Did you actually have a favorite when you made this? Is your favorite restaurant public information?
This is why I set all my security question answers to a single answer for low security stuff and random pronounceable strings (in case I ever need to read them to a support person) stored in KeePass for high security stuff.
I once tried to perform an internet banking task only to find out I had to call in by phone and enable it first. So I did, and I was asked a few security questions about my data, one of them was: what's the name of your spouse?
I gave the name, was asked to repeat it, so I did, they informed me I was wrong.
I still don't know if they had a name with a typo in the records, a maidem name, or maybe they didn't even have her name (don't remember telling the bank about marital status) and it was some sort of trick question where I was supposed to answer I'm single (even though I wasn't).
I had a similar situation just recently with an old Gmail account. Despite knowing the password, Gmail wants me to answer the security question or log in from a place I logged in ten years ago or list folder names (which didn't exist the last time I used that account) or ...
The whole point of this misery was to recover my Steam account to play a few games. Fortunately, Steam lets you recover your account if you can provide proof of ownership (like CD keys of physical copies).
I don't want Google to be the safekeeper of my digital identity.
Story time: I have been trying for 3 years to figure out what I wanted to hint at with "If it's not this one then it's the other one" as a secret question. I thought I was a clever boy not choosing the usual predetermined "what's your mother's name ?".
It most certainly is but... it doesn't work. At that time I had some kind of semantic combinations for passwords but it doesn't compute for that website.
Comcast's password recovery is pretty weak. I just did it last night. They ask for your zip code and your favorite sports team. If I have a Boston zip code there are likely only 4 options for favorite sports team.
I might have the answer for you- the security question on some of my unimportant shared accounts where a question was required is "what color is my VGA cable?"
I had a similar thing happen with my Battle.net account. I forgot to transfer over my authenticator backup code when I switched password managers last time. I had to send them a photo of my driver's license next to my face and another one of it next to a physical newspaper with the date on it. This seems like a much better process for recovering accounts that matter.
Some people would be happy to pay for a leaked copy of those photos, for use with any other company that would accept only the "face/license" photo as sufficient proof.
I think having a recent date on the newspaper prevents reuse of the images. I'm not a Photoshop expert, so maybe that's trivial to change. When I look at my support ticket history on the website, the ticket attachments are gone, so hopefully they're shredded after the support person views them.
Very easy? I'd say "possible" at best. And now you've got access to a battle.net account. Took a heck of a lot more work than asking someone for username/pass in a live chat, and what you gained access to is worth a heck of a lot less. Plus Blizzard actually does keep backups and records and will be able to fix the situation for the account owner.
I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
I think you are a little confused here. There's no way for Blizzard to authenticate those pictures, you can take literally anyones passport and just swap the name on it.
> Took a heck of a lot more work than asking someone for username/pass in a live chat
This might be true in a world without photoshop, but that's not the world we live in.
>I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
In my personal experience, they'll very rarely insist upon receiving those photos. My battle.net account isn't even under a real name and despite that I've had the authenticator added and removed several times.
It seems like a general principle that the less important something is, the better the security probably is. Steam, for example, is really paranoid, constantly asking for verification whenever it thinks I'm logging in from a new computer, bugging me nonstop to set up 2FA, e-mailing me with alerts, etc. Meanwhile my bank does straightforward username/password authentication, with the bonus that the password is case insensitive and silently truncated to eight characters.
It's not that Steam is paranoid for no reason. People keep virtual items on their accounts which attackers can sell for real money, potentially yielding hundreds to thousands of dollars from one account, and so attempted Steam account hacking is rampant. Of course, that's nothing compared to the amounts stored in bank accounts...
Not to mention an account with saved billing info could be used to sell "extra copies of games I can gift you for a few bucks" since Steam allows gifting. What, you mean you didn't mean to buy 10 copies of "DARK SOULS III Deluxe Edition + Steam Controller Bundle" for $115 each to gift to all your friends and family?
I don't know what battle.net is so I was speaking generally. Since pictures don't have security features on them it wouldn't be too difficult to photocopy your own id, change the name/address, print it out and glue it on a plastic card. You now have an Id that looks good enough for photo verification. It's a lot of work but if the steaks are high then it will be done. There's also the case when these pictures get leaked, there's a lot of people who have scans of my id. Or just photoshop the pic after its taken.
There's services on the darknet where you can buy these fake Id picture/scans as well.
To be fair, that's a game account. I realize some MMOs can have really real-money valuable characters/items, so this argument can break down, but the security should be different from an MMO and a VPS solution or a bank.
All of my security questions are passwords. I once had someone at a bank ask "Wait, your mother's maiden's name has a number in it?"
"Wait, you actually answer security questions that any of your friends can guess honestly?"
Magic Online simulates paper Magic. You buy packs to get cards, each card is its own individual digital object which can be traded and sold, card sets go out of print or have limited runs, and rare promos are released. It even has its own digital currency, "tickets", which are reasonably easily converted to cash.
Magic Online accounts can be worth tens of thousands of dollars.
And at least back when I played, if your account got compromised and all your cards were liquidated, the response from customer support was pretty much "that's too bad, shouldn't have let your account get compromised". I seriously doubt they ever added 2FA or anything either.
> Magic Online accounts can be worth tens of thousands of dollars.
It might cost 10k+ dollars to make a behemoth account, but I don't think they are worth that much. It's basically fake internet points, I can gain these for free in this very comment.
> To be fair, that's a game account. I realize some MMOs can have really real-money valuable characters/items, so this argument can break down
You want to know how bad it can break down? I imagine the worst case scenario, for so many reasons, actually happened and was mtgox.com. It started out as a Magic: The Gathering Online Exchange (from what I understand) before it became the now infamous Bitcoin exchange that was hacked[1] and massive amounts of money was stolen. I don't know for a fact that old accounts before the pivot to Bitcoin still existed and worked, but it's not inconceivable that they would. One hopes they adopted much better security compared to when they were a trading card exchange (if it wasn't already exceptional at that time), but there could very well have been a time when it was gaining traction for financial type services but didn't have good account safeguards.
1: Or whatever. From what I remember that's a story convoluted and with enough conspiracy theories it's worth a movie.
Are you sure? Wikipedia[1] seems to indicate it at least got into a beta, but it also implies that the domain was what was reused for the bitcoin exchange, not the code base. That said, the references on wikipedia point to the Internet Archive, which shows a placeholder page that says it's in beta, but there's little there to indicate there was ever anything working. It also says it was used to advertise another card game later, but that was just a link to another domain.
I guess that's a long-winded way of saying you are probably right.
Then again, bitcoins were worth so little in 2010 that I could imagine the account security on exchanges back then being comparable to hobbiest exchange sites, so the spirit of the comment may be valid even if the specific example falls down. :/
I recently changed my phone support password at work to "aaah, f*, I'm not sure is it.." after listening to all my previous support calls and realising that was what I answered with 9/10 times. I suspect its only a matter of time before someone else accidentally guesses it. Its only for my regular user account, for my admin accounts I need to get another domain admin to reset the password, there is no process for anyone to exploit, just an audit every month.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
And apparently I am the nacho King.
What are the chances that you'll remember it in the future? You're already two years into remembering it for the next decade after you found it out again.
Very much this. As a long-time Namecheap customer, this thread caught my attention, but my layman's conclusion is that it doesn't really sound like they did anything worse than any other host would have done.
As someone who forgets his passwords on a regular basis, I'm kind of glad that there's no such thing as perfect security...
I would prefer being stuck without being able to login to my server, compared to potentially having customer support allow someone else to access my server.
The answer to "I don't know my password and I don't know my security question/answer" is, "Sorry, for security reasons we can't help you access this account, you'll need to create a new account." This isn't a problem for banks, why is it a problem for tech companies?
> This isn't a problem for banks, why is it a problem for tech companies?
Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID. That would not really work for most online companies.
Other methods which involve sending in copies of ID and/or letters signed by appropriate notaries would fail due to human engineering too because your average tech company isn't going to have people sat ready who are capable of accurately verifying this information.
The other problem is PR: due to the lack of another option the average person who is locked out of their account will instantly turn to twitter/facebook/any-where-else-they-can-post and scream as loud as they can that they've been mistreated by company X. Many other average persons will take this at face value without checking the fats and start avoiding company X, or worse bombarding them with communication in support of the inconvenienced user.
> Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID.
Not always true! Less than a month ago I needed to login to my Wells Fargo account. Unbeknownst to me, they had been doing some 'upgrades' and there were some glitches. After a frustrating period, I decide I'd just go to the physical branch 1/4 mile from my house and get this fixed!
On site, the bank personnel have access to exactly the same system that I did. (At least they knew there were glitches and sorta how to work around them.) I had two accounts, one for a credit card that I rarely used and my mortgage. Turns out, if you have a credit card then the new system requires a piece of information only found on the physical card - the onsite employees couldn't get around and neither could their call-in tech support!!!!
Point is - for log-in purposes - don't assume going to a physical branch will be any more helpful!
Since I didn't have the credit card with me ('cause rarely used) I canceled the rarely used credit card and was able to login shortly thereafter.
Banks don't try that hard. One of my bank is happy to resend me a password by snail mail with an account ID reset by phone.
Also, rechecking the ID of a user can be as simple as asking for a new token payment by the same credit card as used by the account. It's not infailable, the CC can be compromised as well, but it should be way better than what we have now.
Actually, this could be an interesting and lucrative side business for banks, identity verification. You could have varying levels of verification, requiring varying levels of authenticating documentation and numbers of employees to review and vouch that could then be used to provide a certificate of verification for a service.
E.g. namecheap.com generates verification ticket item requiring valid identification and SSN that Bank of America then uses to verify your identity for $30, and vouched for the identity. Meanwhile Goldman Sachs generates a verification ticket requiring much more strenuous authentication, and the bank charges $200 for (with increased insurance, etc), which satisfies the much higher validation standard the Goldman Sachs requires to authenticate you for your ritrement account with over $X in it, etc.
With MTGO above I had maybe $500 in virtual stuff on my account. As the gatekeeper I'm not sure that would go over well.
With servers a similar thing. Say my only copy of a database is on my VPS. May have a business value of $50k. Can't really just say no unconditionally. Need some process to unlock..
There are at least two such services in Germany WebID[1] and PostID[1] (not to be confused with the older PostIdent that requires identification at a post office).
I haven't used PostID yet but with WebID you basically have a Skype video call where you show them your ID.
I have a few bank accounts with banks that don't have branches. To "verify" your Id they ask you questions from your credit report - which can be problematic. "What was the payment and term on a loan you had 5 years ago?" Fuck if I ever knew what the payment or term was, I didn't care when I took out the loan, I had my own payment schedule (I think if you can't pay back a loan [with the exception of a mortgage] in a year or two you really can't afford the loan...). Some of the stuff I just plain can't remember!
The best was when they asked me which model of car I had owned... and listed two cars that I had owned... I could only select one.
These records can be flat out wrong too. The DMV associates a car with my address that I don't own, for example. I think this happened because the owner never changed their address with the DMV. Or someone could have just fat fingered something which gets populated to other databases with data sharing.
I agree. With banks, if you need to prove identity remotely, you need to get a medallion signature and a notary. Takes time and money. I assume that web businesses would be happy to have to do that in return for robust security not easily broke via social engineering. I think it would even be a competitive advantage. If you want cheap and easy (and insecure) then you can use a competitors offering.
Well you can go to the bank with your ID if everything else fails
Of course that might not be even necessary as there have been reports of people withdrawing money or wiring it somewhere with not even that (but the bank has legal responsibility)
We have had this at Amazon AWS. We had 2FA, one phone call was enough to disable 2FA. The only thing they asked were the last four digits of our credit card.
Too bad you can't ask for the CVV code, and run a dummy $1 transaction. Anyone could have the last four of the card number, but the person is much more likely to have the card itself with the CVV.
Disclaimer: I use AWS extensively. Please do this.
If you're using the payment method as auth, you should be locking it out as an auth method for X days after a change has been made, and emailing/SMSing the contact regarding the change.
Yeah, that's the only thing amazon seller support asks me for verification as well. Either last 4 of credit card or last 4 of bank account - which is even stupider - anyone who you wrote a check out to has your bank account number.
A reporter at Wired had his digital life destroyed (including personal mac wiped) by social engineering, using little more than info from one account (last 4 digits of credit card from Appstore account) to socially engineer Amazon customer support - http://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/
Edit. Had some details wrong. It was a reporter at Gizmodo, and basically a collection of data was gathered from various online services (mailing address, last 4 digits of CC) to ultimately social engineer his Apple account and remote wipe his computer. A number of factors were involved, but ultimately he was done in by not having 2FA on google, and not backing up his machine.
I mean from a system level perspective. The trouble is that they have policies that are optimized for people shopping on a site with a money back guarantee not for hosting critical infrastructure.
Lastpass will let you remove 2FA using the attached email address. I do not believe it requires you to enter codes from the 2FA to disable. It makes me uncomfortable despite my email having two factor.
Another vote for gandi.net here. I transferred all my domains to them during the past years and have never been happier.
Security is almost bomb proof with IP restrictions, GPG keys, 2FA and one little checkbox in their settings I like a lot: "This setting allows you to authorize or disable password resets from the login screen."
I think a good practice is to try to social engineer your own account and see if you end up getting it.
I agree on the 'customer support backdoor,' with some caveats. I worked for a small IT Services provider, and we a policy of "don't do dangerous stuff that an unknown party asks for unless a known party verifies it." But sometimes someone gets fired, and there are no known parties, especially with cloud services.
I realize there has to be a way to work around 2FA, for situations like loss of device, terminations, etc, but that should have some known policy way of verifying (say, you must send a notarized or local equivalent letter, or appear in person somehow) identity, especially for a cloud service.
(Totally incidental and you perhaps won't see this, but, r1ch, Conflict Crusher was instrumental in 15 year old me learning how to mod TA, which led me to learn how write BOS scripts, which led me to realize that I liked this programming thing, which led me to my current career/way to support my family. Thanks! :) )
Conflict Crusher, wow that takes me back! Back to that awful Visual Basic UI with random windows metafiles for backgrounds, what was I thinking.. :). Great to hear that I helped you on your path to become a programmer! Modding games inspired me a lot too, especially Quake and TA.
Even better, have the option to disable tech support and get an alert if a reset is requested with the metadata related to the party making the request.
Fairly common for enterprise type apps to have a list of preapproved contact points, not on the list they won't even talk to you. Maybe other places could take this up... not foolproof, but at least adds another layer to the challenge.
I've personally gotten past lists like that a number of times simply by stating that person is not on staff any more, I'm their replacement; legitimately did replace the old point of contact.
Getting people to do stuff on the phone is easy a huge amount of the time.
That's crazy that customer service was able to turn off 2 factor!
Godaddy has gotten really good at preventing social engineering stacks like this. I use 2 factor authentication for my account and customer service can't event talk to me till I give them the code. Don't have that? Need to send them my drivers license and other proof to get the account reset.
The problem I think, is that Godaddy only does hosting so has appropriate controls in place. Since Amazon does shopping and hosting and did the former first they have policies designed for the customers that are shopping.
I tried to call a few times and unfortunately kept getting disconnected from my VoIP line. I ended up using their live chat and the person I spoke to was very quick in recognizing it as a social engineering attack and worked with me to reset all my information.
Following up I had to reset my passwords a few more times to ensure the attacker's session was properly terminated. They identified the IPs accessing my account but they were all proxy IPs from Europe and Israel so not much could be done.
I still have the support tickets the attacker made, and I feel their support staff should have recognized it as a compromised account (they changed the name to "John Smith", bad spelling / grammar, asking to reset the root password, accidentally "lost" their private key and need password logins and root SSH enabling, etc).
Aside from this incident though I've been pretty happy there, the hardware is competitive, uptime is good and the network is solid which is all I really ask for with a server provider.
My hobby: role-playing how I would respond as the CEO if my company was getting skewered on HN. Here is my version!
---
Disclaimer: I'm [not] CIO @ Namecheap
We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.
In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.
We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>
I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
^^ we'll take it. We've been responding for the last 1+ hour to things in real time across several social networks, so we're a little rushed. But thanks for the role play :)
For a "what not to do", have a look how (the CEO of?) FTDI responded after they were caught intentionally "bricking" chips that were detected as counterfeit by the Windows drivers.
Go out right now and get the book "Crucial Conversations". It is BY FAR the best book I've ever read on this kind of thing. It is simultaneously the best relationship book I've ever read and the best business book I've ever read. It goes through the basic principles for handling these situations in an easy to understand way.
> 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
This is not the correct solution. What's to prevent the next new person from making the same mistake?
If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.
This is the kind of 'learning experience' that becomes part of corporate culture and future training. When someone says 'Why bother with all this?' the response can now be 'Read this writeup of how ONE person NOT doing this correctly cost the company a ton of marketing $$$ and STILL left us with a black eye with our more technically-savvy customers.'
And how many people at Namecheap do you think aren't aware of this by now?
But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
That's part of the whole issue. It wasn't simply an issue of retraining. The entire company is well aware of this issue and is using it to improve, not simply to reprimand a single person.
Putting #1 as #1 looks like bitter deflection. You do it elsewhere in the thread too, saying that lack of 2fa on the email account opened the door to this. You should be well aware both that most security issues end up being perfect storm of circumstances, and that attackers can and will target multiple points in the chain. Relying on #1 as the spearhead of your apparent defense here is tantamount to admitting that you are relying on the security of people's email accounts as part of your own security process, which is wild.
You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.
This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
With #3 - ideally your systems should not allow you to break established procedure. Mitigate the risk by not giving the support staff tools to shoot yourself in the foot so easily. This could be achieved with peer verification or some other mechanism (lots of ways if you think it through).
Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".
In this case, the support staff wouldn't even needed to be trusted in the first case.
This is a great point. The software should be modified to not allow the employee to even make any modifications to the account without the correct credentials.
Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
People make mistakes. The customer support person was probably just trying to be helpful and not fully aware of all the ramifications. This is unfortunate but presumably there has been some retraining.
Note: I have no direct or indirect relationship with Namecheap at all.
I guess take Matt up on his offer where he said this: "Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place."
I love namecheap but 5 sounds like victim blaming. Come on.
EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.
EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.
EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
Low end hosting doesn't generally have backups, because it's well, cheap. Extra overheads make the price increase, then you're not cheap and can't compete at that end.
Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
Furthermore, is Namecheap authorized to copy their clients' data by their terms of service? If not, automatic backups may bypass totally-reasonable expectations that other users have. Backups can potentially be a threat vector, for example. There might be many reasons why one of Namecheap's clients might say "you copied this data?! and now I have no control of the environment the backup lives in?!"...
An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.
That's true, although we should be fairly concerned about a company using very cheap hosting on VPS with no form of encryption storing anything sensitive. You may also be breaching PCI DSS (fwiw) doing that.
In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.
Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.
> Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.
Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.
Imagine you just lost two servers you can't replace, or you're a potential customer reading this thread, and are afraid of the same.
This is what they read as the company's response to this loss:
"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."
Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?
Now consider this reinterpretation of the statement:
"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
I am not sure why you would respond to an accusation about victim blaming by reiterating the exact thing that caused the accusation. You might want to reconsider continuing this particular aspect of discussion for PR reasons. It's not an argument you're going to win.
Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?
The issue is that Namecheap was the one that fucked up here, and now is not the time to emphasize "you should really be prepared for us fucking up in this manner". It's victim blaming. It looks shitty. The argument I refer to isn't "you should have offsite backups". The argument is that Namecheap is implicitly victim blaming, and they're not going to convince many people that they aren't.
Eh.. I don't really agree that this is victim blaming. But then again, I find that I disagree with most uses of the phrase "victim blaming". Pointing out that somebody did something sub-optimal, while still acknowledging the mis-deeds, mistakes, etc. of other parties, is not "victim blaming" in my book. It's just pointing out the truth.
I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.
I don't know where any of you live, but saying recklessness is "victim blaming" sounds like a first world privilege. Yes, in generally in the first world, screaming for your rights can actually work.
In other worlds however, the problem is usually too widespread. You might get a lot of attention, comiseration, etc. but in the end, being reckless goes against survival. People who point this out should not be shushed for pointing out what you need to do to survive.
Its amazing to see that this "victim blaming" mentality is growing in Brazil. Violence here is out of control. You might get mugged/shot/kidnapped for no reason, or not displaying any wealth. Having been kidnapped myself, and chatted with the kidnappers, they do look for signs of wealth before pouncing. Therefore, yes, the victim does has an ounce of control over their risk and it's not wrong to point that out.
It does not solve violence, and attackers will just look for other victims regardless of their reward estimate. However, would you tell your children not to not show affluence/vulnerability in shady places just because you don't want to "victim blame"?
>I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid?
Yes, this is the textbook example of victim blaming. Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged. I am admittedly not the best at describing this because up until recently I had the same thought process as you. I would encourage you to find better explanations than what I can offer and be willing to have your beliefs challenged.
Yes, this is the textbook example of victim blaming.
Then "victim blaming" is a meaningless concept and we should quit using it. Because if I choose to do something stupid, I do bear some responsibility for the outcome, even if somebody else violates my rights. That doesn't absolve the other party of course, which is my point. That is, you can blame the perpetrator of a crime while also pointing out that the victim could (possibly should) have done things differently.
Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged.
It isn't "blame" for the actions of the other person. Why wouldn't you point out the stupidity of knowingly putting yourself in a dangerous situation?
Victim blaming is used to vindicate a perpetrator of a wrong doing. That isn't being done here, nor in the mugging example.
You can say a victim is stupid without giving any vindication to the person in the wrong.
Namecheap are saying "be responsible for your backups, but yeah we screwed up on our security policy" - They are seperate things, that the victim here has conflated, but are seperate problems (in regards to namecheaps offering).
It's not victim blaming. It's simply a reiteration that it helps to have this in place if you are specifically opting to rent/lease a server that does not offer it.
I don't believe you properly understand what victim blaming is or the argument I am making here, hence the reason I recommended you and your CIO don't bother continuing trying to discuss this. You're giving people reason to dislike Namecheap for no gain to yourself and your brand.
Agreed. However, is this messaged anywhere in your documentation or setup instructions? Do you provide instructions how how to set this up with a 3rd party or list of 3rd parties?
Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.
"Better be safe than sorry" - namecheap for when you lose your stuff on their services.
I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.
That comment I made refers to data integrity across platforms. You should be smart about data, no matter where it arises, if it is important to you.
For example, let me give you a look at what my Windows hard drive looks like.
My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.
Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.
So yeah, I repeat, better to be safe than sorry. Your mileage may vary.
Re: Your edit - just a note, it is very clear here that in points 1-4, Namecheap has acknowledged a mistake. That's exactly why there was a lot of training (and retraining) internally to ensure this mistake does not recur. But we do acknowledge it is an isolated incident. That doesn't mean it's not less important - we're fully aware of what happened here and it will not recur.
Let's not throw personal attacks at me (and the tongue-in-cheek "congrats for being promoted to executive!" comment). There's plenty of remorse and there's plenty of acknowledgment of mistakes here. That said, as we acknowledged elsewhere, we're responding to the matter across several different platforms and specifically say we're rushed in trying to get out some basic insights behind what happened and transpired. A more well crafted blog response for all to see (this time from the CEO) has been published to https://blog.namecheap.com/social-engineering-issue/
It's a common practice. I don't see how it is personally offensive to you. My description was for the CIO and execs in general, yet you insisted they were for you. So maybe you should stop making tongue-in-cheek comments. In fact, I am moving my domains off of Namecheap because I don't think Namecheap is very good at handling customer relations, particularly on social media.
(the reply link didn't show up before, so I don't know if you saw my response posted right after this)
To be fair, your third edit was only for me. And that was the only comment I was replying to.
As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).
It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.
Regardless, to fix this PR disaster, I suggest you add some strong and perhaps just as importantly modern security features in the future that would regain you good will with HN types (and therefore everyone else).
And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.
The offer of a free year of hosting is nonetheless a paltry joke. The high road here is to acknowledge that customer may choose never to host with you again and still go above and beyond in attempting to make it right to them, e.g. by offering a full refund for the last year of hosting they'd paid for or the like.
Tell me one registrar where you can be sure that this will not happen and I will move my domains today. I wouldn't even care if I have to pay 100$ per year for a domain.
Not trying to be snarky, but the biggest lesson here seems to be "don't operate without off-host backups". Cheap VPS providers don't typically offer that sort of thing as a standard feature. Even when they do, the backups would be on the same infrastructure, and easily wiped from the same (compromised) console.
You could have just as easily lost all the data in an accidental way, with no malice or 3rd party involved.
That said, I do empathize, and it's disappointing that a major player like namecheap would be so easily socially engineered.
Yeah I'm pretty sure the data loss is not a namecheap specific problem. Speaking from experience I lost an entire website (zero backups including app code - it was a horrible technical debt perfect storm situation no VCS etc) in the huge TigerMate inmotionhosting hack back in 2011 so this is nothing new. It was a PITA but the site was nothing special so I wrote it off as a valuable lesson.
Example of on HN expecting everyone (newbies and all) to know who you are. This happens with DANG and SAMA comments as well. Back when PG used to comment also happened. Look at their profiles, really no explanation of who they are here:
Why is it so hard to put info in your profile or to put a footnote in your comments for the newbies? Would you have your business act this way? Reply to a person's inquiry and not say who you are and what you do? Of course not.
I've reviewed all 16 of your comments on the page and beyond sawing a single person at Namecheap didn't follow policy and blaming the user in question, I don't see anywhere that you've stated there's an issue with controls.
Am I missing something, or is Namecheap saying they didn't do anything wrong?
Was interesting that the victim said they used 2FA for everything they considered important, but not email. I guess their email provider doesn't provide it?
I don't see how that can be the biggest lesson. Someone at the service provider bypassed their own protocols in order to hand control of the system over to an unauthorized user. Even with local backups he would have needed to restore the servers because Namecheap royally screwed up.
Yes, you should always have more backups than you need. But wouldn't you be moving to a different provider after something like this anyway?
I think another lesson in security is that 2FA are sometimes completely broken because of real use-case where the user lose his second authenticator device. This is one of the reason I'm really iffy about setting up 2FA on my own accounts.
Did you make it through to the part of the article where he says "my biggest personal lesson is to make off-host backups"? Not sure why you're making this post.
Social engineering in tech has been around since before Kevin Mitnick publicized it and went to jail (unjustly). Why do we keep making the same mistakes over and over again as an industry? We NEED UNIFORM security standards with ALL trusted companies with customer support, where we have tiers of support, and 1st tier doesn't have any access that could compromised security. Similar to ISO standards.
This means there can't be any "impedance" mismatching that can be used from one service to another. For example, one company gives out the last 4 digits of the credit card, and the other uses the last 4 as security info.
What we need is a uniform security standard and training for ALL customer support personnel so that you can use Apple to break into Amazon, or Digital Ocean or Namecheap. The staff need to be trained to never succumb to social engineering ever, and in fact make it impossible for 1st line support to reset anything. Have any security information get passed up to second tier support who are extremely well-trained. Etc.
And have this standardized so that there is incentive for customers to look for this certification so that we don't have to keep suffering the same mistakes over and over again.
FWIW, I've only read Ghost in the Wire, but all he admits to there is that he broke into systems and download the source code för various systems. Nothing destructive, nothing he made any money off.
(the book is great, by the way, highly recommended)
Ever heard of a plea deal? He was obligated to 'admit wrongdoing' in order to get a reduced sentence. Sure, he could have stuck to his guns, but, jail has very bad internet.
I meant in his book and speeches he has since given he has admitted doing more or less what they accused him of doing.
I'd definitely agree that the solitary confinement was cruel and unusual. I'd also agree that the law wasn't mature enough when he was charged so he was charged with proxy-laws.
But ultimately he did do what they said he did, and some of it was pretty messed up. He would definitely be charged today with computer crimes (or generic crimes) for many of his exploits at the time.
I would imagine owning up to his "crimes" helps his business / brand. If he disclaimed all credit, he would be seen less an an expert in his field of work.
Do you know anything about the case? They said he shouldn't even use the phone and spent time in solitary because they said he could cause a nuclear war. The entire case against him was horrifying to anyone that cares about human rights.
As a business, it costs more to piss off your customers regularly because they can't get into their accounts than it does to refund the rare victim of social engineering.
Sounds exactly like what I would expect from a lazy, shortsighted business who believed that a successful social engineering attempt wouldn't cost them more money.
I hate to break it to you, but "uniform" security standards that are out there in the open would be like a whole can of worms.
That is like showing someone "here's a lock and what's inside of it."
In time, someone will pick that lock.
Uniformity is what you don't need, nor would you want to know the nuances of how security and privacy are handled at a company so that you know exactly what holes need to be exposed.
You can't standardize security. It's way too risky.
You're basically advocating for security through obscurity.
A standardized process design could be carefully examined and improved to plug the holes, so all you're left with are implementation bugs. You'll never get there with a thousand disparate processes: they'll have design and implementation bugs, as well situations where system compromises data used to secure another.
Plus, standardized processes would allow implementation of more expensive processes. For instance, you could have a higher-grade fallback "prove who you are" process that involves going to some designated office in-person with all the right documents. Not even Google would pay to setup such offices in every city, but if Google, Amazon, Online Banks, etc. all would use it, it might be possible.
That's exactly what I'm advocating. Standardization would expose millions of people to policies that can be exploited in time. It's better for all of us not to know.
I know, this is an issue some would disagree with. I do not think it's safe to standardize at all.
"Here's a lock and here's what inside it, but you still can't break it because you lack a separate secret of no mechanical relevance" is the only way to make a system secure. If it is possible at all, you WANT a system where knowledge of all the mechanics do not allow people to crack the safe.
See encryption. A secure crypto mechanism is not vulnerable to disclosure of its mechanism. The key is the secret. The mechanism is not. Having a secret mechanism that relies on its own secrecy means your main weakness is someone explaining how it works to the outside world.
RSA is perfectly well-known. So is rot13. rot13 is crap, disclosure or not. RSA is not as crap, disclosure or not.
Of course you can standardize security. This means that the same authentication methods would be used across the industry, and the agents would be trained to not leak those details through social engineering, etc. It would mean that there would be standards with respect to what information tier 1 agents have vs tier 2 agents, etc, with proper separation of duties, so that poorly trained tier 1 agents wouldn't have the ability to be socially engineered.
Your method of security through obscurity simply doesn't work because hackers will figure it out and exploit impedance mismatches between vendors.
So he is using 2FA for all the important accounts but for the most important one (the email which he used to register an account at all these services) he's using a weak pw and no 2FA? Am i missing something here? Yes they did not follow protocol but why would one not use 2FA for such an important email addy?
I thought the same thing! Although he is definitely right to complain about Namecheap, the biggest takeaway is, your email is the most important service you have on the internet:
> I’m pretty careful to use 2FA for any service that I consider important
The email being compromised opened the door to his email being compromised. The door to his Namecheap account being compromised was apparently already wide open.
No, OP didn't have 2FA enabled on their namecheap account. It was namecheap's fault for improper handling of the social engineering attack but OP could have protected themselves by having 2FA
I remember when NameCheap launched the "security notifications" feature where it would email you whenever there was a login or activity on your account. I noticed that logging in on the mobile site didn't trigger any emails. When asked, they replied that the mobile site was just a beta version.
It doesn't help that the front door is securely locked when the back door is not! :-/
Crazy. I have trouble figuring out how you'd even program it that way. Is it not obvious that security notifications belong in the authentication layer that everything uses, and not in platform-specific front-end code?
At the end of the post he seems quite snarky and bad mouthing namecheap's security for things that aren't even their fault or even security issues.
> The VPS panel allows full serial console with only a login/password (no 2FA required or possible)
Yea that's because it's a serial console, if you want 2FA or something then that's a matter for your operating serivce. A serial console is literally like you're plugged directly into the machine.
> They send out your VPS panel login/password in plain text emails when you sign up, and when you reset the password. So if you ever failed to delete one of those emails completely and someone gets into your email…your totally screwed…
To be fair this is pretty standard. It's your job to secure your passwords once they've given them to you. If they're storing it in plain text then you can complain but this basically sounds like you're complaining that they're not encrypting emails. Sure they could only show you it once when you boot it up. But since this action was done via customer support they would have to give you the password some how. To your email address the most secure other than the chat which can be by an attacker like it was in this case.
> VPS can be irrevocably wiped within seconds without any prompts or confirmations just by the click of one button; whether the server is turn on/off it doesn’t matter.
This isn't a security issue. A UX issue yea, but it's not even that big of a deal. It's in an area you won't be that often and where you know you're doing admin related thing.
> They keep no backups, even to cover hardware or security failure.
This isn't a security issue. It's your job to back up your stuff not a VPS provider.
> And of course the icing on the cake is that they ignore 2FA and are willing to send out your username/password to anyone that asks.
> Yea that's because it's a serial console, if you want 2FA or something then that's a matter for your operating serivce. A serial console is literally like you're plugged directly into the machine.
Which, once the machine is up, is basically just exporting getty and exposing the system login prompt. Why couldn't the system login prompt require 2FA. Mostly for historical reasons I'm assuming, but just because it's serial doesn't mean it isn't just some software on the other end.
What's this crowd think of this idea for solving this problem?
1) Offer an option to opt-out of all automated account recovery. If set, no more email resets, support PINs, or similar. This would be targeted at people truly care about security and have no issue with "forgetting passwords" (i.e. you use a password manager and you're not an idiot about backups).
2) Offer in-person, manual recovery. To participate in this you'd need to pre-register with full contact details (name/address/etc) of the valid people who could use this feature. The person would have to physically come to the office of the company, present two (or more) forms of identification. To add further security, you could add a mandatory wait period between initiating a reset and it taking effect (ex: min 7 days). That way a combination of fake ids and social engineering could (in theory) be stopped by getting an alert that "You initiated a manual reset of your XYZ account. Did you actually do this??"
EDIT: For #2 you could also add a non-trivial fee (say $500) that would need to be charged and cleared in advance of the person showing up.
We do something similar at Silent Circle. In your recovery options, there's a page with a high-entropy secret key and a QR code that you can print out to use if you ever forget your password.
There's also a checkbox that says "don't ever recover this account" (i.e. the "I have a password database on Dropbox") checkbox. Checking that box actually disables password resets on the admin interface, so your account is pretty much dead if you lose the password.
OT question about Silent Circle: I was just looking at your website, and I noticed that you cannot ship to PO Boxes. Is this a security feature (eg, no government knowledge of the recipient) or a logistics issue (eg, FedEx/UPS can't deliver to PO Boxes). I would imagine it is pretty hard to get service for your SIM card without revealing your identity to degree.
I'm not actually sure about that (I assume you're referring to shipping a Blackphone?). The Blackphones are a semi-separate division that I don't have much contact with, unfortunately.
weaker form of #2 would be a "send a registered letter" reset mode, in various forms (e.g. here in Germany there is a type of letter where you have to go to the post office and show matching ID to send it). Or require a (possibly named) notary to validate the request, or something along those lines.
If you announce sending the reset letter there is also time for the account owner to prevent the reset unless an attacker manages to isolate them from all notification channels.
This is more or less how NearlyFreeSpeech works. You can set what level of identity proof you need to reset your password, or disable reset entirely. They seem pretty serious about it!
The most significant security problem with Namecheap is really this: It only takes a 4 digit PIN to perform any action on an account through live chat (which seems to be outsorced to Eastern Europe), even if the account is protected with a 2FA... All you need is the PIN, and an attacker can do anything to the account.
That's not good verification. It takes a couple of minutes to produce convincing fake ID scans, and they aren't going to have anything to verify them against.
And presumably they wanted you to send those photos to them as an unencrypted email attachment, right?
As someone mentioned above, they publish a gpg key for sending this data.
As for fake IDs, yes, it is certainly possible to create them. But when it is so much easier to socially engineer your way into another service like Namecheap, it creates a disincentive for going after Gandi (and other similar hosts).
No security measures can be foolproof, possibly short of sending someone to your home to take a DNS sample, but at least they're trying for a better solution.
Other comments in this thread have indicated that Gandi will take GPG-encrypted emails have have published their public key for this purpose: https://wiki.gandi.net/en/gandi/documents
Faking ID scans adds a whole layer of law enforcement on top. I'm uncertain about the situation in the US, but in germany the fake itself is punishable by law (up 10 ten years). It also creates more traces to look at and creates work. You'd also need much more information to create a convincing fake id scan of your intended victim. It's all about increasing the amount of work for the would be attacker.
But producing fake scans isn't covered by this law, scans aren't even an official document. In fact, it is illegal for a german company to ask you to send them scans of official documents.
> You'd also need much more information to create a convincing fake id scan of your intended victim
To fake a good enough passport scan you'd need your victims name. That's all the rep is going to have.
That was a few years back when another well-known registrar only required the last 4 digit of the customer's credit card, and would even help them guess if they didn't remember.
I sent the ids by fax (yeah a few years back, I still had a fax machine).
I thought asking for id's + phoning on the number listed in the whois database was a good cross check, especially back then.
Most will require a password reset email, I'd say that's significantly better than asking for ID scans.
Edit: Since I'm getting some downvotes I'd really like to know how one could possibly argue that asking for ID scans is better than email resets. You can't really forge the ability to receive email at an address, but you can very easily replace the name on an ID scan.
Even if the email account is compromised it's still stronger proof of identity than ID scans.
An attacker can't just pretend to be able to read your email, such ability is too easy to conclusively prove. To be able to read your email they need to hack you somehow.
But for a fake ID the attacker only needs to throw your name in a PSD and they're good to go.
You can get AWS customer support to reset your password if you know the last 4 digits of the credit card used to pay for the account. This is the same info that's printed on any credit card receipt.
If they have your bank account number for whatever reason you can also use last 4 of the bank account number. Your bank account number is not secret by design and most people only have one.
They publish a GPG key to use for this purpose, which puts them leaps and bounds ahead of most other hosting/domain providers who do identify verification.
I've had process issues like this with them; their CEO is responsive on email/Twitter and the email alias on this page: http://www.gandi.net/no-bullshit gets things fixed. They're not perfect, but they are quite human.
>Without context, anybody could say the same thing about anything. Care to share more?
Sure! Gandi received an abuse report regarding someone using one of my domains to scan for open dns resolvers. I informed Gandi that there was nothing I could do about this and expected that to be the end of it. Instead, they suspended my domain and started demanding that I send them ID proof.
As I needed the domain back I sent them a redacted photo of my id card, after which they demanded to see the full id. I decided to terminate my relationship with them.
I feel that this was absolutely unacceptable and likely unlawful behaviour from them as they had absolutely no need for that information. This wasn't a whois dispute.
FWIW I painlessly transferred five domains away from Gandi (ironically, to Namecheap), and I was never prompted for anything like a scan of my passport.
Didn't Cloudflare just launch a domain registration service for its high profile clients which can't afford downtime due to a support worker making a mistake like this?
Does Namecheap claim to take backups? Even if they do you should be taking backups as well if you care about your data.
I do agree with more login forms needing to support 2FA. At this point I wish almost everything did. It is a bit more hassle but is easy to manage for me at least.
> but on the way out decided to click the conveniently located “Re-install” button next to each VPS. This instantly wipes everything and installs a new OS. Again this action requires no 2FA authentication or any other form of confirmation
This is the same for DigitalOcean. I'm always amazed that clicking "Rebuild" or "Delete + Scrub Data" doesn't require _any_ confirmation at all.
Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place.
Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch.
What do you use to tell whether a chat session is a genuine user or someone successfully using a social engineering attack against your chat operatives? If the answer is "nothing" then you can't know if this is an isolated event or how many of your chat sessions go without a glitch.
Parent's point was, how do you tell whether or not your rep was socially engineered? Only some mistakes get complained about. If you don't have such a method then your "10000 sessions a day without a problem" number is fantasy.
Have you considered making this something that can't be done manually?
First of I can social engineer one of your staff. Regardless of how much you train them. I could also bribe your staff or try to get you to hire a plant. Yea that last one is far fetched but just making a point that as long as someone can manually do these things someone will.
No, it is never okay to compromise security just because "it's good when you forget your password". There should be no way around this for any reason; a bypass via SE or any other mechanism is a failure of the company, end of. If you forget your password or you do not have your 2FA/security questions available, tough, you should lose access.
For the sake of "workarounds for legitimate users", that just translates to "a security hole".
Special pleading is a logical fallacy and if it works on a support rep, that rep has failed at their job.
Since the CIO (and another employee) are here. Why are you not offering support for Google Authenticator? Last time someone asked for it was 2 years ago[1] and still no sign of the feature. Cheap prices are good to have but combining that with more security can only add value.
Its been months since I wanted to write a detailed summary, but the notion that "namecheap is hackers best domain registrar" is not valid anymore!
About year ago I noticed DNS changes on many of my there-parked domains. Upon reaching via Chat (no phone support so that angry customers cannot vent off) I was told that they cannot help me cause Im not the owner of the account! Upon full verification even with CC on file and telling them purchase history going back to 2009, I was still denied the access. As it turned out, all hacker needed to know is my public WHOIS info to take over my account!! That was insane! Only continuance of threats from my side that I will plaster it all over the net made them change their mind, which again is a breach of trust - what if I was actually the hacker??
What really made me start moving domains to NameSilo (Im not affiliated) is that upon doing a thorough research, I found many cases where Namecheap gives up on fighting for peoples domain! I seen names like nanotmz where company was building some sort of magnetic devices and TMZ came in and threat to sue Namecheap if they dont shut the domain down. That's where I found similar cases for NameSilo and learnt that they stand their ground and would not give up on your domains, even if are threatened with legal action.
I'm out of Namecheap completely as of last month with last SSL expiring.
Namecheap has had two factor authentication and was the first provider to have it. Knowing public whois would not grant someone access to anyone's account at Namecheap. They'd still need to know your Namecheap username, your password, and your PIN, and if you had 2FA, that would need to be provided as well.
If you have specific information, you are welcome to contact us with full details. Policies and procedures are in place to ensure no one falls victim to social engineering and as you call it, "intimidation."
skj, there's a lot more to policy and procedure than just human intervention. We're fully aware of this valid concern and are committed to security on both human and technical sides.
I would like to see something where a postcard is mailed and a phone call, each with half of the code needed for a reset. Postcard should not be sent using a method that supports forwarding so an attacker cannot setup a mail forward.
Customer support should not be able to see anything about these accounts except for a reset button. I do expect to be charge a fee for a reset if I need to use it. This would need to be rate limited to prevent people from dosing an account through it.
Hey Bill - when I was remote staff at AOL in the 90s, we had SecurIDs too (which is basically a key fob with 6 numbers that changed every 60 seconds, pretty reminiscent of 2FA on phones/Authy/Google Authenticator). Problem is if you lose that, you're not able to get into your account...
I would like to see more control options for the end users.
Let users chose if their account can be recovered through a password reset form, let users leave 'secret memos' for support which can't be viewed on the site later, let the users decide if their accounts can be altered in any way by support staff, etc, etc.
In 2014 I was able to bypass .htaccess restrictions of a site hosted on Namecheap using social engineering through the live chat... Twice.
All I had to do was tell the chat operator I was having some issue with .htaccess and .htpasswd until they offered to delete it temporarily. In the few minutes between them deleting it and reuploading it, you're free to do whatever you want.
I reported this and was told it was resolved (I guess with new policies).
I'd give money to a VPS, or what have you, that had a stated policy along the lines of, "here is a recovery key, here is the 2FA setup, here's how you recover your password with those items if you forget. If you call about account recovery and you do not have $REQUIRED_ITEMS, our service reps have been instructed to hang up on you. If you lose access to your account without $REQUIRED_ITEMS, you have lost access to your account permanently because we have set procedures from which we do not, under any circumstances, deviate. With a name like Ft. Knox VPS, our customers value the security of their accounts. As security is our priority, we have chosen to trade convenience and possible loss of data for the elimination of a large threat surface. If you regularly lose your car keys, don't use a password manager, or get angry when companies won't just email you your password instead of making you reset it, we're probably not the right choice for your VPS needs."
we have set procedures from which we do not, under any circumstances, deviate
Fair enough. There's a subset of people who want that and would agree to that beforehand.
But, and here's an important "but", do not just display a checkbox and an Agree button. Instead display a sentence something like this:
I UNDERSTAND THAT I WILL PERMANENTLY LOSE
ALL ACCESS IF I FORGET MY PASSWORD AND
MY RECOVERY KEY
In order to agree to the terms, make the customer type out:
I UNDERSTAND THAT I WILL PERMANENTLY LOSE
ALL ACCESS IF I FORGET MY PASSWORD AND
MY RECOVERY KEY
BTW, don't allow "paste" into the reply field, try to enforce actual typing of characters.
After you've done that, if a customer is later locked out, that customer will STILL be mad at you. It will be the providers fault that the customer can't get access. No matter how explicitly the customer agreed to the terms.
That's human nature. I don't care how explicit you make it, you simply can't win this one.
I don't know... Apple did this for a while, and it backfired on them. The average user (even technically savvy user) simply wasn't competent enough to understand that they really truly would be locked out forever.
I by no means intend such a system be sold to average consumers. As you point out, it would be doomed to failure. More for the hard-core "I'd rather lose my customer DB than watch it walk off in the hands of someone with a good sob story" crowd. And I'm not 100% sure that there is large enough, bold enough type to forestall some customer saying, "but I didn't think you meant it!"
The lesson here is you should always keep your own off-site backups - especially if you don't pay for a 'managed' server.
There will always be rare occasions such as this, but considering how many customers Namecheap handle, I don't think we should be seriously concerned. I'm pretty confident lessons will be learned.
I'm leaning more and more towards treating the email address that you use to register for business-critical services as secret.
It's a level of security-through-obscurity, yes, but that doesn't mean it's wrong. It means you can keep that address monitored well. You could make any activity on it send a page, for example.
"On April 9, 2016 I had an email address compromised, with the attacker brute-forcing a weak password."
Namecheap is obviously to blame for the compromise of the VPS, but failing to secure an email account which can be used for password resets is even a bigger fail, IMO.
If you only use 2FA on ONE THING online, make it your email! How could you not consider your email the most important service of all? It is your identity!
I'm going to go ahead and say that he used an email service which does not offer 2FA since every email service that does support 2FA probably has a good brute force protection.
That's bad, really bad. No 2auth can save you from humans who do support.
I also had one of my VPS attacked recently, and I feel for you.
But the name namecheap says "cheap". Maybe they are indeed cheap? I'm not sure the same would have happened with say HE. You pay, but you know what you pay for and get in return.
Personally, I am thinking about moving from a "manually setup" distribution to a "no ssh but deploy", so as to ease reimaging in the future. This way, if a server is compromised, all I have to do it to start the install of a new one.
Any suggestion for tools to do that with Debian distro? (yeah I could write a shell script, but I think there must be better tools out there)
> Any suggestion for tools to do that with Debian distort?
If you write apps, package them as Debs.
If you need to configure other Debs, make config packages with config-package-dev [1] from the DebAthena project.
Create a metapackage that depends on your software + config packages, and your setup process just needs to be "add private apt repo, apt update, apt install <metapackage>".
> No 2auth can save you from humans who do support.
Well, conceivably the second factor could be used to generate crypto key material which is used to decrypt/unlock one's record, so without the second factor even support couldn't read & edit one's record.
Nothing can stop support from deleting & recreating a record though.
The comments remind me of the time I tried to get a copy of my credit report from one of the big 3 agencies back in '09. One of the authentication questions was, "What is the name of your mortgage company?" My house had been foreclosed during a divorce 5 years earlier, and of course the mortgage had been sliced and diced about 15 times by different companies during the heyday of mortgage-based derivatives before the '08 crash. I finally gave up trying to get a copy from those guys.
I finally gave up trying to get a copy from those guys
I had a similar experience. IIRC TransUnion was the problem one. They wanted something like 3 credit card numbers as part of the identification. But I only had two active credit cards. Fortunately I was able to find an old cancelled one in a drawer and they accepted the number!???
The problem is that you're not their customer, you're just an irritant that the federal govt demands that they give "free" information to.
They treat their real customers much better. E.g., go to a used car dealer. Give them your SSN and they'll have your life story in front of them in about 10 seconds. They get good service because they're paying for it.
I'd love to have an option on services where I define a X-hour wait period for manual password resets. That is, "oh, I've lost my email account and I need to reset a password so I have to access my account through pleading over Live Chat... they can do that but there's an X-hour wait period before you will gain access to the account."
Which is great until the customer actually needs to access the account.
$CUSTOMER calls in, their nameservers are down and nobody has the account password. Do you think the management at $CUSTOMER is going to accept "hey we need to wait 6 hours to get our site back up because namecheap wont allow us in"?
Let the customer set it during sign-up as part of the password reset process. You set up your email address and password, let them choose $HOURS for last-line-of-defense password reset.
I don't see any perfect answer here. Ultimately you need a way to recover your account when you've lost all of the "somethings you have" and you've lost your "something you know", but then that allows a social engineer access to do the same. So let the user decide during sign-up.
Incidents like this remind me of Blizzard's policies vis-a-vie their 2-factor auth system.
I don't know if it's changed since this happened, but in the early-ish days my friend's phone broke and he lost his ability to generate 2FA codes. Blizzard was happy to remove his 2FA once he had made a photocopy of at least one (maybe two?) forms of ID and (I think?) some evidence he owned the credit card paying for the account. Once he mailed that in to Blizzard hq, some human confirmed the info and they removed the 2FA. If Blizzard can do that, and they're protecting MMO characters, certainly other providers can do so as well (maybe for an increased fee as I realize it's more expensive than online chat).
IIRC, people hacking into Blizzard games resorted to compromising users' computers and capturing the 2FA codes in flight - then logging in and changing the credentials before the user could react. That's a much higher bar to clear than the one here.
The whole point of social engineering is that, no matter how good the protocol you use, if there is a point of failure involving a person (like a tech support guy with access to a console), then it will fail.
The only solution is to remove these sort of powers from your general tech support guy and let them in the hands of a few highly technical, well trained, well paid staff members (presumably managers?). Of course, I made the blind assumption that your average support staff member is not paranoid enough, but based on my acquaintance with a few guys in the business it seems to me that the salary is not high enough to expect well trained technical stuff doing support.
I might be wrong, and in that case I would gladly know which companies employ such well trained staff, so that I can move my servers there.
Okay, so just for fun, I tried to do the same with my Google Account:
Login: my email address
Password: forgot it
"Please note that without your phone, the recovery procedure will take 3 to 5 days". hopefully, this means that there will be many many checks to avoid social engineering
On another note, there 2FA seems to broken as well. I once received a text about resetting my Instagram account from the same number that they send me to authenticate my login session. I've never had an Instagram account.
It's true :) We hear about it on Twitter all the time that the same number we use for 2FA is also texting 2FA codes for the other services I mentioned.
The real problem here is customer support not following established procedures. Maybe simulated social engineering attacks to test compliance should be part of SOP, like some companies do simulated phishing attacks.
I agree with most people that calling to reset password should be a service that can be entirely disabled, or at least require 2FA.
But think about this. All of this could have been avoided if they had the simple policy of calling you back. The only thing you would have to do as a user, would be to keep your number up-to-date.
Of course this also requires that you should never be able to add a number using the phone though, but this makes sense, since they can just say: "To do that you just have to sign in and click on..."
I transferred all my domains away from namecheap several months ago when the Ukrainian based live chat support was adamant they couldn't send an "ACK" message and have my domain transfer out automatically, and instead had to wait nearly a week for the "AUTO-ACK" to process. They lied to me and insisted ICANN require a five day wait even when I linked the ICANN documentation stating otherwise.
I've been battling with Namecheap for over a month now just to try and get the registrant details updated on a handful of .com.au domains I own. Their "support" is hands down the worst I've ever experienced, and the moment I'm able to move every domain I have away from Namecheap to a registrar that actually gives a shit, I'm going to do just that.
If it's "Whats your mother's maiden name?" and they let you reset it in the browser, it's a bug.
But if they send you an email (in my case to Gmail, that has 2FA turned on), then it is a feature, because then you'd be required to either 1) intercept the recovery email (and get the password reset URL) or 2) know the format of the password reset URL and just happen to guess mine after brute-forcing every possible link (assuming there is no timeout for the URL or anything else like that).
I had an interesting thought (literally as I was reading your comment) about improving "forgot password" emails, albeit only likely useful for the technically minded:
Have the customer provide an SSH/GPG public key, and store it with the account.
When a password reset is requested, encrypt a random string using said public key, and email it to the email for the account.
An attacker who may have breached your webmail is then reasonably unlikely to also have your private key to decrypt the string.
Follow the link (which didn't necessarily need to be encrypted) and enter the string you decrypted to reset the password.
On a related note: do any/many sites with 2FA, require the 2FA code to do a password reset?
That's basically what TOTP/HOTP authentication tokens are, which many sites (including Google, AWS, Github) etc use for 2FA - https://en.wikipedia.org/wiki/Google_Authenticator.
When you set it up, the service provider creates an 80 bit secret key, which you enter into your local device (or some implementations create a QR code) and then whenever you log in you need to provide a 1-time password from the app.
I'm aware of 2FA using (T|H)OTP, my thought was that a GPG/SSH key can be stored in a secure and yet reasonably easy to use way, effectively offline (i.e. add a passphrase and store it on a USB key or similar).
With a 2FA code, you either a) use the same code they use for regular logins, or b) require them to find a way to securely store the (T|H)OTP secret and then add that information to a 2FA app when they want to do a password reset.
I realise the pubkey concept is more than most people would bother with (or even be able to get through on their own), and I think the first 2FA option is definitely better than no extra security at all on password resets, but my thought was about increased security for those who are particularly paranoid/security conscious.
If you enable 2FA on an account with a service, and subsequently lose access to 2FA otp's (e.g. phone lost/wiped/etc) and lose access to (or never kept) the recovery codes, you generally lose access to the account.
This is similar, but with the benefit that you can keep the key secure by default - i.e. put a passphrase on the key and then store it wherever you like.
In reality, if you fail this "recovery" method, my next suggestion would be a billing based one (i.e. talk to a human, get confirmation of previous invoice details, what is being billed for, how it's paid for, etc)
It wouldn't happen to you. As you may see by other comments here (written by @matthewdrussell), this was an isolated incident that occurred specifically to an already-compromised email account. Still, we could always do better, and there have already been many meetings and policy improvements that have resulted from this single incident. We always take these opportunities to improve.
Yes, I work for Namecheap, but that's probably implied by my comment.
I use them for domains, but I wouldn't host there. (nor any other registrar, and if Starbucks started selling burgers, I doubt I'd buy one of those either)
I think this is a very good point that I also overlooked when I first read the article. If someone hacked my gmail account, I honestly am not sure if there would be any account of mine that would be safe. Anyone using the Internet today has to put utmost care into protecting their email address and most email providers enable you to do that fairly easily.
There was an article here a few months ago promoting logging in via email token as the only way to log in instead using passwords. Because as you said, 99% of websites allow you to reset the password anyway if you control email, so why bother having insecure passwords? If I remember correctly then that article was fairly well received.
And this is an argument for making sure your email provider has two factor authentication to avoid having an external breach that could give someone access to accounts that do not support 2FA.
* If they happen to be using authy for 2FA and you have the Authy app on your phone, it will use that instead of sending an SMS. You could also just have it send to Authy's Chrome extension.
* Consider setting up a Google Voice number to receive the SMS.
Hmm ... I have my VPS' set up so that I can only log into them using a private key, but I hadn't thought about the possible security flaws in my provider's control panel. Time for another audit!
If you can't secure your email, why would you be surprised when your servers dissapop?
I understand that there should have been more layers beyond this and all, but really, what is the point if you're vulnerable across several OpSec levels?
the first error was buying server space from namecheap. good domain server but they get hit frequently being a midsize provider of services, they have enough bait and not enough people to protect it.
I don't use their hosting, but I have a lot of domains with them. Anytime I've had an issue with the settings on a domain, they've been quickly resolved.
Just a few weeks ago I was getting a domain set up with Amazon SES, and one Daria P. helped interpret Amazon's docs to get it verified with them, and explained how I was using the dig command incorrectly to inspect the domain's settings. It's rare you see a support person do that at any company.
The domain registration process isn't automated from my experience but it works well. However, I think the meta is (I think many people will agree) to not mix domain and hosting with the same provider. For example, if you get your domain from namecheap, you should not do hosting at namecheap. Therefore, the argument is that if you want to buy a domain name from Namecheap (as they're pretty decent), you shouldn't do hosting there.
I wish there was a way to disable the "customer support backdoor" in all these kinds of services. I've started to deploy full disk encryption to all my servers now so if the attacker does manage to get into the management account the server itself is still protected from single user / rescue mode / etc.