What a mess, but does highlight the fact that you know, when you gather a lot of information on people, you become a target. They really just do all the hard work, and make it easy for the criminals.
Obviously, a huge blunder of government, totally irresponsible and reckless that such a massive breach was even possible. And it happened in dec 2014, and they only just now found out. A team of amateurs could do better than that.
I know data security is hard, but maybe if the government spent money on proper protections of people's data instead of building data centers to spy on it's citizens, this wouldn't have happened. But it's clear that's not what their priorities are.
Really, just disgusted by this.
That said, what did they do wrong? what should they have done, that they didn't do? Getting hacked seems like an eventuality at some level. What can an organization do to protect such sensitive information, or at least reduce their exposure and the amount of data that is able to be leaked before detection?
Seems like you'd have to partition up your data at some level, maybe encrypt it at rest; but I don't know how far one has to go.
>I know data security is hard, but maybe if the government spent money on proper protections of people's data instead of building data centers to spy on it's citizens, this wouldn't have happened.
The Gozer Principal: in Information Security you get to design the weapon that will be used against you [1]. Don't build a tool you are unwilling to hand to your greatest enemy.
For example what happens when foreign governments steal the domestic bulk surveillance data? I bet the NSA accidentally hoovers up all sorts of top secret information that is just accidentally sent over the wire or non-classified data that could do great damage to US interests. Or what happens when a foreign government gains access to the tools used to perform this bulk collection? They could inject fake traffic or hide traffic for strategic deception campaigns.
Collect it all is a strategically empty slogan, it represents a serious risk to US national security, but on the other hand it is a wonderful Rice Bowl [2] for the NSA.
>What can an organization do to protect such sensitive information, or at least reduce their exposure and the amount of data that is able to be leaked before detection?
* Keep it offline/airgapped.
* Store the most dangerous data on paper with hashes replicated online to insure integrity.
* Delete information you don't need anymore.
* Do not have a centralized repository of data to reduce risk of catastrophic exposure.
There are always trade offs between usability, functionality and security.
Thank you for this. great write up, and totally on point.
> * Keep it offline/airgapped.
Agreeable, but what about physical access? What about compromised individuals, or nefarious agents?
> * Store the most dangerous data on paper with hashes replicated online to insure integrity.
Interesting; i guess that reduces the accessibility of the information, so it's harder for you and harder for someone else. Interesting, and in line with your comment about the Gozer Principal (is there anything more than a tweet about that?)
> * Do not have a centralized repository of data to reduce risk of catastrophic exposure.
Also a great point too; and pretty contrary to how we typically design systems.
> what about physical access? What about compromised individuals, or nefarious agents?
Always a risk and a very hard problem to solve since someone always needs access and that person can be compromised. Detecting and preventing this is what counter-intelligence is for, but counter-intelligence is never a guarantee of security.
You can trade off usability for security to slow the speed at which an attacker can copy everything giving you more time to detect the activity. This only makes sense if they information is exceptionally important to protect. That being said, even paper can be stolen in large quantities. Jonathan Pollard managed to steal quite a bit of classified paper from the US during his tenure. The Israeli's needed multiple highspeed copying machines to keep up with his output [1].
One could imagine a system in which records are stored in multiple places and people only have access to the local files. No one agent could compromise the entire system. Then again, you might be so secure you protect the information from yourself and may not be able to "connect the dots" which is probably what you have the information for in the first place.
As Snowden suggested, "The Two Man Rule" [2] can also make enemies spies jobs harder. While it is true a determined attacker can overcome it by gaining the other parties credentials, such actions raise the risk the spy being caught.
A very silly idea:
I've been told on older nuclear armed submarines the device used to launch the missiles must be carried from one part of the sub to another to begin the launch sequence. The device was extremely heavy and awkward requiring multiple people to move it and making the movement of it very obvious and loud. Thus, they accidentally achieved an additional "Two Man Rule". Perhaps if very secret information was encrypted and then stored on heavy large objects which could only be read/decrypted at another location in the building than it would be difficult to quickly steal and decrypt many of the files. A computational version of this would be a cipher that requires massive resources (memory, CPU) to decrypt. This is likely more fantasy than reality, but at least enemies spies would have to break a sweat.
>Interesting, and in line with your comment about the Gozer Principal (is there anything more than a tweet about that?)
I've been working on a blog post on it, but I don't really have enough content to flesh it out and make it worth reading at the moment.
The two-man rule is a good idea - not fool-proof, but I suppose nothing really is.
And the story about the heavy device on nuclear subs; that's brilliant! Although I'm not sure a computational version would be a ciper that needs a lot of resources to decrypt, unless that is something that even people who have valid access need to expend resources for in order to access.
It really seems like ease of access is the enemy; that's a pretty fruitful take away.
I do hope you share your blog post about the Gozer Principal on HN when you're done, I look forward to reading that.
all of your points totally undermine the benefits of centralization and computerization. perhaps that is the point and that there are some things that should just not be digitized or made "easy to do," but security people also recognize that security is often not the end goal. creating a usable system where the reward outweighs the risk is the goal. if the reward, despite this risk/vulnerability, is still very high, then we'll probably keep doing it.
your points are what you would want to do if you wanted to make an ideally secure system, but nobody wants only an ideally secure system...
> … but nobody wants only an ideally secure system
Indeed! The ideally secure system would be one which doesn't exist/doesn't have ANY interface, much like the perfect computer which doesn't perform any IO with the rest of the world.
"The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards... and even then I have my doubts."
100% agree! Typically you do not want to trade off usability for security, but there are situations in which you do. The question then becomes how much do you trade off and are there solutions that give you similar gains but don't involve trade offs?
I'm just proposing some tools which might be useful when security is significantly more important than other considerations.
I don't think that this is a case where you want to trade off security for usability, and I don't think that generally those cases exist. You always want more usability. The problems that OPM has are massive, their solutions need to scale. Solutions like what you propose make their solutions stop scaling.
Security people (and armchair security people especially) think that security is the end goal. Security is never the end goal. The mission is the end goal. Security is only useful in so much as it helps you achieve the mission. Sometimes security needs to get out of the way when the mission needs to get done. It is always a tradeoff. The personnel clearance system is already such a tradeoff, a background investigation system that needs to scale to millions of people. Good luck making it "ideally secure."
There were data hygiene and application/network security best practices that OPM should have followed. In hindsight, they would have been way cheaper than the response to a breach like this. Responding by taking all the systems offline would most likely be far more expensive to the system as a whole than a future breach.
> There were data hygiene and application/network security best practices that OPM should have followed
Like what? Can you reference anything?
Your comment criticizes security but, this is a HUGE leak. You can't just sweep this kinda data under the rug for the sake of usability. That's just laziness. The mission matters and is important, but if you can't protect the data that people give you, you shouldn't have it at all.
This was a government agency; there is NO REASON for any kind of security tradeoff.
A company with CC info? Ok, yeah that's a different story with different tradeoffs. But this is the kind of data breaches that can cause vast amounts of harm to individuals and the nation; people's lives are put in danger by this leak. You wanna tell me that it's OK to sacrifice security for the sake of usability in cases like that? Would you feel the same if it was your life that was now at risk?
They had a responsibility to protect the data they held AS WELL AS to serve the mission. not one or the other. both. it is irresponsible to take risks with data that is not yours.
You know that there is no perfect security, right? So do you imply that, since security is so important, government should stop function? Anything government does decreases security, even if so slightly.
With FISMA no one builds even remotely secure systems, nothing anyone here would even want their name associated with. And this is because under FISMA government executives can "except risk", and they don't have to justify why.
So when your agency needs an application to do X, and you will face consequences if it doesn't get spun up, and to do so requires you cut a lot of security corners, but you won't face any consequences for doing so, you're going to cut those corners. Especially if not cutting those concerns means delays in rolling out that system, or spending a ton of money to fix all those security problems. The state of information security in the government is atrocious for this reason. It's not that complicated. There is no real incentive to secure systems, and very real insentives to not do so. You just issue the ATO and accept the risk. It's up an running and everyone is happy. If it's not and running people are pissed. It gets owned, people shrug and say "well nothing is totally secure".
>Security people (and armchair security people especially) think that security is the end goal.
This only seems to be your mistaken perception of what security people think. I'm a security analyst and I've never met anyone in my industry that believes that security is a goal that can be achieved. Security is a process that will never end.
>I don't think that this is a case where you want to trade off security for usability, and I don't think that generally those cases exist. You always want more usability.
If you believe that security is always a tradeoff(which you stated) and you believe that there is never a situation where you would want to make such a tradeoff, you are basically saying that there is never a situation where you would want security, which is ridiculous. There is always a balance between usability and security. In most cases, the scales will tip towards usability by a large margin, but there are times where a significant hit to usability in the name of security is the right decision.
> when you gather a lot of information on people, you become a target
The uncomfortable truth for many HN readers is that this is also true for all of that user data that businesses are currently aggregating: from the data people trust an online service to manage to the modern version of "library records" we call analytics. Thee is value in all of this, or it wouldn't be worth paying to collect/store it.
Unfortunately, even if the data is collected with the best intentions, the fact that the database exists creates a target that must secured. It also creates an "attractive nuisance" for governments holding national security letters and PRISM. Those are troubling enough, but gathering data about people has another risk that is rarely addressed: moral hazard. There is always the temptation to sell that data to the highest bidder, which we have seen happen many times as businesses look to find new ways to "monetize" their users.
I suspect a lot of this data that's commercially collected, directly by services and indirectly by advertising and analytics networks, will eventually end up in the digital storage closet, forgotten and eventually discovered by criminals just rooting around. Especially after acquisitions.
For the sf-86 data it really should have been stored on an air gapped network, encrypted, with physical security. That type of data is far too valuable to leave connected to a publicly accessible network in any way.
It's really bad. I was a Russian major and spent my junior year there. I subsequently went on to jobs in the military / IC that required an SCI clearance as well as a couple of additional SAP screenings.
I carefully listed the Russians I knew under penalty of perjury. I've lost touch with most of them. I wasn't trying to turn them into agents, and they were patriotic Russians who liked me despite, not because of, me being American. The fact that they might be getting FSB attention now is sickening.
The fucking government, man. It really blows your mind sometimes.
To expand a bit on that: any job that requires you to list the names of random people that you've had contact with in the past should be avoided like the bloody plague, there is nothing that those people have done to warrant you putting their name into some form and subsequent database with unknown consequences for the people you decide to list.
They're not sheep to be offered up on the altar of your ambition to rise up in the ranks, absolutely nothing good could ever come for them. So if the penalty is perjury just walk, that way you don't perjure yourself.
I agree. I've had Secret clearances at two different companies (aerospace/defence contractors). One of my lessons is that anyone who gets a clearance draws a lot of people into the process with them. It's a bad and creepy side effect, and even though this mostly involved college friends getting grilled by Federal Agents, I wouldn't do it again.
In fact, I'll go so far as to say that "clearances" are a symptom of the pathology of the security state. Lots and lots of highly cleared people have gone wrong (Pollard, Ames, Hansen, I'm sure there's more), so the process isn't really all that helpful. For a corporation, having "cleared" employees does two things:
1. It raises a huge barrier to entry for competitors. A smaller company just can't afford the specialized record keeping and record keepers, so only lumbering dinosaurs have cleared employees.
2. It means that the corporation can say, just like my kids, "I didn't do it!" as an excuse. An employee goes bad? The corporation didn't do it, because the Defense Investigative Security Clearance Organization cleared that traitor. It's yet another excuse to not have a relationship with employees, just like drug tests. Managers can just fill out forms, they don't have to know or like or have meaningful conversations with employees.
You list relatives, roommates, long term relationships, places you've lived, schools you've attended, and jobs you've had and then people who can confirm and validate those. Further - in regard to foreign nationals - you have to list and describe ongoing relationships and any foreign officials you've met.
This not "I was standing next to this guy at Starbucks" and closer to "I lived with, worked with, or dated with this person for X months."
> This not "I was standing next to this guy at Starbucks" and closer to "I lived with, worked with, or dated with this person for X months."
Yes, exactly. Those people that you randomly meet on your walk of life and who take on a role of some significance, and who clearly were NOT consulted about being included in some foreign country's databases on contacts with someone who is now part of the intelligence apparatus of one of their enemies.
What could possibly go wrong?
That person that you dated for X months is now potentially a target for a foreign intelligence service, just like that person that you lived with or worked with. And if they are as paranoid as your country and one of them happens to also be employed in their intelligence services and forgot or intentionally withheld similar information they are quite possibly in trouble.
Only they weren't random people, they were people he knew. And if you knew anything about getting a Clearance in the States a lot of it is based on how truthful you are on your SF-86 - List you use to be addicted to methamphetamine and you might still get a clearance, lie about that one time you smoked pot and you lose any hope of getting a clearance.
It's done to smoke out any "snowdens" and "mannings" who are trying to get a clearance for things other than wanting a job. (Not that manning or snowden joined to leak intelligence, but many have tried and many have been rejected).
It's also a way to gauge if you can be black-mailed to leak secrets. If you are open about your dark secrets (e.g., addicted to meth), then the chances of a foreign agency using that information to turn you into an asset are lower so there is still a chance you might get that clearance. If they dig something like that up on their own (which there is a fair chance they will), then you are guaranteed denial since it means you are a higher risk to black-mail as you were not willing to divulge such information.
Way to go to mis-interpret that: random people as in 'people that you simply come in contact with during everyday life'.
It's not as if any of those people had a way of controlling who they came in contact with. Life is built up out of tons of coincidences and who you know is rarely a matter of deliberation, far more often it is random chance that causes you to know one person and not to know another.
Random is the Russian I sat next to on a flight to NYC and never talked to / connected with again.
Random is NOT the guy I went to school with in Moscow and would fly back to russia for a wedding for.
The OP didn't say which group he'd classify his friends in, but if he thought it was pertinent enough to list them, than he had a close enough relationship to warrant listing them. Cause when OPM/FBI find out that you went to school for a year in Moscow and you didn't list any acquiescence's, they'll raise some flags and find what you are hiding.
The form actually asks uses terms like close, personal relationship so it not just listing random people. And to put the shoe on the other foot, what if it were your job to vet a person working on plans for a new nuclear reactor. Don't you think you'd like to know if the person had a college roommate from [insert hostile country here] and continues to maintain the relationship with visits, phone calls, etc? Maybe relevant, maybe not, but you'd be negligent not to check it out.
If you rely for your security on having people who do not maintain relationships with the people from their past then your security has already failed.
I see the red scare is still alive and well, which 'hostile countries' would those be? How does what someone does in their time off impact your ability to let them work on the plans for your nuclear reactor?
Do you really believe any of the leaks from the past 40 years or so were because someone had a college roommate from a hostile country (whatever that is) and who maintained their relationship?
On the other hand: If all the decent people refuse to work for broadly 'evil' organisations, the organisations are going to stay that way without there being anyone within them who might resist bad things.
I participated in a conversation a few years ago, where one person said "I've quit gmail, so I'm not worried about them collecting my data." Another person said "Yeah, but you correspond with a lot people who use gmail. So they're collecting your data anyway."
This is pretty standard for even the most trivial of national security related background checks. They want to know that your associates are also legitimate.
This comment is over the top. The vast majority of people wouldn't think twice about listing foreign nationals they have had contact with. Maybe It shouldn't be like that, but it isn't like he turned their names over to the gestapo.
You have to stop thinking about it from the point of view of the party they voluntarily turned that data over to and start thinking about it from the point of view of the people whose names were turned over to parties currently un-identified.
And then the Gestapo may be closer to home than you might like, for instance the FSB or the Chinese equivalent.
And for those people the consequences are well outside of the sphere of influence of the people that initially reported contact with them.
Databases like these on internet connected computers should not even exist, to see them fall in the wrong hands is absolutely in-excusable and to voluntarily aid in their creation is irresponsible at best.
I've been living abroad for years, in multiple countries. I've had close relationships with scores of foreign nationals. I wouldn't even know where to begin.
Not sure whether you're aware, but "foreign national" has a specific meaning in this context.
It means a representative -- military, ambassadorial, intelligence etc. Not sure whether it includes any public / civil servant. There's often a bit of judgement call required.
Perhaps if the press had actually reported on the Department of Interior's antics[1][2] surrounding Cobell v. Salazar[3][4], we could have brought to public a discussion of our governments handling of sensitive data. A whole department was removed from the internet[5], should have been a wake up call for data handling. I don't remember it being covered in the tech press of the time.
5) This also for a time included all Native American colleges including those that were buying their own line and charted by the tribe and not the BIA. It left students without access to distance learning classes and research beyond small libraries. It was hellish on students.
OK - Anyone for a funny story regarding filling out an SF-86?
As part of the clearance process, your co-workers are interviewed regarding your work-habits, perceived integrity, etc. We had one woman, "Mary", in the office who was a bit of a busy-body, listening in on phone calls, other people's conversations, etc. One day she overheard another young woman, co-worker "Jane" talking on the phone regarding meeting her boyfriend John at the airport. In order to embarrass him, Jane and a friend were going to dress up like hookers, hang all over him etc. Only Mary didn't hear the whole story and became convinced that Jane was really involved in prostitution and was going to meet a John at the airport. So when investigators were working on Jane's clearance, Mary flat-out told them that Jane was a practicing prostitute on the side. I'm sure these investigators hear it all, but I can only guess that this was a memorable interview. Of course when the investigator confronted Jane with the accusation that she was a hooker, she flipped out. Mary and Jane's relationship was never quite the same after than..
A lot of investigators are completely incompetent and the job has one of the lowest pay of any cleared jobs out there. My investigator didn't realize I was male until he had interviewed a friend of mine and corrected him.
This is a huge breach, and it will have repercussions for a generation. Nobody thought of the Office of Personnel Management as security critical. Previously, OPM has been criticized for not being computerized enough. OPM exists as a unit to centralize personnel records across agencies, all of which once had their own systems. Their retirement operation is still paper-based and located in a mine in Pennsylvania.[1]
Apparently, they succeeded in centralizing security clearance data. Then, of course, it had to be made available to all the security agencies. Remember the demands after 9/11 for "tearing down the walls" between the law enforcement and security communities? That means lots of people able to access databases in other agencies. Of course, people will want to access the data from the field on their mobile device.
Not just that, but how do these disparate government agencies verify that so-and-so checking into the local agency office is actually cleared for SECRET or TS or whatever?
They have to look it up somehow. That lookup will likely involve a computer database, and the pathway to reach that database will likely involve the Internet.
Practically all the rest of this sad story follows immediately, because the whole strategy of how the government handles computerized records in general is all screwed up.
Even after this I'm not sure it will get better... the trend in government is for inexorable centralization of related information. At the same time there's incredible demand to have those work-related systems available online and all the time, so that people can work while on duty travel, or from home.
Obviously there are technical things that can be done to mostly have our cake and eat it too (VPNs, redacted mirrors/views of the sensitive central database to be made available across the public Internet, etc.). But no one gets promoted in the government for doing that, and much of the talent is at Google or Facebook or Silicon Valley anyways :P.
Well if this includes the entire SF-86 database, then I guess it includes me. And I realize there's nothing I can do about it, so I guess I'm not sweating. Life goes on. If it really was the Chinese behind it, then the data likely won't ever end up dumped on pastebin or wherever.
The SF-86 form gets very, very personal, so I can imagine that some folks will be panicked, but reading my form would be a yawner. Maybe I need to get out more :-)
> If it really was the Chinese behind it, then the data likely won't ever end up dumped on pastebin
What if the calculation of both maximum effect and maximum deniability is exactly that: Create a shadowy, possibly composite hacker persona (a "Satoshi Nakamoto"), release a bunch of signatures and generate publicity, and then roll out the database, verifiable by the previously released signatures.
The first sentence of the article I linked to:
"The Chinese breach of the Office of Personnel Management network was wider than first acknowledged, and officials said Friday that a database holding sensitive security clearance information on millions of federal employees and contractors also was compromised."
I truly hope this will make all NSA employees that have worked on mass surveillance infrastructure come to understand the importance of privacy, and reconsider their participation in the similarly intrusive but far more large-scale crimes that their own organisation is guilty of.
First thing to pop in to my head after reading your comment is this Upton Sinclair quote: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."
It's a bit like wishing slaughterhouse workers would start to consider the animal's feelings and spend more time thinking about them.
I hope so too, just like I hope I get to fly around in a UFO because that would be pretty cool. Not really expecting that to happen though.
It would be fascinating for American citizens to be attacked (cyber or otherwise) because they work for the federal government and another branch of that government messed up and allowed this database to be breached?
The more likely way in which this will be used is by targeting those abroad who have been in contact with the US intelligence community. A bit like what they accused wikileaks of doing with the cablegate release.
I don't get why this is a big deal. If the people don't have anything to hide, they shouldn't be worried. That they're so concerned is highly suspicious and indicative of loose morals. China is just protecting its national security and has a right to do so.
I think you're missing the point, and would mind if we all went through your financial, medical and personal records. These records are made available to OPM under what are supposed to be strict privacy controls, because it's very very personal information. Everything from divorces, psychological counseling, drug history, you name it. You open up every secret in your life to scrutiny to demonstrate that despite all that you can be trusted. None of that is anyone's business, and it supposed to be protected and only available to a small number of people for a period of time to determine if you can be trusted.
Everyone has things in their lives they'd rather not have made public because it's nobodies business, and this compromise just betrayed the trust all those people put in the US government.
And this, boys and girls, is why you DELETE valuable, sensitive information when you don't need it anymore.
But, deleting information might result in an error of commission which would have your signature on it rather than an error of omission which has no one readily blameable. So, no one in the organization will ever sign off on it.
Or you could archive it on an air gapped network and delete it from all systems connected to the internet. Seems like a relatively simple procedure that I imagine is in use all the time with sensitive data...
FWIW, I recently had to get the records of a medical test that was performed when I was a teenager. All I had to do was fax the hospital a form with my date of birth, approximate year the test was done, and a signature, and they sent the records to me in the mail. Absolutely terrifying.
I'd rather have my health records public than information that could be used to compromise my financial security, considering the US doesn't enforce squat against credit credit bureaus who provide blatantly false information.
NSA security breach was several orders worse. Instead of getting all the sensitive information of 4 million US citizens with some ties with their government, it got sensitive information of 4 billion world citizens, and keep getting it because the backdoors, mass information collection, network interception and so on is still running. The elephant in the room is not just big, but pretty smelly too.
Doesn't this breach pretty much invalidate anyone who has ever had a security clearance? A bad actor who got ahold of the data could find people in sensitive positions and blackmail them with the sensitive information in their security clearance history. How can anyone be trusted going forward?
If you have something you can be blackmailed over you can't get security clearance. The info, however, does facilitate identity theft and KBA type auth. It'd be easier to pose as someone who does have clearance, which undermines the system.
They collect the info in order to obtain anything a foreign operative could use to blackmail you. The kicker? OPM also stores the results of the Polygraphs. Were they accessed? I don't know.
Fragile data such as this needs to not only be prevented from being stolen, but also needs to be of no use to a hacker even if it is stolen. Only then can we truly be robust to error.
The sheer number of people with clearances who are now at risk of blackmail and other untoward influence has everyone saying how terrible it is. OK. Obviously.
But what about the obvious fix: Pull clearances from everyone who does not need one. I mean really NEED. There are hundreds of thousands of schlubs with clearances only because the paperwork they have access to is classified higher than FOUO. And that classification is the product of self-importance and ass-coverage.
This probably sounds very awful but part of me really really really hopes that the vulnerability at the source of this was caused by one of the NSA programs to undermine security. Maybe then the sheeple will wake the freak up.
The flip-side of this story? Imagine what the NSA is doing. Imagine how much information the NSA is slurping up, on everybody, 24x7. Now... assuming their databases are an even more attractive target to hackers/criminals. Now assume that the folks who design/build/maintain/operate the NSA's are just as human as you and I, and therefore, are still prone to making just that one "oopsie" kind of mistake in their defenses. When that happens? All that data they slurp up falls into the hands of the hackers, criminals, people who mean you harm, etc.
Only part of the danger of what the NSA is doing due to the "what if government turns evil" scenario.
The other danger is the "what if hackers/scumbags/criminals get hold of it" scenario.
Only one of those scenarios has to happen, in order for it to hurt you. And the NSA has the very biggest pot of gold at rainbow's end, PII/fraud/blackmail-wise, of any of these systems to date. Contemplate that. Fear that. Take political action. Make day-to-day choices based on that.
I sure this will put me on some list, but you are right. They have a ton of information. It's probably just a matter of time before they get hacked. The question is by whom and for what purpose. It would be interesting if it was done as an act of civil disobedience.
Im sure there was a USB backdoor open somewhere. So who's fault is it if the US can't protect its own data? Blame others! That seems to be the way they operate!
Why for christs sake do they even collect this data in the first place? This is not a database on felons or potential terrorists ... why does the government care about the neighbours of their employees???
It's for security-sensitive positions. They ask questions to your neighbors about you to see if you're hiding anything.
Heck, just to apply to the bar I had to list eight character references plus a contact at every employer I've had in 13 years. It's pretty standard practice for trust-sensitive jobs to ferret out people who have something to hide or are lying on their application.
This is a database of potential Snowdens. The "intelligence community" is a strange, puritanical, paranoid sort of place. It wouldn't be so bad if it hadn't grown so preposterously large.
Note that this is far larger than the "intelligence community". This would include everyone from janitors who empty the trash in secure facilities to accountants, to mechanical engineers who design pumps for nuclear facilities, to web designers who write database front-ends, etc, etc, etc. And to say all of these folks are "puritanical, paranoid" is a very limited viewpoint.
If you filled out an SF86 it's because the "intelligence community" demanded it. That doesn't mean you're personally puritanical or paranoid; you're just subjected to the requirements of people who are; people who have altogether too much power and influence these days. And now those requirements have come back to bite you.
As more clarification, this is not just relevant for the "intelligence community", i.e. the three-letter-agencies. It would also apply to folks at various National Laboratories, Army bases, NASA, etc. And even for universities doing government-sponsored research. When I worked at Cal-Tech associated Jet Propulsion Laboratory, plenty of people had clearances.
We are using different definitions of the words "intelligence community". To me, if you have a clearance then that makes you part of the "intelligence community" regardless of whether your salary is paid by NSA, NASA, a defense contractor, a national lab, the Army, a university, or whatever.
If your definition of "intelligence community" includes NASA or random low-level soldiers just trying to keep their planned operations out of the hands of their adversaries, then I'd submit that your definition of "intelligence community" is functionally useless. Just say "clearance holders" if that's what you mean... there's already a very precise definition of "intelligence community" as it pertains to the U.S. anyways.
Different country, similar use case (clearance) - guy I worked with says a great way to get a proper fake identity is to assume one of a dead child from years ago when documentation wasn't as precise as it is now. (gives you a real, registered birth certificate) That's where family/neighbors information comes in - agency asks them and they say "sure, I remember, he died so young"...
I've been through this multiple times. Basically they just come out to your neighborhood and ask your neighbors if they've noticed anything suspicious going on. Strange behavior, odd visitors, unexplained affluence, etc. Takes all of 5 minutes. If it were my job to vet people, I suppose I'd be doing the same sanity-checks to make sure I wasn't dealing with an unstable person: drug addict, wife-beater, etc.
I've been on other side of the coin as well. Agents came to me asking about another person. They tell you up front that it's voluntary to speak to them and that anything you say can get back to the person under investigation if that person asks for the investigation records. All in the open.
Obviously, a huge blunder of government, totally irresponsible and reckless that such a massive breach was even possible. And it happened in dec 2014, and they only just now found out. A team of amateurs could do better than that.
I know data security is hard, but maybe if the government spent money on proper protections of people's data instead of building data centers to spy on it's citizens, this wouldn't have happened. But it's clear that's not what their priorities are.
Really, just disgusted by this.
That said, what did they do wrong? what should they have done, that they didn't do? Getting hacked seems like an eventuality at some level. What can an organization do to protect such sensitive information, or at least reduce their exposure and the amount of data that is able to be leaked before detection?
Seems like you'd have to partition up your data at some level, maybe encrypt it at rest; but I don't know how far one has to go.