Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Has anyone worked at the US National Labs before?
410 points by science4sail on Jan 17, 2023 | hide | past | favorite | 228 comments
I have spent the last 10 years working for FAANG companies, but nowadays I find their performance-review and promotion obsessed cultures to be really draining. Worse, those negative feelings seem to be leaking into my personal life and slowly alienating friends and family.

Therefore, I've been pondering a change of pace. The classic HN answer is of course "create/join a startup", but I've also been looking at areas more adjacent to scientific research.

One option that has come up is the US Department of Energy's national laboratory network[0]. From what I understand, the pay is 33-50% of FAANG, but they do seem to have interesting projects (e.g. the nuclear fusion facility that was recently in the news).

Has anyone here worked at one of them before? What is/was the day-to-day like?

[0] https://www.energy.gov/national-laboratories




I worked for Sandia. Pay is pretty good by almost any standards except FAANG. The glory days where every staff member got a real office with a real door are over (shared offices are the norm) but it's still a pretty decent work environment.

Things don't move fast, as another commenter said. In my area of work, projects tended to last 1-3 years and you'd be on several projects at any given time. In general, it is ICs rather than managers who run the projects. Your manager might say "Bob over in 9876 has a neat project that could use somebody like you, send him an email if you're interested".

You have to acknowledge that the core mission of the DOE National Labs is nuclear weapons. You might not ever come in contact with the mission, but it is there. They have strong HPC programs--because HPC as we know it is basically driven by the need to simulate nuclear weapons. Some people have moral objections to this, and that's fine!

I thought it was a good place to work, all in all.

Edit: I'd like to stress that probably the biggest advantage of the labs is the opportunity for self-directed work. If you can convince somebody (external sponsors, internal R&D funding committees) to give you money, you can work on just about anything. If you can't get funding of your own, you are still more or less able to choose what you work on.

Your work environment will depend highly on which group you're in. Some groups look like a university department without the students: you work in the unclassified area, you publish papers, you can even open-source software (with some effort). Other people spend their whole day in a windowless SCIF working on very sensitive stuff which they can never, ever discuss outside of a SCIF -- but while their public visibility is nil, their impact is arguably greater.


I'm a computational physicist at Los Alamos and would echo these sentiments.

Note that there are two main types of DOE labs: NNSA (Sandia, Los Alamos, Livermore) and Office of Science (Brookhaven, Berkeley, Oak Ridge, Argonne, ...). Although the former is more focused on "nuclear weapons stockpile stewardship", there is still much basic science at all DOE labs, especially where computer science meets physics and other domain sciences.

Perhaps relevant to HN, I would mention the Applied Computer Science group at Los Alamos, which is in hiring mode (https://www.lanl.gov/org/ddste/aldsc/computer-computational-...). Besides supporting computational physicists in code development efforts, this group does a variety of researchy things like designing programming model, doing compiler development, building ML models, especially with an eye towards large scale scientific computing. The pay at a DOE lab is less than FAANG (PhD student interns might be around $80k/yr and starting staff scientists maybe $130k/yr), but the tradeoff for some people would be the research-flavor of the work, and the flexibility. Many of the LANL codes being developed are open source, for example. Other DOE labs have similar computer science divisions. For example, Oak Ridge, Argonne, and Berkeley all have "leadership computing" facilities.


Curious, is there room at LANL for a senior full-stack generalist engineer who hasn't necessarily been doing published research work? I've looked at LANL Careers and can't gauge how biased you guys are toward research backgrounds.


LANL definitely needs and uses generalist engineers. I interned there as one, many years back. Standard CRUD-app stuff to help the lab do its main goal of nuclear stockpile stewardship.

But it will help if you enjoy research culture. The location is rather isolated and you will probably be friends / co-workers with people doing research. (I say this because not everyone likes being around PhDs; I've met several people in software who disdain academics.)


Think they need any janitors/maintenance techs? I can't do much but I can relocate to Oak Ridge fairly easily and I can surely unclog a toilet and clean up blood and paint some baseboards pink for breast cancer awareness month and reset some tripped breakers and change your backup generator fluids and swap out a jiggly door handle and refill the printer paper and repair the reserved parking sign that fell over and set rat traps on the drop tile ceiling space and swap out a malfunctioning automated transfer panel and buff the floors and refill/repair the vending machines and hang the pictures of the boss on the wall and upgrade the circuit breaker to accommodate all the space heaters and check/replace the fire extinguishers and AEDs and rekey the RFID door readers and I dunno, whatever else that "engineers" don't even realize needs to be done. HMU I got an expired TS but nothing precluding a renewal except for (currently) slightly disagreeable ideas that I don't express beyond internet fora.


Do you have an insight into the career trajectory in the labs for international students with a PhD from an US university?

I have heard it’s easy to get fixed term positions and incredibly difficult to get a permanent one.


But you can only work for these labs if you're a US citizen, am I right? :) Also are there any such labs or outposts in Hawaii? Would be a great beautiful place to live


You need to get security clearance for most roles - even a joint citizenship may be problematic:

Sandia is required by DOE to conduct a pre-employment drug test and background review that includes checks of personal references, credit, law enforcement records, and employment/education verifications. Applicants for employment need to be able to obtain and maintain a DOE Q-level security clearance, which requires U.S. citizenship. If you hold more than one citizenship (i.e., of the U.S. and another country), your ability to obtain a security clearance may be impacted.

Applicants offered employment with Sandia are subject to a federal background investigation to meet the requirements for access to classified information or matter if the duties of the position require a DOE security clearance. Substance abuse or illegal drug use, falsification of information, criminal activity, serious misconduct or other indicators of untrustworthiness can cause a clearance to be denied or terminated by DOE, resulting in the inability to perform the duties assigned and subsequent termination of employment.


I work at Berkeley Lab and there are tons of internationals here. I think it differs between labs - just an hour away at the Livermore Lab there are a lot fewer internationals because of what they do. We don't do anything classified and this lab hands out visas like they're candy.


Inspiring thank you!


Is that true for all national labs or just Sandi/all DoE labs? Also, where did you copy paste this from!!? :)


One of the interesting looking jobs on Sandia's job page:

https://sandia.jobs/jobs/


Ah, yes, I see, for instance, under: "R&D Nuclear Engineering (Experienced)", https://sandia.jobs/albuquerque-nm/rd-nuclear-engineering-ex...

It looks like the exact text of your copy above:

Sandia is required by DOE to conduct a pre-employment drug test and background review that includes checks of personal references, credit, law enforcement records, and employment/education verifications. Applicants for employment need to be able to obtain and maintain a DOE Q-level security clearance, which requires U.S. citizenship. If you hold more than one citizenship (i.e., of the U.S. and another country), your ability to obtain a security clearance may be impacted.

Applicants offered employment with Sandia are subject to a federal background investigation to meet the requirements for access to classified information or matter if the duties of the position require a DOE security clearance. Substance abuse or illegal drug use, falsification of information, criminal activity, serious misconduct or other indicators of untrustworthiness can cause a clearance to be denied or terminated by DOE, resulting in the inability to perform the duties assigned and subsequent termination of employment.


Sandia has a site on Kauai, but it's for very specific purposes: https://www.sandia.gov/locations/kauai-test-facility/


Cool, thanks!

edit: wow from the looks of their capabilities that site seems like a mini NASA.


Worked at Sandia National Laboratories from 1990-2013 in ABQ/NM, after 10 years in the Silicon Valley (CA Bay Area). I was recruited as a PM in Nuclear Energy to lead joint, cost-shared programs with the US Power Industry. Worked in tech. leadership roles in many areas including Homeland security, Intelligence, Nuclear Deterrence and Emergency Response Readiness, Global Security and Leading a remote test site within a USAF Base. All Missions of National Importance. Had opportunity to help shape national policy and strategy, team with allies and work with some of the smartest technical minds. Great R&D resources and funding. It's world's largest engineering R&D organization with current annual budget of >$4 billion and a workforce of ~15,000 professionals.

Having a MS or Ph. D. from top ten schools used to be requirement for entry. Salary is competitive, with excellent benefits including 10% 401K match since they no longer offer pension. No stock options, profit-sharing and other tech industry perks. However, excellent work-life balance, stability, working in different tech. areas with best minds w/o having to relocate and start over again, and wonderful quality of life, especially in NM if you love outdoors.

Employees are not government employees. It's a government owned contractor operated (GOCO) FFRDC, a non-profit. Like any big organization, there is a fair amount of beaurocracy and people issues to deal with. Having a team of former "A" students and ranking them is not conducive to teamwork although for all large projects or initiatives, it's a must. Multi-displinary teams range in size from a few to 100's and are spread out in many locations around the country. Opportunity to interact with Wasington lawmakers and agency e ecutives.

Recommend exploring opportunities at Sandia.gov


I found this whole thread really interesting. Maybe I have no idea but I’m wondering about interlab collaboration culture: is there like competition and maybe some lack of cooperation or grudging cooperation between labs? Sort of in the same sense that you get that stereotypical lack of information sharing between different government departments in other fields? Or it’s much more collaborative than the government bureau analogy (modulo classification/ project restrictions)?


National Security Labs, by law, do not compete with US Industry. However, they compete with each other so the nation benefit from best, innovation and solutions. They also co-operate in many areas including nuclear weapons. LANL & LLNL design the physics package whereas Sandia is responsible for systems engineers, safety, security and reliability. Rarely there is direct competition for funding since they have assigned missions.


>Salary is competitive

Competitive to the rest of government, sure. Its a joke given the total industry standards though.


I've worked at a few DOE sites albeit as a consultant not an FTE, Fermi, Los Alamos and a couple others. You are correct that the work is interesting and a like academic atmosphere, it was the academic atmosphere part that I found off putting. Where I worked there was very much a hierarchy and if you didn't have a PHD your opinion didn't matter much, you just did what you were told. Having a couple of decades of experience and being brought in to spin them up on their system and being talked down to on a daily basis like I was a Sophomore in college was really annoying -that said I'm a consultant so I get paid to be annoyed by the people who hire me. Aside from that I liked working there, the tech was cool, as far as the moral issues I don't ask nor do I want to know about what I'm working on -I don't have a need to know.


Having worked in R&D at a big pharma for 17 years and in university HPC groups for 10, I echo these comments. I've found that corporate R&D is essentially investigative (rather than production driven), and as an IC computational scientist (or data scientist) you will move from project to project over the years, usually working solo 95% of the time. You will have some opportunity to propose projects, especially in partnership with scientists (in pharma, those are chemists or biologists or work to improve manufacturing). But without a PhD, advancement along the technical track will be limited.

If you're not embedded in a science R&D group, you will be lumped in with general IT staff where tech support, database management, or software development of products are prized by management, but tackling individual R&D questions is not (though it is tolerated by IT brass since they know investigation is critical to finding & improving drugs).

I found the same limitations when I worked in R&D-based military contracting (or for US gov't FFRDCs for various agencies). There it's more important to develop a strong relationship with the gov't client, irrespective of the academic degree you have.


Sounds similar to what I mentioned in comments a couple months ago, about science organizations: https://news.ycombinator.com/item?id=33673259

(I'm not familiar with DoE sites. And I actually had a good experience (other than the pay), as a high-end federal consultant doing challenging technical work, reporting to operations research PhDs at the Director level, who respected what I could do. Where I've seen and heard of problems is other places.)


> as far as the moral issues I don't ask nor do I want to know about what I'm working on -I don't have a need to know.

I guess that's also a solution but you know that you contributed something, right? I believe that even if you ignore it the effect of your actions is your responsibility, even if somebody else told you to do it. And I don't even condemn it per so I just find the ambiguity puzzling.


What I do is pretty low on the stack so it could be almost anything, I just assume it will be used for good.


But you DO need to know, right? At some point there is a line?


I did an internship at Sandia this past summer, specifically on the HPC programs. Being a PhD student, the pay for that internship was excellent, and the cost of living in ABQ is low. I lived somewhere that I could bike to my office. My manager was excellent and a technical staff member and really understood how to manage other technical staff; no pointless meetings, hands-off, and tried at every turn to shield us from boring administrative non-sense.

The work was interesting, though my experience was very skewed as I was working on a solo project. I met weekly with my "mentor" to discuss where I was and if I needed support, but I was working nearly entirely solo. Like the parent comment here says, from what I learned the norm is that you work on many projects -- my mentor certainly was. Even in the department meetings it was clear that while we were unified under a general theme, each person in the department was working on their own, many projects.

My work was entirely unclassified, and my understanding is that most of the people in my department worked on projects with similar levels of classification. My office building looked like every other building and infosec and opsec requirements were pretty mild; wear your badge, don't photograph things, don't tell people any specifics about what you do.

I was offered to stay on as a full-time intern as part of the hiring pipeline and if it weren't for that it's in ABQ I would have strongly considered it -- the work was very interesting and also like the parent comment says, you are largely in control of what you do there. It's a lab first, not a defense weapons company, so research is the name of the game for the department I worked in.


> I was offered to stay on as a full-time intern as part of the hiring pipeline

This is a big deal and it's something people currently in school should consider: At least at Sandia, we loved the internship programs because it was a really well-defined way to do a trial-run with potential employees. After 3 or 6 months, we'd have some idea if the intern was actually any good, and they'd know if they liked the environment or not. This is important because hiring sucks and firing really sucks.

So if you're in school and you're looking for an internship, consider the labs. Also consider that, in my experience in a CS group at Sandia, startups and FAANGs really liked to hire Sandians away, so it can be a good springboard even if you only spend a couple years there.

Also look into the Scholarship For Service program.


Not all DOE national labs do weapons stuff. For example, Fermilab (FNAL) is pretty much all open science research. FNAL doesn't even have fences and the guard booth just checks to make sure you have a drivers license if you're driving. Argonne and Brookhaven are a bit more on the defensey side but not as much as say oak ridge or sandia.


Idaho National Lab and Oak Ridge National Lab are pretty focused on power reactors these days. I have a lots of friends at Argonne National Lab, and some of them work on entirely open source exascale reactor projects like:

https://github.com/openmc-dev/openmc (ANL)

https://mooseframework.inl.gov/ (INL)

https://www.anl.gov/mcs/nek5000-computational-fluid-dynamics... (ANL)

Cool shit.


Having spent some time inside Fermilab's campus, I got the impression a large portion of their purposes in watching visitors was for their own safety more than the security of the facility.


I worked, as a civilian, for part of the R&D arm of the Navy, but it has a very similar feel to what you have posted. I would 100% agree with everything you have said, except the pay.

One thing I will add is getting tools and resources to do the work you want to do can be a challenge. There can be times when getting through the process to buy a $75 multimeter can be more difficult then the $16,000 signal generator.


I also worked as a civilian for the Navy as an EE (b.s.). I think my pay was ~50k (in the mid 2000s). I don't know about the national labs, but my experience working for the navy was that funding was a fight, especially everything being siphoned off for the Afghanistan war. The equipment was pretty up to date (the computers were not the latest, but decent), the furniture was a mismatch collection of liquated stuff, lab reports were published to a confidential library, and we were on flex time, which made the hours great.


The greatest secret of government work is the hours. With the federal holidays, work celebrations at restaurants or bars, long lunches, you already work 30 hours a week.


My friends who worked there say that the job was great but living in a remote science outpost made dating unbelievably difficult


The town of Los Alamos is beautiful, but it's small. It can be a great place if you already a have family that enjoys outdoors activities. Many people prefer to commute from Santa Fe (45 min drive with mountain views, negligible traffic). Certain groups allow a flexible hybrid home/office working mode.

https://www.usnews.com/news/healthiest-communities/articles/...


I was only a guest scientist at LBL, but LBL is practically in Berkeley.

But yeah, generally I assume this is true, though you may just find that you need a car and that if you have one it's not so bad. For instance, NREL (where I was also briefly a guest scientist) is in an incredibly gorgeous area near Golden which isn't super small but everything is still very spread out and it'd take a while to walk anywhere for sure.

I guess it also depends on your definition of "middle of nowhere" I suppose. Golden is hardly Oakland, but I am pretty sure I could find people to date as long as I included Denver... and if you have a car in Colorado you will find that Denver is considered close to a lot of things you might not at first consider it close to. It's only an hour drive from Boulder (where I lived and drove to NREL as needed).


Guest Scientist makes you sound like a celebrity, that kinda tickles a part of my brain that makes me smile.


In this case it just meant I was being paid by someone else and convinced them to let me do stuff on their campus. But glad it made your smile ;-)


If you are talking about Los Alamos you can commute in from Santa Fe, it wasn't bad, about 45 minuets each way. In my opinion it's kind of expensive for being in the middle of nowhere, you don't get much bang for your buck house-wise. Social wise I couldn't imagine living in LA for an extended period of time if I were single.


Took me a moment to realize that you meant LA = Los Alamos, not LA = Los Angeles :)


You’d think there would be plenty of pretty young Russian and Chinese women looking to meet handsome American nuclear scientists.


NetworkX came out of LANL circa 2004, it probably is still being developed from there


Is there a scientific arm of research I could go into as a software engineer that, by its nature, can’t be transmogrified into murdering millions of people?


Medicine generally has the opposite objective, and has plenty of funding and impact.


At national labs they need software engineers to build internal tools for scientists in different fields. DOE labs are not all murder building machines, in fact most aren't. They do all kinds of research depending on the location. DOE national labs are a big tent.


Where's the fun in that?


Are you allowed to publish papers (for non-sensitive basic science/CS)? Do you have to relocate?


I published papers and I published open-source software, but not everybody does. Depending on your work, you may have to relocate; if you are doing any classified work, you should count on it. I also think there's a much clearer path to starting on-site and then after a few years transitioning to a fully-remote position -- one problem is that it's not 100% clear where you're going to end up when you first hire on, whether you're going to work on classified or unclassified projects etc.


Is there a lot of work left in nuclear weapons?


Plenty! Among other things, they literally decay over time, and that's fundamental to and completely inseparable from the way they work. Plus for some reason people always want them to be smaller and cheaper and more reliable.


Plus, we're not allowed to actually blow them up to test any more, so they do all that in simulation on the world's biggest supercomputers. This leads to work in HPC system software, kernels, compilers, programming languages, HPC libraries, and so on.


ZFS-on-Linux (now OpenZFS) comes out of LLNL, for example.


probabl a lot of that stuff is very sensitive but are there any good books people can recommend on that type of work at the intersection of programming/ (nuclear) weapons development?


Disappointing that it's so rare for knowledge workers to get their own offices nowadays. About 7 years ago, I did a 6 months contract where I had my own office and it was great. Nowadays, knowledge workers are treated like cattle. There's just so many of us. Knowledge and intelligence used to be precious, now they're treated as cheap commodities.

Also, there are a lot of people who are working in knowledge fields who have no curiosity at all - They seem to be doing it entirely for money and/or status - These people are absolutely awful to deal with if you care about the work.

They couldn't care less for the work and are all too happy to throw the project under the bus to protect their status and keep playing the role of the big boss in a bad movie.

Stakeholders couldn't care less either; so long as they can keep finding investors to pour money into their pointless projects, they don't care if the project leaders are incompetent.


>Also, there are a lot of people who are working in knowledge fields who have no curiosity at all - They seem to be doing it entirely for money and/or status

What's wrong with doing something for money? You make it sound like a bad thing. Do you think the janitor cleans your toilet because they find the work so intellectually stimulating? The work needs to get done, and most people would rather watch TV or go on a walk or play a sport or read a book or countless other things instead of slaving away at someone else's project, so society uses money to motivate them to devote their time and attention to that thing. You sound like one of those people who think that knowledge workers should work for peanuts just because the work is so interesting.


Nothing wrong doing certain things only for money. We've all been there. But if someone pursues knowledge for the sole purpose of getting money or power, their knowledge ends up being corrupted because they are driven by self-interest and not by truth. You can't see the truth if you are overly focused on your personal self-interests as the two often don't align. It creates a bias and corrupts your knowledge.

Then later you will teach your corrupted knowledge to others...


I worked at LANL [0] for five years and enjoyed the hell out of it. I was a Research Technologist, which is basically an R&D Software Engineer.

You will find a research group within a division to work for. For example, mine was the Computational Earth Science group (since renamed) within the Earth & Environmental Sciences division.

You will be working with a handful (3+) of research scientists as their supporting engineer. On some projects, you may be doing machine learning work in Julia. On others, you may be coding a fluid dynamics simulation in FORTRAN or C++. On others, you might be doing data analytics in Python. It's highly, highly variable, depending strongly on the PIs you're working with, and can change as frequently or infrequently as you wish (within reason).

Ultimately, I did the reverse: went from a DOE lab into a FAANG company. My reasons are particular to me, but if you're at all interested in a slower paced, more varied and collaborative environment, you can't do much better than working for the labs.

For context, at LANL, I was making ~$100,000 / yr with 3 YOE (circa 2019). This is in northern New Mexico, with such a low cost of living that this amount of money goes about as far as $150k+ up in the Boston area (where I am now).

[0]: https://www.lanl.gov/


LANL is also in an absurdly beautiful place, and Los Alamos is a very pleasant little town.


It is, I was just there this past weekend. The town itself is very small, and like some have said its getting more pricey. I can talk about this area for hours as I had a job offer from a startup spun out of LANL a few years back, I ended up not taking the job and sometimes regret it.

But for Los Alamos itself I would consider it on an "island" so to speak as the surrounding communities(except santa fe) are not great, see espanola. There are reservations around the area and they have their own issues(see drugs alcohol etc - the drive from the valley to Los Alamos requires you to turn on daytime running lights due to many DUI's) but the town is extremely friendly and safe just not alot to do if its not outdoor related.

Like some have said some LANL employees commute in from Santa Fe which is a nice town, it skews very old and rich(on the north and east sides) and if you have a family is not ideal(almost all the families live on the south side of town). Overall the area has excellent food(northern new mexican food is incredible!) and for outdoor enthusiasts it really can't be beat. Home prices in Santa Fe have risen alot in the past few years like most nice outdoor areas. But I think you can't go wrong with the area if you don't mind the few downsides.


New Mexico can be a challenging place to live, Los Alamos even moreso. Like someone else mentioned: married with kids? Great place, but even then Los Alamos is isolated, it’s a 45 minute drive to Santa Fe which itself is not huge. For the national labs you will also need to maintain a clearance, and the DOE is somewhat regressive in its policies around past drug use and mental health treatment.

There is very little in the way of night life. There are activities but this is a place where people live their whole lives and you often have to know someone to get involved. There is a paucity of health services. The airport is small and has had its routes reduced dramatically over the last 15 years. Los Alamos real estate is dated and expensive.

On the other hand the state is beautiful and there are endless outdoor activities.


Strong emphasis on "little". If you're married with kids, it can be great. Much harder for single young adults, especially men since the gender ratio at the lab is unbalanced. Most people in town are employed by the lab or contractors for the lab.


Definitely. The town always gave me 'Stars Hollow' vibes, if you've ever seen Gilmore Girl.


It’s much less affordable than it used to be. Lots of transplants seem to be driving prices up.


If you're talking about housing and property prices, that's a long-standing supply issue related to building a town on a mesa...


That just makes it a factor which long term diminishes the labs abilities and people. Consequences priced-in as it were.

But the grocery stores and surrounding supportive industries like medical/dental are also going to reflect the towns population.

I do love Los Alamos but knowing what a large city is capable of helps, too.


>low cost of living

Except for the nightly drive to Santa Fe.


If you live there, sure. There is also Los Alamos, White Rock, Jemez, Española, etc.


It gets only a sliver of attention compared to Los Alamos and Santa Fe, but Española is actually effectively another LANL company town--the lab is the town's biggest employer and a large portion of the lab's workforce lives there, so e.g. sharing the commute up the hill would be easy. And compared to either of the others (or any FAANG city) real estate is cheap (like 3+BR 2000+sqft on half an acre for < $500K).


Did some time at LANL as an R&D engineer in the non-global security skunky areas, though wound up leaving for reasons not pertaining to the work. Participated in several projects involving Sandia and LLNL.

Pros: - Pay was excellent, especially for the area - Incredibly beautiful country - Very interesting work - Infinite well of taxpayer dollars for equipment and materials - The best job security one can find - Crippling bureaucracy enforced a remarkably safe work environment

Cons: - Crippling bureaucracy made it difficult to move quickly and hit tight deadlines - Internal politics (intra-lab and inter-lab) often adversely affected decision making and program success - Living in a company town - An inability to remove demonstrably problematic employees - A Q clearance limits certain extracurricular activities

Personal experiences with LANL were all over the place and highly, highly dependent on which group one works with. I was very lucky to get in with a group of wonderful people and immediate management that firewalled most adverse developments from higher up the food chain. This is not a common experience but organizational mobility is relatively free, so you can move to work and groups that are attractive.

Worth noting for those coming from private industry: the national labs are institutions first and foremost, not businesses. Organizationally and operationally they exist in a very different mindset and within very different value systems than FAANG-like orgs. The adjustment can be a bit jarring.

My work at LANL will likely be the most interesting and most fulfilling work I'll have done: every day was an adventure into the unknown. The work/life balance was also excellent. If you're a naturally curious person and have an inclination for basic science I'd recommend taking a look at the labs. If you have specific questions feel free to drop them here!


Throwaway since I'm going to say some negative things.

Overall I agree with almost all the positive things people say in other comments - the national labs have a lot of very smart and kind people, there are interesting things (or at least interesting ideas), and if you're at home at a university, you'll find a lot of kindred spirits.

But I made the opposite move you're considering and quit my national lab job and moved to FAANG. Why? Because I wasn't a scientific superstar with a clear vision, and IMO my field (applied math, computational science, etc.) seemed to asking (by which I mean funding) people to do software engineering without many engineers and at the same time still be academics: scientists and mathematicians who spend a lot of time writing grants and trying to publish papers, etc. This made me feel like a liar, writing the grants, and a hack, writing the code. Not to mention that like all academic-type and "interesting" jobs, you are supposed to be happy with the idea that you're going to expend a lot of your free time and energy, and perhaps not be paid quite as much as you would if you weren't pursuing your (supposed) passion.

Industry is for sure a whole other pile of bullshit, but don't assume that the performance review and promotion stuff is any more draining than the things you will have to do to get funding (you'll likely either be spending a lot of time writing proposals, or you'll be funded by weapons money). Don't make trying to get away from money and politics your reason for moving, though there are plenty of good reasons. Good luck!


I interned at Sandia around '04-'05. That was a little bit before the current "FAANG" thing we have now (IIRC amazon was turning itself from a bookstore into an e-commerce startup, and google was still not being evil) but certainly it was already less glamorous than getting a Microsoft or even Apple internship.

My recollection is that the whole experience depended on which group you were in, and mine was fortunately very chill. Smart, friendly people who arrived and left more or less at the same time every day. Lots of matrixing and loaning of people from different orgs -- I had the feeling that if I were making a career there I would wind up slowly drifting around between projects.

The biggest surprise to naive me-in-my-early-20s was that "Department of Energy" is a euphemism for "Department of Nukes." Nuclear stockpile stewardship was a large portion of the activity there, and so a lot of your colleagues will be people who are at least vaguely comfortable with that.

There was a ton of "basic research" too -- some high-energy group had a daily experiment that would deliver a "whomp" of a shockwave around 3:15pm most afternoons, there was a room temp. fusion group, lots of interest in assisted driving cars and unmanned aerial vehicles... you just had to appreciate that all the first applications of all this tech was going to be military.

Also the security clearances.... the joke was that the "L" clearance stood for "Lavatory pass" because in our building until you got one, you needed a line-of-sight escort at all times, even in the bathroom. Even for the "L" the process was quite onerous, and I understood that the 'Q' clearance held by nearly all full-time staff was even more burdensome. I heard stories of people waiting for their clearance getting stuck in rooms with nothing to work on. One person in my group basically got sent offsite to some "think tank" or something for several months while he waited for his clearance - I only met him once the whole summer, at a conference.


The biggest surprise to naive me-in-my-early-20s was that "Department of Energy" is a euphemism for "Department of Nukes."

It also surprised Rick Perry when he got to be Secretary of Energy (he thought it had to do with oil)


Haven't seen anyone mention a non-DOE lab, so figured I'd weigh in.

I interned twice with MIT Lincoln Labs, which among other things, helped build and deploy Radar for WWII which turned into building/managing the technology for Air-Traffic Control, and then turned towards space.

They are primarily a DOD-associated research lab (even located on an US Air Force Base), and so most of the projects have some military-oriented mission. Their mission is entrepreneurial-minded (which I found cool), in that they do the "basic research" and prototyping to prove viability and then the DOD turns over the project to a contractor to make feasible.

While I was there I worked in their GeoIntelligence and Natural Language groups, doing research which I'd ultimately come to understand as being relevant for Project Maven (year 1) and PRISM (year 2). While I'm sure as an intern my contributions weren't directly related to or otherwise leveraged for these programs, in hindsight it was clear that this was the bigger picture that the work was contributing to. Take from this what you will.

Most of the anecdotes that I've read through in the comments mirrors my experience. However, one thing I see missing was how opportunity was "metered" out. Each group I was in was organized like a research lab and the level of your academic progression limited (or opened) your ability to get access to specific projects/work. Their pay scale was also dictated based on this as well. So if you have a BS, your ability to "move up", doesn't exist, but it does if you have PhD.

Ultimately, I was given an offer to work there, but ended up taking a SWE position in the Bay Area because I wasn't interested in continuing my education and felt like my ability to have a career progression at MITLL would have necessitated that.


One highlight of working at LL as a radar software contractor was getting to go to the regular lectures they had on various topics related to the lab's activities. But of course the big downside is not being able to talk about what all the lectures were on and having to avoid some of the topics they were on now.


Yeah - and it wasn't even necessarily on the tech stuff either - the talks are definitely the things that have stuck with me the longest after working there (and it's been 10+ years!)


LL are the best radar designers on earth, full stop. They do some neat cyber research, too.

Don’t work there, but work very closely with them.


The glass ceiling for non-PhDs sounds like a caste system, not the meritocracy that the broader MIT-ish community sometimes professes.


I have no idea how this plays out in the US national lab system being discussed .. but a PhD is literally "my first independent solo original research project obsessed over and submitted to critical peer review".

It's the apprenticeship for doing independant solo research and, in general, makes perfect sense that someone have demonstrated a capability for this work prior to being given the reins to resources on the scale of millions, tens of millions, etc.

It's hardly a "glass ceiling" (ie. we profess equality but don't promote women or people of colour or not our religeon but never say why) when it's a stated requirement to, say, first pass an apprenticeship prior to becoming an certified electrician and hired to wire a nuclear weapon.

It's an actual formal staging in a meritocracy, the only issue would be if those that might gain a PhD are denied the opportunity to do so ... (a somewhat tangential issue that might have more play in various places).


Counterpoint: a PhD is a ~five-year performance review from an organization that doesn't pay very well.


A defining characteristic of a PhD is that the definitive review is from an external examiner who is paid by a different organization than the candidate, and the review is the personal, professional opinion of that researcher, and not that of their organization.


This is definitely not true in the US, at least in physical sciences. Typically your PhD advisor (who is obviously in the same organization and gets paid with the same funding source as the student) has almost entirely all of the say in a PhD defense. There is another faculty or two from the same department on the committee (who work on different stuff, possibly funded from somewhere else). And, IME as a mere formally, there is often another faculty from a separate academic unit (department) who's just along for the ride to give the appearance of oversight but doesn't really know what's going on. They'll ask a softball question to remind everyone they are there.

Sure, anyone in the committee can grill you as a sort of hazing ritual, but the reality is that your PhD advisor won't let you stand for defense unless you are almost 100% sure to pass it.

Source: have attended probably a dozen thesis defenses (including my own).


In the UK you can normally submit without your supervisor's approval if you insist (and have survived the programme long enough to actually have a thesis written). The actual viva will most likely be one internal academic (not your supervisor) and one external and your supervisor won't be present.

That said, it's a pretty crappy idea under nearly all circumstances -- if your supervisor doesn't think the thesis is passable and discourages you from submitting it then it's quite likely the examiners will agree with them. And getting a terminal MPhil isn't exactly a badge of honour...


So there’s no external examiner? That’s very surprising to me, but it certainly refutes my claim.

I agree the supervisor should almost never allow a defense to take place that you won’t pass. But I can’t imagine a school passing a candidate if the external gives an unfavorable report.

I’ve also been involved in many PhD defenses, every one of which had an independent external examiner. Computer Science or closely related, not physical sciences.


The PhD advisor is in the room and takes part in the defense in the US? Or is this just the committee?

It sounds like such a conflict to have them actually present at the defence and actively taking part.


Counter-counter-point that poorly paid five year review doesn't have to be poorly paid or take five years - that's a function of countries and their approach to education etc.


Notwithstanding both degree inflation and the notion that "the exception proves the rule" -- a system that would rule out someone like Freeman Dyson would seem flawed.



In that NYPost article Dr Natalie Gosnell isn't discussing the PhD requirement for advancement within the US National Labs .. so you may have to expand your point if you want to make one.


[flagged]


Please don't post in the flamewar style. It's not what this site is for, and destroys what it is for.

https://news.ycombinator.com/newsguidelines.html

Edit: you've unfortunately been breaking the site guidelines in other places recently too, e.g. with personal attacks and using HN for ideological battle. We end up having to ban accounts that do that. I don't want to ban you, so if you'd please review the rules and use HN as intended, we'd appreciate it.


I was specifically talking about a PhD requirement as a precursor step in a research career.

> Academia is a shocking hive of nepotism and corruption wherever you are.

Also isn't something discussed by Dr Natalie Gosnell who has her Dr. and a position. Her complaint (in the article you linked) is that " astrophysics .. is paralyzed by “systemic racism and white supremacy” ".

There's no obvious mention of nepo babies in that link that you selected.

My own preference is for astro tangential projects such as

https://www.sarao.ac.za/media-releases/2022-africa-data-scie...

which are off shoots from the larger SKA project initiated here back in the 1980s.


It's largely part of the same rotten thing. You think rampant “systemic racism and white supremacy” and sexism has absolutely no relationship to the presence of nepotism or glass ceilings? Or that its presence in one part of academia has no bearing on whether or not its present in academic-adjacent industry, who could not be expected to have any idea about the problems in academia? Come on.


Its really not, have you ever spent any actual time in academia ?


Yes. And I'll let my link speak for itself, do you deny the claims made by Dr Natalie Gosnell?


Work at MIT main, and had job offers at MITLL.

MIT-LL is way more stringent on degrees no matter what your experience is. B.S is relegated to technician work for upwards of 90%+ of your time no matter how long you have been there. Masters is generally a step up from that, and in the engineering that could mean middle of the road in the hierarchy. But PhD is the only way to really move up beyond that except in the rare cases of the fabrication group.

MIT main campus is slowly heading in the same direction over the past decade, at least in the science staff positions. There are three major titles, research specialist (B.S.), research engineer ( masters), and research scientist ( PhD). Unfortunately in the past 5 years, they changed the requirements of research engineer to require a PhD. And honestly, I have more recently been involved in the HR/hiring side of main campus, and though with the right department there is some wiggle room... There isn't that much you can do when the office of the VP of research says otherwise.

So unfortunately to move up with Anything but a PhD, you need to shift into some type of admin or adjacent role.

To exemplify this, I had a coworker who had worked in a research group as a research specialist for half a decade, coauthored like 15 papers while there, and had a BS in physics. And said he couldn't get a promotion no matter what he tried because VPR wouldn't let his boss do it without an advanced degree (PhD). So he left for a different department as some kind of lab manager of a large research group. Which to me is crazy, you have a dedicated and knowledgeable worker, who wants to stay and advanced the research and group... But because they didn't spend 6-8 years on a PhD, their only way to advance their career is to go into management of a lab - giving up on any research work...


That's sad.

FWIW, I was a Research Scientist there, a bit over a decade ago, despite having only MS degrees, no PhD.

I knew of a few Research Scientists there who didn't have PhDs, so I didn't think it that unusual at the time I was hired. At one point shortly after arriving, I did suddenly wonder whether some rule had been bent, or there was a hiccup in some process that was supposed to block that, but I still got an appointment renewal after a year.

As the recipient of getting to work for a great PI, and of a title that sounded impressive to my parents, I can't complain about that.

(I might've been lucky that time. I'm not a fan of degree/class/caste ceilings. For one one of many reasons... There's the tragic story of a dear friend, who was a lab tech at another research university, in a field that had a degree glass ceiling among the technician ranks. Her supervisor and lab director sounded very supportive, and said she was the best technician in the lab, but their hands were tied on promoting her to a higher technician rank. She couldn't stomach the doctorate-level degree debt load that the supervisor was encouraging (she was poor, already had debt, and no family safety net), though it would've leapfrogged her over the role she sought. So she tried earning affordable transferable credits, for the gatekeeping for the next rank for which she was already qualified, while working full-time and living in lousy conditions. It killed her at around 30. Her lab did a memorial service. I got invited, but I didn't go. Besides being devastated myself, I was sure the supervisor and director already felt awful, and I had nothing to say in a memorial service context to certain cliquish technicians who bullied her for being meticulous about science protocols, and perhaps for aspiring above her station.)


I've spent most of my career at them, with a bit of time in industry in between time at the labs. (For context: me = ~20years post Ph.D. in EECE, started career at one lab, later landed at a second lab where I still reside.) Labs are awesome environments: it's not too hard to get involved with projects that are pretty cutting edge scientifically, which is exciting if you're a science nerd like me. It's also nice not to have layers of management over me dictating what I do - the scientists play a major role in running the show, so you have much more control over where you end up going. That said, they are an environment where it's most easy to dictate research direction if you have a PhD or in some cases, a masters + lots of experience.

The pay isn't competitive with the giant Silicon Valley companies, and computing tends to be a little less bleeding-edge than the other scientific domains. The only place computing is at that bleeding edge is in the HPC world since the labs typically have machines like nobody else, so there is a lot of research to do in terms to utilizing them well and programming them.

The only other complaint I see for the tri-labs (SNL/LANL/LLNL) is that pretty much everyone is expected to hold a Q clearance (roughly equivalent to DoD TS + CNWDI). That can be an obstacle for some people. Not a really difficult process - lots of paperwork, interviews, patience while it goes through the system, and then the periodic renewal process and occasional random drug tests.

I personally love working at the labs and plan to stay for the rest of my career. I don't optimize my career around maximization of take-home $. For me, I want fair pay doing something I really feel like I get excited about in an environment where my employer treats me pretty well. The labs give me that.


I interned at Sandia Livermore and Los Alamos in college, then worked at Sandia's main site for a few years before moving out to the Bay Area to work in the more dynamic world of consumer electronics.

The labs are not for everyone, but it's the perfect job for some. If you want to work with fantastically smart people and don't mind following a lot of arbitrary rules, it can be a lot of fun. Most of my coworkers intended to spend their entire careers there.

Just like anywhere else, a lot of the day-to-day experience depends on the group you work with. In general, it's somewhere between a university campus and a defense contractor, and the mix is different for each project. The good part is that once you get a security clearance and make some friends in other groups, you can move around.

There might be some culture shock. Most employees have to be US citizens, so the labs are probably less diverse places than you might be used to. And you will really be hitting the brakes while you wait for a security clearance.

I'd say look at the job postings and give it a try! It didn't end up being for me, I don't regret the time I spent at the labs. And it's tough to beat the work-life balance. You can't take a lot of the work home, and most people take every other Friday off (9/80 schedule).

But do consider the location carefully. For example, Sandia and Los Alamos are both huge and have a huge variety of projects, but you're stuck in Albuquerque or Los Alamos which can be limiting unless you really enjoy hiking.


Just a heads up that most of the interesting work will require a "current DOE security clearance". Many positions at places like LLL or any DOE lab really are going to require the more intense Q clearance.

Sometimes prior clearance can be negotiated and a well qualified candidate who is likely to clear might be accepted and placed in a holding pen until they clear but I'm not sure what the backlog is now or even if they do that anymore. At the very least you will almost certainly need to be a US citizen proper.


When I worked at Sandia, most of us were hired without any clearance but with the expectation that we would get one soon.

It is not difficult to get a Q clearance, just annoying. You have to fill out a massive document listing everywhere you've lived in the last 10 years (a real hassle for a recent grad) and give all sorts of info about people you know. They will drive out and interview people.

There was plenty of work that did not require a clearance, but so much of the sites are cleared-only that it just makes your life easier to have it.

edit: oh yeah good point made in the dead comment below, if you've smoked weed in the last 7 (? something like that) years you're gonna have to tell them. Even if it was legal in the state where you did it. I've heard it's not a deal-killer these days, but they want to know and you will have to stop using it. There are not a ton of ways to lose your job at a National Lab but failing a drug test is one. Do not toy around with it.


> It is not difficult to get a Q clearance, just annoying.

Will be if: you do drugs (including pot), have debt problems, have dual citizenship esp. if you have made use of it in some way.

> You have to fill out a massive document

SF-86. https://www.opm.gov/forms/pdf_fill/sf86.pdf


These are the forms that the OPM leaked some years ago, BTW. So there's that risk also.


Oh yeah this is the other thing for sure. In my lab I didn't need a clearance for the work I was doing, but my boss had one and his later grants were going to require it. He asked me to come back and I declined because I didn't want a clearance.

It's not just annoying to get it, it's kind of intrusive to keep it. You have to notify the government when you travel to foreign countries, if you have contacts with foreigners of certain nationalities when you are at conferences, you have to be careful with your social media presence (many people with clearances don't have one at all for this reason), no drug usage even if it's legal in your state, etc. I don't like the feeling of those kinds of restrictions so I said no to that lifestyle.


Is this something you can apply for prior to getting a job?


Certainly not. This is a government security clearance, not an industry certification. Zero need to distribute these to anyone who doesn't specifically require it.

That said, it stays with you afterwards, so you can take another job that requires clearance at a different organization/company (provided you have the right level of clearance for the job). I think it expires after some time.


Lets say one wanted to start a defense startup. What do you do in this case?


It depends on whether the tech is itself inherently sensitive or whether it will just be used for defense among other uses.

A lot of defense stuff falls in the latter category in which case you can start by needing to hire some sales engineers with security clearances who meet with the defense customers -- mostly contractors themselves. The devs don't necessarily need clearance.

If you are making something sensitive -- like a new nuke -- then you will have trouble going the start up route. IN-Q-Tel, DARPA, these programs were designed to fill the gaps.

But if you are making an AI for drones, for example, or some new polymer that has good materials properties that would be useful on a tank, then you can ease into it as these have multiple applications and you just need your SEs to meet with the defense people and say "hey, we have this cool tech, are you interested in it?" and then if they are interested they will work with you on clearances by helping sponsor key employees and letting you know what their requirements are and how you can meet them.

In many cases an acquisition by said contractor is your exit.


No, a company or organization must sponsor your clearance. You cannot start the process on your own.


Not that I've heard: it's not unusual to have the start of the job delayed by a year or so while someone checks in with everyone you've lived with / worked for for the last 10 years.


I worked at LBNL, at the Molecular Foundry. The day to day was a mix of typical nanoscience work (chemical synthesis, electron microscopy, etc) and work in support of the user facility. In my case that involved consulting on projects involving our users (design of high-throughput screens, teaching spectroscopy, etc), setting up and maintaining instrumentation, and developing workflows for our chemical synthesis robots.

I liked the work and really enjoyed getting to be a consultant on many projects. Turnover is massive among the researchers because there are few permanent positions, and most groups are heavy on postdocs since graduate students tended to be primarily on campus (UC Berkeley).

If pay is a concern, look closely for the open databases of salaries. At LBNL there is the "book of tears" at the library under the cafeteria, listing every employee and their salary. The exact amount you get varies wildly with the department: prior to unionization in 2016, the range was from 20k to 125k annual salary for postdocs. I hear they raised the floor to NIH levels at least, but I assume they did not make NERSC take a paycut.


I worked at Argonne National Lab for two years as a web developer for a few projects (https://afleet.es.anl.gov/home/, https://energyjustice.egs.anl.gov/).

The people I worked with were super smart in their fields, but were pretty bad at writing code / handling data outside of Excel so they usually hired interns to help with code-related stuff. Some of the divisions had full-time software developer teams, but I was the only software developer in mine.

The pace was extremely relaxed, deadlines were not tight at all.

I worked remotely, but came in to the office a few times a month. The campus is beautiful, as it is right inside a nature preserve. Everyone there is doing scientific work, so it feels like a real scientific think tank atmosphere and I loved it.


I spent 4 years at Fairchild in the bipolar memory division in Mountain View, 15 years as a professor at Portland State, and 15 years working at LANL.

On the plus side: At Fairchild I worked with a technology that made the fastest circuits at the time. I designed some great circuits and new technologies. At Portland State I had some great students, and at LANL I had great colleagues.

On the minus side: At Fairchild status = number of subordinates. I had tenure at PSU, but the institution didn't have a use for me. At LANL success = spending money. I ended up supporting code for the nuclear weapons program. The software infrastructure and software practices were out of date. We were not allowed to update things because that would threaten security. The quality of much of the code was appalling, as was the quality of the science and mathematics behind it.


I'm closing in on 20 years at the US Naval Research Lab. Which, contra the name, mostly works on basic research - AI, chemistry, meteorology, space weather, and a cutting edge spacecraft engineering group.

It's a pretty good gig. The working environment is, AFAIK, pretty similar to the DOE labs. The pay is lower than FAANG, but the work life balance is a whole lot better, the research facilities are terrific, and the research money is a lot better than, say, in academia. We tend not to recruit young engineers that well. We tend to have more success recruiting folks who have been in industry and have come to realize that paycheck size is not the only thing to look for in a career.

There are a handful of non-DOE government labs around that are similar. Not all NASA labs, but NASA/Langley and NASA/Ames are research focused. There are also FFRDCs, which are a bit more towards the applications side (hence, a bit closer to industry than academia). JPL, APL, MITRE, etc. They all have different focuses depending on how comfortable you are being close to the military-industrial complex (MITRE) or not (JPL).

Also, as a suggestion completely from left field: the State Department is always looking for people with an engineering background to work in the foreign service. They do a lot of negotiations around and provide assistance in engineering-related fields, but engineers don't tend to view the foreign service as a career. So if you sign up you get to bypass the normal seniority-based foreign service career path, and you get assigned to an American embassy somewhere overseas (usually, as I understand it, in Europe or Asia). They cover your living expenses on top of your salary. If you love to travel, it's a thing to consider.


You've covered the FFRDCs well, but another potential avenue of interest are the University Affiliated Research Centers (UARCs) -- APL is actually one of these as opposed to an FFRDC. Another well-known UARC is GTRI. It is my understanding that UARCs are more flexible in how they can compete for contracts. FFRDCs are prevented from competing against private industry and the amount of work they have is directly controlled by congress through allocation of an annual ceiling of staff years of technical effort (STE) [0].

[0] https://www.gao.gov/assets/gao-22-105278.pdf


I'm not sure my response qualifies, but I see no other responses yet so:

I interned in high school at Brookhaven National Lab working on a team that analyzed STAR (Solenoidal Tracker at RHIC) data from RHIC (Relativistic Heavy Ion Collider). I didn't contribute all that much as a high school intern but the program director said at the end that he liked the high school program because he wanted to help funnel and bring people back to help build up the labs.

My experience was that everyone there was extremely smart, but all post-doc and top scientists in their field (the team I worked on was looking for Anti-Alpha particles from gold-gold particle collisions that also helped create Quark Gluon Plasma). So I'm not sure their relative need for regular software engineers.

In terms of bureaucracy, you're still working for the government. The scientists all complained about the layers of government bureaucracy but were mostly okay with it. High-tier science moves at a pretty slow pace; coming from a tech background you might not be used to the slow pace around the actual physical construction of some of these devices, let alone the fund-seeking, approvals, testing, runs, and data collection. and 33-50% is a hopeful estimate. Let's say one is a 500k a year senior/staff SWE at FAANG. at a similar level of experience, one's pay would be very lucky to break 150k.

So fascinating science, layers of bureaucracy, slow moving stuff, PhD's in their fields, and reduced pay. Again I was only a high school intern, but I spoke with the scientists about their experiences so take my recollection with massive salt. I walked away from the summer fascinated by the work and I had a love of physics at the time; but I also left (this was 2010 IIRC?) watching the world of tech explode at a massive pace and thought that I didn't like physics enough ( I had spent my junior/senior year of high school doing a capstone project on theoretical physics and having taken a lot of physics classes). When I went to college the next year, I tried a few engineering courses, and switched to CS. I'm glad I made the switch.


There's a pretty decent need for software engineers at national labs and they occasionally do some cool stuff. If you've used ZFS on Linux for example, you've used something produced by LLNL. They do some pretty massive software projects and have a huge number of software testers. The ones I've interacted with were pretty exceptional.


BNL has a bunch of software engineers doing cool data infrastructure stuff. The current project lead for matplotlib works there!

https://www.bnl.gov/staff/tcaswell


I work as a software engineer at the JHU applied physics lab and absolutely love my job.

* Pay is very comfortable to live on in the area

* The large majority of my teammates are self motivated and driven which keeps me motivated and on my toes

* I get to constantly experiment with new tech, work on prototypes, and pursue work I'm interested in.

* Almost all of my work is software development but it's rarely pure software work. I'm almost always working with other SMEs and helping them develop their ideas into code

If you're a curious, hard working person it (and I imagine other UARCs) are great places to be


I know a lot of people who work at APL (but not SWE), love it, and have no plans to leave. Definitely seems like a great option for someone in OP's situation.


And Baltimore still has a low cost of living for an East Coast city but isn't so isolated, as say national labs like LANL and Sandia.


Speaking from my experience. All comments here are my own.

- Exceptional work/life balance that you will not find anywhere else.

- I started at 100k fresh out of masters program, at 5 YOE I was 150k. Goes up steadily YoY.

- The labs operate like a variety of small businesses. This is because there are many projects and funding sources from a variety of customers.

- From what I hear some labs are super relaxed. Like Los Alamos. The working environment is unlike any other.

- It is typical for teams to be in the same building but have no idea about each other. Overlap and rework is common, but is improving.

- Performance reviews depends on the lab and whatever review process du jour HR wants. Where I am you are in competition with your peers. Limited bonus money. So you’ll need to go above and beyond your peers to get it.

- day to day is: you work on one or more projects that last anywhere from weeks to years and report to that projects principle investigator. The PI will interface with the customer and get funding.

- for software development we need to go through strict security processes that dictate what libraries and dev tools we can use. We use self hosted versions of popular tools like Mattermost and Gitlab. No cloud. GovCloud is typically too expensive for most customers unless your working on very well funded projects.

- Managers are hands off and mostly there to ensure corporate compliance activities get done. E.g training, timesheet, perf reviews,etc.

- High level of autonomy. So your expected to be knowledgeable in your area, able to learn quickly and able to work with and network with others to deliver results quickly to customers. For example, you might be tasked with implementing an algo a staff scientist came up with in C++ for an ARM board. It doesn’t matter if you haven’t done that before. Your expected to learn C++, get a demo out, and maybe you can pull in some colleagues you previously worked with who are experts to help.

- There is a political structure in place and reputation is important. While it may not be as intense as FAANG, and underperforming is likely not to get you fired, if you consistently underperform folks will remember and your reputation will be permanently ruined. Which will result in not being picked for more desirable projects. And likely shunned. I’ve seen it happen a few times.

The lab was going to be a brief stint on my way to FAANG but will likely turn into my career.


I was a postdoc at Lawrence Livermore National Lab (LLNL) from 2013-2016, working on computational particle physics (lattice QCD). Pay was extremely high (for a postdoc) but not high by comparison to other bay-area employers. What you get instead is reliable job security, and some sense of civil service (as most of the programs are federally funded).

LQCD is kind of funny because it doesn't YET have anything practical to say about nuclear physics, which is what the lab cares about, but it will someday. So I was pretty insulated from all the weapons+complex integration stuff; my work was 'pure research', which is not that common (though it is more common at the postdoc level, which the lab views as a way of recruiting talent). But unless you can find your own funding (usually from a DOE grant), you're working on something that advances the lab's programs. I can't bring myself to work on nuclear weapons, which is why I didn't stay [there's a lot more to LLNL than that, of course, but it's what my field funnels into, broadly speaking].

The computational expertise for HPC is really unparalleled, especially at Livermore (and Oak Ridge, which I've only visited). They're consistently pushing the envelope in terms of high-performance machines which can address scientific questions that require extremely tight coupling between computing resources, rather than a cluster, and they have a lot of experimental architectures and things like that. LLNL publishes a lot of open-source software; if you've used a cluster in a scientific setting you might be most familiar with SLURM or spack.

The day-to-day can be a bit surreal. At the defense labs people with enormous machine guns thoroughly check your badge on the way in. On your walk to the cafeteria you might pass a beach volleyball court that's inside the superblock [an extra-high-security area where they've got plutonium etc.], next to a machine gun turret. Very few employers have teams that regularly win SWAT competitions.

The food was fine. No luxuries like free snacks or anything else I'd seen my tech-company friends enjoy. No dogs allowed. LLNL has a lot of employee organizations for sports, charities, exercise, etc. Transportation around the LLNL site is via sporadic shuttles but more practically there's a bike share, which is just a bunch of bikes you can leave anywhere (on the sidewalk / by a building).


I spent 7 years at JPL, which is run mostly like a national lab, based on this comment [https://news.ycombinator.com/item?id=34414812]

Is it a change of pace? Yes, but not in workload. You can easily work yourself to death if you let yourself. And politics happens everywhere -- usually in the form of missing the good projects. But the promotion frenzy is minimized, there's a sense of greater purpose in all the projects that is impossible to replicate anywhere else, and the technologists run everything. Managers help connect, but they don't typically determine your day to day priorities. In that sense, you can continuously shop around for good projects and teams without any formal change of position.


I second this comment. I spent the first 8.5 years of my career at JPL as well. My first project out of college was writing testing support tools for the Mars Exploration Rovers (MER) Project. I sat 10 feet from a small window into the Spacecraft Assembly Facility (SAF Building 179). Got to watch the rovers take their first steps. Got to do a ton of interesting things, from Satellite Radio Software Stack Testing (QA), to Project Planning Systems, to first steps into Cloud Computing. Got to fill out forms that placed some of my work into the National Archives.

Also don't forget there's other FFRDCs (https://en.wikipedia.org/wiki/Federally_funded_research_and_...) in the US.


My dad works in the Molecular Foundry division at LBNL / Lawrence Berkeley Labs (Dept of Energy and UC Berkeley), and loves it. He started there 40 years ago working in electron microscopy and oversaw the transition to digital imaging (you’d be surprised how much code they write). Good work life balance (he comes home for lunch everyday), a pension (rare these days!). His favorite part of the job is the revolving door of very smart people using/visiting the lab and getting to interact with so many ambitious (and not yet jaded) younger grad students from UC Berkeley.


I am 99% sure I know him - shares your same first initial? Tell him hello from me (TC Pekin), I always liked talking to him during some down time, and the feeling was mutual. As a grad student I always liked talking to him and hearing about all the history and his time at NCEM!


Congrats on your Zeiss job!


I haven't, but outside the DoE (but still in national lab land) my brother worked at APL and I interviewed there (after hearing all the praise he had for it). I loved interviewing with them. Everyone, even when they severely outclassed my own education level (I have a bachelor's in journalism and am a self-taught software engineer, and I was talking to a couple of PhDs with many years of experience), were super personable, humble, and passionate about their area of expertise. They made me feel like I was already part of their team even when I was just interviewing (they even went out of their way to make international interviewing work since I live abroad). If they'd made me an offer, I almost certainly would have taken it. Great people and by far the best interviewing experience I've ever had (though in retrospect, interviewing for a position requiring clearance while living abroad was probably an uphill battle).

As others have said, the work was highly self-directed. As for the need for software engineers, it was definitely there according to my brother. The scientists he worked with were capable in their field, but they needed someone capable of translating their models into something that would execute on a computer. I don't know what exactly he worked on, of course, but he's an ML specialist and was pretty interested in CUDA programming around that time, so maybe that's a clue what kind of skills he was applying.

Anyway, maybe something to check out similar to the DoE network.


I worked at LBNL as a research scientist and spun a startup out of there. It was a great place to work in some respects, lots of really smart people, awesome brainstorming, seeing Nobel prize winners around. But if you like getting things done quickly it’s quite a challenge due the the bureaucracy (ex ordering simple things could take an extra couple weeks going through lab purchasing). I once described it to a friend who worked at a FAANG company and they said “Oh so it’s like working at a big company but without the advantages of a big company”.

Working at our startup almost feels like working at the lab (ie we have scientists and are doing hard tech), but we can also move fast and don’t have the bureaucracy. So maybe consider working at a hard tech startup with a heavy science base!


I used to work at National Renewable Energy Lab in Golden, CO. Everyone in my office had engineering or science degrees. The projects were all fascinating. There were almost no people with clearances (nor any need for them) in Golden - I wanted to get one back then in order to break into the Defense industry. If you are able to get one, I strongly recommend getting one. If you have an engineering degree, I strongly recommend looking into getting your PE license. It might be worthless to you, but as a federal employee, it will get you a bump up in pay. I think that in the future, there will be some sort of licensing for developers (or at least for team leads), so this would be useful.

Among other things, NREL has the lab that tests all of the windmills in the US. There is this squirrely little canyon that gets really weird winds. Almost every day, so you don't have to wait months to get gale/storm winds.

There will be a number of safety training videos shown when you start. We made fun of them: "rattlesnakes are not your friends" or "you and your fire extinguisher". There are lots of paths for walking and rattlesnakes like to get warm in the sun. Please don't pet them. I know they're cute (well, I think they're cute), but they are pretty scared of anything big enough to eat them. Even though we made fun of the videos, everyone was a result of someone else getting badly hurt, so they're important.

The biggest threat to the group I worked with was Congress. Due to a hostile R-Congress, the budget for DOE was cut by about 30%. I expect similar things this year from the crowd in this Congressional session. The excuses for the cutbacks at that previous time was hostility towards Energy Star and conservation programs of any kind.

Public transit to/from the NREL campus is available. Denver has a pretty good public transit system.


I've known several people who worked at LANL, and they reported you might sit around for six months or more waiting on bureaucratic approval for your project. They all agreed: very slow moving, lots of red tape, otherwise fine.


I had a bad experience. I have top tech companies on my resume and they made the role look cool and I was trying to slow down a little . They instead put me into support . The codebase wasn’t in source control ; it was just a few scripts in any case . They didn’t have a sane deploy. The people that they had me working under had been there for decades but were not good . They talked down to me and did not give me tasks that aligned to my experience. I thought I would ruin my career if I stayed so I left. Much happier with a principal role where I’m at now .


I worked at LBNL (in Berkeley) and it was great. It's like academia but with no teaching responsibilities. Yes, the pay was lower and the expectations were roughly the same as MAAA (Microsoft-Amazon-Apple-Alphabet) but you didn't get fired if you didn't meet expectations.


LBL is a special place. It's still managed by UC without the involvement of any other orgs (aside from the DoE). It's basically just another UC campus, but research oriented and federally funded. The other labs can be quite different, managed by LLCs that involve a few universities and some private companies.

Things might have improved now that Bechtel is out of the picture, but for many years LBL was hands down the best lab to work for, purely because of the management situation.


I wonder if the same thing can be said about Argonne (run by U of Chicago)?

My manager at LBNL worked there because he was kicked out of one of the other labs (run by Batelle or Bechtel or one of those) for refusing to take a drug test. They said he'd fit in well at LBL- and he did.


I also worked at LBNL in the same department as dekhn. In hindsight it was a pretty nice gig, it did have great work life balance, and the problems that we solved were interesting. The people and culture was great. It was my first real job.

Shipping software at the lab usually meant manually creating a .tar.gz file and uploading it to our web page.

Now I work at a medium sized ad-tech company in industry, which is better for me, in terms of pay, autonomy, and solving interesting problems(quickly!!). The work life balance isn't as nice as the lab.

I think that the LBNL could be a great place to work for a few years before I retire.


It's been a minute since I was an intern there, but haven't seen this one mentioned yet. I spent a summer at Pacific Northwest National Laboratory (PNNL, https://pnnl.gov) and really enjoyed my time. It seemed like compensation was pretty good, especially considering the cost of living in the area. There were a number of times when we needed some help from a subject matter expert, and we could go down the hall or to another building and speak with someone who recently published on the topic. There was a lot of interesting work going on, from national security issues to storing nuclear waste, etc.

The campus there was also different than many other national lab campuses in that it's an open campus and doesn't have the military entrances that others have. It felt like the culture was much more laid back than the FAANG and other corporate cultures that OP mentioned, but perhaps more bureaucratic as well. Again, I was an intern, so didn't have much visibility into that aspect. Overall, definitely a positive experience and I could see myself there if things lined up right.


I have a friend who works at Sandia. I remember him saying the pay is lower and there is way more work life balance. There is less of a sink or swim attitude around perf there. I got the sense you can come in, do your thing, and be out by 4 or 5 - everyday.


My dad works at a national lab. The other side of this is that underperformers stick around longer, and working with them can be frustrating.


Fascinating thread. Love reading the oral histories. I heard some tales of covered up nuclear accidents happening perilously close to Manhattan at Brookhaven National Lab back in the day, and its clear from the fusion results, government science is still the largest research outlay. If you are comfortable with payscale, lack of advance, essentially academic role in natsec context, then its a dream job! I feel like the only people I know anymore who have multi-decade careers for the same entity (DoE/PPPL) are gov scientists ;)


Your experience will vary strongly depending on which PI you work for.

Expect a very different work culture. If academia is cozy to you, you'll fit in fine.


I've personally done both Sandia (2004-05) and LANL (2006-10). I'd do LANL again, but Sandia was very political with dull work. Interesting ideas, but dull work. So much bikeshedding. So slow moving.

I did workflow management systems, environmental controls in labs, and lightning prediction software.


Since nobody's said it yet: if you want to talk with people working in nat'l labs, a good place would be the US Research Software Engineer association.

Go to https://us-rse.org/join/ and fill out the form to get a Slack invite :)

edit: obviously all the people commenting here are really helpful but you'd get a wider range of perspectives there


I’m surprised how many people here who have worked in these facilities don’t seem to understand why they should never reply to a stranger on the internet who’s asking this question.


This isn't the 1960s at the height of the Cold War. I'd guess a fair amount of respondents are just trying to recruit people to come work at DOE National Lab jobs.

Personally I think the US needs a new independent energy research program if it wants to keep up with China - the DOE's record on fossil fuels is a bad joke (look into their "Clean Coal" partnerships and FutureGen, the fabled zero-emission coal-hydrogen nonsense plant they poured billions of dollars into). Their renewable energy programs have always been miniscule in comparison (to be fair, any major DOE efforts towards renewable energy tech would have led to threats of budget cuts by Congressional politicians in the pay of the oil/gas/nuclear/coal sectors).

The high-energy physics and materials sciences divisons are not bad and have a decent track record, however. I suppose people will always be needed to maintain the nuclear stockpile and deal with the huge piles of highly hazardous waste, as well. Sounds pretty boring to me.


As a counterfactual to this, go on linkedin and type "sandia national labs" into the search bar and see how many results come up with people freely listing them as their employer.

This isn't the movies, you're allowed to tell people if you work at a national lab. You just may not be able to go into detail on what your work entails.

I work in a classified environment, but if I didn't intend to remain anonymous on HN I'd have no problem telling you who my employer is.


I interned at Lawrence Livermore for three summers and a fall circa 2010, first building internal web apps for IT and then building open source websites for the climate group. As someone who needs a large amount of autonomy, I wouldn’t do it again.

My experience was that 80% of the folks are lifers, for better or worse. Work-life balance was great because everyone’s got kids and soccer games and traffic to beat home, but it was still very much a “butt in chair” system, not a “get work done” one. People regularly do “10-40” (10-hour days M-Th) or “9-80” (9-hour days M-Th, every other Friday off) schedules.

Compared with startup life and even FAANG, everything moves slowly. It’s a combination of real security, security theater, and the red tape that comes with being funded by taxpayer money. That plus the extra rules from just being near the nuclear stuff (can’t drink at lunch, guests need a background check, relationships with foreign nationals must be disclosed) made it a slog.

The most hilarious example, heard only through stories: there are two internal networks, the normal one everyone uses and the high-security one that has no internet access. They are physically separate—no cables crossed, no SSH tunnels, no bridges, nothing. But you have to transfer data between them sometimes, so you would remotely load your data onto a tape drive, and there was a tech you would ping who would eject the tape, run it over to the other network, and insert it in.

As with anything rote, the process was soon automated with a shell script. The script would take a couple of minutes to execute, and your data would be transferred. Entire workflows were built on this script. And around noon every day the latency would skyrocket, because the tape runner was on lunch.


The story about how data had to be manually transferred between the unclassified and classified networks might seem odd to people not used to how this world works, but it's extremely common to this day and absolutely necessary in certain scenarios. If you have a network that must be absolutely secure, both to data extraction and the injection of malicious software, airgapping the network and tightly controlling writable media is the only option. See STUXNET [1].

These days there are some solutions where the high-side and low-side networks are only pseudo airgapped with things like data-diodes or cross-domain guards [2] but these are really only implemented in scenarios where the frequency of high-side to low-side transfers (and vice versa) necessitates it and/or in relatively low-security (but still classified) networks.

1. https://en.wikipedia.org/wiki/Stuxnet 2. https://en.wikipedia.org/wiki/Cross-domain_solution


I worked at National Center for Biotechnology Information at the National Library of Medicine at the National Institutes of Health. It was a nice entry level job, pay was OK, environment was fantastic (low stress, extremely collegial), but without an MD or a Phd (many had both) my path there was relatively limited.

One other thing, I was absolutely blown away by the number of outrageously brilliant people I got to work with. Really some next level people.


FYI- there is also the US Digital Service. My interviewer was a former SWE manager @ Microsoft who wanted to help people in a "civic duty" type of manner.

https://usds.gov/


In addition to USDS(which has issues being under the White House), there is 18F and PIF at GSA which can also be good choices.


I work in infra for Energy Sciences network (https://www.es.net/). You can think of us as a highly performant ISP for DoE lab sites including both NNSA + Office of Science.

I worked for Peloton, Pivotal, and few other startups and have to say I love working for ESnet. The entire org is primarily composed of software, network, and systems engineers and the mission is incredibly focused on engineering and sciences. I love not pushing for quarterly profit goals (although we do have our quarterly OKRs), the amazing work life balance (I have yet to work more than 50 hours a week and end up putting in 25-35 solid hours of work in), and not have to worry about being put on PIPs or at mercy to economical forces that are affecting the FANGS.

Yes, the pay is nowhere near the FANGs and the engineering stack is a few years behind but you get back sanity back. I've been able to put more focus on getting my life back after letting myself go during the pandemic and I feel so much better health-wise and mentally.


I'm starting at Pacific Northwest National Lab on Monday in the data science and analytics group. I am coming from a 100% corporate background, and I am expecting a non-trivial culture shock to work overwhelmingly with academics and government folks. The reason I joined is I'm highly motivated to work on climate and energy projects, and it seems like an incredible place to do it.


I worked part time at LBNL in Berkeley for a few years around 2000, writing software and working closely with physicists (related to my PhD/postdoc work).

The fact that they would hire me half time (with benefits) should already say something positive about the place. It's a beautiful location and they have fascinating things going on. Sadly, I was laid off after a few years during a RIF, but got hired back six months later as a consultant (for more $$), which set me on quite a good career path for awhile after that.

I think the national labs (and similar scientist/engineer positions at universities) can be a good option compared to industry jobs, if you can find them and your salary requirements aren't too high. You generally don't have to teach, though depending on where you are, they may like to see academic-adjacent measures of progress like publications, talks, etc. My experience involved far less corporate-style BS than I later experienced in industry, but some very big egos. YMMV.

Good luck w/ your change of pace, I hope it goes well for you.


I'm surprised more people haven't brought up NASA? I was offered 180k/yr salary in 2020 doing fullstack development. While the offer was brought through a recruiting agency, I believe it was a full time position as a NASA Employee - I was surprised by the competitive salary, maybe worth looking into for others interested in the space.


doubtful it was as a NASA employee (i.e. US Govt). It was probably as a contractor. I believe (someone can correct me if I'm wrong) that NASA employees are on the GS schedule, as GS-15 step 10 in the DC area, only gets ~176K (after cost of area increase). i.e. a very senior (but not executive) manager with many years at that level.

The value of working at a national lab (vs say a military one) as an employee is that when one works at a military lab as an employee, one is a govt employee and on the GS schedule (which makes it hard for them to compete). When one works for a "national lab", one is an employee of the lab which is under contract with the US govt, but they are not paid according to the GS schedule.


yeah, looked up the old email thread. It seemed like a nice deal - if anyone is interested Bay Systems Consulting, Inc. contracts with Nasa Ames Research Center. Hope it helps someone out.


Just to add a couple places to your list that are similar to the national labs:

Simons Foundation (offices in NYC and Berkeley) Allens Institute (Seattle)

OpenAI would be more research focused as well

I have no affiliation with any of these nor have I worked at any of them, but I'm also looking towards a career in scientific research.


This thread is totally government/national lab heavy so I made a separate one about private research institutes: https://news.ycombinator.com/item?id=34417257


I highly recommend it- the culture is much better than academia or industry in my opinion, having experienced all three. You get to work on big projects that directly target big problems in the world, with the best equipment and facilities in the world. There is less politics vs academia, and less careerism- people mostly love what they are doing, and are excited about it. Generally great benefits, pay, and work-life balance.

Also, the national labs model gives you a chance to work on big teams with people from diverse backgrounds, which is a lot of fun. For example, a software engineer may find themselves working day to day with physicists, biologists, etc. and learning enough about these fields over time to make novel contributions to them.


I worked as a grad student in LBL/UC Berkeley for 5 years. At the time I didn't want to make a career of it, but if I had to go back to a national lab, that might be the one. The culture was department dependent, but at the Molecular Foundry was pretty good. The campus is gorgeous, it's not isolated like other national labs at all, pay is low compared to tech for the Bay Area, but not pennies, and the conversations and people you can meet are fantastic. Always lots of new faces with students and visitors, but the core group that I interacted with were all very kind, helpful, hardworking, and just fun to talk to about science! I can definitely recommend it.


I worked at ANL, which didn't do any weapons research, and loved it. Definitely had a lot of bureaucracy so if you haven't worked enterprise be prepared. Things can move slow. I knew some people at several other national labs and a few in DOD. I got the sense that it was the same over there as well.

That said, it was hands down the most interesting place I've ever worked. I don't think it would have been nearly as interesting if I were a remote worker though. Half the fun was taking to the "lifers" there. Learning the lore of old experiments and asking them to show you around on lunch breaks.


Currently at Amazon, did a physics fellowship with the DOE and completed a practicum at LLNL. I decided against the labs mainly for location reasons. I have nothing but nice things to say about the DOE labs and the people that work there.

The pay is good for government work. You can continue to do fundamental science research. You get great benefits, and they do some wicked cool science and engineering there (read about NIF and Z-pinch). I didn’t get to see any of the wild stuff behind security, so I can only imagine.


Briefly worked at Johns Hopkins APL.

- PHd professors were basically seen as demi-gods. It was their way of the highway, down to the micromanagement level, and they make terrible managers

- Worked seemed pointless. "Here is some money to look into this, this will likely never get implemented"

- Implicit push for producing positive results rather than saying this won't work.

- Just dreary environment overall.

Also worked at NavAir at Paux River.

- salary was absolute shit for the amount of technical knowledge/work required. I feel like setting up the instrumentation (hardware/software) for an experiment in a wind tunnel is orders of magnitudes harder than writing a web service, but maybe thats just me.

- Got to work on some cool things, but gov culture and nepotism is too much. People with military experience automatically put ahead career wise above those without. No thanks.

>but nowadays I find their performance-review and promotion obsessed cultures to be really draining. Worse, those negative feelings seem to be leaking into my personal life and slowly alienating friends and family.

As someone that has worked for FAANG for 6+ years, this isn't an issue with FAANG but an issue with you. Its fairly easy to move around FAANG (although probably not right now) to find a niche job where you can cruise control on your salary without anyone bothering you too much.


> US Department of Energy's national laboratory network

As someone who had all their personal information stolen in the famously known OPM data breach, think twice about if you really want to do the background check with SF-86 for a Secret or TS clearance. It's a real pain in the ass. Even if you have an absolutely squeaky clean record.

https://www.google.com/search?client=firefox-b-d&q=opm+data+...


Not relevant to the present but my dad worked at LLNL in the early 70s. We lived in Santa Cruz and he commuted to Livermore, so best of both worlds (good job, nice place to live). This would be insanity today since in 1970 San Jose metro was a fraction of today's size. Was hired straight out of UCLA undergrad (UCLA's very first CompSci cohort) so req's were different back then. Also grew pot and had long hair so I honestly don't know how he got a job there.


I worked at Fermilab (FNAL) as a Diploma/PhD student and really enjoyed it. The atmosphere was collegial and the science was interesting. You feel like part of a community, and everybody is enthusiastic about their work (and for some reason, triathlon).

As somebody else mentioned, FNAL is very open in contrast to the other national labs. The guards are very friendly, and tourists come in to see the buffalo and the architecture.

(One potential downside is that your experience in industry does not count much and it might be hard to get in without a proper academic career. That is not a problem with the national labs, but academia in general.)

I had this letter from the lab founder Robert Wilson on my wall, which reflects the spirit well: https://www.fnal.gov/pub/today/images/images09/WilsonBureauc...

> Dear collegues,

> an all to common failing of large institutions is to fall into the bureaucratic morass - complicated procedures, red-tape and all that. That's terrible.

> Let's try hard to keep the good old can-do informal spirit of Fermilab alive! I ask each of you to be intolerant of creeping bureaucracy

>

> Bob Wilson


I'd avoid them at all costs. There is one in my hometown. (Software Engineering Institute)

It's employees love to stroll off onto online dating sites and list of things like drug use, foreign ties, or... other things that would preclude a clearance, then act aggrieved that someone like this poster thinks people who want to weaken the internet under the guise of national security shouldn't also obstruct them from private employment and have done so since the days of "don't ask don't tell".

I interviewed with CERT a couple times... and I got a PA MMJ card after rather than have one more job interview treated like a free consulting session, and I suspect the solarwinds breach happened because so many people who shouldn't have clearances use their positions to obstruct perfectly good candidates because they feel threatened if the tech savvy at risk youth of Appalachia are lifted out of precarity.

Do not work for these people -- stay in the private sector then get a barista gig or something when you have a nest egg.


Worked at Argonne for ~5 years, initially as a postdoc and then some as a scientist. I LOVED working at the lab - you get to do pretty cutting edge stuff and Argonne being near Chicago was pretty cool. The culture is pretty collaborative!

Yes, you earn less than the private sector, BUT, you get to work on important stuff that makes a difference (sometimes) and with some very smart people. Plus, you never get bored. Labs like Argonne have plenty (~95%) of open science stuff, so you don't need a clearance.

Finding a job there is a challenge though and luck plays a large part.

The only drawbacks, besides the pay and location (for other remote labs) is the funding. If you want to pursue your own research agenda, you need to find funding, which can be challenging. The lab overhead rates are pretty steep, so it gets annoying. If you don't want to be a PI, you will need to find a PI who has the funding to support you. How secure that funding is and for how long depends. But if you have a skill set that can be used across fields and domains (e.g., applied stats or data science), then it gets easier to choose PIs and interesting projects.


As an aside, if you are an international PhD student, thinking of doing postdoc, National labs provide much better pay and work/life balance compared to Universities. However, you don't get any priority or positive consideration in your Green Card because you work in US National labs (EB1A). You will be treated the same way as working/studying in a random foreign university.


There's also NIH which does lots of computational biology and medicine:

https://datascience.nih.gov/

https://hr.nih.gov/jobs

https://irp.nih.gov/


In a very roundabout and unhelpful way to your problem: kind of. I've got patches contributed to ZFS. After project management was turned over to them it occurred to me that my code likely touches their nuclear weapon simulations - especially because much of it was optimization for the IBM power processor. So thats weird.


The national labs are pretty good, as is Research Facilitation at most of the R1 research universities. You might find good leads through the Campus Research Computing Consortium. We're still in the early days of developing these positions, but it is starting to snowgball.

https://carcc.org/


Reminds me that I started re-reading The Cuckoo's Egg over the holiday break and need to keep up the habit.


There’s a cool accelerator that is partnering with Oakridge and out of TN that might be a good middle ground:

https://www.techstars.com/accelerators/industries-of-the-fut...


I was a summer intern at LBNL as a student. Beautiful views and very smart people (and very big computers).

The cafeteria is way less fancy than tech company cafeterias. :-)

As a few posts have pointed out, there are national labs that do only unclassified work (LBNL is one). So you don't have to get a clearance or be prohibited from accessing lots of places or conversations, and you don't have to work on weapons. You do still have to sign a loyalty oath as a state government employee (the lab being managed under contract by the University of California), something that became highly objectionable to me in retrospect.


> The cafeteria is way less fancy than tech company cafeterias. :-)

The cafeteria at Sandia in Albuquerque always had extremely good posole, though.


The posole is a thing of the past. The latest food contract switch over didn't come with the recipe as I understand it.


Maybe some of the scientists there could reverse engineer a sample. :-)


I thought that loyalty oaths were not allowed in the UC system? I recall there was a brouhaha many years ago where they were requiring employees to disavow communism; several people refused and were fired. As a result, the UC system adopted a rule that political litmus tests would not be allowed when making hiring/firing decisions.


A long story! :-)

There was indeed a famous controversy about this that people commonly feel the professors (who were opposed to the oath) "won". I remember attending a conference at Berkeley that was celebrating the 50th (?) anniversary of the controversy, and the attendees were all talking about it from the perspective of the oath having been defeated. I got up and said something like "why are you all celebrating? there's still a loyalty oath and it's still being enforced!". Which didn't go over very well.

I used to know the details much better (having read several books about the controversy), but more or less as I think I recall it now:

There was an attempt to create a special oath for university employees (based on suspicion of their potentially having Communist sympathies), and the university employees rebelled against this. Part of the compromise to resolve the controversy was to have an oath for all California public employees (some form of which had already existed beforehand), and to say that people employed by UC had to take that oath, which was in some sense not singling them out as suspicious. (It might also have had wording that was somewhat less objectionable to the protestors, as there were many different formulations proposed for these oaths by different parties.)

Later, there were court cases which held that "negative" oaths were not OK (for academic employees of public institutions?) but that "positive" oaths were OK. (Like "I do not believe in Communism" vs. "I support the principles of the constitution".) So, the oaths used in state university systems were narrowed accordingly. (The "negative" forms are akin to what you refer to, which were interpreted as a kind of political litmus test, while the "positive" forms were not -- though I think they ought to be.)

Most academic workers did not find it as offensive to take a "positive" oath that other public employees were also required to take, and so this iteration of the compromise has lasted up to the present day. There were some people in the original controversy who didn't accept the compromise and left academia (I think Fred Cody, the founder of Cody's Books in Berkeley, was one!), or went to private institutions instead. However, these were apparently a distinct minority.

Nowadays a couple of people per year in the UC system (and CSU and community colleges systems) find the oath objectionable, often because they are Jehovah's Witnesses, or anarchists, or very serious about civil liberties in the abstract. (One person I heard about was the child of Japanese-Americans who had been required to take a loyalty oath on being released from the internment system in World War II, and found the whole concept extraordinarily offensive.) I've heard from many of them over the years. There are also some Protestant groups who do not believe one should take any oaths, but they've mostly been mollified by the "swear or affirm" version which has always been available in this form of the oath. Foreign citizens have also routinely been granted an exception because of concerns that the oath could jeopardize their non-U.S. citizenship (!).

Some people also worry that the oath could make signers liable to conscription (or limit their ability to claim conscientious objector status in case of a draft). There are also court cases indicating that it shouldn't be interpreted this way, but it remains a concern for a number of people.


I live in NM and meet quite a few people who work at Sandia Labs. I am sure it's not perfect but they seem pretty content. It's definitely not a high stress place. If anything, it's too slow and relaxed for some people.


Look at FFRDC's (Federally Funded Research and Development Centers) as a whole. There are quite a few of them and depending on the type of mission, some will have very dedicated software development teams.


I worked at Argonne for a year. Depending on the teams you're working with, it can be exciting or mind numbing boring. I spent a lot of time building test documentation systems, so my job was boring, but i got to walk around the campus, explore buildings, meet scientists doing cool and boring projects.

Life seemed to move slower here, nobody was up in arms about timelines to sell products to new customers for lower advertising costs, and though annual budget processes were mundane, that was for the directors to deal with.

The commute was the worst part. But the people and work made up for it.


Experiences can vary a lot. For example, we're bootstrapping a CS research group at SLAC; our goal is to do fundamental CS research. Depending on where you land, that may or may not be typical of your experience. A lot of labs are science-focused (which they should be, but sometimes it comes at the cost of awareness of the CS side).

(We're hiring by the way; a bit stale but [1] is still relevant.)

[1]: https://news.ycombinator.com/item?id=33424570


I worked at LANL for a few years as a contractor, building out their SCM tooling.

I loved every minute of it and, by proxy, got to see some really really interesting projects and meet some really interesting people.


Johns Hopkins APL is also a good option. Closer to a large cities than many of the Labs with the same type of work and benefits. Also lots of opportunity for subsidized graduate degrees from JHU if interested. Similar things to say about MIT Lincoln Lab or CMU Software Engineering Institute.

All of the Labs or these DoD Research centers typically require Security Clearances so that means you need to be a US Citizen and have a squeaky clean background and it also precludes things like using Cannabis.


> All of the Labs or these DoD Research centers typically require Security Clearances so that means you need to be a US Citizen and have a squeaky clean background and it also precludes things like using Cannabis.

I’m pretty sure LBNL doesn't require a security clearance (both from it being a DOE non-weapons lab, and because I knew someone who worked there who almost certainly could not have gotten a security clearance.)

On the other hand, federal (including contractors, like the labs) workplace policy on drug use reflects federal law, even where states have removed their own laws against certain drugs, and even where, as is the case with some cannabis use, federal law enforcement policy withholds enforcement where states have decriminalized.

(Also, the labs are mostly major-university-run and so have policies reflecting mandatory policies for federally-funded universities on drug and alcohol use.)


Interesting, wasn't aware LBNL had different requirements.


I'm not sure how you would end up on the fusion program as a FAANG engineer.

Software engineering is vastly different than the physical sciences (I've worked in both).

I worked a bit with the Sandia and some folks from national labs on the corporate side in the energy industry. The only software engineers I was aware of were ones that did IT work (Integrated workday, salesforce, etc). The scientists and mech/chem/EE engineers I met were the ones doing all the physical sciences work.


Always had a grass is greener mentality of the Labs but i suspect there are certainly their own set of challenges to overcome. Also i imagine if you are trying to level up competition is subtle but fierce at the higher levels given that its zero sum for the positions. I imagine multi year to decade long political campaigns for top positions in the orgs.

Staff levels probably pretty good if your motivation isn't to move upwards and focus on whatever your interests are.

These are my speculations.


> I find their performance-review and promotion obsessed cultures to be really draining

I don't even work at a FAANG company but this resonated with me so much.


Great thread! I noticed a bunch of the comments are from Sandia/LLNL/LANL all of which are mostly focused on the National Nuclear Security Administration side of the Department of Energy which is focused on the various aspects of maintaining the nuclear stockpile of the US.

I worked at Oak Ridge National Laboratory in High Performance Computing and did not work with anything directly nuclear at all. The HPC efforts of the DOE are under the Office of Science (separate and at the same level as the NNSA) which is focused on more wider scientific impact and application than just nuclear. The Office of Science has a number of program offices that focus on all different kinds of science from basic energy sciences/physics to biological/environmental and scientific computing (where HPC is funded in DOE).

I agree that the work/life balance is great and it is definitely a slower pace than what you would find in industry. The lab system is huge and there are plenty of opportunities but on the Office of Science side I like to break it down between what I think of as a research group and a user facility.

Working in a research group is much like academia, they mostly require a PhD and from what I could tell performance is judged on publication output. These folks also write grant proposals that come from DOE program offices for funding their own research. In some cases I have seen these groups employ non-PhDs to be computational scientists and write code.

The user facilities are long-running projects funded by the DOE at the labs to provide specific capabilities to researchers, sometimes just for DOE scientists but a number of them are open to scientific researchers all over the world. This is where I have the most experience where I worked at ORNL's National Center for Computational Sciences on the Oak Ridge Leadership Computing Facility (OLCF). These projects are generally well funded and have all kinds of interesting challenges to solve. For example, the OLCF has consistently deployed the number one supercomputers on the Top500 list and it offers those computational resources to anyone through their allocation program INCITE which supports many different computational modeling and simulation experiments. Other examples of user facilities at ORNL are the Spallation Neutron Source and the High Flux Isotope Reactor.

One thing I have noticed since moving from ORNL to industry is that the sense of shared purpose does not extend as far in the lab system as it does in company. What I mean is that with the small research group and with a user facility like the OLCF there is shared purpose with the people in those groups but it does not go much beyond that. A lab is generally made up of lots of different research groups and a few facilities but beyond the drive for "Science!" there is not a lot of shared purpose or collaboration at a macro level. The analogy I use is that a lab is a bunch of small dinghy boats that are all generally moving in a similar direction but a company is a single ship with a specific purpose driving it forward.

Overall I loved my experience at ORNL, I learned so much working with so many smart people and made friends that I will have for life.


The projects are more interesting and feel like they are actually benefiting humanity, rather than making money for some random company. Though there can be a lot more bureaucracy and friction. Pay is certainly less, though sometimes the benefits can be better.

There's also annoyances coming from political things, such as the budget not being done on time so no one gets paid or there are furlough days.


My high school internship was at PNNL. It paid well for high school! Mostly in experience---that was where I learned Visual Basic.


The first 7 years of my career were at the doe labs. So grateful for that time, but I’m happier in industry.

At least where I was at, a sky-high tolerance for bureaucracy was critical to personal happiness. You will work among the brightest people you ever met. But also some colleagues will be 100% checked out. Aspire towards those that inspire, not the working retired.


I've worked at Sandia Labs as a software developer my entire career, so I can't compare to FAANG or SV in general. Obviously I like it, or I wouldn't have been here for 20 years.

Their job classification system is such that you will want a Masters degree. I joined with a BS in CS, and until I got my MS, I was categorized as a technician and was paid the same as say someone who soldered and assembled electronics - just over half of what someone with masters in CS would get with simular rankings. I've heard it has improved since then, but there is still stong bias towards those with a masters. For many engineering jobs this makes sense, but it is out-of-touch for computer programmers and security researchers.

Apart from that while the pay is less than SV, all the labs except LLNL are in parts of the country with much lower cost of living as well, so the pay is pretty darn good for the area IMO. Benifits are good, but not exceptional. Work-life balance is great. I've hand a handfull of month-or-two long crunches in a 20 year career where I had to work 60 hours a week. The rest of the time I work my normal 40 hour schedule and go home. They have standard, 9-80, and 4-10 schedules as options (which nearly all managers will approve). After years of having every other Friday off it would be hard for me to go back to a normal schedule.

The actual work varies wildly with the project you are on. Nuclear Weapons work is extremely slow and process heavy as you might imagine, others are more nimble. Projects I've been on have varied from solo development writing software for the engineering next office over, to small agile teams on quarterly releases, to 5-year waterfall development cycles which a huge team. I've done everything from microcontroller software for sensor systems, realtime streaming data processing, desktop data analysis software, web tools for managing data stores, to pure algorithm research. And that is just a small sample of projects going not even touching supercomputer simulation or security work, that I have no experience with. I feel like I have had a great balance of interesting and stimulating technical work and necessary grunt work. Needless to say it is hard give a single "this is what working at the labs is like", and individual experiences will vary.

There are some differences from industry that are independent of the project, particularly around security. It is not uncommon for software development to be done at the unclassified Official Use Only level, but (production and test) data to be at the classified level, which is done on separate networks (or stand-alone computers). Moving between the two environments can be a time sink. Getting approval to use third-party libraries on classified systems can be a very slow process (weeks at a minimum) depending on the network. If the generic security plans won't work for your project developing a custom one can take the better part of a year. There are many security processes for which I completely respect the purpose, but am flabbergasted at the inefficiency of the execution. Contributing patches back to open-source projects is painful enough that it is rarely done. There is some third-party software that is prohibited (like JetBrains due to connections with Russia), and cloud based tools (on the public internet) are obviously not allowed. You need to be constantly mindful of what you type/say to maintain OPSEC and avoid leaking classified onto OUO systems, or leaking OUO to friends and family.

They are allowing WFH now, but most managers for most jobs will want you start on-site to help acclimate to the security culture, and to live in town to be able to come on-site to work in the classified environments when needed. You will need to apply for a security clearance once hired, and some projects are better than others at finding meaningful work for you to do while waiting for the security clearance to be granted.

As far as ethics go, on one hand you won't be asked to write dark-pattern advertising-driven manipulative spyware. On the other, most work will be related to defense applications directly or indirectly. There are some projects related strictly to energy generation and power-grid security and the like, but they are the exception. The best way to advance your career in the Labs is to move around between departments every several years, so you will be limited in your options to do that if you have reservations about defense work.


If scientific computing interests any students in the NC triangle area, we're looking for a Fall intern to do some platform engineering: https://unc.peopleadmin.com/postings/247397


My brother-in-law works at Sandia. He likes the pension, the work, and the quiet atmosphere. He doesn’t like the commute, and you still get a weird performance review culture, at least in his experience. He described a PIP process that sounded unpleasant.


Have worked at Sandia for 12 years. Will probably retire in another 20.

Pay is good. I make $145,000. That is low for Sandians with 12 years experience I think, but have had some atypically low points in my Sandia career. You can make more in 'private industry' (other defense contractors) even in Albuquerque but you will lose work-life-balance

Benefits are very good. 3 weeks paid vacation. 2 weeks unpaid. Flexible work schedule: normal hours, or 9/80, or 4/10. Generous 401k match, plan supports roth mega backdoor, HDHP+HSA available. Good WFH was slow to arrive, but corona fixed that. Nobody has ever disturbed me on vacation or implied I should not take one.

Location is ok. Abq is high crime and NM is a poor state with poor outcomes but it is very rugged beautiful. Don't knock LCOL, it provides wonderful peace of mind, and the bad parts can be easily avoided, but maybe you won't like it. Relative to NM, Los Alamos is outlier with very good outcomes (crime nil, public schools among best in nation), because it is a place that only exists due to LANL. Sandia CA is option but while Sandia Abq pays well for Abq, Sandia CA pays very bad for CA. Californians often poached by FAANG.

Clearance means govt will look up your ass, and often. You must report all 'meaningful' foreign interactions, including friends/family. Investigations occur every five years. Random drug tests: get a phone call that tells you to go to the medical facility and piss today, or you're fired. 'Forgetting' to pick up your phone only holds them off for so long. Some (rare) clearances have worse requirements: must report all dual citizens not just foreigners; random polygraphs; must ask permission to leave the country; but again, these are rare. Easy to opt out of such a clearance and it will hardly limit your opportunities at all.

Project work varies. Nuclear is the mission but Sandia has expanded to wider govt contracts. Nuclear weapons, non-nuclear weapons/military, CIA/FBI/NSA partnerships, all are possible. If this is against your morals, there are other options, but probably better to work elsewhere. Would be like working for Google while hating ads. Myself, I am okay with thise things. I have not encountered govt abuses, nor any projects I consider inherently immoral, but perhaps I am naive.

I have had good projects and bad projects. Sandia is broadly 10+ years behind the curve at software engineering. There are pockets that are better, but eg version control is spotty in some places, lots of crufty old codebases. I think a symptom of being primarily an elec/mech/chem eng shop since the 50s. But there are 12000 people and if you find the right department, combined with the job benefits it is heaven. I have, and will stay till I retire. It's the perfect job for me.


Why not join a private company like Helion or Commonwealth if you're interested in fusion?


I worked at LANL for awhile. Los Alamos is beautiful, northern New Mexico is lovely, but lab work is 99% bureaucracy. This was years ago but I needed 8 separate signatures to bring an iPod classic into a non-classified building. Insanity.


I help do recruiting at LANL if you are looking for something specific, just let me know.


I did this thing: https://youtu.be/hq03MsP1MPI . It was tons of fun and a good intro to life at Los Alamos.


I have heard universally positive things from my friends working at NREL, Oak Ridge, and Argonne.

You might also want to check out the US Digital Service, which might be more aligned with traditional SWE skills.


I worked for LLNL. Got to see the NIF. Super cool projects. You should seriously consider it, however they also work on bombs and other things that are destructive to society.


I know someone who works at LLNL and it sounds pretty typical but he has to go on trips every so often that can span anywhere from a week to a month for testing their designs and such.


Jeff Bezos grandfather (a major figure in his life) was director of Sandia.

My daughter really wants me to work for NASA.

My point is just that this sort of work has a “wow” factor to it.


I worked for NOAA, technically part of the Department of Commerce. Very different culture to tech, but sounds like what you're interested in.


I spent about a decade working for one of the NNSA labs. I think they're a good place to work, but it's a very different work environment than startup culture. Most of what I'm seeing in the discussion is pretty accurate. I'll add a few things in case they're deal breakers.

It's likely that you'll either be required to or pressured into obtaining a security clearance. Have a look at Standard Form 86 (SF86) and see if you're fine with these kinds of questions. Depending on how comfortable you are with this line of questioning, do note there will be more digging especially if you've lived abroad, had prior drug use, had prior legal issues, or a variety of other items that pique their interest. Failure to obtain a clearance is grounds for termination in most circumstances.

A clearance comes with certain responsibilities. You will have to report certain kinds of travel even if on vacation. You will have to report certain kinds of contacts even if not work related. You may not be allowed to freely publish even non-technical documents or books without prior approval depending on the level of clearance. There is a way to accommodate the lab in a way where you can mostly live your life freely. Most of this is just paperwork. However, you do give up some of your autonomy.

There is a proprietary innovation form they will push you to sign that will assign to them all innovations even on your personal time without company resources. Likely, this is not valid in California. Not sure if this is currently negotiable. They were pretty insistent in the past.

Pay for a starting Ph.D. was a little over $100k about 10 years ago. It capped out for most people at around the $130-140k region after 10 years of experience. Likely a bit different now, but not tremendously so.

While it depends on the group, internal politics heavily favors pedigree and degree level. Meaning, you are treated better if you have a Ph.D. even if it's not really necessary for your position. I'm not sure I would consider a job there with less than a masters. Practically, this means you're more likely to be the PI with a higher degree. More likely to be selected for promotion. More likely to be able to move to management if that's of interest. More likely to obtain internal research funding. To be sure, I don't agree with this, but it's a reality of the culture.

Things like time card fraud are taken very seriously. Meaning, if you work extra hours, you will be compensated for it either in terms of flex time or pay with overtime. Management doesn't like to pay overtime, so this means that you'll likely be out on time most of the time.

It's a professional workplace where people come to work and then leave. As such, no alcohol on the facility, no gaming tables, or sleep cubicles or cutesy architecture. It's not that people are completely serious the entire time, but they are professionals and there is a pretty strict separation between what's considered business activity and personal activity.

The research projects are great. There's easier access to grant money than in academics and it provides the opportunity to work in areas that are not necessarily commercially viable, but have broader impact. Getting access to the money either means being good at proposal writing, and having the right pedigree, or making friends with someone who is and likes to farm out work to you. When I say money here, you're not going to make anything more on the grant. It means being able to buy out your time to work on this particular project.

That's already probably too much, so I'll stop here. In short, I think they're a great place to work if you can fit into the culture with them. They tend to play by the book professionally and tend to hold by their agreements with you. However, there is a lot of paperwork and you will necessarily give up some of your personal autonomy.


tinfoil hat, but couldnt this be an easy way for foreign agencies to easily find whos working in a sensitive industry


Bad take. These people are not prohibited from saying where they work, a Linkedin search would turn up thousands of Lab employees.


Yes. The lack of simple security practices in this thread is jaw dropping.


throwaway - I worked 10+ years at a bay area lab (out of college) without a MS/PhD but background in physics. Did software mostly. I left about a year ago to work at a major bay area tech company.

I’ve worked with people at FNAL, LBNL/NERSC, SLAC, JLab, LLNL, BNL, Argonne - as well as NCSA, CERN, NASA, INAF/IN2P3, tons of universities and NSF research facilities.

I get paid now, but it’s because there was always chances to learn and challenges out in the national lab systems.

my tech complaints are that the existing IT people in the national lab system really are old school. Most computing is done in batch farms/grid, and if your not that you have to be creative for your resources. Cloud isn’t really something that gets used. Build systems are ad-hoc. Lab overhead rarely pays for anything software for the purposes of software development. People are generally smart but really conservative to new tech. The NNSA programs are a bit more flexible than department of science programs (Sandia/Los Alamos might be most famous). On the office of science side there’s a lot more cross collaboration between labs.

Simple things, like a $20/mo cloud service, are almost impossible to pay for. Expensive things, like a $400k purchase from Dell, is a piece of cake. Nobody is probably going to provide you a k8s cluster or anything, so running your own stuff/third party things is painful. Every national lab has a stupid opinion on which container tech to use and it’s never just docker or podman. Hope you like Singularity and Charlie Containers.

That said, I still miss it. I will probably go back at some point. I don’t miss having to run so many things, fight central IT on small things, deal with purchase. I miss the people. Age distribution is bimodal, probably peaking at 27 and 55 would be my guess for many labs.

Remote work is probably okay with Office of Science jobs, depending how close you might be to hardware. Probably not the case with NNSA work.

It’s easy to write/collaborate with software as open source among all the California labs. LLNL is the most impressive there by a large margin. Berkeley software is under BSD usually.

I was making 165 ish and now I make about 350k total compensation. Senior software would probably be around 145-185 at the very highest in bay area. If you can be an architect on a significant data intensive experiment with 200MM+ funding you might be able to get 200. Might get a 20k boost at LLNL if you are under NNSA related things.


Side question: How are these positions affected by things like government shutdowns and other Washington bullshit?


It depends on how your projects/team is funded. I’ve been on teams where a shutdown had a pretty immediate impact (people had to be shuffled around to other projects in the short term). I’ve also been on teams that get all their funding at the start of fiscal, so a shutdown doesn’t impact them until they need to be funded again.


These days it seems like management tries to keep a buffer so they can operate for a little while during a government shutdown. That being said, there are a variety of funding sources and it wouldn't be fair for some teams to be allowed to keep working while others could not, so they'll typically shut basically the whole lab down once they run out of extraordinary budget measures.

Also, if there is an actual shutdown Congress usually retroactively pays the federal employees who were furloughed but it doesn't typically apply to contractors. (DOE labs are staffed by contractors rather than government employees so that they can pay more than the government pay scales)


Dont work directly for the national lab, work for the contracting company. Pays alot better and same work.


There are also FFRDCs and other similar organizations outside of the DOE. For example, JPL is a NASA FFRDC.


Former neighbors had good things to say about Sandia. Albuquerque isn't so great though.


What are the qualifications to get a software job at a national lab?


Likely depends on the lab and the project.

For example, here's a software developer at Idaho National Lab - https://inl.taleo.net/careersection/inl_external/jobdetail.f... which is bachelors + 5 years professional.

Same lab, Cybersecurity Researcher https://inl.taleo.net/careersection/inl_external/jobdetail.f... which has bachelors + 0-2 years.

Over at LLNL Embedded Software Developer https://www.llnl.gov/join-our-team/careers/find-your-job/all... is "just need a bachelors"

while Software Developer https://www.llnl.gov/join-our-team/careers/find-your-job/all... asks for a masters.


Thanks.


From what I remember from some old all-hands slides, in the R&D groups (so not facilities, not administration etc) there was a pretty even breakdown of 1/3 PhD, 1/3 MSc, 1/3 BS, and like 2% with no degree.


I worked at LLNL for a little over 6 years, and recently took a job at a "real" software company. Other peoples comments here are completely valid so Im not going to rehash them all, but they all seem to be focusing on the positive so here's a few take aways from my time behind the curtain (taken with a grain of salt, there was a reason I left):

1. Extremely high bureaucracy, very poor facilities management. Expect a terrible shared office in a temporary building with asbestos. Dont expect to be reimbursed for your travel expenses for like 3 months, and you're very likely going to have to travel for conferences (i.e. fly to DC to make your sponsors happy).

2. The old joke around LLNL was, "hey, you know how many people work here? About half." Half the people you work with will be the smartest most productive (and friendly!) people you've ever met, the other half are also smart but have realized that their productivity has no bearing on their advancement and so have decided to not give a fuck. Since hiring is such a nightmare, its very hard to get fired. From a program managers perspective, they'd rather keep people around who massively underperform then fire them since getting a replacement could take years. The upside is that if you want to get paid to do nothing, this is an amazing place to work!

3. You dont need a clearance to get hired, although it helps, you get to do the incredibly invasive FBI-agents-knocking-on-your-moms-door clearance process after starting, and then again every 5 years for the rest of your time there. The upside is that you get to make really fun Qanon jokes with all your coworkers. Be ready for random drug tests as this is a Federal facility and Cali's pot laws have no affect. Eat a gummy at a party? kiss your career goodbye.

4. If you're cool working on nuclear weapons, you're set for life. If you dont want to operate the gas chamber at Auschwitz (how I see nuke people), then your funding will perpetually be in danger, and you will likely spend more time chasing grants than writing code. (Nearly) Everyone you work with will be totally OK working on a tool with the express purpose of killing 100MM people. I had friends/co-workers who couldnt talk to their spouces about what they did during the day. The nuke people are also very enthusiastically pro-america in a creepy way that always set my teeth on edge. All diversity/inclusivity programs here forcibly killed by the GOP, which leads me to:

5. Politics affects everything. Government disfunction is annoying enough, but if there's ever a fight over the federal budget or a government lockdown expect it shutdown your work as well. If you work on something politically sensitive (hello climate program!) expect your funding to be on the chopping block. Real shit, when Trump came into office my entire program changed its name to exchange "climate" with "earth system science" to try to run under the radar.

6. (LLNL Specific) Livermore is very expensive (about the same as the rest of the Bay, but still pretty expensive by anyones measure) with none of the things that make the Bay nice. No BART stop and massive traffic means more than 2 hour round trip to Berkeley and back. Your Bay friends are not going to come out to visit, and in terms of travel time Sacramento is closer than SF. Unlike the rest of the Bay, its temperature is not regulated by the ocean so it regularly hits 115 degrees for weeks on end during the summer.

All that said, I still (mostly) enjoyed working there. The reason I left is because during covid I moved to SoCal and after the climate program got its funding reduced the HPC/Nuke people who wanted to hire me onto their team wanted me to come back into the office (there are certain terminals for accessing classified material that are fixed in place and cant move).


So what you're saying is that you'd have been a Auschwitz guard if they allowed you to work from home? Me thinks the nuke guys dodged a bullet with you.


One thing to notice: None of the people here who have worked at these labs seem to have paid any attention to their security briefings.


I worked at Argonne as an intern, and then Idaho National Lab as an intern and then as a full time research associate for a further 18 months.

The pace is definitely much slower, and in fact you'll find a lot of people just barely doing their part and kind of hiding in the woodwork, so to speak, which annoys some people to see. You also see that in big corps too of course.

I was given an enormous amount of responsibility for being right out of college, which was basically just determined by my boss who had won a grant to do some specific type of research, so he pretty much decided how to spend the money. In retrospect with more experience, I was underqualified for the job, but I had a lot of fun and learned a ton.

My boss and his colleagues had to account for their time in something like 15 minute increments, because their time was billed out to their projects. Now, I don't think they actually tracked their time that closely, they more realistically were probably just billing time according to the money they had in each pot to keep the projects going on time.

Which brings me to my next point - the labs as far as I could see run as a form of contracting business. They compete for funding from other agencies, like the military, DOE itself, BLM, etc., to do either basic research or applied research. So grant-writing and competition for the next grant is very important. Each principal has to have many grants in the pipeline at all times to make sure their funding doesn't dry up at an inopportune time. This can be stressful I think for people, depending on your role in this process and your personality.

In some cases you'd just be working for someone else who had secured all the funding for many years, and in that case you'd never have to worry about that aspect.

I see some other comments that not having a PhD means no one listened to you... that wasn't my experience. My boss only had a masters, I had a bachelors, and we pretty much did what we pleased, and decided how to do it ourselves. I'm sure it totally depends on the area you are in and your working group though. I think our group was small enough that none of the PhD scientists gave a damn what we were doing, plus my boss had gotten the funding with his colleagues (all of whom were masters degrees, no PhDs in the group at all), so no one could really say boo to them.

I ended up leaving because if you don't at least get a masters you get capped really quickly in what you can do. In my case they didn't have a permanent role for me at all with a bachelors, I was on some kind of extended internship of sorts. I decided to go into industry instead of going for a higher degree.


I worked at Brookhaven National Laboratory back in 2010. There we very nice perks that I noticed immediately,

* access to large interesting tools and projects

* Big big budgets that often take good long looks at their decision long term making

* Generally very smart people around and interesting community events unlike working at a corporation.

* Perks and benefits well above and beyond working at corporations (often in lieu of higher pay that corps would offer)

I worked on the LHC and RHIC projects on the data processing side and it was pretty great. The laboratory itself was very run-down and not maintained well in many areas. Conversely because you're often shortstaffed you get enormous opportunities to play with various aspects of technology and head up interesting projects.

Day to day it was a lot of coffee breaks and discussions about where to go for lunch that day. There's a 'hurried' aspect that probably exists at one of the FAANG's.

Last bit of advice, most labs are either DOD or DOE managed facilities that are often managed by a large consortium, in BNL's case it was Battelle. They are often run this way to keep workers from being Federal workers with more rights and perks. Different labs are run by different consortium's, something to be aware of.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: