I'm the product lead on cloud.gov... Thanks for noticing us! There are other Cloud Foundry deployments, but what makes cloud.gov special is the focus on ensuring federal agencies are actually able to use it. Federal compliance for a cloud service provider is a tough bar to clear, and without it most agencies are simply unable to take advantage of capabilities the rest of the world now takes for granted. That in turn impedes improvements in the many services the government has to offer. We've just reached the "FedRAMP Ready" status, which is a signifier of confidence that cloud.gov will make it through the exhaustive auditing process to come. Best of all, everything were doing is open source, including all the compliance work, so others will be able to follow in our footsteps. AMA!
What sorts of jobs are available there and what's the salary like? I currently do devops at a large Internet content company near Dulles and I've got federal government experience from many years ago.
We're recruiting actively across many disciplines. 18F employs a huge range of amazing people with diverse histories from both inside and outside of government. Everyone is an impact junkie. Salary is competitive with industry through a special hiring authority although capped at government levels without bonus, stock, etc. There are intangibles that come from actually being able to improve the way government approaches technology and adopts user-centered design, agile culture, and DevOps that are totally unique. https://18f.gsa.gov/join
For what it's worth, I applied to work with 18F late last year. I'm a web developer with a wide array of skills - front-end, back-end, and even UI/UX design, product design, and some marketing and management - with about 8 years of experience, mostly doing independent consulting with a couple years at a SF startup in there. I'm also very civic minded, doing a bunch of pro-bono work and (since) getting involved in community activism.
I had two phone interviews in which both interviewers asked almost the exact same questions (which struck me as odd). Then I had a third phone interview in which the interviewer - who had a law background, as I recall - seemed very disappointed that I had not worked on Google-sized teams before. Maybe he was having a bad day, but his tone made it sound like I was wasting his time. That was the last I heard from them.
Sorry, this probably isn't the right place to comment on this, but I just wanted to share my experience with you. I was somewhat disappointed in the process, even though I LOVE the mission of 18F. I was prepared to up and move to DC, and honestly take whatever salary you offered - I was in it for the mission, not the money.
I've actually started working with some Code for America folks here in my community - independently of CfA - on some city open data projects. They're awesome. :-)
What are the security and nda requirements? I recently got an invitation to a show and tell meeting (a recruiting event) but I didn't end up going. I'm afraid of having to sign onerous agreements, similar to a "never tell anyone that this was bad". I unfortunately fell victim to that at a startup, and I want to maintain my freedom to praise or criticize the government and politicians as appropriate.
I also applied. It took about a month for someone to send me a two sentence 'thanks but no thanks' response. No interview offered.
I'm no rockstar, but I do have nearly a decade of experience, half of which is gov. contracting in the beltway. And my resume has no problems attracting recruiters from Amazon, MS, etc.
The whole process (admittedly only a few emails back and forth) felt very amateurish - as if their recruitment was run by some non-technical recent grads - similar to a lot of NGOs in the area.
My experience has all been with teams and companies < 15 people, and he seemed unhappy about that.
The startup I worked on had 12 people at its height (before it was bought by a large tech company), and I was the first employee hire so I was in a position of some leadership.
He kept asking me strange questions that presupposed I knew how to negotiate with large entities like government agencies... as a developer.
That interview - plus the other two "groundhog day" interviews - were the strangest interview process I've ever experienced.
> He kept asking me strange questions that presupposed I knew how to negotiate with large entities like government agencies... as a developer.
Perhaps that's part of the skills that are needed to succeed at 18F. It sounds like these folks are in a sort of consulting position where they help other government agencies with technical projects and policy efforts. Take a look at the 18F Innovation Specialist GS-14 and -15 roles: https://pages.18f.gov/joining-18f/pay-grades/ - based on your background, they might have expected you to fall somewhere around, I'm guessing, GS-14 (1). Some of the qualifications required are:
> Knowledge of and expertise in driving and implementing technology solutions that overcome significant challenges resulting from complex or bureaucratic environments, or technically difficult problems
> Skill in oral communication to present sensitive recommendations to higher authority, to obtain compliance with policies from activities nationwide, to articulate positions/policy of vast technical complexity, and to represent the agency on task forces
> Comprehensive knowledge of and expertise in all stages of product or business development, and ability to lead complex technology and policy initiatives from inception to implementation
Bureaucratic environment is right there in the job description :-) More seriously, this sounds like a reasonably senior technical role in which one would likely interface with other agencies in the way you're describing. GS-15 is even more demanding. While it's disappointing that the interview expressed condescension at your lack of expertise in this area (and expressed anything other than professionalism), I can understand given these qualifications why they'd probe into those skills. It appears they're looking for technical leaders, and not exclusively heads-down individual contributors -- this makes sense to me given their mission. These positions seem to be about influencing the government through policy and technology initiatives, and influencing other government agencies (which is harder than influencing one's local environment), not just delivering technical projects.
I am not trying to excuse the interview experience that you had, just to be clear. I'm just making an observation about the kind of challenges they appear to have, and the kind of qualifications they might be looking for in candidates to tackle them. Innovation Specialist GS-14 and GS-15 sounds like pretty interesting roles. I am personally glad that they expect such leadership from technical specialists; this kind of broader influence is key to career growth as a technical person past a certain point. Organizations that don't expect this and foster this in individual contributors are organizations where you need to move into management in order to keep moving up.
However, it sounds like they could have done a lot better job communicating with you respectfully and professionally, as well as conveying what they're looking for.
(1) I know nothing about 18F beyond what I've read on these sites, nor about government pay grades. I'm just taking a guess based on your industry experience, and by comparing the 18F job role levels to the qualifications expected of candidates with similar background in private industry.
Thanks for your thoughts, I think you have a decent point there.
My experience does include running front-end at the startup I worked at, and even being part of the three-person group that decided on the direction of the product (with the two founders). My communication skills are battle-tested from years of contracting (and a liberal arts university background), but I do lack experience dealing with huge bureaucracies and I lack experience with business development. So maybe that was it.
That said, if your theory is right that they were slotting me in as a GS-14, they could have communicated with me about that difficulty. I would have been totally fine with whatever role they wanted to give me - which I made clear in the "groundhog day" interviews - because I just wanted to help. :-D
Look at job ads. Many, many companies of all sizes ask for experience working with massively distributed architectures, "over 1,000 servers" and things like that.
FWIW I built and worked on a server farm of upwards of 2,000 physical servers with myself and one other. (with minor labour support from another team of 3 people when needed).
it's possible to work in a small team and be effective even at large scale.
If that happens to you in the future, be honest and say that the other interviewer already asked you the same questions. Not doing so probably worked against you, as someone probably realized at some point that the same questions had been asked, and then they looked at each other in confusion wondering "Why didn't this guy say something?"
The funny thing was, in interview #2 I did mention that that they were asking me all the same questions, once it became clear that that was happening. The interviewer was like, "Oh? Really? Well this is our interview process," and then kept going. And these were 30-45 minute phone interviews, too. It was really strange.
That doesn't even make any sense. Why would you intentionally ask people the same questions two times? Something is screwed up over there. I'm surprised you stuck with it after that; I would've turned down going any farther with it. If their interviewing is that broken then you know the work must be too.
The way we operate is that agencies come to us to do projects (or we initiate shared services like cloud.gov), so we don't exactly get to pick anything we want. That being said, there is a lot of freedom within the projects we have to work on things that interest you.
I'll let @bmogilefsky describe the jobs. @18F as a whole hires engineers, designers, product managers, content writers, journalists, folks with non-traditional cross-functional backgrounds, etc.
Salary depends on job grade. See [1] for an explanation of the grades within 18F.
Then see the GS pay scale [2] to figure out the pay for your grade in your region.
18F positions are limited to 2 years, renewable by another 2 years.
Many people (self included) leave the private sector for this duration to support the mission of 18F. After the term is over many people will return to the private sector.
I'm OK with a pay cut to work on meaningful projects that benefit the American public. I'm not worried about returning to the private sector if/when I choose to do so.
That's a bit out of our control. We are able to hire quickly because we use a "specialist" hiring authority, but that comes with downsides like what you describe. I'm a developer and am making more than I did at any previous job, if that helps.
18F and other groups in government offer a great potential for impact, but we don't have the same flexibility in compensation policy because of universal federal rules.
Speaking personally, I think there's a lot of room for policy folks to dig into how gov't can better hire and retain skilled talent, but that's not 18F's function.
I can't give a comprehensive answer as to how we hire, but I can offer my personal experience: I was able to finish my last year of part-time law school when I started at 18F. I was working full time, and everyone was very respectful about having a life outside of work.
In the shorter term, do check out https://micropurchase.18f.gov. It's where we post very short term contract opportunities ($3,500 or less, and tasks typically take up to a week). I also happen to be a dev on this project, so feel free to reach out here or at micropurchase@gsa.gov if you have any questions.
It was enough of a problem that the reaction was to probably take it down and take a hard look at everything before being it back up. Maybe a hosted rocket.chat will be the replacement to allow them more granular control over security.
How do you see innovations in software able to reconcile bureaucratic processes while not remaining susceptible to scalability and trust issues?
I could easily see how the government's business process could be at conflict with the commercial sector's business process. Colluding the two in even a single Open Source project would seem to be illogical.
We can't control the decisions they make, and wouldn't want to... Each agency has their own CIO, and needs to be able to make decisions about stacks based their needs. Compliance requirements for running a service in public are so huge that agencies have conservatively stuck to ancient options, or farmed it all out to vendors. Our goal is to make the operations, deployment, and compliance aspects of service delivery trivial so they can put more of their resources (and those of the vendors they pay) into the improvement of the services they provide rather than sinking a huge portion of their budgets into redundantly addressing compliance and deployment concerns. And of course, use modern tech.
Good luck with that. I was only a government contractor and the amount of blue badges that argue how the other agency is doing it wrong / stupid and they would never use their stack is insane.
You're right, everyone is on the hook for their own agency, and with such strict regulations they are very conservative about using each other's stuff, which is effectively delegating decisions and responsibility to others that may get them in trouble.
This is a major reason for cloud.gov going after the FedRAMP JAB P-ATO recognition. "JAB" is the Joint Authorization Board comprised of the CIOs of the Department of Defense, Department of Homeland Security, and the General Services Administration. Having a triple-sign-off from three CIOs under a consistently applied set of standards is the highest social proof you can get in government that will convince other agency CIOs that it is OK to use your stuff at their agency. Normally it's vendors that go through this program... We're among the few to do it for a government-developed-and-operated service, and the first to do it for something as generally useful as a PaaS.
The other aspect is making sure everything we do to deploy and document the platform's compliance is open source and subject to scrutiny, so they can check for themselves... and ideally contribute in areas they think it could be better, of course!
You're fighting the good fight and seems you have a good path. Curious how far you guys make it (it almost seems like everyone is against using "the other guy's" stuff but the vast majority of the time it would save millions). Good luck!
We're in the process of moving to GovCloud, but that's a relatively small part of the overall compliance...fun...that goes into getting a service FedRAMP-approved.
The "Contact us" section is actually a "Subscribe to our newsletter", I would look into changing the copy there and/or providing actual contact details.
Yep, I just got Azure Gov FedRamped on my project and it was some serious gnashing and pulling of teeth. Writing all those CMS and IRS procedure docs was a great, but arduous experience. There's a larger story here though involving application outside the US, which is what I'd like to pursue after my current project. It definitely will not include Azure unless that becomes a more cost effective platform. Would love to chat with you guys about some ideas.
>I just got Azure Gov FedRamped on my project and it was some serious gnashing and pulling of teeth. Writing all those CMS and IRS procedure docs was a great, but arduous experience.
If you're pulling teeth in regards to FEDRAMP, you can join ##GRC on irc.freenode.org with fellow teeth grinders. Its a chat channel with 20+ Security Auditors and System Administrators dedicated to discussing enforcement, regulations, and systems administration for FEDRAMP and other compliance frameworks.
There's also the brand new subreddit called /r/FEDRAMP that started a few days ago.
> are simply unable to take advantage of capabilities the rest of the world takes for granted
A major argument in favor of PACER is its high-availability. Hopefully this makes it easier to build a better system with the same high-availability but a much better UX.
I'm about to join 18F (in a few weeks), working on cloud.gov.
I worked on the Google Cloud Platform team around the time Compute Engine and Big Query were launched (but spent most of my time on App Engine).
A few weeks ago at the Cloud Foundry Summit the folks from cloud.gov.au gave a keynote. Cloud.gov got some nice shoutouts - the Australian counterparts reused a lot of the cloud.gov work, thanks to the transparent open-source approach.
I work in GSA as well, and I have a lot of trouble getting software approved that you use all the time, even getting nginx approved took a long time. Forget about vagrant, virtual box, recent versions of python, etc...
How were you able to get ATOs to operate most of your systems and work through these policies?
Edit: they ignore a lot of the rules that GSA (among others) mandate the rest of the government follow. One great example is cloud.gov, operating on the public Internet without finishing the fedramp approval process. It also doesn't comply with the TIC requirements GSA pushes on the rest of the .gov.
The hard problems in government IT are being able to pay enough to attract top talent and dealing with all the authorities and oversight that parent refers to. If you remove those two legally-required obstacles, you're cheating. That's not necessarily a bad thing, but comparing GSA's pet project to normal government is disingenuous at best.
I’m Noah Kunin, the Infrastructure Director at 18F/GSA.
While the Department of Homeland Security (DHS) owns the Trusted Internet Connections (TIC) policy and controls (https://www.dhs.gov/trusted-internet-connections) we’ve been working hard with DHS teams to clarify and improve implementation guidance.
We hear you - loud and clear - and understand there’s a lot of frustration.
Check out our updates with one of our pilot partners, Amazon Web Services:
Don't get me wrong, I really like the stuff you guys are doing, but I think 18F and the rest of the Gov operate in two different realms of bureaucracy and feasibility. Your management is on board with this stuff, while for the rest of the Devs in the government it's an uphill battle from the bottom.
There is still a large gap between software that is on the approved list and the stuff you use, but as I am sure we both know, there are waivers and blind eyes for that. If someone really wants it, it will happen.
You guys are paving the way with your tools and process. But, please be careful, one serious security breach and it's kaput. For example; the incident with slack. I know it was trivial and people chose to blow it out of proportion, but regardless, it will now be harder for me to get slack approved for our team.
(I know very little about this world) why do you need approvals to use open source software? Certainly the purse strings are tightly held, but why is use of free software restricted?
The way I see it, there are multiple grades of free software. There is difference between downloading a linux distro or apache project and downloading some dude's python library from pip.
Any software used on government systems has to go through an accreditation process. This is to prevent you from pulling in some random code that may have security vulnerabilities or backdoors. Just because it is open source doesn't mean it is secure.
As an example, we've had to run security scans on jquery for god's sake and justify every occurence of random number generation to make sure it wasn't used for anything security related....
There are other things like warranty, support, size of the community, etc...
Overall, GSA is one of the best agencies when it comes to open source. It has definitely come a long way, but still a long way to go.
> Overall, GSA is one of the best agencies when it comes to open source. It has definitely come a long way, but still a long way to go.
I think it's never been a better time to get approvals/clarifications/etc for open source at GSA. GSA's CIO posted a supportive comment on this issue here, w/r/t the White House's proposed source code policy (and 18F's comment on it), and reinforced that GSA has an "open source first" internal policy for the enterprise on the books:
Of course, policies only give people the space to spend energy making the policy really mean something -- which I encourage you to do. Find me on 18F's GitHub or GSA email any time if you have ideas you want to talk about.
This is really cool. 18F has been doing a lot of amazing work, it seems like the possibility of government tech not totally sucking may be on the (somewhat distant) horizon.
:) Thanks! I also work at 18F, but not directly on cloud.gov. Govtech is a huge market, and I wouldn't expect any individual effort to turn the entire ship. Hopefully, though, we can provide a space to experiment and build with modern solutions and practices to make them more "normal."
I'm especially a fan of developing in the open - you can see our commits and issue discussions and keep track of how we're doing as your civil servants. The other thing I love is that all of our work is in the public domain or CC0.
As someone who develops for a Gov't agency, this is the wrong move. So much money has been spent on mimicking the private sector innovators. Millions and millions of tax dollars spent on these type of projects.
The focus really needs to be on building a process to better integrate private cloud services and gov't applications. I mean they already are for the most part: https://aws.amazon.com/compliance/fedramp/
You listed two IaaS provider and one SaaS provider. cloud.gov is a PaaS provider. There is no FedRAMP PaaS available to agencies, and PaaS is the level of abstraction needed to accelerate delivery of services the government itself provides.
We also leverage the fact that AWS GovCloud already has FedRAMP status as an IaaS to accelerate our delivery of cloud.gov (which sits on top of it), so we're piggy-backing on the money and effort already spent and eliminating the millions we would otherwise spend by having the government run the IaaS layer.
In other words...
> The focus really needs to be on building a process to better integrate private cloud services and gov't applications.
Actually, there are several FedRAMP PaaS' available to agencies. A quick look at https://www.fedramp.gov/marketplace/compliant-systems/ lists several compliant PaaS offerings, including one (Autonomic Resources LLC – ARCWRX) based on OpenShift. It's great to see activity in this space.
Acquia is FedRAMP accredited as well. Acquia is optimized for Drupal. Drupal now powers 40% of .gov sites in the US (and growing). It's a fully managed and supported platform, which offers not only the CLI integrations that developers appreciate but accessible, responsive GUI DevOps tools and 24/7/365 application support.
If you want a hosted Cloud Foundry, my employers run one in Pivotal Web Services[0], aka PWS, aka p-dubs.
IBM BlueMix includes a CF instance. There's also AppFog and AnyNines.
Naturally, I think ours is the best. It's usually the most up-to-date, based on the internal monitoring I've seen. We upgrade PWS to the latest Cloud Foundry release usually within a week of one being tagged.
My favourite part about it is that thanks to BOSH, basically nobody ever notices that we upgraded the entire platform underneath them.
You can certainly host it yourself on AWS, GCP, Azure etc.
The general services administration is restricted to only assist federal government agencies, so you wouldn't be able to use the version being managed for the federal government.
Hi Bret, long time fan-first time commenter! In the govFresh excerpt that led me here you state the 'Federal compliance for a cloud service provider is a tough bar to clear'. Have you discovered the data policy map is in line with US corporate standards (e.g. Microsoft data security) or separate and distinct? I'd like to understand to what degree.gov innovation into the cloud is at parity with existing, more public facing standards or rewriting what we know and observe today.
Haven't read every comment but a quick ctrl+f didn't find any keywords on the topic;) thanks for the insight!
We are using AWS GovCloud for our base IaaS layer, but we run a PaaS layer on top based on Cloud Foundry... That's the actual cloud.gov service being provided, the location on AWS GovCloud is just an implementation detail.
Cloud.gov is a PaaS based on Cloud Foundry. The idea is that it can run on pretty much any IaaS provider (as long as the Cloud Foundry plugins are written), certainly not limited to AWS.
at first I was going to make fun of them for choosing "Cloud" half a decade after that buzz word reached its peak, but then I looked at the service and it is pretty cool
