I really hope that this portends advances in other government agencies, as well. The judiciary, for example, is in massive need of an overhaul on this scale. Each of the 13 federal courts of appeals have their own methods for publishing opinions on their web site, and it's often difficult to keep track of opinions published in each. Additionally, the 94 district courts also publish opinions, so it's difficult to get access to those, without paying for an aggregation service (Lexis or Westlaw). Justia does a good job, as well, but they only provide pdfs of the opinions. We're also familiar with the PACER data mining issues surrounding Aaron Swartz, so it's almost impossible to freely access and analyze all of that data.
And then there are the state courts of last resort, intermediate appellate courts, and trial courts...each having their own way of publishing decisions. In short, it's often really difficult, if not efficiently impossible, to gain access to public records to perform high-level analyses on them. This is so unfortunate, because at our disposal right now are extremely powerful computers and amazingly robust algorithms that could perform really complex, revealing, and important analyses of litigated issues.
I'm one of the developers on the Federal Register 2.0 project. Thanks for the great comments; we'd love feedback on how to improve the API or the site in general. In addition to the API client, the whole site is open source and on GitHub: https://github.com/criticaljuncture/fr2
A beautiful website designed for usability, an API for exporting public data, an open source website available on Github, and feedback through Tender Support; this is by far the greatest .gov I have ever seen. Thank you for your hard work!
Can you explain more about your team? Are you an outside company working on contract or are you federal employees? What process and team structure allowed the Federal Register site to be developed like this when so many other .govs are backwards and languishing? How can other federal websites follow your lead?
We're a team of three SF developers (Dave Augustine, Bob Burbach, and myself) working on FederalRegister.gov in our spare time (we have day jobs at airbnb.com and wested.org).
Two years ago we entered a developer competition run by the Sunlight Foundation to use content from the newly-released data.gov clearinghouse to build an open source application; we created http://GovPulse.us, and took second place; traffic grew and we kept working on the site.
Six months later, the Government Printing Office and the Office of the Federal Register contacted us; they wanted to know if we could take GovPulse and expand on it to create a new face for the Federal Register. We jumped at the opportunity, quickly formed a company, and got to work. Last July we launched Federal Register 2.0 and we've been iterating on it ever since.
It's hard to say what the lessons are; but we're definitely not typical government contractors. Largely I think it comes down to having the right people in the agencies who really want to make change, and having strong leadership from above encouraging such change. And I don't think its surprising that the Office of the Federal Register is on the cutting edge for openness--the Federal Register is one of the earliest and most important open-government institutions; and when trying to cut through red tape it doesn't hurt to have a staff comprised largely of regulatory lawyers.
I add to what everyone has said here and elsewhere: terrific work. I would love to discuss with you guys adding a commenting component to meaningfully organize public comments, particularly on proposed rules.
I am a lawyer with a tech background and had ~10 years of experience with (comments on) Federal Rulemaking at the Natural Resources Defense Council. State of the art is still often hundreds of printouts of electronically submitted comments.
I am blown away by how impressive & awesome this is.
And that is on both sides of the coin: implementation itself looks to be very solid (Rails, JSONP, yay!) AND the concept itself is amazing to me. We get 'our' apps (and data) in a format that is open for us to not only use but to contribute to and improve upon!
I definitely hope to see more of this type of stuff in the future...
Any chance you're going to work on Grants.gov? It's a perpetual problem for my family's business; we do grant writing for nonprofit and public agencies, and dealing with Grants.gov is a perpetual pain.
Just a quick heads up: 404 on "About Us" link on the "Tutorials, History, and Statistics" page. Also, the "User Information" link there gives me a certificate warning in Firefox.
But otherwise: great site, really exciting to see stuff like this happening.
Grep gives a count of 551 federal agencies. Wow. Each one with millions to billions in annual budget. It'd be interesting to start cross referencing this versus budget and media coverage. You could identify things like "the most well-funded agency you've never heard of".
And that list seems to give every new federal rule published in the last day. At 9am Eastern, looks like we already have 20 new federal rules, applicable to every man, woman, and child in the US. Surely some people are now violating laws that just entered effect today. And the amazing thing is that a rule is actually a pretty high bar to clear, much harder to get than a simple guidance document.
The second link is not all rules, it also includes notices. There are certain actions that require a notice be issued to the Federal Register when taken by a government entity. There are probably more notices than rules for a given day.
This may seems pretty boring for most people, but as I and anyone else who uses federal gov't websites with any frequency will tell you, they usually suck. The federal register has had a surprisingly awesome website for a while now, and I'm glad to see this API too. Even if the Washington websites made it as easy as possible to navigate data, there is still a mind boggling fuckton of data, I hope this API is a sign of things to come.
It would be cool if design like this would become a basis for other gov sites. But it Will never happen because of the structure of departments (who going to pay for the continued development of the base platform? There is not to my knowledge a central IT budget for the Fed gov) but it would be cool.
I can say that at first glance the Federal Register API is far ahead of DoL which requires API keys, shared secrets, HMAC SHA1 encryption (for _every request_) and other assorted annoyances that made writing the wrapper a pain (it works quite well and easily however). The API in the OP looks much less annoying.
(edit: I've also got a repo with avery rough Clojure wrapper for the same API. Its very rough as it was a self-teaching exercise but it does work: https://github.com/mattdeboard/clj-usdol)
The goal was to use an Auth framework that would be strong enough in case clearance was given to make more sensitive transactions possible. The hope is that at some point developers would be able to do more then simply consume data, but there are lots of policy hurtles to overcome for that to happen.
Also, the U.S. Government Printing Office makes available bulk data for certain publications in XML format. See http://www.gpo.gov/fdsys/bulkdata/ for more information.
I have yearned for a day where government and enterprise systems are simple RESTful/ish, JSON, javascript and HTTP friendly APIs instead of bloated SOAPware... Beautiful simplicity.
I really hope developers do cool things with this. There are so many parts of the government that appear apathetic or backwards that when something like this comes along, I feel this visceral need to give them positive reinforcement for fear they'll get discouraged and revert to their old ways.
And then there are the state courts of last resort, intermediate appellate courts, and trial courts...each having their own way of publishing decisions. In short, it's often really difficult, if not efficiently impossible, to gain access to public records to perform high-level analyses on them. This is so unfortunate, because at our disposal right now are extremely powerful computers and amazingly robust algorithms that could perform really complex, revealing, and important analyses of litigated issues.