FWIW the greater Department of Labor (of which the BLS is a subordinate unit) has a pretty okay web API with several "SDKs" (most of which are API wrappers for particular languages). Weirdly, no Python wrapper yet but I wrote one, as well as another for Clojure
"This Dataset contains historic data (last 10 years) for the most common economic indicators. More information and details about the data provided can be found at http://bls.gov
Given the crazy complicated XML schemas people come up with, I'd say you are lucky it's just plain text, and you can parse it without needing a library.
Ironically, at work we buy a bunch of BLS datasets from a third-party vendor. Their format is even worse: an opaque binary database and a win32 DLL that reads it. 1000 lines of Haskell later, it sort of works...
(This is mostly because they chose to represent dates as integers, and they have four possible date types: yearly, quarterly, monthly, and daily. 1901, in their exciting world, could mean "year 1901" or it could mean "month January 1919". That's nice work, Lou.)
I work for BofA, but I'm pretty sure I'm the only one that uses Haskell.
I used Haskell because I hate Windows, and Haskell lets me do all my work in a maximized Emacs session without having to install anything other than Emacs and the Haskell Platform. "When life gives you Windows, don't make lemonade. Use Haskell."
I don't doubt this guy's assertion that the data could be formatted better, but if I were working on something like this I'd just be glad that I could access the data at all.
And 64 lines of code... bfd? That's much better than the bad old days of custom binary data formats where you have to write a thousand line data loader in C, worrying about issues like lsb vs msb, whether the number storage is properly following IEEE 754, etc.
I guess I'm just...old, because my first response to 'I had to write 64 lines of Python code to get this data into shape' is to shake my head and think "Kids these days!".
My point was not that this was absolutely hard, but rather that's way too hard for the totally mundane thing I was trying to accomplish---namely getting public data from a US-funded statistical agency into a form suitable for further statistical analysis (which is the whole reason they make this data public).
Also, I don't see how it's ironic that I used Python---it seems irrelevant. Whitespace in Python has a well-defined, commonly understood purpose; using tabs or spaces to indicate hierarchal data relationships is not at all standard, presumably because it creates messy dependencies across rows of data.
The reason indentation to imply structure doesn't make sense for datasets is that in data analysis you are constantly subsetting and sorting data at the row level---something you generally don't do to the lines of a compter program.
Yup, worked on this very data problem (bad data sources, especially gov) for a long time, but gave up as I didn't see it as the road to riches. At the end I hit upon an AI solution I'm sure will work, but I never finished it, still on the shelf, though.
I'm guessing that's the goal, but it's only going to be as good as the inputs provided by the different agencies.
For example, if you look for the time series I was after on data.gov, the "Download CSV" button brings you to...the BLS page, with all the problems I discussed.
Can you give some brief examples of what you might do differently? Particularly, how would you solve the problem with fewer lines of code and/or complexity?
I'm suggesting the Python code is not bad at all, so this headline is just confusing. Personally I'd just use regex replace to convert the data rows to CSV directly, not as thorough as the Python code.
Why am I down voted? Semantic Web does indeed solve this and many more problems. And I'm the only one to mention it. The semantic web has standards for data, standards for the semantic mark-up and ability to connect multiple datasets together. It also allows for standardisation of vocabulary and sharing and reuse.
https://github.com/mattdeboard/python-usdol https://github.com/mattdeboard/clj-usdol
edit: A couple of the BLS's databases are available through the DOL API:
http://developer.dol.gov/BLS_Numbers-DATASET.htm
"This Dataset contains historic data (last 10 years) for the most common economic indicators. More information and details about the data provided can be found at http://bls.gov
http://developer.dol.gov/DOL-BLS2010-DATASET.htm
"The Bureau of Labor Statistics produces occupational employment and wage estimates for over 450 industry classifications at the national level."
That last one is actually a pretty interesting dataset. I pulled it to tinker with at work last week.