Django has the right response to this issue: disable entity handling altogether. Last I checked, this was not the Rails response; you can pass doctype declarations in with XML in POST requests, and they're honored (this was clearly the case during YAML-debacle #1, because we took advantage of it, but I think it remains the case now).
There is no good reason that core XML handling in frameworks should be dealing with dynamically defined entities at all.
One of the reasons that things get extracted to gems is specifically so that people who use it can improve stuff without the whole process of contributing to Rails (and its release cycles) getting in the way.
For example, ActiveResource[1]. Now that it's a gem, the people who use its features can actually take over the maintenance, and release versions independently from Rails proper. It's gotten a lot more contributions lately now that it's independent.
It is also a way of burying hair-on-fire security issues, and of encouraging plugin-style development for what is properly considered a piece of core functionality for Rails.
Not the same thing, he's talking about how the xml are parsed, not whether they are parsed or not. If you install this new gem in RoR, the xml parsing would still do what tptacek is talking about (processing external entities), which is something that should be off by default because you almost never actually want it.
I understand that, but it's the default that's important here. If this gets merged, Rails will no longer be automatically parsing XML from parameters by default.
Yes, the gem that's extracted could also be checked over for things like the processing of external entities. I'm not familiar with that part of the code, so I can't tell you what it does or does not do.
Sorry, I read an opinion from your comment immediately upthread, which is that you believe that simply not parsing XML is a reasonable step towards handling XML securely. I disagree with that opinion.
> you believe that simply not parsing XML is a reasonable step towards handling XML securely.
Ahh, this may be the difference, sorry. I don't think that it's a reasonable step towards handling XML securely. I _do_ think it's a step towards not exposing people who don't use XML to attacks on their site via XML-parsing code paths.
One of the worst parts of the recent security... situation was that people who didn't even support YAML or XML for their API were still vulnerable. It's these people this helps, not people who actually do use XML.
I totally agree that this isn't useful to people who are actually using XML, except for my comments about quicker releases and fixes by detaching it from Rails.
It's not just billion laughs, this fixes related but different entity-expansion attacks, as well as fetching of external resources (DTDs or entity definitions), which is a totally separate issue from billion laughs.
It's probably out of spite. At an OSS project I worked on previously they'd always hack into competing projects when they trashed the project's security.
If you live in a glass house, don't go around throwing stones.
That's 100% completely wrong. We've worked with Rails' security team in the past to coordinate releases fixing similar issues, and they've helped us out with discovering and analyzing issues of our own.
They're a great group of people and I've got nothing but good things to say about our work together.
Yet another reason why open source is fantastic: friendly competition can actually be friendly. It rocks.
Or if there's actually a history of web frameworks doing behind-the-scenes coordination and watching each other's backs... oh wait, that's not as good a gossip narrative. Never mind.
For a second I got scared. Then I saw the fixes and realized that I don't have to panic-update. It is great that the issues are fixed but none of them seem very dangerous. DoS can always be done and I doubt many people use the XML serializer/admin log. Not sure about how many people use get_host(), but I imagine its not a lot of people.
I've always wondered: What stops someone from uploading a 10 GB file using Django's upload forms? The file could even be very small when gzipped, it sounds like a reasonable and easy DoS attack.
I too upgraded and everything went smooth except got a "Bad magic number" error[1] while running south migrations. The fix is to delete all .pyc files (even those inside the virtualenv)
It seems particularly crazy to me that many people find out about security releases (or any releases) like this. What happens when one doesn't make the front page of HN?
Apologies for the shameless plug but at BundleScout we've been working on a tool to notify you when things like this happen[1]. It's not perfect, but it's a step up from this I'd hope.
We don't just announce this stuff on HN; there's a whole lot more:
* Post security release announcements to our blog.
* Post announcements to django-announce, a very low-traffic mailing list specifically for security announcements.
* Post announcements to django-users, a mailing list with over 20,000 subscribers from all over the Django community.
* Post announcements to django-developers, where most of the people who hack on Django hang out.
* Publicize these releases on Twitter, Reddit, HN, etc.
Further, we work with people who re-distribute Django (e.g. as part of RHEL/Fedora/Ubuntu/etc.) and people who use Django in high-risk areas to get them early access to security notifications [1].
So yeah, we make a lot of noise about security issues.
BundleScout looks nice, but it kinda sucks that you'd make people pay to hear about security issues. We're going to continue to do our best to make sure people find out about them quickly and for free.
(Disclaimer: I'm the other guy behind BundleScout.)
The Django project does a fantastic job with announcements. (So does Rails.) And if every open source project had the infrastructure to make that much noise, the world would be a better place. But of the 100k or so Python/Ruby/NPM projects I know about, only a small handful do, or could even manage it.
And if they did, would you want to be on every one of their mailing lists?
Bleach, for example, has more users than I have twitter followers, so I doubt most of them heard about v1.2.1. (Out today! Go get it!)
> but it kinda sucks that you'd make people pay to hear about security issues
Well, like you said, there are other ways to hear about updates, at least from the big projects. This is the convenience of not needing to join dozens of mailing lists and/or check dozens of websites periodically.
Would you maybe consider making a distinction between security releases and regular run-of-the-mill update releases?
I see a real value in a service to let me know that there's a new version of one of the dozens of things I depend on. But like I said, it seems janky to rely on a for-pay service to find out about security vulnerabilities. So what about making the security notification free to all users, with the non-security update notifications a for-pay addon? Seems like a good way to strike a balance between developing a self-sustaining business and helping out a community.
Second, I think the severity of XML external entity injection is understated in the advisory. This class of vulnerability isn't just a DoS in general. Does something about Django specifically limit the impact to just a DoS?
Yes: Django only uses XML in one very limited place, the serialization parsing framework, which isn't exposed publicly in any way. It's used for data backup/restore as well as for test fixtures, but that's it. Further, the same framework also supports JSON, and it's extremely rare to find a Python developer who's going to chose XML over JSON. So in terms of Django itself the attack surface is extremely small.
That said, if you're parsing XML in your own code you should be looking at defusedxml, also announced today: http://blog.python.org/2013/02/announcing-defusedxml-fixes-f.... So maybe we should make a bigger point about that? If you've got suggested edits to the announcement let me know, happy to incorporate 'em.
You're right, thanks for pointing this out. Although (as jacobian mentions) Django's exposure in general to these attacks was quite limited, we still shouldn't describe these XML vulnerabilities as DoS-only.
I'm working on updates to the release announcement and release notes for 1.3.6 and 1.4.4 to make this clearer.
The impact varies with the platform and parser being used, and with the extensions installed. I don't know the specifics of ExpatParser (the one used by Django), but in general, the "careless" parsing of unsanitized user-supplied XML might lead to:
- DoS
- Disclosing of sensitive files (in general, the XML has to "validate". In PHP, it's always possible to read any file accessible by the process parsing the XML).
- Making arbitrary network connections (with this an attacker can portscan a network, attack vulnerable services that would be protected by firewall, and/or use the vulnerable server as a restricted kind of proxy).
Edit:
As a clarification, I'm not implying that any of the above (apart from the DoS issue) is applicable to Django. I'm just saying what can possibly happen (and often happens for PHP apps).
The vulnerability we fixed is not just entity expansion, it's also accessing external entities. Although this is a DoS vector, I think it's true that it could be used to gain information about the presence or absence of files on the local filesystem, or even potentially information about their contents (given access to the results of XML parsing).
You can also use it to reference arbitrary remote URIs. Suddenly your XML parser is making outbound connections to anywhere the attacker specifies, looking for more XML to process.
On some Windows systems, you can use a UNC path (\\share) and force the parser to try and connect to a share that the attacker controls, which will cause windows to hand over the user's hash during the handshake.
After the recent Rails issue we added a PagerDuty service which is subscribed to django-announcements. When it gets a message from the list it filters the subject for "security" and alerts the on-call engineer who can evaluate the severity (as it applies to us).
I'm old fashioned, I guess. I deploy my apps on Debian stable, and they (like most distros) have a competent security team which stays on top of all this stuff. I have a script running on all my servers which looks for available security updates to my installed packages every few hours and emails me in the event.
Sidenote: This is one reason why I avoid virtualenv and its ilk for my apps unless I absolutely have to use them (e.g. need a newer version of something to get a feature I can't live without) -- I don't toss aside the good work of vigilant security pros lightly.
Your tool doesn't currently show the last security update, according to your website last update was 2 months ago. Django wise, there is the Google group which you can be follow for all updates , you can also subscribe to the RSS feed of django's weblog.
Your tool seems interesting, i like the idea of being able to keep a list of all the software i use in a single place and get notified when a new update comes out(had a similar idea myself), however i would need some kind of reassurance that such an application is reliable and wont let me in the dark for some important update.
Typically, the updater runs about once a day. If you take another look, it will show the latest security update :) We're working very, very hard to guarantee that no updates go missed and ensure that you receive a notification within 24 hours of any update going out.
And while, yes, you can watch the RSS feed or Google group, do you really want to do that for Django and the other 29 libraries you use?
It shows the update now, it didn't when i posted my reply, however its still fast enough compared to when it was announced by django's official communication channels.
A bit off topic but i have some thoughts on the website since i might be interested to using it in the future: It would be great if you could lower the 'within 24 hours' to something like 'within 2 hours'. I understand the difficulty of that since you are tracking over 100.000 packages according to the homepage but a 24 hours dilation is a bit too much, i don't deploy code yet in any high traffic sites which could be subject to a 0day security attack but still i find the dilation too much to consider it as a viable option for the future.
Is the XML expansion issue truly just a DOS issue, or is it possible for attackers to get feedback (either directly or on subsequent requests) on the XML parse result and thus read local files via a SYSTEM entity?
AFAIK the XML entity expansion attacks are DoS-only, but the other fixed vulnerability (external entities) may be usable in some cases to gain information about local files (again, presuming Django's XML model deserializer is exposed to untrusted input, which it is not by default and should not be). I'm working on updating the announcement and release notes to characterize that more accurately.
I don't see the usual strong admonition to upgrade ASAP. (For example, the December 10 updates said, "All users are encouraged to upgrade Django immediately." This bulletin does not.)
From a quick skim of the 4 fixes, perhaps not all Django sites may be at risk. For example, if they are: already checking the 'Host' header or not using it; not using Formsets; not using multiple tiers of admin users; not using Django's XML (de-)serialization.
But, I'm least confident of that last one. Do all Django installs use Django's XML deserialization for something internal/everpresent? Or is this just a risk for users whose apps explicitly do extra XML operations?
These aren't "drop everything, take the site down, upgrade now" issues, so yeah we'd like to save the tone for that sort of thing.
... but seriously, please do upgrade as soon as possible! There's really no such thing as a "minor" security issue when it's 2am and you're being woken 'cause the site's down.
No, Django's XML deserialization is not used automatically. It is only used for mass importing objects to the database, which most sites would allow only admin users to do.
Ah yes, the billion laughs.
http://en.wikipedia.org/wiki/Billion_laughs