Ok, so to verify: the academic article I referenced, in your opinion, is incorrect? I would love a more-detailed response (hell, it sounds like it might be publishable!).
I would also love it if you could actually respond to the relative merits of the author's "developers will hack randomness out of their projects" belief (which I claim only happens to idiots) vs. my alternative "developers will not realize the security of the libraries they are using is reliant on an unrelated set of decisions made by the system integrators" (which apparently happens to engineers who don't sound like idiots).
(Also, I appreciate it adds rhetorical strength to your comment to point out that key streams don't "run out of key" and urandom doesn't "run out of entropy", but I never stated that, never believed that, and would not now claim it: those statements are entirely orthogonal to any of the arguments I made, and seem to have only been included to make your comment sound like it responded to mine.)
I guess put shorter: care to respond to anything I said? Anything at all? :( All you've done is restated something from the article... (I mean, you certainly don't have to respond to me, but then I question the decision to leave a comment in the first place: my comment's are long because I had to respond to all the substantive points in the article, not just state my position :/).
You both quoted and interpreted my post from a few weeks ago; I don't think you're far off, but I made a stronger claim, which is that developers should actively avoid /dev/random. I'm just clarifying.
I don't understand how people who understand the relationship between CSPRNGs and (say) CTR mode can believe that there's merit to using /dev/random after urandom has been seeded.
Clearly my argument is predicated on the idea that /dev/urandom's implementation doesn't guarantee randomness is seeded, and it is not possible as the developer to guarantee that randomness will be seeded: neither the library developer writing the cryptographic library nor the developer using the cryptographic library often has any control at all over the systems on which their software will execute. You keep preceding your restatement with "if" and "once" and "when" with relation to this seeding having completed, but are providing no way for the developer's code to know that this has worked; in comparison, /dev/random always works.
And again: I have not, am not, and will not, state that /dev/urandom runs out of entropy. I further did not state that your old post claimed these things: I only stated that your old post pointed out the seeding limitation of /dev/urandom. I continue to not see why you keep stating that. Nothing I have said, either explicitly or implicitly, claims that when /dev/urandom "runs out of entropy" it causes a problem: I would never claim that, as even if I thought I believed that (and I do not) I do not consider myself qualified to say that (hence why I rely on arguments from practicality only).
It is easily possible to "guarantee" that /dev/urandom is seeded; just seed it from /dev/random. It's a 1-line shell script, or 10-or-so lines of C code.
I think it's a rubber chicken security measure, but if it makes you feel better, that's fine.
What's not fine is relying on /dev/random as a source of entropy for your application, as opposed to seeding urandom.
Is there anybody here who knows enough Linux internals to say that it has sense to actually seed /dev/urandom by copying to it from /dev/random? I'd believe it would be enough just to read some limited number of bits from /dev/random once to be sure that the pool is populated according to the /dev/random estimate. Then it should be enough to continue reading /dev/urandom. Anybody?
(However even reading a limited number of bits from /dev/random on the first call of your code can freeze the router with no mouse and keyboard (or anything else if this is done often enough). So I still consider the above practice a bad strategy for a library writer. The person who designs the initial OS boot however should think about it. The reason that even Linux kernel doesn't do this automatically even once at the start of the whole system is because the kernel developers know that they don't have access to enough entropy unless the hardware somehow provides it. I believe having RDRAND type (not in the sense of alleged backdoor but in the sense of a known hardware source) of entropy in every CPU or chipset is actually a good thing solving an actually existing problem. We are far from that state still, unfortunately.)
As saurik mentioned before trying to seed /dev/urandom by reading once from /dev/random relies on an implementation detail, i.e. the fact that both share the CSPRNG and therefore seeding /dev/random also seeds /dev/urandom. IMHO it is better not to make this assumption but assume that /dev/random and /dev/urandom are completely independent because the implementation may change.
There may of course be a specification stating that this relationship between the two has to be this way but then many things have been carve in stone and changed later. So unless there is a compelling argument to prefer implicit seeding of /dev/urandom over doing it explicitly, I would definitely avoid it.
If you want to wave a harmless rubber security chicken over your software, explicitly seeding urandom is trivial. What's dumb is continuously exposing yourself to /dev/random's weirdness --- once anyone has successfully read from /dev/random once, you know from that point on that urandom must be seeded.
Reading from /dev/random does not necessarily ensure that /dev/urandom is seeded because it relies on an implementation detail. But reading from /dev/random once and using that numbers to explicitly seed /dev/urandom by writing to it will work without relying on implementation details. That is why I would prefer doing it explicitly by writing to /dev/urandom instead of implicitly by reading from /dev/random although it makes no (big) difference given the current implementation. But this is what you suggested elsewhere anyway.
Sure, but if /dev/random supplies me 256 bits without blocking, then the random pool behind /dev/urandom must have been initialised with at least that much estimated entropy by that point too, right?
> It is easily possible to "guarantee" that /dev/urandom is seeded; just seed it from /dev/random. It's a 1-line shell script, or 10-or-so lines of C code.
Neither today's article nor any of your comments today (before this one) nor what you wrote a while back makes the statement "the correct solution seems to therefore be that when your program starts you should seed /dev/urandom from /dev/random and then continue to use /dev/urandom".
So, is this the correct option? All everything has been saying so far is "do not use /dev/random, use /dev/urandom", which I maintain is actively-harmful advice and should probably be retracted by everyone saying it: making it into a black/white issue is not just silly but will lead people into a situation where they've made confusing mistakes.
This statement so far seems to be well-argued (if nothing else, no one, including yourself, has provided any argument against it; in fact, you seem to presuppose it in all of your statements, assuming that someone smart enough to realize that this is a problem has gone to the trouble of making certain /dev/urandom is seeded sufficiently, despite all evidence to the contrary):
"What's not fine is relying on /dev/urandom as a source of entropy for your application, without first seeding urandom."
Your version, however, seems to only have one argument going for it: that under load a system using /dev/random might block. If the choice is down to using /dev/urandom or /dev/random, I have argued in various ways that this should not be considered as serious of a problem. So far no one (including yourself) has provided any arguments that I'm wrong except to emphatically restate the premise. Like, seriously: I've now spent hours arguing with you, and you haven't actually made any arguments :(.
I also, however, have stated (such as in the other part of this thread I linked to while responding to your other comment) that I think that there are opportunities for a more nuanced discussion of this. Humorously, the solution I came up with was different than yours (I suggested blocking on /dev/random once to make certain /dev/urandom is "ready") and int thinking about it might not work (it relies on an implementation detail of /dev/urandom that I realize I don't know), which frankly just goes to underscore my argument: you and this author telling people "do not use /dev/random, use /dev/urandom" is harmful advice that should apparently be replaced by "use /dev/random to see /dev/urandom, and then use /dev/urandom for subsequent random usage", lest people either 1) use /dev/urandom in a way where it might be used before it is seeded or 2) hear about this issue and then enact some incorrect workaround.
> under load a system using /dev/random might block
The most interesting thing I've learned from this discussion is a comment of Dan Bernstein's, quoted by the OP:
> Cryptographers are certainly not responsible for this superstitious nonsense. Think about this for a moment: whoever wrote the /dev/random manual page seems to simultaneously believe that
> (1) we can't figure out how to deterministically expand one 256-bit /dev/random output into an endless stream of unpredictable keys (this is what we need from urandom), but
> (2) we _can_ figure out how to use a single key to safely encrypt many messages (this is what we need from SSL, PGP, etc.).
> For a cryptographer this doesn't even pass the laugh test.
If I understand this correctly, what DJB is saying is that the idea that reading random numbers from /dev/[u]random decreases the amount of entropy in the pool is ludicrous. So the real bug here is that: once the pool reaches a threshold level of entropy, even /dev/random should never block again, because the entropy in the pool is not decreasing.
() It should be seeded from keyboard presses, mouse movements, etc. (i.e. real entropy sources) at OS installation time (if there's a persistence mechanism) or at boot time (if there isn't)
() If called before being seeded, it should hang, but in practice this should never happen
() It should never hang thereafter
() If we accept DJB's argument that adding entropy later can be harmful (and since we've already agreed it's unnecessary), we shouldn't do it
() There should be a standard protocol by which a freshly cloned VM can be told it needs to reseed
So, how do we convince the kernel devs of this, and put an end to this whole argument?
I don't think that's the only logical conclusion one can come to after reading DJB. Periodically rekeying has the beneficial effect of adding some forward secrecy to your RNG, so losing the state once doesn't necessarily permanently bone you.
People are working on a better Linux random driver.
Do you know if those folks are working on making /dev/random faster*? I'd really like to see even traditionally "non-cryptographic" RNG's (e.g. rand()) just use the kernel's CSPRNG.
No developer is making a mistake by using urandom. Every Nacl user, for instance, is relying on urandom. Every Golang user is relying on urandom. Those are two crypto libraries with impeccable pedigrees.
I don't understand the rest of this comment. You build an argument about I didn't write "use random them urandom". But I did; I just didn't word it the way you seem to want me to have. Specifically, I wrote: "But [the problem is] also easily fixed in userland: at boot, seed urandom explicitly".
Developers should use urandom to the exclusion of all other CSPRNGs.
> Specifically, I wrote: "But [the problem is] also easily fixed in userland: at boot, seed urandom explicitly".
That is not "use random then urandom": that is "configure your computer differently"; the person who has the job of following that advice is no longer the person working on the secure software, but the person building the computer; it might even be the end user, whom we have absolutely no reason to believe has any clue how to secure anything (including his lunchbox). Now more people are involved, we have reason to believe that these new people do not understand this issue well, and there is a possibility of failure mode.
This failure mode seems to only be being defended due to some minor performance loss on startup of a select subset of processes. Can I hope to assume you also encourage people to not use stack protector, because the 1% global performance loss is unacceptable, as clearly we should simply trust developers everywhere to write programs that don't have stack overflows? It is exactly the same argument: relying on someone to do something correctly, someone we have evidence to show doesn't do that thing correctly, because we are afraid of some minor performance cost associated with guaranteeing that the problem simply cannot ever happen.
> No developer is making a mistake by using urandom. Every Nacl user, for instance, is relying on urandom. Every Golang user is relying on urandom. Those are two crypto libraries with impeccable pedigrees.
And apparently OpenSSL, as stated in the paper that analyzed the wide-spread cryptographic fail that happened a year or two ago.
Then I don't understand the point you're trying to make.
You have argued that people should always use /dev/urandom, as opposed to the common understanding that /dev/random is safer to use if you need high quality randomness, due to the cold-start issues with urandom.
You said that "Not using urandom has caused way, way, way more real-world security problems than any bizarro embedded systems cold start entropy problem ever has.".
I asked you for an example of security problems caused by using /dev/random instead of /dev/urandom. You provided a link with no such examples.
The common understanding about /dev/random is incorrect.
I'm still not reading anything about "long lived keys" in your clarification here.
If you are concerned about cold-start entropy on first-ever boot, then seed urandom from random explicitly at boot. There is no reason ever to use /dev/random in crypto code, and you will find that the best crypto code doesn't.
I think I'll end my part in this discussion right here.
> I don't understand how people who understand the relationship between CSPRNGs and (say) CTR mode can believe that there's merit to using /dev/random after urandom has been seeded.
If I were more cynical, I'd say that the NSA is spending money on giving such people a loud voice, so that they can filibuster attempts to actually fix things across the board. ;)
The article you cite cites a Peter Gutmann article [1, Section 3.6] that details what programmers will do when things don't work as expected.
Additionally, what the article recommends (Section 6.2) is that OS generators behave like FreeBSD: block only until enough entropy has been gathered, and then block no more. That is achievable on Linux using tptacek's suggestion, but of course a native solution would be better.
I did not claim that no one would do something so stupid as to hack the code to remove randomness: I am claiming that those people are so "beyond help" that trading them off against the people who simply fail to realize that decisions made during boot affect their system security (which sounds like a problem even a well-meaning and careful developer could fall into) should take precedence, if that is seriously the only substantial argument. (Also, yes: it would be great if Linux were fixed; if and when it is fixed in a way developers can rely on, then of course that would be a great argument against everything I've said: as far as I know, it hasn't. Hence the authors of the article I referenced recommending the importance of libraries use more secure mechanisms by default, including /dev/random.)
- Using /dev/random occasionally blocks applications, but that's acceptable because it's only a small high-throughtput subset of them, and those users can deal with it themselves.
- Using /dev/urandom never blocks, but a small headless subset of devices will in some situations generate predictable streams. This is unacceptable.
I fear the assessment of the damage done by /dev/random blocking is understated. When it comes to generating long-term keys, yeah, sure, use /dev/random if you want; it doesn't matter how long that takes. But in online applications, these blocks can make an application unusable. Also remember than CSPRNGs are not only used for short key generation: generating CBC IVs is a nice example of something relatively high-throughput that really should not block.
You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available.
As an aside, these arguments also put some amount of faith in the entropy estimator's quality, i.e., that /dev/random blocks only when it should. This may not be the case, as shown by [1] in (admittedly) somewhat contrived conditions.
> You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available.
The good workaround for /dev/random is trivial: you use /dev/urandom instead; if you are sufficiently frustrated that you hardcode /dev/random to a file of static data, you probably have made numerous other horrible mistakes. Yes: I maintain that this developer is an idiot who is "beyond help"; it is harder to make a static file for this than to use /dev/urandom.
This can also be a configuration file option: "potentially blocks, but always works" vs. "will never block, but you'd better have setup your system correctly". Even better, the developer could go to steps to verify /dev/urandom is ready (such as reading from /dev/random and writing it to /dev/urandom) before using it receptively in ways that could block.
> I fear the assessment of the damage done by /dev/random blocking is understated.
When /dev/random blocks you know it happened, and a FAQ entry can guide you to a reasonable solution. (And again: given that you can apparently seed /dev/urandom from /dev/random, a solution that the advocates of /dev/urandom are still unhappy with for some reason, it seems like there's no excuse to end up with /dev/random blocking; instead, what everyone should be encouraging is to seed /dev/urandom from /dev/random and then use /dev/urandom.)
Alternatively, the people who end up building systems that do not have enough entropy at boot have no way of knowing this happened, and are sufficiently divorced from what is being executed that they are unlikely to even realize that someone is relying on something they shouldn't. This is where "sane defaults" come in: on Linux, it seems like software should not rely on /dev/urandom until the software proves somehow to itself that it is "ready".
> instead, what everyone should be encouraging is to seed /dev/urandom from /dev/random and then use /dev/urandom.
I have no problem with this whatsoever; it's probably overkill for most users, but it's failure-proof. What I object to is using /dev/random all the time under the misconception that it is somehow more secure.
I'm confused. /dev/random is not in fact more secure than urandom. Can you restate your argument here?
Developers use bogus userland libraries because the man pages tell them (wrongly) that urandom is unsafe for cryptography, and when they use /dev/random, their application randomly hangs. Exactly why would you ever recommend that a developer attempt to use /dev/random?
During boot of the system, /dev/random will block until there is sufficient entropy while /dev/urandom will not: it will just return, as documented in detail by the paper I referenced, a deterministic stream of "random" data. The researchers document that during 1000 trial boots, the only entropy in the returned data was the position in the stream.
This is just asking for system integration errors, such as the wide-spread issue with headless routers generating keys at boot. Hell: even the desktop systems tested by the researchers were only just barely managing to run the key generation late enough that the keys weren't running into the "deterministic data returned from /dev/urandom" problem.
Yes: this is fixable by making certain the system has enough entropy sources and doesn't attempt to do anything with /dev/urandom until enough entropy has hit the system to make the data returned actually-random. However, in practice, that isn't happening, it isn't happening often, and it is trivially defendable against by using /dev/random.
As it stands, the only reason anyone has come up with for using /dev/urandom instead of /dev/random is "occasionally /dev/random blocks". The question is then whether the "sometimes the system blocks under load" is more or less dangerous of a default than "sometimes keys are generated that are actually the same as every other device".
Now, maybe you believe that the former issue is more important than the latter: that people building systems under load are more likely to break them by replacing /dev/random with a file full of constant "random" data (rather than replacing it with, for example, /dev/urandom) than people generating keys at boot without realizing their device is nothing more than a CPU with 16kB of flash and 32MB of ROM attached to a dead network.
If you do feel this way, (and it isn't at all clear to me, as you still seem to be beating the "/dev/random doesn't generate better data than /dev/urandom iff the random entropy pool has been correctly seeded" horse, which I maintain was dead long before I arrived with my argument) I encourage you to read the other comment I've written elsewhere in this thread showing why this seems like a poor choice of default behavior.
This is 6 grafs of concern about the Linux first-boot process that can be addressed by 1 line in an rc script. Seed urandom at boot from /dev/random, and then forget about /dev/random.
It's a real problem, Thomas, particularly in headless VMs cloned from a freshly-installed image.
On these machines, reading /dev/random at boot time can hang indefinitely. Your "1 line in an rc script" will make the OS appear to be broken.
It's possible the problem can be ameliorated by a combination of approaches. Educating sysadmins is clearly important. I can imagine a userspace process monitoring /dev/random at boot time and screaming if someone tries to read from it and it blocks. (Though on a headless VM, "screaming" in such a way as to reliably be heard may be nontrivial.)
The point is, work needs to be done here, by a variety of people, including the distro publishers. Getting that to happen is going to take some advocacy.
As a rather contrived example (VirtualBox, without keyboard input:
[voltagex@arch ~]$ cat /dev/random | dd of=/dev/null
^C0+0 records in
0+0 records out
0 bytes (0 B) copied, 8.91021 s, 0.0 kB/s
[voltagex@fedora ~]$ cat /dev/random | dd of=/dev/null
^C0+1 records in
0+0 records out
0 bytes (0 B) copied, 8.09669 s, 0.0 kB/s
voltagex@saucy:~$ cat /dev/random | dd of=/dev/null
^C0+1 records in
0+0 records out
0 bytes (0 B) copied, 8.31898 s, 0.0 kB/s
And then after installing haveged, the following:
voltagex@saucy:~$ sudo service haveged start
voltagex@saucy:~$ cat /dev/random | dd of=/dev/null
^C0+81630 records in
20406+0 records out
10447872 bytes (10 MB) copied, 2.92772 s, 3.6 MB/s
Is installing haveged the right thing to do here?
Of course if I'm completely misunderstanding what you're saying, please correct me but yes this seems to be pretty bad.
For the billionth time: is the user actually doing this, and does the user or system integrator realize they should do this? Apparently, the answer is no, based on real-world systems that don't do this. In some situations (such as an Android phone) it might not even be possible to do this (locked bootloader). Why risk this when the solution is so simple? I maintain that you are encouraging a situation where this kind of mistake happens, when there seems to be a negligible cost to working around it as a developer, protecting your users from their own stupidity and protecting your users from the stupidity of their system integrator.
No common library is supposed to protect the user from something that should independently be done by the system, once. It's more than non trivial. Consider modern boot systems which do initialisations in parallel.
I would also love it if you could actually respond to the relative merits of the author's "developers will hack randomness out of their projects" belief (which I claim only happens to idiots) vs. my alternative "developers will not realize the security of the libraries they are using is reliant on an unrelated set of decisions made by the system integrators" (which apparently happens to engineers who don't sound like idiots).
(Also, I appreciate it adds rhetorical strength to your comment to point out that key streams don't "run out of key" and urandom doesn't "run out of entropy", but I never stated that, never believed that, and would not now claim it: those statements are entirely orthogonal to any of the arguments I made, and seem to have only been included to make your comment sound like it responded to mine.)
I guess put shorter: care to respond to anything I said? Anything at all? :( All you've done is restated something from the article... (I mean, you certainly don't have to respond to me, but then I question the decision to leave a comment in the first place: my comment's are long because I had to respond to all the substantive points in the article, not just state my position :/).