A few years ago I worked for a gift card manufacturer and I was tasked with creating random PAN numbers for a gift card product for a large high street retailer (sadly no longer with us). I read up on it and couldn't find any decent guide as to how to produce CSPRNG - and to be fair, there was very little information in 'layman' terms about what CS meant in this context. As I couldn't be certain that any of the software routes available to me were secure I ended up purchasing this http://www.protego.se/ to generate the random numbers.
I am not convinced that random numbers were required for the security element of the gift cards, but I don't believe there were any reports of any fraudulent activity based on unknown numbers.
Developments in this arena over the last 10 years would have made my job a lot easier!
> I read up on it and couldn't find any decent guide as to how to produce CSPRNG - and to be fair, there was very little information in 'layman' terms about what CS meant in this context.
When you say "produce CSPRNG" do you mean writing a generator or simply generating numbers? Nobody[1] should be writing a CSPRNG and app developers should be using language/OS functions (ex: /dev/urandom).
> As I couldn't be certain that any of the software routes available to me were secure I ended up purchasing this http://www.protego.se/ to generate the random numbers.
Wow did you really look at that page and say, "This is going to be easier and more secure than using /dev/urandom?". Reading the rest of the info on the page, http://www.protego.se/trnsup.htm, does not give me a lot of confidence in the product. The explanation page of how it works is just a bunch of scare info against using a software solution.
[1]: Okay obviously someone has to do it but you all know what I mean...
I mean just generating the random numbers, not writing a generator. At the time I only had access to Windows machines, there were no linux boxes on site and nobody in the company (including me - at the time) knew enough about hack it yourself linux boxes to know what to do, I'd never even heard of /dev/random.
The functions available from microsoft were only PRNG and not Cryptographically Secure (as far as I can remember). Even if they were 'Cryptographically Secure' not having a background in randomness theory makes it pretty hard to work out what is actually needed.
So faced with uncertainty over using VB6 to produce random numbers and finding a product that claims to produce truly random numbers I chose to mitigate the risk by transferring it. If there had been a problem down the line related to the nature of the randomness of the numbers then I would have had recourse to the third party vendor that owned the risk.
So in the same way you suggest no-one should write a CSPRNG, I decided that given my level of knowledge, the documentation available to me and the uncertainty that created we shouldn't trust ourselves to produce random numbers for a particular purpose. What if they weren't random enough? What if someone worked out a way to predict the numbers of cards that had been loaded with value but were still sat in a warehouse? Massive risk, mitigated for £150. Now that's value.
> Wow did you really look at that page and say, "This is going to be easier and more secure than using /dev/urandom?". Reading the rest of the info on the page, http://www.protego.se/trnsup.htm, does not give me a lot of confidence in the product. The explanation page of how it works is just a bunch of scare info against using a software solution.
While the page doesn't look as cool as your average SV hardware startup, and their white paper is more or less blank, don't dismiss them that easily. According to linkedin the guy beyond the company has been involved in ASIC TRNG designs at some large companies.
Also, from what I have heard there are multiple open source hardware entropy generator designs that together with minimal software processing can generate pretty good random. Now, this is not NSA-proof randomness but probably better than anything you can get from /dev/random and more than enough for, say, gift cards.
> While the page doesn't look as cool as your average SV hardware startup, and their white paper is more or less blank, don't dismiss them that easily. According to linkedin the guy beyond the company has been involved in ASIC TRNG designs at some large companies.
It doesn't take much to look "professional" and having HTTPS on a page that is meant to sell a security product is table stakes.
> Now, this is not NSA-proof randomness but probably better than anything you can get from /dev/random and more than enough for, say, gift cards.
I disagree. Not being able to vet how it works or seeing the code running on the chip is not better than /dev/urandom. The latter is "more than enough, for, say, gift cards".
Heck if it's good enough for SSH and GPG keys, it better be good enough for gift cards!
The README of the gem that the article recommends ("sysrandom") provides a good explanation [1].
And here is the contentious Ruby issue wherein the Ruby core developers insist on cargo culting according to an outdated man page despite comprehensive refutation from a number of experienced security engineers [2].
That said, the arguments against OpenSSL seem to be largely theoretical at this point ("has been a source of vulnerabilities in Ruby ..."), so the misgivings about `SecureRandom` may be slightly exaggerated.
What a disaster of a bug report, on so many levels.
Speaking as a foss maintainer, to other foss maintainers out there: When you're systematically adopting a defensive attitude towards someone who believes found an important flaw in your project, you're the one being rude. You're refusing to hear people out. I see this awful attitude far too often.
On Github, it's even worse nowadays where a lot of maintainers immediately fall back to locking the bug report so that they don't have to think about it anymore. At least here there is some form of discussion going on. But that's with a lot of fingers in a lot of ears, and nobody trying to be practical...
There is huge undiscovered land of material for cross-cultural and comparative research in communication in these developer forums. Ruby and Linux mailing lists could provide material for number of really interesting studies in many fields.
Ruby community has hint of Japanese culture: design aesthetics, simplicity, importance of politeness, saving face and rigidity.
Linux community has hint of Finnish culture: practicality, "pig energy" and "management by perkele" combined with 80's competitive hacker culture.
They have a /dev/random interface which (literally) randomly blocks, disrupting programs, and a man page that instructs the Ruby community to use /dev/random and not /dev/urandom.
/dev/random is the blocking pseudorandom number generator in Unix-like operating systems and it provides only the entropy that can be obtained from environmental noise. I don't see how it can work any other way.
What is the manpage we are discussing here? I think getrandom(2) and random(4) are quite good.
/dev/random keeps a counter that is effectively random that tells the interface to block. /dev/urandom has the same mechanics as /dev/random, the same inputs, and the same cryptographic constructions, but does not have the random blocking counter.
/dev/urandom is the interface you want.
/dev/random is unsuitable for production code because, again, it randomly blocks, which can freeze your application. Worse still: applications that use /dev/random tend to "work around" the problem by using /dev/random to seed a userspace RNG that doesn't block. That's not just inconvenient but overtly unsafe.
> /dev/random is the blocking pseudorandom number generator in Unix-like operating systems
This is not the behavior on FreeBSD and OS X. On those systems, once a sufficient amount of entropy is estimated to have been gathered since boot, /dev/random will not block again.
Linux's /dev/random behavior (1) relies too heavily on accurate estimates of entropy and (2) implies a simultaneous fundamental mistrust in the ability of cryptographic methods to expand several hundred bits of entropy into a few megabytes of data indistinguishable from random noise and yet simultaneous belief in the strength of cryptographic algorithms for expanding maybe a kilobit of long-term key material into terabytes of data indistinguishable from random noise. (The rationale for blocking is to produce better long-term keys.)
Linux's /dev/random rekeys its internal state at a fixed entropy estimate. If state is compromised and the estimate is too optimistic, it will never recover from state compromise. If the estimate is too pessimistic, state is left vulnerable longer than necessary.
Bruce Schneier and Niels Ferguson's Fortuna for a design is more robust and consistent. A series of entropy pools all collect entorpy at the same rate, but are emptied at exponentially longer intervals. Eventually, one of the pools used to rekey will have enough entropy to recover from state compromise, and it doesn't rely on the dubious practice of entropy estimation. Now, estimating how long recovering from state compromise will take requires estimating entropy, but with a Fortuna construction we can prove that recovery will eventually happen as long as entropy is actually being collected.
If you're using TLS with AES-GCM, a Fortuna construction built using AES has an added advantage that an attack that recovers Fortuna state is almost certainly directly applicable to your TLS setup. In other words, you're depending on fewer algorithms, the failure of any one of which dooms you.
FreeBSD uses Fortuna for /dev/[u]random. OS X uses Fortuna's predecessor Yarrow, which still has the weakness of relying on entropy estimation, but at least it has two pools (one with a pessimistic rekey rate) instead of Linux's single rekey rate.
The LRNG is fine. It could be faster, and it could be simpler and more coherent to facilitate formal analysis. But the underlying task that we want an OS CSPRNG to do is not complicated.
I think it would be a bad idea to forklift out the LRNG in favor of an entirely new design.
It's possible that all Linux really needs to do is fix the man page, and perhaps do something in the kernel (rather than in OS distributions) to solve seed-at-boot.
> /dev/random is the blocking pseudorandom number generator in Unix-like operating systems and it provides only the entropy that can be obtained from environmental noise. I don't see how it can work any other way.
There is no reason to block; there isn't even any good way to estimate the entropy of environmental noise.
The only sane thing to do is to seed a CSPRNG with environmental noise, reseed with environmental noise as available, and never block.
I think the maintainer has a point in saying that the `man 4 random` page should have been updated if its giving wrong advice. I just checked and on my system that manpage still says all that wrong stuff about /dev/urandom. Why is it still unchanged?
Because Linux man pages are sometimes dumb. I can give you a more precise answer but I don't think it will improve the thread.
You should, however, take my word for it. Or, better yet, take Daniel Bernstein's word for it. Take Adam Langley's implied word for it (he's behind crypto/rand in Go). Take Thomas Pornin's word for. You can take Thomas Pornin's word on almost anything! Take Filippo Valsorda's word for it. Or Thomas Huhn's.
The man page is wrong. Should it be fixed? Yes, it should be fixed. But, no excuses. You can't introduce security problems out of obstinacy about man pages.
> That said, the arguments against OpenSSL seem to be largely theoretical at this point
I would disagree. This is a recent paper on the state of OpenSSL's userspace PRNG, for example, finding several issues: https://eprint.iacr.org/2016/367.pdf
> uby issue wherein the Ruby core developers insist on cargo culting according to an outdated man page despite comprehensive refutation from a number of experienced security engineers [2].
Anecdotal case here. I did a research about usage of secure random number generation few year ago and gave up with OpenSSL. They used indirection upon indirection and it was really hard to understand what would be actually called.
Therefore I was really not surprised when problems related to OpenSSL random generation started to pup up later on.
In this case, that's far from true. Many people have contributed correct documentation. Whoever is in charge of these things continually rejects it because of their mistaken beliefs about PRNGs.
I always use SecureRandom to generate unique IDs, but not to actually secure things - am I to take it that it would still be wise to switch to Sysrandom in that case as the vuln is in going through OpenSSL to do it (rather than the generated random not being random etc)?
I can't help but feel this sums up the difference between these two ecosystems:
Cryptographically Secure Randomness in .NET (C#)
The generally accepted solution is to use System.Security.Cryptography.RNGCryptoServiceProvider
Cryptographically Secure Randomness in Node.js
Don't use crypto.randomBytes()
An unfair cherry picking perhaps, but an illustrative example.
on the whole, redefining 'int' in Python is not a great idea:
>>> int
<type 'int'>
>>> int('10')
10
>>> int = 10
>>> int('10')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
Really hurts the article's credibility to intersect cryptographic domain knowledge with technical programming knowledge. That said, I did learn something from it, but I'll be doing a bit more research...
- Java 7 and below are, unfortunately, still in common use, despite being end-of-life. There is a config file to be edited, or a switch to be passed to the java executable to switch the RNG source away from /dev/random, the pre-8 SecureRandom default.
- Some obvious note about how client-side Javascript is a lost cause, in case anybody's tempted
Rust has no out of the box crypto, specifically because we want to take our time and get it right. The rand crate can generate random numbers, the ring crate is the most promising crypto library overall, IMHO.
I'm not sure your point about Java 7 is correct. The docs (http://docs.oracle.com/javase/7/docs/technotes/guides/securi...) state that it is a SHA1PRNG, seeded with "true random". So the first call may block if it goes to /dev/random on Linux, but it should be good-to-go after that. I think this is the same behaviour as Java 6, but I don't know about previous versions.
If you have a fresh VM without a seed file, that isn't connected to the internet and has had little in the way of devices, and you immediately begin generating keys before the normal file system operations stir the generator, then the deterministic nature of the generator says you'll get a predictable key, or more likely, end up generating a key that's simply not unique.
Just by being online and running for a few seconds will cause enough entropy to put the generator into a safe state, so this rarely happens in practice.
In fact Ubuntu comes prepackaged with pollinate[0] which makes this basically a non-issue for everyone that's not wearing a tinfoil hat.
Device interrupts will feed the generator so that the output from the CSPRNG is indistinguishable from random.
There's lots of stuff happening on a network that's impossible to guess and replay. This is also called noise.
Keep in mind that the CSPRNG generator only needs a random seed, then it can produce psuedo random numbers all day long. Linux will continue to reseed as more interrupts occur.
Linux will also save a 512 byte file on shutdown and seed the generator with it on the next boot up, so you immediately enter into a safe/random state.
+1 to this question. If the rand_linux strategy linked above decides to fall back to rand_unix, it can result in a userspace AES-based generator [1] running on output from /dev/random
A big part of the point of this blog post is avoiding the OpenSSL userland CSPRNG, which has not proven to be consistently safe. Far better would be the snippet of Common Lisp that reads random from getrandom(), and then /dev/urandom.
Since the article doesn't want a userspace CSPRNG (which is probably a good idea, even though the Linux PRNG design is embarrassingly bad), better to just read from /dev/urandom or make the appropriate syscall.
(ql:quickload :isaac)
;; FIRST TIME (one of both)
;; using (random), which on sane implementations should rely on /dev/urandom
(defvar *isaac* (isaac:init-common-lisp-random-seed))
;; using /dev/urandom directly
(defvar *isaac* (isaac:init-kernel-seed))
;; Generate bits
(isaac:rand-bits *isaac* 30)
ISAAC is supposed to be secure, but I don't know enough about cryptography to judge it.
The main reason to use OpenSSL in secure-random is to get a seed - not all platforms have /dev/[u]ramdon. OpenSSL has code to initialize a seed from platform-dependent APIs
We can't condone any solution that depends on userspace PRNGs like OpenSSL instead of the kernel's CSPRNG.
There are probably worse examples of userspace PRNGs that attempt to be cryptographically secure than what OpenSSL provides, but we wrote this to encourage good habits. Reliance on OpenSSL's PRNG is definitely not a good habit to get into.
The only time you'd have to worry about this is during boot. Ideally, whatever you're trying to do at boot time should be moved to post-boot time, when the kernel has taken care of all the concerns for you.
I don't know what the answer is to "If you want to generate secure random numbers during boot before urandom is seeded, how would you do it?" I assume it's "Seed urandom, then use it."
That's precisely why getrandom(2) is preferred on newer kernels, and the following hacky workaround from the article is given on older kernels:
For software that runs during the Linux boot, poll /dev/random until it's available.
This means /dev/urandom has been seeded and you can safely read from /dev/urandom for
all your cryptographic purposes. Don't read from /dev/random.
1) Define "enough" in terms of each and every specific piece of hardware, combined with the mixing heuristics and functions in every kernel of interest. It's one thing to say you want N bits of entropy; it's quite another to accurately quantify it. (Count the number of times you see a word like "assume", "estimate", etc, in papers discussing kernel entropy estimation, and you'll never look at entropy estimation the same way.)
2) If you managed that and were able to conclude in the negative for a specific combination of kernel, kernel revision, and hardware models and revisions, what then? Are you going to recreate all that infrastructure in your app or through a gargantuan dependency along the lines of egd or havaged (which we've already learned from experience is a poor idea)?
My point being, the question is arguably irrelevant. If available entropy at boot time (usually first boot, as the shutdown/startup process recycles entropy pools) is insufficient, better to pressure vendors (software and hardware) to address it than to add another layer of cruft or a complex mitigation. The fix will probably be out in the next year or so (indeed, with RDRAND has been out for years); the hack will linger for far longer, perhaps eventually becoming the weakest link in the chain.
For similar reasons I think the article's as well as libsodium's choice to only use the most recent arc4random implementation as shipped in recent OpenBSD releases is misguided. Every other *BSD provides arc4random, and arguably applications should always prefer to use the libc's arc4random implementation. Eventually they'll be upgraded, at which point your application will have a useless dependency adding at best needless complexity and at worst vulnerabilities.
Once you look behind the interface of arc4random, you're obliged to look behind the interface of /dev/urandom, etc, and I doubt it will look much better. Until the recent overhaul (last month), the PRNG infrastructure in Linux relied on SHA-1, which isn't much better than RC4, especially when considering that the RC4 used in the classic arc4random implementations discarded the first 1k of output and reseeded regularly. SHA-1 hasn't been recommended for cryptographic purposes for several years, and while you could make pragmatic arguments for why it was still acceptable, you could do the same for the particular use of RC4 in older arc4random implementations. I don't see libsodium skipping /dev/urandom for kernels before 4.9. At some point you have to stop bending over backwards to support platforms with poor QoS and leave things as you found them as long as they're not utterly broken. If experience with egd and, more generally, the boots-and-suspenders approach to security is any measure, classic arc4random is well within the bounds of reasonable. Simplicity is the rule of the day; experience has taught that complexity is invariably a net loss in security, especially when the weakest link remains unchanged.
Not really, you can always improve upon an existing PRNG by XOR'ing the random number stream with the output of your own PRNG without thereby weakening security. So if you think /dev/urandom does not produce enough entropy, rolling your own PRNG with additional entropy sources (e.g. mouse movements) on top of it cannot harm if you use XOR for combination.
However, you can also feed the system-wide PRNG with additional entropy, and for many applications this might be the better choice.
I would add, while this is mostly accurate (XOR is perfectly fine for retaining 'randomness' in the sense you're talking about - even if the second stream isn't random the output will still be as random as the initial) there is the big issue that you don't want to accidentally reuse an entropy source.
For example, imagine /dev/urandom is already using mouse movements as an entropy source. If you then decide to also use mouse movements, you may have weakened the randomness of the resulting stream because now some of the random bits get reused.
In the worst case, if you were the only user of /dev/urandom and output the exact same sequence of bits from the mouse movements, an XOR would completely remove all randomness from the mouse movements (All 1's get flipped to 0's, all 0's stay the same). Even without the worst case though, it could still influence a pattern into the bits due to the reuse.
> Not really, you can always improve upon an existing PRNG by XOR'ing the random number stream with the output of your own PRNG without thereby weakening security.
Unless an attacker gets to acctively control your PRNG results.
Further advancement in secure random number generation across languages will probably lead to more "don't roll your own crypto" reminder posts in the near future.
But it's still a great feature to have in every programming language imho.
The recommendation for windows is hopelessly out of date. On Vista and up CryptGenRandom is exported directly from Advapi32. That said... just use the CNG functions (BCryptGenRandom[1]), which is what both RtlGenRandom and CryptGenRandom use under the covers. BCrypt.dll is going to get loaded either way.
-- edit -- In case it's not clear I really, really agree with this article, I was having a rant about people I've worked with who make things needlessly complex, not the link, which is great advice!
Just use /dev/urandom
I can't believe how many crappy engineers I've encountered who are resistant to this advice. The onpy coherent objection I've ever heard is that "It never blocs therefore it can provide random numbers with insufficient entropy", which has been adequately refuted many times now.
But still I encounter experienced, senior guys doing all sorts of Heath-Robinson crap with reading tons of entropy from TPM chips, feeding it to openssl or other systems as a seed and using those instead. It's completely unnecessary adds complexity (which in turn decreases security) and just...
What's really frustrating is people being convinced that random numbers are hard and therefore that they really need to do all this 'hard' stuff. I guess it flatters their egos.
As long as it's not the first system startup, on a headless machine with little in way of devices. Then you run the risk of generating a number without a seed. But the risk is low, since things like disk IO read times and other such will seed the generator unpredictable ways.
Linux is smart enough to squirrel away some random on reboots so that it can properly seed the generator when it starts up again.
FWIW in Tcl it's fairly easy to generate random numbers from /dev/urandom on Unix-like systems[0].
It involves opening /dev/urandom, reading 8 bytes from the device, binary scan of the returned value, converting from the hex format and scaling the result to fall within a specified min to max range.
The tcllib has other PRNG procedures, and there's the "Random" package ISAAC implementation. Whether output quality of any of these procedures is adequate probably depends on the purpose.
> and scaling the result to fall within a specified min to max range.
This is actually incorrect. Unless the size of the range is a power of two, this gives non-uniform probability. The bias might be acceptable if it's small enough, but it's something to be aware of, and, if you want to have something robust, to avoid.
A general safe strategy is to check if the raw random value is below the maximum multiple of the size of the range that's no greater than the number of possible values for the raw random value, and if not, generate a new one and repeat. (In principle, any smaller multiple (>= 1) would work as well, but taking the maximum gives the smallest number of retries.)
> reading 8 bytes from the device, binary scan of the returned value, converting from the hex format
It is not necessary to round trip through a hex encoding. Tcl's binary scan is capable of converting an 8 byte value directly into a 64-bit signed or unsigned integer. You can even request big or little endian interpretation of the bytes.
Check the "w" conversion flag in the man-page. For unsigned you'd add the "u" flag. For big-endian conversion use the "W" flag instead.
Reading this story, I was curious about ISAAC, so I posted a link to its webpage in another post (https://news.ycombinator.com/item?id=12441451), hoping people would give their opinion on it. Do you have experience with ISAAC?
That's the same link provided in the Tcl wiki. I haven't actually used the ISAAC generator since the urandom is considered reliable. However, urandom is only available on Linux/BSD systems, on Windows alternatives are necessary.
From the comments on the ISAAC home page, the ISAAC and ISAAC-64 versions would appear adequate for most uses, but a reader more experienced in CSPRNG analysis might have more information.
It's a ~$50 USB dongle, opensource and available for shipping. Anecdotally, I personally used it for an RF project that need uniform randomness and exhausted /dev/random on the target hardware. It worked fine, if a bit slow for my very specific usecase.
That said, if you want to go even cheaper, seriously consider if you can just stick with /dev/urandom. See http://www.2uo.de/myths-about-urandom/ for an interesting read on why that might just work.
I remember him from the CipherShed project that was trying to reboot TrueCrypt. Discussed securing the build system & distribution. He seemed smart. I have no experience with Tindie, though, so I'd appreciate feedback from anyone that knows about them. He also includes BOM, etc on the GitHub so someone can source and build it themselves.
I'm not normally a paranoid person but EM radiation seems like it's something that it would be easy for someone outside your house in a van to manipulate.
It should be basically impossible for them to manipulate the low bits of the signal. Then you hash it, which makes any bias irrelevant as long as enough bits are random.
You realize modern Intel CPUs include a TRNG, with the RDRAND and RDSEED instructions, for no extra cost. Can't get cheaper than that, if you already have one.
The potential of an NSA backdoor in it is a popular, and entirely possible, conspiracy theory, of course. You'd have to decide for yourself, and compare it to your trust in other options.
I had already suggested (to Dion Hulse) that this antipattern of pinging wordpress.org to define the salts during setup be removed in favor of always generating them locally.
I don't know if/when they'll implement that (no matter what benefits a security fix might provide, WordPress always prioritizes "no BC breaks even on 0.0001% of the installed base" first), but if you're looking for a more secure CMS I happen to be building one here:
As general rule, before you start using language provided secure random number generator, you should at least informally verify the full chain of the generator on multiple platforms.
I am not convinced that random numbers were required for the security element of the gift cards, but I don't believe there were any reports of any fraudulent activity based on unknown numbers.
Developments in this arena over the last 10 years would have made my job a lot easier!