Probably the lesson I'd take away from this example is that libc specifically often has surprising shared state, and you have to be really careful about using it from multiple contexts. And of course, you should avoid hammering a mutex from multiple threads -- but you already know that.
All the different flavors of libc, apart from the outdated and cryptic naming style, have many hidden flaws like this one, heavy multithreading was not a central design feature when it was designed, and there are many instances of global state, with workarounds to avoid bugs, at the cost of potential serious performance issues.
My opinion is that the standard libc should be used much more as a fallback than a default.
And this is especially true about random number generation, there are much better generators out there, and some of them are only a few lines of code.
That random number generator depends on global mutable variables without any protection against concurrent access. It isn’t suitable for production use.
I assume the response of the original site depends/depended on the user agent and only showed a HTML page for user agents it recognized as a browser, which doesn't include the IA bot.
not supporting multi-threading is not a design flaw, it's a conscious design choice. Multi-threading is hugely overrated as the UNIX kernel already distributes workload across processes and processors. It's a performance hack.
Threading is indeed a performance hack - the whole idea of multiple processes sharing an address space is pretty gross and it's very unsurprising that it leads to many bugs - but there's also a reason why it was universally adopted despite the problems. I do think there's a fair argument that libc's design was not a mistake; the thread-unsafe functions worked very well in the environment they were designed for and it wasn't until decades later in a very different environment that they became a problem. Nonetheless, they are a problem now.
Shared memory address space is a couple (IIRC) orders of magnitude faster to access than any IPC mechanism. Even on modern very fast machines IPC will add noticeable overhead if you tried to replace threads with processes.
The question would be how much data are you sending on how frequently. You're still advertising performance. Even so, go ahead and setup shared memory between your processes. Better to explicitly share part, rather than implicitly share full.
What does that mean? Do you mean optimize a piece of code to take advantage of multiple cores? Or do you mean describe multiple independent threads of execution. The latter works best when it has it's own memory address and you assume that "if then else" works as you would expect.
