Hacker News new | past | comments | ask | show | jobs | submit login

malloc + memset takes about .24 seconds per gig on my machine. So be sure that your dataset is really, really sparse before you try this.



I'm not sure if your comment suggests that .24 seconds is fast enough not to worry about it, or not fast enough.

In my time at IBM doing very performance oriented C development, saying that a server operation only took .24 seconds would get you laughed out of the development meeting as the operations were designed to take 12 microseconds.


The article is about latency; if you use its techniques and your dataset is not sparse, you will use more time and space overall. But, you'll have your first item in memory sooner.

There are few operations faster than writing a 0 to a memory address.


Writing 0 to a register is, and that is what this is trying to compete with in practice. It's an approach that scales beyond 32 or 64 members for certain kinds of operations.


Allocating and initializing a huge array of memory at start-up can take a second, and subsequent server operations take 12 microseconds. Nobody's talking about allocating a gig of memory every time a server gets a request.


One thing to consider is we also aren't using small development boxes. These were Power7 zSeries servers with usually 16 cores (8 in hardware) and up to 32GB ram. You would be surprised what these machines are capable of. Of course, I'm not saying I can allocate a gig in .24 seconds and set it to zero, but I'm not saying its not possible either (I simply don't know as I no longer have access to these machines).


Well, why not? I'm sure you could put a gig of space to use calculating an extended social network or what not. 0.2 seconds could be the difference between expanding use up to a couple of gig per request vs rewriting from scratch or not implementing the feature altogether.


Also for people new to C programming, don't forget about calloc!


It's worth noting that on modern OSes, the OS will forcibly zero out memory when it's allocated from the OS, so if you're allocating a really large sparse set (large enough to not come from existing free heap RAM), and don't intend to clear it a lot, sparse sets won't save any time, and calloc (which can skip the zeroing if it's getting new pages from the OS) will probably be a better idea.


.24 seconds isn't fast at all.


Per gig. How long will it take to do the rest of the processing to fill a 1G hashtable?


This is a bitset we're talking about, not a hash map of any kind.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: