Hacker News new | past | comments | ask | show | jobs | submit login

Should be titled "without using malloc" or something. It's pretty much impossible to do anything with user input strings unless you have some sort of memory management. Maybe you have one giant buffer that you use for strings in a particular session, but that still involves memory management because even if you allocate the buffer at startup, using one buffer for multiple strings is just a type specific arena allocator.

As someone who works on low level database code, this approach is going to be EXTREMELY wasteful. Instead, it's much better to instead have a more sophisticated understanding of memory use by subsystems and have some concept of required memory and caches. Each system allocates the minimum memory it needs at startup and then treats the rest of memory as caches which can be emptied as necessary to preserve fairness among all the various subsystems.




> Should be titled "without using malloc" or something. It's pretty much impossible to do anything with user input strings unless you have some sort of memory management

Title says "without dynamic memory allocation" == what means without malloc in this context, so static memory allocation ... where you do your own kind of memory management in fixed size arrays or more sophisticated if you want. It didn't say without memory management? (And article explains also why no strings or other unknown/arbitrary length stuff..)


Also, the article mentions the only entities are two fixed size types: accounts and transfers.

There are no user strings in TigerBeetle. :)

If anyone wants to see what you can store in accounts and transfers, check out the reference docs!

https://docs.tigerbeetle.com/reference/accounts

https://docs.tigerbeetle.com/reference/transfers


That's wild. So every column is fixed width?

Is anything nullable, or for like the user_data column, do you just use 0x0 to indicate no record?


Yep, every field is fixed width. Nullable would just be zero but N zeroe for the width of the field.


> using one buffer for multiple strings is just a type specific arena allocator.

Sure, but there's a lot of advantages of entirely controlling memory allocation in-process.

> this approach is going to be EXTREMELY wasteful.

Could you elaborate why? Sure, you have to do a bit more juggling in-process, but I would imagine this would be potentially more efficient than having conventional mallocs.

In the environment it is designed to operate in, there is usually a fixed memory limit per process, so you might as well grab all that up front.


Thanks!

Joran from the TigerBeetle team here.

> I would imagine this would be potentially more efficient than having conventional mallocs.

Yes, in our experience, static allocation means we use sometimes 10x less memory. For example, TigerBeetle's storage engine can theoretically address up to 100 TiB with less than 1 GiB of RAM for the LSM in-memory manifest, which is efficient.

Because we think about memory so much upfront, in the design phase, we tend not to waste it.


I think it should be noted that TigerBeetle is an extremely specialized database. It's not a general-purpose DB for storing arbitrary data, all it stores is numeric financial account information.




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: