Hacker News new | past | comments | ask | show | jobs | submit login
Building a database buffer pool in Zig using io_uring's new fixed-buffer mode (gavinray97.github.io)
158 points by gavinray on Oct 15, 2022 | hide | past | favorite | 23 comments



Aha! I was thinking about trying this exact same approach recently, so I'm glad to see I wasn't insane! Very clever.

I have been "in my spare time" (that is, never finishing) building out a page buffer mgmt system based on the one in the Umbra paper (see https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf) which has a kind of novel approach where they essentially allocate the same physical address at multiple virtual addresses but with different block/slab sizes so that they can reduce fragmentation and still support dynamic page sizes. Anyways, it seems this author is also on the same "page" (sorry) as me but focusing more on exploiting `io_uring`. Nifty.

I'm working in Rust, not Zig, but a lot of nice stuff to read through here.

I will have to read through this in more detail later when I have more brain-space available.


  > I'm working in Rust, not Zig, but a lot of nice stuff to read through here.
FYI, Datadog has a Rust library for scheduling things to run thread-per-core with io_uring

It'd be really useful for DB use cases:

https://github.com/DataDog/glommio


Nifty. I started from the part of the Umbra presentation that initially struck me as most interesting -- the allocator portion (where he talks about repeatedly MMAP_ANONYMOUS on the same address range to reduce fragmentation, etc.) I stopped working when I started pondering the bits about pointer swizzling and how I'd make that work nicely in Rust, and never got as far as the pieces that would actually do the syncs to disk.

I also came across another fellow who was inspired by the optimistic locking btree presented in LeanStore Umbra: https://github.com/NeowayLabs/bplustree


Jinx, I'm also working on this, as a tutorial series!

I had the pleasure of hearing Neumann present the Umbra architecture on the CMU DB series webinar recently.

I'm slowly working through re-implementing the Coroutines async buffer pool paper. Blocked on trying to properly implement and understand "Morsel-Driven Parallelism":

https://db.in.tum.de/~fent/papers/coroutines.pdf

I have the async, io_uring buffer pool written in Kotlin using it's native Coroutines but the scheduling and morsel stuff I'm still trying to understand


I watched the same CMU lecture, very cool.

Are you getting paid to work on this kind of thing, anywhere? Do you want to be?


  > Are you getting paid to work on this kind of thing, anywhere?
My dayjob is a fair bit higher level. I work on a sort of query engine on the JVM that translates an internal query IR/AST into dialect-specific SQL + other languages so you can query anything with the same frontend language:

https://techcrunch.com/2022/06/28/hasura-now-lets-developers...

I think it'd be neato to work on a database, but my dayjob is also pretty interesting to me.


One note: I mistakenly thought that fixed buffers were part of the 5.19 changes for provided buffers

Jens Axboe corrected me on this, fixed buffers have existed for a while apparently!

https://twitter.com/axboe/status/1581392910687567873?t=gx_tQ...


Aren't pages in a buffer pool usually in a different representation than the on-disk format?


[flagged]


People say that, yet it is not anywhere near Rust. But yes, I do wish people could now keep Zig submission under control. I dont want Zig community becomes another Rust community.

Zig

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Rust

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...


> I do wish people could now keep Zig submission under control. I dont want Zig community becomes another Rust community.

I don't think anybody has control over what gets posted on HN or not. People act independently and to whatever fancies them. Should the author of this blog post have avoided using Zig to keep the frontpage of HN more balanced? Obviously no. Maybe the curators of HN could prevent Zig posts from reaching the front page but they too have their own motives (eg having the frontpage reflect the topics of interest of the collective HN readership) and it might just be that these are Zig's 15mins of fame.

So I don't think your request can be fulfilled in practice and I don't think that Zig's popularity (or lack thereof) on HN will impact our community. I would recommend judging both Zig's and Rust's respective communities for how they are in their own social spaces (chatrooms, meetups, conferences) and for what they achieve on a concrete level (ie quality of the software they make), more than for what reaches the frontpage of HN with the brand name in the title.


I'm in neither community, so what does that even mean? Why?


> I’ve never heard of a company actually using it

Depending on your definition of "company" and "actually using it", https://oven.sh might qualify?


[flagged]


What do you mean by "won't even compile"?

  $ cat main.zig 
  const std = @import("std");
  
  pub fn main() void {
      std.debug.print("こんにちは, {s}!\n", .{"東京"});
  }
  $ zig run main.zig
  こんにちは, 東京!


> (Commenting from a throwaway account because they are going to lynch me otherwise.)

No, you're just using a throwaway account because you know it's going to eventually get banned, just like it has already mostly happened in the recent flurry of similar posts that you and your unsophisticated friends have made.

https://old.reddit.com/r/programming/comments/y4rgv7/zig_dev...

https://old.reddit.com/r/learnprogramming/comments/y1zuzv/is...

https://old.reddit.com/r/AskProgramming/comments/y1zwan/is_t...

https://stackoverflow.com/questions/74040564/is-the-zig-prog...

https://news.ycombinator.com/item?id=33183818

Plus also sneaking into our communities:

(requires joining the main Zig Discord server) https://discord.com/channels/605571803288698900/102362568632...

Unfortunately for you we play more than enough Secret Hitler to know when it's time to yeet somebody out of the server right away. Can't wait to meet your new alt, maybe next time you will be able to post more than one message before I ban you :^)

> They have a whole Discord server where they upvote and promote posts about Zig.

We have more than one Zig Discord server where people share links, most of which do get posted on HN. If you think that would be enough to game HN, you really have no idea about anything you're talking about.

> (The program won't even compile if you try to write 'hello world' with non-ASCII characters.)

Wrong. https://zig.godbolt.org/z/dane73Wsr

> Also the language is nowhere near a 1.0 release so companies are reluctant to use it in production environments.

Uber is using zig cc:

https://www.youtube.com/watch?v=SCj2J3HcEfc

Bun is written in Zig and has VC funding:

https://bun.sh/

TigerBeetle is written in Zig and has VC funding

https://tigerbeetle.com/

Zig is not yet v1.0 and we do recommend caution to anybody that wants to invest in Zig, as they will have to deal with all the breakages that are expected in a healthy pre-1.0 language.


> Uber is using zig cc

One guy at Uber has set up zig cc in a CI system. Not 1 line of Zig is written at Uber.

> Bun is written in Zig

Bun is pre-alpha software developed by one guy. You can't even install Express with Bun right now.

I hardly call these production-ready code examples of Zig.


[flagged]


Yep that was me. Not interested in replying to any of the points above? Also your comment is already flagged, I would jump into a new alt if I were you.


Where did Zig come from?

Never did I hear about it and suddenly people talk about it.

What happened


If you want an overview of the language and the Zig Software Foundation:

https://www.youtube.com/watch?v=Gv2I7qTux7g

https://ziglang.org/zsf/


Portland, USA apparently (from info at https://twitter.com/andy_kelley).


Popular language of the moment. Rust, Go, Elixir. :)

Next few months will be "built X with Zig" with hundreds of upvotes.


tbh, i don't even think that promotion a product based on their language is a selling point (unless it's a lib of course!)

But this whole `built with X` is in my opinion just some 'i try to appear cool by using x language' and i don't really like it.

Don't get me wrong, i absolutely love rust, zig and go, but i believe, that you shouldn't pick a technology based on such subjective matter. A language is just a tool to solve problems, and you don't really want to screw in a screw with a hammer, do you?


Zig does look great


It was first released in 2016 so it's not ultra new but it's getting more attention at least here.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: