We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.
As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.
Stack: Typescript/React, Go and PostgreSQL.
To apply, email alexander [at] hatchet [dot] run, and include the following:
1. Tell us about something impressive you've built.
2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.
Yeah we're using RabbitMQ for pub/sub (but are considering getting rid of it) and Postgres for the actual task queue. There's some more about that here: https://news.ycombinator.com/item?id=39643940.
Ah, great catch - I just pushed an update to match the query from the article. I wasn't looking at much except for the WindowAgg line, thank you!
I tried with a similar indexing strategy - it did make a very noticeable difference, breaking at about 40000 enqueued tasks instead of 25000. I left indexing out of the article because it can open up a different can of worms with performance degradation over a longer time horizon.
I also tried with an `ORDER BY RANDOM()` across group keys first, which does help with fairness but breaks the "deterministic" element.
So it's still doing a seq scan on tasks when I'd expect it to join using the PK. It must be getting tripped up by the redundant filter on queued status. Try removing that.
I ninja edited my previous comment, but if you move the LIMIT to the 2nd CTE, that should fix your issue with workers not getting work. If you do that and add the other index I think in principle it should be able to do everything by maintaining a priority queue of the heads of each partition (which are each pre-sorted by the index now). idk if pg does that though. If it does, then that portion of the query should be streamed, so you don't need to try to limit it early to avoid a sort of 10k elements when you only need 10. Then if you remove the redundant QUEUED check, it should be doing everything else through indexes.
Basically, if doing this manually I'd expect the "good" solution to do this in a way where starting from an index, each row is streamed (i.e. no full sort) with logn complexity. So I look at it from a perspective of "how do I get the database to do what I'd do by hand?"
I created a gist with these recommendations - certainly an improvement, but I don't think it gets around the `WindowAgg` running across all 10k rows: https://gist.github.com/abelanger5/5c1a75755072239716cb587a2.... Does this accurately implement your suggestions?
Happy to try out other suggestions, but I haven't found a way to get a window function or `JOIN LATERAL` to scale in near-constant time for this queueing strategy.
It looks like now it does still only pull 100 rows out of the sort (so moving the limit into the 2nd cte didn't hurt). It isn't doing all 10000 rows now though, which is interesting. By any chance, do you have 9200 different tenants? If so that makes sense. What I suggested would work when you have a small number of tenants with queued work (it scales n log n with tenants with queued work, but log n with amount of tasks that a single tenant has queued). So if you're currently testing with many tenants queueing at once, you could see how it behaves with like 20 tenants where one has 9k items and the others have ~50 each. Then it sort of depends on how your distribution looks in practice to know whether that's acceptable.
You could also probably do tricks where individual workers filter to specific tenant IDs in the first CTE (e.g. filter group_key mod num_workers = worker_id) to reduce that cardinality if you expect it to be large. Or you could e.g. select 100 random group_keys as a first step and use that to filter the window, but then that needs another partial index on just `group_key where status = queued`.
Edit: I see it's still doing a seq scan on tasks. There is a PK on id, right? It knows there's only 100 rows from the rest of the query so it seems odd to me that it would decide to scan the table. You could try putting a hash index on id if it's refusing to use the btree index I guess. Or it might change its mind if you add 1M SUCCEEDED tasks or something.
Another thing to consider is that out of the box, pg's default config for the planner is tuned to like 20 year old hardware. You need to tweak the io costs for SSDs and tell it you have more RAM if you haven't done that. See e.g. https://pgtune.leopard.in.ua/ for better starting values.
No, this assumes you've already split your tasks into quanta of approximately the same duration - so all tasks are weighted equally in terms of execution time. If each of the tasks have different weights you might be looking at something like deficit round-robin [1], which I've looked at implementations for in RabbitMQ and we're thinking about how to implement in PG as well [2].
My gut tells me that it would often make sense to jump straight to shuffle sharding, where you'd converge on fair solutions dynamically, in a lot of cases. I'm looking forward to that follow-on article!
We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.
As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.
Stack: Typescript/React, Go and PostgreSQL.
To apply, email alexander [at] hatchet [dot] run, and include the following:
1. Tell us about something impressive you've built.
2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.
> Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs...
Yeah, it's the latter, though we'd actually like to support the former in the longer term. There's no technical reason we can't write the workflow/task and read from the same table that you're enqueueing with in the same transaction as your application. That's the really exciting thing about the RiverQueue implementation, though it also illustrates how difficult it is to support every PG driver in an elegant way.
Transactional enqueueing is important for a whole bunch of other reasons though - like assigning workers, maintaining dependencies between tasks, implementing timeouts.
> why take funding to build this (it seems bootstrappable? maybe that's naive)
The thesis is that we can help some users offload their tasks infra with a hosted version, and hosted infra is hard to bootstrap.
> why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups?
I think Cloudflare is an interesting example here. You could probably make similar arguments against a spam protection proxy, which was the initial service. But a lot of the core infrastructure needed for that service branches into a lot of other products, like a CDN or caching layer, or a more compelling, full-featured product like a WAF. I can't speak for YC or the other dev infra startups, but I imagine that's part of the thesis.
> a lot of the core infrastructure needed for that service branches
> into a lot of other products
Ah, I think I see what you mean--the goal isn't to be "just a job queue" in 2-5 years, it's to grow into a wider platform/ecosystem/etc.
Ngl I go back/forth between rooting for dev-founded VC companies like yourself, or Benjie, the guy behind graphile-worker, who is tip-toeing into being commercially-supported.
Like I want both to win (paydays all around! :-D), but the VC money just gives such a huge edge, of establishing a very high bar of polish / UX / docs / devrel / marketing, basically using loss-leading VC money for a bet that may/may not work out, that it's very hard for the independents to compete. I have honestly been hoping post-ZIRP would swing some advantage back to the Benjie's of the world, but looks like no/not yet.
...I say all of above ^ while myself working for a VC-backed prop-tech company...so, kinda the pot calling the kettle black. :-D
Good luck! The fairness/priority queues of Hatchet definitely seem novel, at least from what I'm used to, so will keep it bookmarked/in the tool chest.
It can be used as an alternative to dagster or airflow but doesn't have the prebuilt connectors that airflow offers. And yes, there are ways to reuse tasks across workflows, but the docs for that aren't quite there yet. The key is to call a `registerAction` method and create the workflow programmatically - but we have some work to do before we publicize this pattern (for one, removing the overloading of the term action, function, step and task).
We'll be posting updates and announcements in the Discord - and the Github in our releases - I'd expect that we document this pattern pretty soon.
We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.
We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.
As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.
Stack: Typescript/React, Go and PostgreSQL.
To apply, email alexander [at] hatchet [dot] run, and include the following:
1. Tell us about something impressive you've built.
2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.