Show HN: iWF – A new “workflow as code” execution engine

[qlong-seattle]

https://github.com/indeedeng/iwf

It's built on top of Cadence Temporal(Similar to AWS SWF, Microsoft Azure Durable Function) by expert of Cadence/Temporal to provide clean and simple API/interface for development.

* Easy to understand and get started

* Easy to make changes -- no non deterministic errors and no need to write versioning code.

* Easy to write and maintain unit tests

* Easy to monitor and operate

* Provide all power of Cadence/Temporal（scalability/reliability)

Look at the samples for how easy it is, and try it out!

[revskill]

A bunch of words without any simple implementation of simple Todo application + PostgreSQL + RestAPI to get started.

Too complicated to do so ? So i can't use this due to complexity.

[doctor_eval → revskill]

For whatever it's worth, I feel like this might be pretty interesting, and I maybe even have a need for something like it, but needing to spin up an unfamiliar database (elastic) is pretty much a show stopper for me.

[qlong-seattle → doctor_eval]

You don't have to run elastic.

Cadence/Temporal let you run only with database for majority of the feature/power. ElasticSearch is only for supporting the searching functionality(advanced visibility).

https://docs.temporal.io/visibility

https://cadenceworkflow.io/docs/concepts/search-workflows/

Same as in iWF, you can use a Temporal service without advanced visibility.

E.g. this doc is showing how to run with Temporalite which doesn't have advanced visibility with it.

https://github.com/indeedeng/iwf/blob/main/CONTRIBUTING.md#o...

Again, you still have 90% power and the only thing missing is searching for workflows using "search attributes".

You can still store data in search attributes but they are just not searchable.

And later, you can migrate to a Cadence/Temporal service that has advanced visibility enabled.

[doctor_eval → qlong-seattle]

Ok thank you for that. This is a bit above my pay grade right now but I will see if I can wrap my head around it in the coming days.

[qlong-seattle → revskill]

There are samples from the README:

https://github.com/indeedeng/iwf-java-samples#product-use-ca...

https://github.com/indeedeng/iwf-golang-samples#product-use-...

[ctvo]

> Cadence workflow is powerful, but hard to learn and use. Though documentation and samples look straightforward to write a HelloWorld or a demo, many Cadence users run into issues to write complicated workflows and maintain/operate it in production.

This is the problem statement from their design document. I'm not sure I agree with it. This appears to be an abstraction over defining Temporal / Cadence workflows.

[qlong-seattle → ctvo]

Maybe you can start with looking at the sample for solving the same "subscription" use case:

Temporal:

https://github.com/temporalio/subscription-workflow-project-...

https://github.com/temporalio/subscription-workflow-project-...

iWF:

https://github.com/indeedeng/iwf-java-samples#product-use-ca...

https://github.com/indeedeng/iwf-golang-samples#product-use-...

Note that you need to consider making changes to workflow -- in Cadence/Temporal, you must use versioning like: https://stackoverflow.com/questions/61298415/how-can-i-versi...

And this versioning/determinism constraints is just a small part of the learning curves for complicated project

And I am not the only one pointing this out. Also see the article from the principal of Azure Durable Function team:

https://medium.com/@cgillum/common-pitfalls-with-durable-exe...

[ctvo → qlong-seattle]

I think being straight forward about how iWF solves the issues you and others have brought up straight on the landing page will do you a lot of favors.

- Dimension A in Temporal, in iWF

- Dimension B in Temporal, in iWF

...

- Here's when you want to use Temporal directly. Here's when you should use iWF.

I'm still unsure how iWF solves the items brought up in the blog post you linked.

[qlong-seattle → ctvo]

Yeah this wiki page has exactly what you want to see here:

https://github.com/indeedeng/iwf/wiki/How-to-migrate-from-Ca...

[mdaniel]

Congratulations on your release!

I have 2 interrelated comments: the "If you are not [familiar with Cadence]" link <https://github.com/indeedeng/iwf#if-you-are-not> seems to go to a WIP Google Doc, so I would suggest a more tl;dr version instead of linking out to Google Docs. Related to that, it would be great if the "supporting docs" were either in the repo's wiki or in the repo, because it has been my experience that Google Docs (and its Slides friend) are 404s waiting to happen when the owner forgets they're shared or Google "does a Google" and bans accounts

Maybe I'm not the target audience for this because I don't already know what pains Cadence causes, as that seems to be about 80% of the message here, but I do love workflow engines and thus want to understand how this solves Cadence problems but doing independent research into Cadence just to know what makes iWF better is too much to ask, IMHO

[qlong-seattle → mdaniel]

Hello, no you are definitely the target users. Anyone who needs to build long running business/application/process is the target user of iWF.

You are right, there were too many context about Cadence/Temporal in the README. I just made a update to move a lot of them to wiki just for the Cadence/Temporal users.You can check https://medium.com/@qlong/iwf-vs-other-general-purposed-work... for comparison with other workflow engines.

I will move the G docs and slides later.

Thanks for the suggestion!

[swyx → mdaniel]

i wrote a longer intro to the problem domain of Cadence/Temporal here in case it helps: https://www.swyx.io/why-temporal

and you can browse the Temporal youtube (https://www.youtube.com/@Temporalio) for anywhere from 2min - 2hour explainers of what Temporal does, what you use it for, how it works.

(disclaimer - former employee. still a big fan of the entire category - every company needs a workflow engine for anything long running (usually the most valuable/hard-to-test async stuff), and your choices are between build a bad one yourself or take the most industry battle tested one - uber, airbnb, coinbase, netflix, stripe, etc - off the shelf and get tooling/docs/ecosystem/maintenance for free)

[johtso → swyx]

What do you think about using Cloudflare durable objects for long running stuff instead of something like temporal?

[swyx → johtso]

durable objects are just a persistence layer, you still need scheduling, state machine, introspection/querying/analytics (incl search attributes in temporal terminology), signal response, batch transactions etc etc. so you have to understand/internalize that workflow engines are a layer of abstraction above that and if you don't you're going to be handrolling one without knowing it.

[johtso → swyx]

Right.. the tricky thing is when you think your use case is relatively basic, you're a one person team, and you want to decide whether to pull in such a powerful tool, and deploy and manage it.. or take a more DIY approach, have it all just run on demand in the cloud, and not have to learn a bunch of APIs and concepts as they're described by Temporal etc.

cloudflare alarms seem like a decent solution for scheduling. If you want a state machine, you can just use a state machine (xstate).

The durable object primitives are more limited, but therefor seem a little easier to get your head around if you don't need all sorts of complex signalling.

introspection / analytics is obviously a pain to have to deal with yourself. I guess you'd end up saving the state to a database after any action. And you could probably get a reasonable little analytics dashboard fairly painlessly by chucking something like appsmith in front of it?

[rylandgold → johtso]

You can absolutely approach it that way, but AFAIK those are highly integrated but still discrete solutions. Temporal provides a unified model for these things instead of requiring people to stitch things together themselves. Due to Temporal handling all of the things, you also get additional value out of the centralization (you can see the live source of truth for any running business process).

I do agree that Temporal is heavyweight both in terms of the learning curve and deployment requirements.

For disclosure I’m the Head of Product at Temporaly. We know things can be better and we are working on it.

[johtso → rylandgold]

I can't wait for Temporal's cloud offering to work for the "small dev". It would be so wonderful to only need to worry about the learning curve of using the tool, and not the deployment + cost. It would be great to be able to reach for it for any little project that needed robust persistent workflows.

[rylandgold → johtso]

Completely and totally agree! Don't lose faith we will get there I promise.

[qlong-seattle → johtso]

iWF is not just persisting states, but also able invoking/executing states, searching for attributes(like unlimited database indexes), durable timer, resetting workflows etc.

Without Temporal/Cadence, it's 100+ harder to implement iWF. Even using SWF/Azure durable function is possible to build iWF, Cadence/Temporal is much more powerful than the other two(e.g. no reset workflow, search attributes)

[ploppyploppy]

Classic over-engineered "enterprisey".

I'll stick with Argo Workflows thanks.

[santialbo]

If I wanted to build some sort of Zapier-like workflows within my application for my customers to create and run, would I be better using one of these libraries as backend?

[qlong-seattle → santialbo]

I believe so. Welcome to try out. There are only a few concepts you need to learn about iWF and you can start with docker-compose to run a iWF service locally for development.

[Towaway69]

How does this compare to something like Node-Red[0] which is also flow based?

[0]=https://nodered.org

[qlong-seattle → Towaway69]

There are lots of things that other workflow engines cannot do:

Reset workflow : important to recover from bad states for any reasons. You may have bugs caused this or because downstream outages. Reset can let you go back like timing machines.

Search attributes: let you search for anything like having unlimited database index.

See more details in this article: https://medium.com/@qlong/iwf-vs-other-general-purposed-work...

[jasfi → Towaway69]

It seems node-red has a single-threaded execution engine, see: https://discourse.nodered.org/t/node-red-performance-benchma...

However this iWF can execute flows in parallel: "Parallel execution of multiple threads of business".

[panchicore3]

Is it similar to apache nifi?

[qlong-seattle → panchicore3]

Took at quick look at apache nifi. Here is my conclusion but anyone feel free to correct me if I am wrong:

I think they only overlap for a small part but quite different:

* Nifi is focusing on moving/sharing datafile (meaning the size could be big) across the system * Nifi does have some logic control flow to act like workflow, but it's not too closed to be a "workflow system" because it missing a lot of capabilities that a typical workflows should have (see "What they all have in common " in https://medium.com/@qlong/iwf-vs-other-general-purposed-work...)

* iWF is a general purposed workflow engine, could conceptually do data sharing but it is not good at sharing the big amount of data or data files * iWF is more specialized at controlling/orchestrating the business

[LoganDark]

Java and Go only?

[qlong-seattle → LoganDark]

We don't have any plan to build other SDKs at this moment.

But it's VERY EASY to build an SDK for iWF so I would encourage anyone interested to build one. I will provide all the support.

A SDK of iWF is very lightweight. It only need to invoke the two WorkflowState APIs from hosting a REST/HTTP service, based on the worker API schema. https://github.com/indeedeng/iwf-idl/blob/main/iwf.yaml#L215

There is no much tricky things inside the SDK that you have to do. For client APIs to call iWF server(the rest of the API defined in the Open API schema), just need to build a wrapper on top of the generated code from Swagger (and it can be optional but it will just make it easier to use).

As a data point, it only took me a week to build GoSDK, including all the tests, by just opening the IntelliJ IDE and also Goland IDE to translate the Java code into Golang.

The whole community including myself will really appreciate it.