Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours.
Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.
Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.
For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events?
It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery.
Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.
