There are a number of deficiencies with the current system that aren't showstoppers, but are pain points nonetheless. Off the top of my head:
* There's no reasonable way to do cross-batch dependencies (e.g., if process X in batch A fails, don't run process Y in batch B). I've got a few ideas on how I could add this in, but nothing has been implemented yet.
* There's no easy way to visualize what's going on. Airflow has a Gantt view that looks very useful for this purpose, our business users would absolutely LOVE the task duration graph, and the visualization of the DAG looks really helpful too.
* Continuing a failed batch is a very manual process.
None of these are showstoppers because, as you said, this has been running fine for over a decade. These are mostly quality-of-life improvements.
Ah, I understand. That makes sense. If you have business users, then it makes sense to go with something like Airflow because they do make it easier for less technical users to inspect jobs, kick them off, visualize them, etc. The UI makes all the difference for those use cases.
