from Hacker News

Show HN: A just another Cron alternative but with much more capabilities

by yohamta on 7/18/22, 2:46 AM with 2 comments

I have created a Cron alternative that runs DAGs (Directed acyclic graph) defined in a simple YAML format.

Why not Airflow? Airflow and other similar tools are powerful and valuable, but in most cases, they require writing code to manage workflows.

Our ETL pipeline is already hundreds of thousands of lines of complex code in Perl and shell scripts. Adding another layer of Python on top of this would make it difficult to maintain. Instead, we needed a more lightweight solution.

So we developed Dagu, which requires no coding, and is easy-to-use and self-contained, making it ideal for smaller projects with fewer people.

  • by theamk on 7/18/22, 3:57 AM

    "existing tools such as Airflow, Prefect, Temporal, etc, but in most cases they require writing code in a programming language such as Python to define DAGs. [...] there is concern that adding another later of Python code will further decrease maintainability. So we developed Dagu, which require no coding, and is easy-to-use and self-contained, making it ideal for smaller projects with fewer people."

    If the goal was to eliminate project-specific DAG code in programming languages, have you considered writing an YAML adapter for an existing tool? For example, an "generic" Airflow pipeline which reads YAML files and generates tasks for each clause. This would have much less code than writing scheduler from scratch, and none of it would be project-specific.