Dagu

Appearance

Examples ​

Quick reference for all Dagu features. Each example is minimal and copy-paste ready.

Basic Workflows ​

Basic Sequential Steps ​

yaml
steps:
  - id: first
    run: echo "Step 1"
  - id: second
    run: echo "Step 2"
    depends: first

Learn more →

Parallel Execution (Iterator) ​

yaml
steps:
  - action: dag.run
    with:
      dag: processor
      params: "ITEM=${ITEM}"
    parallel:
      items: [A, B, C]
      max_concurrent: 2
---
name: processor
steps:
  - run: echo "Processing ${ITEM}"

Learn more →

Multiple Commands per Step ​

yaml
tools:
  - nodejs/node@v22.21.1

steps:
  - id: build_and_test
    run: |
      npm install
      npm run build
      npm test
    env:
      - NODE_ENV: production
    working_dir: /app

Share step config (env, working_dir, retry_policy, etc.) across commands instead of duplicating across steps.

Learn more →

Reproducible CLI Tools ​

yaml
tools:
  - jqlang/jq@jq-1.7.1

steps:
  - id: transform
    run: jq '.items[] | .name' input.json

Pin portable command-line dependencies in the DAG so workers install the expected binary before running host command steps.

Learn more →

Dependency Modes ​

yaml
# Default graph mode: explicit dependencies define order
steps:
  - id: step_1
    run: echo "step 1"
  - id: step_2
    run: echo "step 2"
    depends: step_1

# Independent steps can run in parallel
---
steps:
  - id: a
    run: echo A
  - id: b
    run: echo B

Learn more →

Control Flow & Conditions ​

Conditional Execution ​

yaml
steps:
  - run: echo "Deploying application"
    preconditions:
      - condition: "${ENV}"
        expected: "production"

Learn more →

Repeat Until Condition ​

Looking for iteration over a list? See Parallel Execution.

yaml
steps:
  - run: curl -f http://service/health
    repeat_policy:
      repeat: true
      interval_sec: 10
      exit_code: [1]  # Repeat while exit code is 1

Learn more →

Repeat Until Command Succeeds ​

yaml
steps:
  - run: curl -f http://service:8080/health
    repeat_policy:
      repeat: until        # Repeat UNTIL service is healthy
      exit_code: [0]        # Exit code 0 means success
      interval_sec: 10      # Wait 10 seconds between attempts
      limit: 30            # Maximum 5 minutes

Learn more →

Repeat Until Output Match ​

yaml
 steps: 
  - run: echo "COMPLETED"  # Simulates job status check
    output: JOB_STATUS
    repeat_policy:
      repeat: until        # Repeat UNTIL job completes
      condition: "${JOB_STATUS}"
      expected: "COMPLETED"
      interval_sec: 30
      limit: 120           # Maximum 1 hour (120 attempts)

Learn more →

Repeat Steps ​

yaml
steps:
  - run: echo "heartbeat"  # Sends heartbeat signal
    repeat_policy:
      repeat: while            # Repeat indefinitely while successful
      interval_sec: 60

Learn more →

Repeat Steps Until Success ​

yaml
steps:
  - run: echo "Checking status"
    repeat_policy:
      repeat: until        # Repeat until exit code 0
      exit_code: [0]
      interval_sec: 30
      limit: 20            # Maximum 10 minutes

Learn more →

DAG-Level Preconditions ​

yaml
preconditions:
  - condition: "`date +%u`"
    expected: "re:[1-5]"  # Weekdays only

steps:
  - run: echo "Run on business days"

Learn more →

Negated Preconditions ​

yaml
steps:
  # Run only when NOT in production
  - run: echo "Running dev task"
    preconditions:
      - condition: "${ENVIRONMENT}"
        expected: "production"
        negate: true

  # Run only on weekends
  - run: echo "Weekend maintenance"
    preconditions:
      - condition: "`date +%u`"
        expected: "re:[1-5]"  # Weekdays
        negate: true          # Invert: run on weekends

Learn more →

Routing Based on Value ​

yaml
type: graph
env:
  - STATUS: production
steps:
  - id: router
    action: router.route
    with:
      value: ${STATUS}
      routes:
        "production": [prod_handler]
        "staging": [staging_handler]

  - id: prod_handler
    run: echo "Production"

  - id: staging_handler
    run: echo "Staging"

Learn more →

Routing Based on Step Output ​

yaml
type: graph
steps:
  - id: check_status
    run: echo "success"
    output: STATUS

  - id: router
    action: router.route
    with:
      value: ${STATUS}
      routes:
        "success": [success_handler]
        "failure": [failure_handler]
    depends: check_status

  - id: success_handler
    run: echo "Handling success"

  - id: failure_handler
    run: echo "Handling failure"

Learn more →

Continue On: Exit Codes and Output ​

yaml
steps:
  - id: optional_check
    run: exit 3  # This will exit with code 3
    continue_on:
      exit_code: [0, 3]        # Treat 0 and 3 as non-fatal
      output:
        - "WARNING"
        - "re:^INFO:.*"       # Regex match
      mark_success: true       # Mark as success when matched
  - id: continue_after_check
    run: echo "Continue regardless"
    depends: optional_check

Learn more →

Nested Workflows ​

yaml
steps:
  - id: run_etl
    action: dag.run
    with:
      dag: etl.yaml
      params: "ENV=prod DATE=today"

  - id: run_analysis
    action: dag.run
    with:
      dag: analyze.yaml
    depends: run_etl

Learn more →

Multiple DAGs in One File ​

yaml
steps:
  - action: dag.run
    with:
      dag: data-processor
      params: "TYPE=daily"
---

name: data-processor
params:
  - TYPE: "batch"
steps:
  - id: extract
    run: echo "Extracting ${TYPE} data"

  - id: transform
    run: echo "Transforming data"
    depends: extract

Learn more →

Dispatch to Specific Workers ​

yaml
tools:
  - astral-sh/uv@0.11.14

steps:
  - id: prepare_dataset
    run: uv run --python 3.13.9 python prepare_dataset.py

  - id: train_model
    action: dag.run
    with:
      dag: train-model
    depends: prepare_dataset

  - id: evaluate_model
    action: dag.run
    with:
      dag: evaluate-model
    depends: train_model
---
name: train-model
worker_selector:
  gpu: "true"
  cuda: "11.8"
  memory: "64G"
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python train.py --gpu
---
name: evaluate-model
worker_selector:
  gpu: "true"
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python evaluate.py

Learn more →

Mixed Local and Worker Steps ​

yaml
steps:
  # Runs on any available worker (local or remote)
  - id: download_dataset
    run: wget https://data.example.com/dataset.tar.gz

  # Must run on specific worker type
  - id: process_on_gpu
    action: dag.run
    with:
      dag: process-on-gpu
    depends: download_dataset

  # Runs locally (no selector)
  - id: finish
    run: echo "Processing complete"
    depends: process_on_gpu

---
name: process-on-gpu
worker_selector:
  gpu: "true"
  gpu-model: "nvidia-a100"
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python gpu_process.py

Learn more →

Force Local Execution ​

yaml
# When default_execution_mode is "distributed", use worker_selector: local
# to keep specific DAGs on the main instance
worker_selector: local

steps:
  - id: health_check
    run: curl -f http://localhost:8080/health

  - id: finish
    run: echo "Ran locally"
    depends: health_check

Use worker_selector: local as an escape hatch in distributed deployments for lightweight DAGs that should never leave the main instance.

Learn more →

Parallel Distributed Tasks ​

yaml
tools:
  - astral-sh/uv@0.11.14

steps:
  - id: split_data
    run: uv run --python 3.13.9 python split_data.py --chunks=10
    output: CHUNKS

  - action: dag.run
    with:
      dag: chunk-processor
      params: "CHUNK=${ITEM}"
    parallel:
      items: ${CHUNKS}
      max_concurrent: 5
    depends: split_data

  - run: uv run --python 3.13.9 python merge_results.py
---
name: chunk-processor
worker_selector:
  memory: "16G"
  cpu-cores: "8"
params:
  - CHUNK: ""
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python process_chunk.py ${CHUNK}

Learn more →

Error Handling & Reliability ​

Continue on Failure ​

yaml
steps:
  # Optional task that may fail
  - id: optional_task
    run: exit 1  # This will fail
    continue_on:
      failure: true
  # This step always runs
  - id: required_task
    run: echo "This must succeed"
    depends: optional_task

Learn more →

Continue on Skipped ​

yaml
steps:
  # Optional step that may be skipped
  - id: optional_feature
    run: echo "Enabling feature"
    preconditions:
      - condition: "${FEATURE_FLAG}"
        expected: "enabled"
    continue_on:
      skipped: true
  # This step always runs
  - id: main_task
    run: echo "Processing main task"
    depends: optional_feature

Learn more →

Retry on Failure ​

yaml
steps:
  - run: curl https://api.example.com
    retry_policy:
      limit: 3
      interval_sec: 30

Learn more →

Smart Retry Policies ​

yaml
steps:
  - run: curl -f https://api.example.com/data
    retry_policy:
      limit: 5
      interval_sec: 30
      exit_code: [429, 503, 504]  # Rate limit, service unavailable

Learn more →

Retry with Exponential Backoff ​

yaml
steps:
  - run: curl https://api.example.com/data
    retry_policy:
      limit: 5
      interval_sec: 2
      backoff: true        # 2x multiplier
      max_interval_sec: 60   # Cap at 60s
      # Intervals: 2s, 4s, 8s, 16s, 32s → 60s

Learn more →

Repeat with Backoff ​

Looking for iteration over a list? See Parallel Execution.

yaml
steps:
  - run: nc -z localhost 8080
    repeat_policy:
      repeat: while
      exit_code: [1]        # While connection fails
      interval_sec: 1
      backoff: 2.0
      max_interval_sec: 30
      limit: 20
      # Check intervals: 1s, 2s, 4s, 8s, 16s, 30s...

Learn more →

Lifecycle Handlers ​

yaml
steps:
  - run: echo "Processing main task"
handler_on:
  success:
    run: echo "SUCCESS - Workflow completed"
  failure:
    run: echo "FAILURE - Cleaning up failed workflow"
  exit:
    run: echo "EXIT - Always cleanup"

Learn more →

Data & Variables ​

Environment Variables ​

yaml
env:
  - SOME_DIR: ${HOME}/batch
  - SOME_FILE: ${SOME_DIR}/some_file
  - LOG_LEVEL: debug
  - API_KEY: ${SECRET_API_KEY}
tools:
  - astral-sh/uv@0.11.14

steps:
  - working_dir: ${SOME_DIR}
    run: uv run --python 3.13.9 python main.py ${SOME_FILE}

Learn more →

Dotenv Files ​

yaml
# Specify single dotenv file
dotenv: .env

# Load multiple files (all files loaded, later override earlier)
dotenv:
  - .env.defaults
  - .env.local
  - .env.production

steps:
  - run: echo "Database: ${DATABASE_URL}"

Learn more →

Secrets ​

yaml
secrets:
  - name: DB_PASSWORD
    ref: prod/db-password
  - name: API_TOKEN
    provider: env
    key: PROD_API_TOKEN

steps:
  - run: ./sync.sh
    env:
      - AUTH_HEADER: "Bearer ${API_TOKEN}"
      - STRICT_MODE: "1"

Learn more →

Positional Parameters ​

yaml
params: param1 param2  # Default values for $1 and $2
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python main.py $1 $2

Learn more →

Named Parameters ​

yaml
params:
  - name: foo
    type: integer
    default: 1
  - name: bar
    default: "2"
  - name: environment
    type: string
    default: dev
    enum: [dev, staging, prod]
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: uv run --python 3.13.9 python main.py ${foo} ${bar} --env=${environment}

Learn more →

Output Variables ​

yaml
steps:
  - id: get_today
    run: echo `date +%Y%m%d`
    output: TODAY
  - id: print_today
    run: echo "Today's date is ${TODAY}"
    depends: get_today

Learn more →

Parallel Outputs Aggregation ​

yaml
steps:
  - id: process_regions
    action: dag.run
    with:
      dag: worker
      params: "REGION=${ITEM}"
    parallel:
      items: [east, west, eu]
    output: RESULTS

  - id: summarize_regions
    run: |
      echo "Total: ${RESULTS.summary.total}"
      echo "First region: ${RESULTS.results[0].params}"
      echo "First output: ${RESULTS.outputs[0].value}"
    depends: process_regions

---
name: worker
params:
  - REGION: ""
steps:
  - run: echo ${REGION}
    output: value

Learn more →

Special Variables ​

yaml
steps:
  - run: |
      echo "DAG: ${DAG_NAME}"
      echo "Run: ${DAG_RUN_ID}"
      echo "Step: ${DAG_RUN_STEP_NAME}"
      echo "Log: ${DAG_RUN_LOG_FILE}"

Learn more →

Output Size Limits ​

yaml
# Set maximum output size to 5MB for all steps
max_output_size: 5242880  # 5MB in bytes

steps:
  - run: "cat large-file.txt"
    output: CONTENT  # Will fail if file exceeds 5MB

Control output size limits to prevent memory issues.

Learn more →

Stream Output to Artifacts ​

yaml
steps:
  - run: "echo hello"
    stdout:
      artifact: hello.txt
  
  - run: "echo error message >&2"
    stderr:
      artifact: error.txt

Learn more →

JSON Path References ​

yaml
steps:
  - id: run_sub_workflow
    output: SUB_RESULT
    action: dag.run
    with:
      dag: sub_workflow
  - id: print_result
    run: echo "Result: ${SUB_RESULT.outputs.finalValue}"
    depends: run_sub_workflow

Learn more →

Structured Step Output ​

yaml
steps:
  - id: build
    run: |
      echo '{"version":"v1.2.3"}'
    output: BUILD_JSON

  - id: publish
    output:
      version: "${build.output.version}"
      versionLabel: "ver - ${build.output.version}"
    depends: build

  - id: print
    run: echo "${publish.output.versionLabel}"
    depends: publish

Learn more →

Step ID References ​

yaml
type: graph
tools:
  - astral-sh/uv@0.11.14

steps:
  - id: extract
    run: uv run --python 3.13.9 python extract.py
    output: DATA
  - run: |
      echo "Exit code: ${extract.exit_code}"
      echo "Stdout path: ${extract.stdout}"
    depends: extract

Learn more →

Command Substitution ​

yaml
env:
  TODAY: "`date '+%Y%m%d'`"
steps:
  - run: echo hello, today is ${TODAY}

Learn more →

Scripts & Code ​

Shell Scripts ​

yaml
steps:
  - run: |
      #!/bin/bash
      cd /tmp
      echo "hello world" > hello
      cat hello
      ls -la

Run shell script with default shell.

Learn more →

Shebang Script ​

yaml
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: |
      #!/usr/bin/env -S uv run --python 3.13.9 python
      import platform
      print(platform.python_version())

Runs with the interpreter declared in the shebang.

Learn more →

Python Scripts ​

yaml
tools:
  - astral-sh/uv@0.11.14

steps:
  - run: |
      import os
      import datetime
      
      print(f"Current directory: {os.getcwd()}")
      print(f"Current time: {datetime.datetime.now()}")
    with:
      shell: uv run --python 3.13.9 python

Execute script with specific interpreter.

Learn more →

Multi-Step Scripts ​

yaml
steps:
  - run: |
      #!/bin/bash
      set -e
      
      echo "Starting process..."
      echo "Preparing environment"
      
      echo "Running main task..."
      echo "Running main process"
      
      echo "Cleaning up..."
      echo "Cleaning up"

Learn more →

Working Directory ​

yaml
working_dir: /tmp
steps:
  - id: show_default_dir
    run: pwd               # Outputs: /tmp
  - id: create_data_dir
    run: mkdir -p data
    depends: show_default_dir

  - id: show_data_dir
    working_dir: /tmp/data
    run: pwd      # Outputs: /tmp/data
    depends: create_data_dir

Learn more →

Shell Selection ​

yaml
shell: ["/bin/bash", "-e"]   # Default shell for all steps
steps:
  - run: echo hello world | xargs echo
  - run: echo "from zsh"     # Override for a single step
    with:
      shell: /bin/zsh

Learn more →

Reproducible Env with Nix Shell ​

Note: Requires nix-shell to be installed separately. Not included in Dagu binary or container.

yaml
steps:
  - run: |
      python3 --version
      curl --version
      jq --version
    with:
      shell: nix-shell
      shell_packages: [python3, curl, jq]

Learn more →

Actions & Integrations ​

Custom Action ​

yaml
actions:
  say:
    description: Print a reusable message
    input_schema:
      type: object
      additionalProperties: false
      required: [message]
      properties:
        message:
          type: string
    template:
      action: exec
      with:
        command: echo
        args:
          - {$input: message}

steps:
  - action: say
    with:
      message: "build finished"

with is validated by input_schema; the rendered template runs as a builtin exec action.

Add output_schema when stdout should be a typed JSON contract. Invalid JSON or schema mismatches fail the step; without an explicit output: mapping, the validated object is available as ${step_id.output.*}.

Learn more →

Container Workflow ​

yaml
# DAG-level container for all steps
container:
  image: python:3.11
  env:
    - PYTHONPATH=/app
  volumes:
    - ./src:/app

steps:
  - id: install
    run: pip install -r requirements.txt
  - id: test
    run: pytest tests/
    depends: install

  - id: build
    run: python setup.py build
    depends: test

Learn more →

Keep Container Running ​

yaml
# Use keep_container at DAG level
container:
  image: postgres:16
  keep_container: true
  env:
    - POSTGRES_PASSWORD=secret
  ports:
    - "5432:5432"

steps:
  - id: start_postgres
    run: postgres -D /var/lib/postgresql/data
  - id: wait_for_postgres
    run: pg_isready -U postgres -h localhost
    retry_policy:
      limit: 10
      interval_sec: 2
    depends: start_postgres

Learn more →

Step-Level Container ​

yaml
steps:
  - id: build
    container:
      image: node:18
      volumes:
        - ./src:/app
      working_dir: /app
    run: npm run build

Learn more →

Kubernetes Job ​

yaml
kubernetes:
  namespace: batch
  service_account: dagu-runner

steps:
  - id: report
    action: k8s.run
    with:
      image: alpine:3.20
      command: [sh, -c, 'echo hello from kubernetes']

Learn more →

Exec Into Existing Container ​

yaml
# Run commands in an already-running container
container: my-app-container

steps:
  - id: migrate
    run: php artisan migrate
  - id: clear_cache
    run: php artisan cache:clear
    depends: migrate

Learn more →

Exec Mode with Overrides ​

yaml
# Override user and working directory
container:
  exec: my-app-container
  user: root
  working_dir: /var/www
  env:
    - APP_DEBUG=true

steps:
  - id: install
    run: composer install
  - id: fix_permissions
    run: chown -R www-data:www-data storage
    depends: install

Learn more →

Mixed Exec and Image Mode ​

yaml
steps:
  # Exec into app container
  - id: maintenance_mode
    container: my-app
    run: php artisan down

  # Run migration in fresh container
  - id: migrate
    container:
      image: my-app:latest
    run: php artisan migrate
    depends: maintenance_mode

  # Exec back into app container
  - id: restart
    container: my-app
    run: php artisan up
    depends: migrate

Learn more →

Remote Commands via SSH ​

yaml
# Configure SSH once for all steps
ssh:
  user: deploy
  host: production.example.com
  key: ~/.ssh/deploy_key

steps:
  - id: health_check
    run: curl -f localhost:8080/health
  - id: restart_app
    run: systemctl restart myapp
    depends: health_check

Learn more →

Container Volumes: Relative Paths ​

yaml
working_dir: /app/project
container:
  image: python:3.11
  volumes:
    - ./data:/data        # Resolves to /app/project/data:/data
    - .:/workspace        # Resolves to /app/project:/workspace
steps:
  - run: python process.py

Learn more →

HTTP Requests ​

yaml
steps:
  - action: http.request
    with:
      method: POST
      url: https://api.example.com/webhook
      headers:
        Content-Type: application/json
      body: '{"status": "started"}'

Learn more →

Wait for Readiness ​

yaml
steps:
  - id: wait_for_api
    action: wait.http
    with:
      url: https://api.example.com/health
      status: 200
      poll_interval: 5s
    timeout_sec: 300

  - id: continue_after_ready
    run: ./run-after-ready.sh
    depends: wait_for_api

Learn more →

JSON Processing ​

yaml
steps:
  # Fetch sample users from a public mock API
  - id: fetch_users
    action: http.request
    with:
      method: GET
      url: https://reqres.in/api/users
      silent: true
    output: API_RESPONSE

  # Extract user emails from the JSON response
  - id: extract_emails
    action: jq.filter
    with:
      filter: '.data[] | .email'
      data: ${API_RESPONSE}
    depends: fetch_users

Learn more →

Archive Extraction ​

yaml
working_dir: /tmp/data

steps:
  - action: archive.extract
    with:
      source: dataset.tar.zst
      destination: ./dataset

Learn more →

Container Startup & Readiness ​

yaml
container:
  image: alpine:latest
  startup: command           # keepalive | entrypoint | command
  command: ["sh", "-c", "my-daemon"]
  wait_for: healthy           # running | healthy
  log_pattern: "Ready"        # Optional regex to wait for
  restart_policy: unless-stopped

steps:
  - run: echo "Service is ready"

Learn more →

Private Registry Auth ​

yaml
registry_auths:
  ghcr.io:
    username: ${GITHUB_USER}
    password: ${GITHUB_TOKEN}

container:
  image: ghcr.io/myorg/private-app:latest

steps:
  - run: ./app

Learn more →

Multi-Container Workflow ​

yaml
steps:
  - id: build
    container:
      image: node:24
      volumes:
        - ./src:/app
      working_dir: /app
    run: npm run build

  - id: test
    container:
      image: node:24
      volumes:
        - ./src:/app
      working_dir: /app
    run: npm test
    depends: build

  - id: deploy
    container:
      image: python:3.11
      env:
        - AWS_DEFAULT_REGION=us-east-1
    run: python deploy.py
    depends: test

Learn more →

SSH: Advanced Options ​

yaml
ssh:
  user: deploy
  host: app.example.com
  port: 2222
  key: ~/.ssh/deploy_key
  strict_host_key: true
  known_host_file: ~/.ssh/known_hosts

steps:
  - run: systemctl status myapp

Learn more →

Mail ​

yaml
smtp:
  host: smtp.gmail.com
  port: "587"
  username: "${SMTP_USER}"
  password: "${SMTP_PASS}"

steps:
  - action: mail.send
    with:
      to: team@example.com
      from: noreply@example.com
      subject: "Weekly Report"
      message: "Attached."
      attachments:
        - run: report.txt

Learn more →

Chat / LLM Request ​

yaml
steps:
  - action: chat.completion
    with:
      provider: openai
      model: gpt-4o
      messages:
        - role: user
          content: "What is 2+2?"
    output: ANSWER

Learn more →

Chat with DAG-Level Config ​

yaml
llm:
  provider: openai
  model: gpt-4o
  system: "You are a helpful assistant."

steps:
  - action: chat.completion
    with:
      messages:
        - role: user
          content: "Explain ${TOPIC} briefly."

Steps inherit LLM config from DAG level.

Learn more →

Multi-turn Session ​

yaml
steps:
  - id: ask
    action: chat.completion
    with:
      provider: openai
      model: gpt-4o
      messages:
        - role: user
          content: "What is 2+2?"

  - id: follow_up
    action: chat.completion
    with:
      provider: openai
      model: gpt-4o
      messages:
        - role: user
          content: "Now multiply that by 3."
    depends: ask

Steps inherit session history from previous steps.

Learn more →

Extended Thinking Mode ​

yaml
steps:
  - action: chat.completion
    with:
      provider: anthropic
      model: claude-sonnet-4-20250514
      thinking:
        enabled: true
        effort: high
      messages:
        - role: user
          content: "Analyze this complex problem..."

Enable deeper reasoning for complex tasks.

Learn more →

Scheduling & Automation ​

Basic Scheduling ​

yaml
schedule: "5 4 * * *"  # Run at 04:05 daily
steps:
  - run: echo "Running scheduled job"

Learn more →

Skip Redundant Runs ​

yaml
schedule: "0 */4 * * *"    # Every 4 hours
skip_if_successful: true     # Skip if already succeeded
steps:
  - id: extract
    run: echo "Extracting data"
  - id: transform
    run: echo "Transforming data"
    depends: extract

  - id: load
    run: echo "Loading data"
    depends: transform

Learn more →

Queue Management ​

yaml
queue: "batch"        # Assign to global queue for concurrency control
steps:
  - run: echo "Processing data"

Learn more →

Multiple Schedules ​

yaml
schedule:
  - "0 9 * * MON-FRI"   # Weekdays 9 AM
  - "0 14 * * SAT,SUN"  # Weekends 2 PM
steps:
  - run: echo "Run on multiple times"

Learn more →

Timezone ​

yaml
schedule: "CRON_TZ=America/New_York 0 9 * * *"
steps:
  - run: echo "9AM New York"

Learn more →

Start/Stop/Restart Windows ​

yaml
schedule:
  start: "0 8 * * *"     # Start 8 AM
  restart: "0 12 * * *"  # Restart noon
  stop: "0 18 * * *"     # Stop 6 PM
restart_wait_sec: 60
steps:
  - run: echo "Long-running service"

Learn more →

Global Queue Configuration ​

yaml
# Global queue config in ~/.config/dagu/config.yaml
queues:
  enabled: true
  config:
    - name: "critical"
      max_concurrency: 5
    - name: "batch"
      max_concurrency: 1

# DAG file
queue: "critical"  # Assign to queue for concurrency control
steps:
  - run: echo "Processing critical task"

Configure queues globally and assign DAGs using the queue field.

Learn more →

Email Notifications ​

yaml
mail_on:
  failure: true
  success: true
smtp:
  host: smtp.gmail.com
  port: "587"
  username: "${SMTP_USER}"
  password: "${SMTP_PASS}"
steps:
  - run: echo "Running critical job"
    mail_on_error: true

Learn more →

Operations & Production ​

History Retention ​

yaml
hist_retention_days: 30    # Keep 30 days of history
schedule: "0 0 * * *"     # Daily at midnight
steps:
  - id: archive_old_data
    run: echo "Archiving old data"
  - id: cleanup_archive
    run: rm -rf /tmp/archive/*
    depends: archive_old_data

Control how long execution history is retained.

Learn more →

Output Size Management ​

yaml
max_output_size: 10485760   # 10MB max output per step
steps:
  - run: ./analyze-logs --format markdown
    stdout:
      artifact: reports/analysis.md

Learn more →

Custom Log Directory ​

yaml
log_dir: /data/etl/logs/${DAG_NAME}
hist_retention_days: 90
steps:
  - id: extract
    run: echo "Extracting data"
    stdout: extract.log
    stderr: extract.err
  - id: transform
    run: echo "Transforming data"
    stdout: transform.log
    depends: extract

Organize logs in custom directories with retention.

Learn more →

Timeout & Cleanup ​

yaml
timeout_sec: 7200          # 2 hour timeout
max_clean_up_time_sec: 600    # 10 min cleanup window
steps:
  - run: sleep 5 && echo "Processing data"
    signal_on_stop: SIGTERM
handler_on:
  exit:
    run: echo "Cleaning up resources"

Learn more →

Production Monitoring ​

yaml
hist_retention_days: 365    # Keep 1 year for compliance
max_output_size: 5242880    # 5MB output limit
mail_on:
  failure: true
error_mail:
  from: alerts@company.com
  to: oncall@company.com
  prefix: "[CRITICAL]"
  attach_logs: true
info_mail:
  from: notifications@company.com
  to: team@company.com
  prefix: "[SUCCESS]"
handler_on:
  failure:
    run: |
      curl -X POST https://metrics.company.com/alerts \
        -H "Content-Type: application/json" \
        -d '{"service": "critical-service", "status": "failed"}'
steps:
  - run: echo "Checking health"
    retry_policy:
      limit: 3
      interval_sec: 30

Learn more →

Distributed Tracing ​

yaml
otel:
  enabled: true
  endpoint: "otel-collector:4317"
  resource:
    service.name: "dagu-${DAG_NAME}"
    deployment.environment: "${ENV}"
tools:
  - astral-sh/uv@0.11.14

steps:
  - id: fetch
    run: echo "Fetching data"
  - id: process
    run: uv run --python 3.13.9 python process.py
    depends: fetch

  - id: transform
    action: dag.run
    with:
      dag: pipelines/transform
    depends: process

Enable OpenTelemetry tracing for observability.

Learn more →

Execution Control ​

yaml
type: graph
max_active_steps: 5         # Max 5 parallel steps
queue: "compute-queue"    # Assign to queue for concurrency control
delay_sec: 10              # 10 second initial delay
skip_if_successful: true    # Skip if already succeeded
steps:
  - id: validate
    run: echo "Validating configuration"
  - id: process_batch_1
    run: echo "Processing batch 1"
    depends: validate

  - id: process_batch_2
    run: echo "Processing batch 2"
    depends: validate

  - id: process_batch_3
    run: echo "Processing batch 3"
    depends: validate

Learn more →

Queuing ​

yaml
queue: compute-queue      # Assign to specific queue
steps:
  - id: prepare
    run: echo "Preparing data"
  - id: compute
    run: echo "Running intensive computation"
    depends: prepare

  - id: store
    run: echo "Storing results"
    depends: compute

Learn more →

Limit History Retention ​

yaml
hist_retention_days: 60     # Keep 60 days history
steps:
  - run: echo "Running periodic maintenance"

Learn more →

Lock Down Run Inputs ​

yaml
run_config:
  disable_param_edit: true   # Prevent editing params at start
  disable_run_id_edit: true   # Prevent custom run IDs

params:
  - ENVIRONMENT: production
  - VERSION: 1.0.0

Learn more →

Complete DAG Configuration ​

yaml
description: Daily ETL pipeline for analytics
schedule: "0 2 * * *"
skip_if_successful: true
group: DataPipelines
labels: daily,critical
queue: etl-queue          # Assign to global queue for concurrency control
max_output_size: 5242880  # 5MB
hist_retention_days: 90   # Keep history for 90 days
env:
  - LOG_LEVEL: info
  - DATA_DIR: /data/analytics
  - DATE: "`date '+%Y-%m-%d'`"
params:
  - name: ENVIRONMENT
    type: string
    default: production
  - name: DRY_RUN
    type: boolean
    default: false
mail_on:
  failure: true
smtp:
  host: smtp.company.com
  port: "587"
handler_on:
  success:
    run: echo "ETL completed successfully"
  failure:
    run: echo "Cleaning up after failure"
  exit:
    run: echo "Final cleanup"
steps:
  - id: validate_environment
    run: echo "Validating environment: ${ENVIRONMENT}"

Learn more →

Released under the MIT License.

Copyright © 2024 Dagu Contributors