Sub-DAGs and Fan-out

Lesson 7 of 10 · 4 minutes

A step can run another workflow as a sub-DAG. The child is a full run with its own steps, logs, and status. Use sub-DAGs to reuse a workflow, or to run one workflow once per item in a list.

The workflow

curl -fsSLO https://docs.dagu.sh/tutorial/dags/tutorial-07-sub-dags.yaml

# Dagu tutorial, lesson 7: https://docs.dagu.sh/tutorial/sub-dags
description: Call a child workflow, then fan it out across regions
type: graph

steps:
  - id: build
    action: dag.run
    with:
      dag: build-app
      params:
        version: "1.4.0"

  - id: smoke_test
    action: dag.run
    with:
      dag: smoke-test
      params:
        region: ${ITEM}
        bundle: ${steps.build.outputs.bundle}
    parallel:
      items: [us-east, eu-west, ap-south]
      max_concurrent: 2
    depends: build

  - id: announce
    run: echo "Released ${steps.build.outputs.bundle} to 3 regions"
    depends: smoke_test

---
name: build-app
type: graph
params:
  - name: version
    type: string
    required: true
steps:
  - id: compile
    run: sleep 1 && echo "app-${params.version}.tar.gz"
    output:
      bundle:
        from: stdout

  - id: publish
    action: outputs.write
    with:
      values:
        bundle: ${steps.compile.outputs.bundle}
    depends: compile

---
name: smoke-test
type: graph
params:
  - name: region
    type: string
    required: true
  - name: bundle
    type: string
    required: true
steps:
  - id: check
    run: sleep 1 && echo "${params.bundle} is healthy in ${params.region}"

The file holds three workflows separated by ---. The first is the parent. build-app and smoke-test are children, named with name.

• action: dag.run runs the child named in dag. params passes values into it. A child sees only what it receives this way, not the parent's environment.
• outputs.write in build-app returns bundle to the parent, which reads it as ${steps.build.outputs.bundle}.
• parallel runs smoke-test once per item, at most 2 at a time. ${ITEM} is the current item.

Watch it run

3.6s / 3.6sSucceeded

announcerunSucceeded

Released app-1.4.0.tar.gz to 3 regions

A recorded run of this lesson's DAG. Select a step to see its log.

Select smoke_test to see the three child runs it started, one per region.

Run it yourself

dagu start tutorial-07-sub-dags.yaml

In the Web UI, the Activity list marks smoke_test with 3 sub-runs. Select it to see, on the Log tab, a JSON summary of each child run's status and outputs.

A child can also live in its own file. Then dag names that file, for example dag: smoke-test for smoke-test.yaml in the same folder.

Try it

1. Add ap-east to items. How many child runs does smoke_test start?
2. Change max_concurrent: 2 to 1. How long does smoke_test take now?

Answers

1. Four, one per item.
2. About 3 seconds instead of 2. The child runs no longer overlap: each starts after the previous one finishes.

Go deeper: Sub-DAGs →