Testing OpenTelemetry Integration

This guide explains how to test Dagu's OpenTelemetry tracing functionality in various environments.

Quick Start with Jaeger

The fastest way to test OpenTelemetry tracing locally:

1. Start Jaeger

docker run --rm --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

2. Create a Test DAG

# test-otel.yaml
otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true
  resource:
    service.name: "dagu-${DAG_NAME}"
    service.version: "1.0.0"
    environment: "local"

steps:
  - id: fetch_data
    run: echo "Fetching data..." && sleep 1

  - id: process_data
    run: echo "Processing data..." && sleep 2
    depends: fetch_data

  - id: analyze_batch_1
    run: echo "Analyzing batch 1..." && sleep 1
    depends: process_data

  - id: analyze_batch_2
    run: echo "Analyzing batch 2..." && sleep 1
    depends: process_data

  - id: aggregate_results
    run: echo "Aggregating results..."
    depends: [analyze_batch_1, analyze_batch_2]

3. Run the DAG

dagu start test-otel.yaml

4. View Traces

Open http://localhost:16686 in your browser:

1. Select "dagu-otel-test" from the Service dropdown
2. Click "Find Traces"
3. Click on a trace to see the execution timeline

Testing Different Configurations

HTTP Endpoint

# test-http-endpoint.yaml
otel:
  enabled: true
  endpoint: "http://localhost:4318/v1/traces"
  insecure: true
steps:
  - run: echo "Testing HTTP endpoint"

With Authentication

# test-auth.yaml
env:
  - OTEL_TOKEN: "your-auth-token"
otel:
  enabled: true
  endpoint: "otel-collector:4317"
  headers:
    Authorization: "Bearer ${env.OTEL_TOKEN}"
steps:
  - run: echo "Testing with auth"

Custom Resource Attributes

# test-resources.yaml
otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true
  resource:
    service.name: "dagu-${DAG_NAME}"
    service.version: "2.0.0"
    deployment.environment: "testing"
    team: "platform"
    region: "us-east-1"
steps:
  - run: echo "Testing resource attributes"

Testing Nested DAGs

1. Create Parent DAG

# parent-workflow.yaml
otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true

steps:
  - run: echo "Starting parent workflow"

  - id: run_etl
    action: dag.run
    with:
      dag: child-etl.yaml
      params: "SOURCE=production DATE=2024-01-01"

  - id: run_analytics
    action: dag.run
    with:
      dag: child-analytics.yaml
      params: "INPUT=/tmp/data.csv"
    depends: run_etl

  - id: complete_parent
    run: echo "Parent workflow complete"
    depends: run_analytics

2. Create Sub DAGs

# child-etl.yaml
params:
  - SOURCE: dev
  - DATE: today

otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true

steps:
  - id: extract
    run: echo "Extracting from ${params.SOURCE}"
    output: EXTRACTED_DATA
  - id: transform
    run: echo "Transforming data" && echo "/tmp/data.csv"
    depends: extract

# child-analytics.yaml
params:
  - INPUT: ""

otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true

steps:
  - run: echo "Analyzing ${params.INPUT}"

3. Run and Verify

dagu start parent-workflow.yaml

In Jaeger, you should see:

• One trace containing all DAG executions
• Parent-child relationships preserved
• dag.parent_run_id attribute on sub-DAGs

Production-Like Testing

Using OpenTelemetry Collector

# compose.yaml
version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14250:14250"
  
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    ports:
      - "4317:4317"   # gRPC
      - "4318:4318"   # HTTP
    depends_on:
      - jaeger

  # Optional: Prometheus for metrics
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s

exporters:
  debug:
    verbosity: detailed
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Start the stack:

docker compose up -d

Debugging OpenTelemetry Issues

1. Enable Debug Logging

# Run with debug flag
dagu start --debug test-otel.yaml

# Or set environment variable
export DAGU_DEBUG=true
dagu start test-otel.yaml

2. Test Connectivity

# Test gRPC endpoint
grpcurl -plaintext localhost:4317 list

# Test HTTP endpoint
curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{}'

3. Common Issues and Solutions

No Traces Appearing

Check:

1. Is enabled: true set in your DAG?
2. Is the collector/Jaeger running?
3. Is the endpoint correct?
4. Any firewall blocking the ports?

Authentication Errors

# Debug with curl
curl -X POST https://otel-collector:4318/v1/traces \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{}'

TLS/Certificate Issues

# For testing, use insecure mode
otel:
  enabled: true
  endpoint: "otel-collector:4317"
  insecure: true  # Skip certificate validation

Performance Testing

1. Baseline Performance

# Create a DAG without OTel
cat > perf-test-no-otel.yaml << 'EOF'
steps:
  - id: step_1
    run: echo "Step 1"
  - id: step_2
    run: echo "Step 2"
    depends: step_1
EOF

# Time execution without OTel
time dagu start perf-test-no-otel.yaml

2. With OTel Enabled

# Create same DAG with OTel
cat > perf-test-with-otel.yaml << 'EOF'
otel:
  enabled: true
  endpoint: "localhost:4317"
  insecure: true
steps:
  - id: step_1
    run: echo "Step 1"
  - id: step_2
    run: echo "Step 2"
    depends: step_1
EOF

# Time execution with OTel
time dagu start perf-test-with-otel.yaml

3. Load Testing

# Run multiple DAGs concurrently
for i in {1..10}; do
  dagu start test-otel.yaml &
done
wait

# Check Jaeger for all traces

Integration Testing

Test with Your Observability Stack

If you have an existing observability platform:

# production-like-test.yaml
otel:
  enabled: true
  endpoint: "${env.OTEL_ENDPOINT}"
  headers:
    Authorization: "Bearer ${env.OTEL_TOKEN}"
  timeout: 30s
  resource:
    service.name: "dagu-${DAG_NAME}"
    service.version: "${APP_VERSION}"
    deployment.environment: "${ENVIRONMENT}"
steps:
  - run: echo "Testing production setup"

Run with environment variables:

export OTEL_ENDPOINT="your-collector.example.com:4317"
export OTEL_TOKEN="your-token"
export APP_VERSION="1.0.0"
export ENVIRONMENT="staging"

dagu start production-like-test.yaml