There is a design that shows up in almost every Airflow estate built by a team that came from Control-M or Autosys — including, at one point, mine. It looks like this: one "scheduler DAG" that decides what should run and when, and one generic "executor DAG" that accepts a JSON job specification and does the work. Every job in the platform is a JSON document; the executor is the only DAG that ever really runs anything.
On paper it is elegant. Jobs are configuration, not code. Adding a job means adding a file, not writing a DAG. The team that builds it usually feels they have out-designed Airflow itself.
I now consider it an anti-pattern — but for a more specific reason than "don't fight the framework." The declared-spec half of the design is right. The mistake is where the dispatch happens.
The shape of the anti-pattern
Airflow sees two DAGs. Your platform contains fifty jobs. That gap between what the system is and what the orchestrator can see is where every problem below comes from.
Why it hurts
Job identity disappears. Every run of every job is a run of the executor
DAG, distinguished only by its conf. "Show me the failure history of job
C" becomes a manual filter over run configurations. Per-job SLAs, alerts,
and success-rate metrics are effectively impossible, because Airflow
aggregates all of them at the DAG level — and the DAG level now means
"everything."
Shared throttles, shared blast radius. max_active_runs on the executor
throttles all fifty jobs collectively. Pausing the executor pauses the
platform. One poison spec pollutes the stats, the queue, and the on-call
signal for every healthy job around it.
Backfill has no native expression. "Re-run job C for last week" is a first-class operation in Airflow — per DAG. Here, the logical dates belong to the executor, not to job C, so backfill becomes a hand-rolled loop of triggered runs with reconstructed confs. This is precisely the kind of plumbing where dates silently go wrong.
Two sources of truth for "why did this run." Airflow's scheduler believes the executor runs are externally triggered; the real scheduling logic lives inside the controller. Every incident review starts by reconciling the two stories. The UI's dependency graph — the thing that is supposed to make orchestration legible — shows you nothing true.
None of these are style objections. Each one is an operational capability you already paid for by adopting Airflow, handed back voluntarily.
Why teams build it anyway
It is worth being honest about the appeal, because the instinct behind the design is sound:
- Jobs should be configuration. Fifty hand-written DAG files drift apart; fifty JSON specs validated against a schema do not.
- The old mental model. Control-M is a scheduling table plus a runner. Teams port the architecture they trust before they trust the new one.
- It ships fast. One executor DAG and a loop is a week of work, and it demonstrably runs jobs by Friday.
The first instinct is correct and worth keeping. The fix is not "write a DAG per job by hand" — that trades runtime coupling for copy-paste drift. The fix is moving dispatch from runtime to parse time.
The paved road: spec-driven DAG factories
Keep the JSON specs exactly as they are. Replace the controller-and-executor
pair with a factory: a loop in the DAGs folder that reads each spec at parse
time and generates a real DAG per job — its own dag_id, its own schedule
or Timetable, its own retries, SLA, owner, and tags, all drawn from the
spec.
The obvious implementation is a single loop over the spec directory,
building every DAG in one file. Resist it. Airflow parses each .py file in
its own processor, and that isolation boundary is worth designing around:
with one loop file, a single malformed JSON raises at parse time and takes
every generated DAG down with it. Wrapping each iteration in try/except
prevents the mass outage but trades it for something subtler — the broken
job's DAG silently disappears. No import error in the UI, no alert, just a
schedule that quietly stops firing.
The robust shape is one thin stub file per spec, with all the real logic in a shared factory module:
# dags/job_a.py — one stub per spec; the factory does the work
from platform_dags.factory import dag_from_spec
dag = dag_from_spec("specs/job_a.json")Now a broken spec produces a visible import error scoped to that one
file — surfaced in the UI banner and airflow dags list-import-errors —
while every other job keeps running. Failure isolation and failure
visibility, both from Airflow's native machinery rather than your own
error handling. As the estate grows, per-file parsing also spreads parse
cost across processors instead of one serial loop marching toward the
import timeout.
Declaration stays central; execution becomes first-class per job. Native scheduling, per-job grid view, per-job backfill, per-job pause, honest dependency graphs — everything the runtime-dispatch design gave away comes back, without giving up "jobs are config, not code."
Three operational rules keep the factory healthy:
- Parse time must stay cheap. Stubs run on every scheduler parse cycle. The factory reads local files only — no database calls, no API calls, nothing network-shaped at import time.
- Keep "add a job = add a JSON" true. The stub costs a second file per job, which invites drift if written by hand. Generate the stubs in CI from the spec directory — a ten-line codegen step — and validate every spec against its schema in the same pipeline, so a bad file never reaches the DAGs folder at all.
- Watch the DAG count. Hundreds to a few thousand generated DAGs are routine. Tens of thousands of tiny, unscheduled, on-demand jobs are a different problem — that is queue-and-worker territory, and one of the rare cases where a shared executor DAG is actually the right call.
Where scheduling logic really belongs
Two Airflow primitives absorb most of what the controller DAG was doing:
- Cross-DAG dependencies belong to data-aware scheduling with Assets (Datasets in earlier 2.x): downstream DAGs run when upstream data lands, not when a controller says so. The dependency is declared by the consumer, visible in the UI, and owned where it is felt.
- Business calendars — trading days, settlement dates, "fifth business day after month-end" — belong in custom Timetables. Written once, reviewed once, reused by every generated DAG that names them in its spec. A controller DAG deciding "is today a run day?" is a worse version of a Timetable in every dimension that matters during an audit.
When a triggering DAG is legitimate
The judgment call, stated plainly: if a DAG's job is deciding when things run, that logic belongs in a Timetable or an Asset. If its job is coordinating one business process across DAGs, a controller can be legitimate.
Three cases pass that test:
- Fan-out over dynamic configs — triggering the same parameterized DAG once per fund, per file, per region, where the set is only known at run time.
- An explicit release train — one business process that genuinely is a sequence of otherwise-independent DAGs, where a single run ID for the whole chain is worth the coupling.
- A migration shim — a controller that mimics the legacy scheduler's behavior so cutover can be reconciled run-for-run against the old system. This one is fine as scaffolding with a demolition date, and toxic as an end state. If you are mid-migration off an enterprise scheduler, you will be tempted to keep it. Do not.
The takeaway
The teams that build the scheduler-inside-the-scheduler are not wrong about jobs-as-configuration — they are wrong about the binding time. Dispatch at runtime hides your platform from your orchestrator; generation at parse time lets one declared spec produce a fully first-class citizen of it. Keep the specs. Move the binding.