One DAG, every environment: the environment-aware DAG factory

Every Airflow estate eventually hits the same question: how does one DAG definition behave correctly in four environments? The naive answers are all bad. Copy the DAG file per environment and the copies drift. Sprinkle if env == "prod" through task code and you have untestable branching in the worst possible place. Push everything into Airflow Variables and your DAG's behaviour is now defined by mutable UI state that no code review ever sees.

The pattern I have come to like — and have run in production — is a small decorator that wraps Airflow's own @dag, paired with a config object that resolves environment settings through a tier hierarchy. The DAG author writes one definition and declares which environments it should exist in. Everything else is derived.

The authoring experience

The whole point of the pattern is what the DAG author doesn't write:

@multi_env_dag(lower_envs=["dev", "uat"], schedule="@daily")
def fund_prices(env=None):
    extract = extract_prices(bucket=env.LANDING_BUCKET)
    publish = publish_prices(role=env.PUBLISH_ROLE)
    extract >> publish

From this single definition the decorator generates fund_prices_dev and fund_prices_uat on the non-production cluster, and a plain fund_prices on production. Each generated DAG receives an env object holding that environment's settings, injected as a keyword argument at build time. The DAG body never asks "where am I running?" — it just reads attributes off the object it was given.

Promotion to production is deliberately explicit:

@multi_env_dag(lower_envs=["dev", "sit", "uat"], prod_enabled=True)
def fund_prices(env=None):
    ...

Until someone flips prod_enabled, the DAG simply does not exist in the production DagBag. That one boolean, visible in code review and git history, replaces an entire class of "how did this get to prod" incidents.

Tiered configuration, not per-env copies

The config side is where the drift problem actually gets solved. Settings live in one module per environment, and lookups fall through a tier hierarchy when an environment doesn't define a value:

A dev-specific value wins if it exists; otherwise the lookup falls back to non (settings shared by all non-production environments), and finally to default (settings shared by everything). Production deliberately skips the non tier — it inherits only from default, so a value intended for test environments can never leak into prod by omission.

The config object itself is mostly a resolution chain behind __getattr__:

class Environment:
    def __getattr__(self, name):
        for source in (self._overrides, self._env_module, self._parent_tier):
            value = lookup(source, name)
            if value is not None:
                return value
        return None

In practice this means the uat config file contains only the handful of values where uat genuinely differs — a bucket name, an IAM role — and everything else is written exactly once. When I have audited estates that copy full config per environment, the differences between files were never the intentional ones; they were the settings someone updated in three places out of four.

Two details make the object safe to lean on. First, required settings are validated when the object is constructed, at DAG parse time:

REQUIRED = ["TEAM", "IAM_ROLE", "DEFAULT_IMAGE"]
 
def _validate(self):
    missing = [v for v in REQUIRED if not getattr(self, v)]
    if missing:
        raise MissingConfigError(missing)

A missing value breaks the DAG import in dev, loudly, instead of surfacing as a None deep inside a task at 2am in production. Second, the environment's identity is part of the DAG's identity — the environment suffix in the dag_id means dev, sit, and uat runs of the same pipeline are separate DAGs with separate histories, separate throttles, and separate alerting, even though they share a cluster.

Standardisation as a side effect

Because every DAG passes through the factory, the factory becomes the natural place to enforce platform policy. Environment-level defaults — default_args, standard params, RBAC access control — are merged into every generated DAG from config:

dag_kwargs = {
    "dag_id": suffixed_id,
    "access_control": env.ACCESS_CONTROL,
    "default_args": {**env.DEFAULT_ARGS, **author_args},
}

The author's values win where they overlap, but the baseline — retries, alert channels, who can trigger what in which environment — is applied uniformly without anyone having to remember it. When the platform migrated between cluster generations, the same seam absorbed the difference: the factory asked the cluster which generation it was (an Airflow Variable, with an environment-variable fallback for local runs) and gated DAG registration accordingly. DAG authors changed nothing.

That is the deeper benefit of the pattern: it gives the platform team a single choke point between "what authors write" and "what Airflow sees", which is exactly where paved-road standards belong.

The honest trade-offs

None of this is free, and two of the costs deserve eyes-open acceptance.

__getattr__ that returns None is a typo trap. Misspell env.LANDING_BUCKET and you get None, not an error. The required-vars validation covers the settings that matter most, but the long tail is unprotected. If I were building this again I would raise on unknown attributes and make optional settings explicit.

Config resolution happens at parse time. Importing config modules and walking tiers inside DAG parsing adds scheduler overhead, and clever tricks like inspecting the call stack to locate the caller's project root are fragile under refactoring. Keep config modules import-cheap — constants only, no I/O — and the cost stays negligible; let someone put an API call in a config file and the scheduler will make you regret it.

DAG count multiplies in non-prod. Three lower environments means three DAGs per definition on the shared cluster. That is mostly a feature — it is the isolation working — but it changes how you read the DAG list, and it argues for defaulting lower_envs to a single shared tier rather than generating every environment for every DAG by reflex.

The test

The pattern earns its keep the same way any platform abstraction does: by what stops happening. Since running DAGs this way I have stopped seeing per-environment file copies drift apart, stopped seeing prod incidents caused by config present in three environments out of four, and stopped reviewing PRs where the diff is "the same DAG again, with two strings changed." One definition, one config delta per environment, one explicit gate to production. The environments differ exactly where someone decided they should — and nowhere else.