If you have been using Airflow for as long as I have, you know the drill with major updates. Usually, it’s a mix of UI tweaks and a few deprecated operators.

Airflow 3.0 is different.

This isn’t just a fresh coat of paint (though the new React UI is nice); it is a fundamental architectural shift. The monolith is dead. Long live the Task Execution Interface.

In this first post of our 30 Days of Airflow series, we are cutting through the marketing fluff to look at the three changes that actually impact how we build systems—specifically for high-stakes environments like Finance and Risk.

1. The Big Split: Workload Isolation (AIP-72)

For years, Airflow had a “dirty secret”: your Workers needed direct access to the Metadata Database (Postgres/MySQL).

If you had a task running a sketchy Python script, it theoretically had a line of sight to your entire orchestration database. In a regulated environment (like a bank), this was a security nightmare. It also meant that if you had 1,000 tasks starting at once, you had 1,000 direct connections hammering your Postgres DB.

What Changed in 3.0:

Airflow 3 introduces the Task Execution Interface (AIP-72) and a new Task SDK.

  • The Scheduler still talks to the DB.
  • The Workers now talk to an API Server via the Task Execution API.
  • The Task SDK is the lightweight runtime workers use to execute tasks in external environments—containers, edge devices, or remote clusters.

The Worker no longer needs a database connection string. It asks the API: “What should I run?” and sends logs back via HTTP.

Why We Care:

  • Security: You can finally run untrusted code in isolated environments (like a separate K8s cluster) without giving it DB credentials.
  • Scale: The API Server handles the connection pooling. You can scale workers to infinity without DDoS-ing your database.
  • Multi-language: Because the Task SDK is a separate, lightweight runtime, non-Python TaskSDKs (starting with Golang) are now possible—opening the door to truly polyglot pipelines.

2. Datasets are now “Assets” (AIP-74 / AIP-75)

In Airflow 2.4, we got “Datasets” to help with event-driven scheduling. It was a good start, but the naming was confusing. Was an S3 bucket a dataset? Was a file?

What Changed in 3.0:

They are now called Assets, covered by two AIPs:

  • AIP-74 (“Introducing Data Assets”) — the core rename and redesign.
  • AIP-75 (“New Asset-Centric Syntax”) — the new @asset decorator and cleaner DAG definition patterns.

Note: AIP-60 (which some older references cite) was a narrower, earlier proposal about standardising URI representation for Datasets in Airflow 2.x. The full rename and redesign belongs to AIP-74/75.

Key changes:

  • New import: from airflow.sdk import Asset
  • Multi-Asset Triggers: You can now write complex logic like “Run this DAG if Asset A updates OR Asset B updates.”
  • External Triggers: The API allows external systems (like a Kafka consumer or a Lambda function) to mark an Asset as “updated” via a simple REST call.

Why We Care:

For our Credit Risk Engine, this is huge. We don’t have to poll for the “Loan Application” file every 5 minutes. The upstream system can just hit the Airflow API, update the loan_applications Asset, and our scoring DAG triggers instantly.

from airflow.sdk import DAG, Asset

# Old way (Airflow 2.x) - Deprecated
# from airflow import Dataset

landing_zone = Asset("s3://risk-engine/landing/loans.csv")

# This DAG runs strictly when the file lands
with DAG(dag_id="risk_scoring", schedule=[landing_zone]):
    ...

3. The New React UI (AIP-38 / AIP-84)

This one is less architectural, but too impactful to ignore.

The entire Airflow UI has been rebuilt from the ground up on React and FastAPI, replacing the old Flask AppBuilder stack (which has been moved to a separate provider package for backwards compatibility).

What Changed in 3.0:

  • DAG Versioning in the UI: Every DAG run is now associated with the exact version of the DAG that ran it — code, task structure, and logs are all version-aware. This was the single most-requested feature in Airflow’s annual community survey.
  • Scheduler-Managed Backfills: Backfills can now be launched and monitored directly from the UI or API, no more CLI-only workflows.
  • Unified Grid + Graph views with improved DAG-to-Asset relationship visibility.

Why We Care:

In a regulated environment, audit trails are not optional. Being able to look at a DAG run from six months ago and see exactly what code was executing — not the current version of the DAG — is a compliance requirement, not a nice-to-have.

Summary: What You Actually Need to Do

If you are migrating from Airflow 2.x, here is the practical checklist:

  1. Update your imports. Move from from airflow import DAG to from airflow.sdk import DAG. Same for Asset (formerly Dataset).
  2. Audit your worker configuration. Workers no longer need DB credentials. Update your K8s secrets and environment configs accordingly.
  3. Replace Dataset references with Asset. The old from airflow import Dataset is deprecated. A simple find-and-replace gets you 80% of the way there.
  4. Minimum version: Airflow 3.0 requires upgrading from Airflow 2.7 or later and Python 3.9–3.12.

The migration tooling (ruff and airflow config update) handles a lot of the boilerplate. Run it early, run it often.

Coming Up Next:

Next up in 30 Days of Airflow: Setting up your local dev environment — Docker, Astro CLI vs. Vanilla, and VS Code setup.