Setting Up Your Local Dev Environment (The Right Way)

Before you write a single DAG, your local environment needs to be right. A bad setup will cost you hours of debugging environment issues instead of business logic. I have seen engineers spend more time fighting port conflicts and missing Python packages than actually building pipelines.

This is the setup I use. It works on macOS, Linux, and Windows (via WSL2).

The Prerequisite: Docker

Both setup paths below depend on Docker, so let’s get that out of the way.

Install Docker Desktop (macOS/Windows) or Docker Engine (Linux). Once installed, give it enough resources — Airflow is not lightweight. The minimum viable allocation for a smooth local experience is:

• CPU: 4 cores
• RAM: 6 GB (8 GB recommended)
• Disk: 20 GB free

On Docker Desktop, you configure this under Preferences → Resources → Advanced. On Podman (a rootless Docker alternative the Astro CLI also supports), run:

podman machine set --cpus 4 --memory 6144

Skimping on RAM is the single most common cause of tasks mysteriously failing locally. Don’t do it.

Path 1: Astro CLI (Recommended)

The Astro CLI is an open-source tool maintained by Astronomer. Under the hood it is a Docker Compose wrapper, but it abstracts away all the boilerplate that makes vanilla Compose painful to maintain. As of Astro CLI 1.34+, it has full support for Airflow 3.

Why choose this path:

• Zero Docker Compose YAML to write or maintain.
• astro dev start spins up a fully wired Airflow environment in under two minutes.
• Production parity: the same image you run locally (astro deploy) goes to production. No “works on my machine” surprises.
• Built-in DAG linting via astro dev parse and astro dev pytest.

Why you might not:

• You are deploying to a non-Astronomer platform (MWAA, Cloud Composer, self-hosted K8s) and want a leaner dev loop without the Astro Runtime image.
• Your team has an existing, heavily customised docker-compose.yml you need to stay compatible with.

Installation

# macOS / Linux
brew install astro

# Windows (PowerShell as Admin)
winget install -e --id Astronomer.AstroCLI

Starting a New Project

# 1. Create the project scaffold
mkdir risk-airflow && cd risk-airflow
astro dev init

# 2. Start the local environment
astro dev start

That second command builds your project image and starts four Docker containers:

Airflow 3 note: The container formerly called webserver is now called api-server. If you use a docker-compose.override.yml, update any service references from webserver to api-server. The CLI (1.35.1+) handles this automatically.

The UI is at http://localhost:8080. Default credentials: admin / admin.

The Project Structure

Astro Runtime 3.0+ uses uv instead of pip for dependency resolution by default — it’s significantly faster. If you rely on pip-specific behaviour (e.g., multi-index setups), add # ASTRO_RUNTIME_USE_PIP to your requirements.txt to opt out.

Essential CLI Commands

astro dev start         # Start all containers
astro dev stop          # Stop containers (preserves DB data)
astro dev restart       # Rebuild image + restart (use after requirements.txt changes)
astro dev kill          # Nuclear option: stop + delete all data including Postgres
astro dev parse         # Lint all DAGs for import errors without starting Airflow
astro dev pytest        # Run tests/dags/ unit tests inside the container
astro dev run dags list # Execute any airflow CLI command inside the scheduler container

astro dev parse is the one to wire into your pre-commit hook. It catches broken imports in seconds without spinning up a full environment.

Path 2: Vanilla Docker Compose

If you need full control, the official Airflow Docker Compose file is the way to go.

# Download the official Airflow 3 compose file
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.0.0/docker-compose.yaml'

# Create the required directories
mkdir -p ./dags ./logs ./plugins ./config

# Set your user ID (Linux only — prevents permission issues with mounted volumes)
echo "AIRFLOW_UID=$(id -u)" > .env

# Initialise the metadata database
docker compose up airflow-init

# Start everything
docker compose up -d

When vanilla makes sense:

• You need to add custom services (a local Kafka broker, a mock S3 via MinIO, a Redis instance for Celery) alongside Airflow. The docker-compose.yml is yours to modify.
• You are on a non-Astronomer deployment path and want zero proprietary tooling.

The trade-off is maintenance. You own that YAML file. Every time you change requirements.txt you need to rebuild the image manually (docker compose build). Every Airflow upgrade means a new compose file to diff and merge. With Astro CLI, that is handled for you.

Path 3: pip install (Don’t Do This)

Yes, you can pip install apache-airflow directly into a virtual environment. No, you should not, for the following reasons.

The Airflow dependency tree is enormous and notoriously fragile. The constraints file (https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt) exists specifically because pip will happily resolve a broken combination of packages. Beyond that, your local Airflow install will not use the same OS packages, C libraries, or system Python as your production environment. A task that works locally will fail in CI. You will waste time you do not have.

Use Docker. Both options above give you container parity.

VS Code Setup

Once your containers are running, here is the extension stack that makes DAG development genuinely productive.

Must-Have Extensions

1. Python (Microsoft) ms-python.python — The baseline. Gives you IntelliSense, linting, and the interpreter selector. Point it at your virtual environment (or the container interpreter — see below).

2. Airflow Extension (Necati ARSLAN) NecatiARSLAN.airflow-vscode-extension — Supports both Airflow 2.x and 3.x. Connects to your running local instance and gives you a sidebar to browse DAGs, trigger runs, view logs, and pause/unpause — without touching the browser. For Airflow 3, the API endpoint is http://localhost:8080/api/v2.

Configure it by clicking Connect to Airflow Server in the extension sidebar and entering:

It also integrates with VS Code’s AI Chat — @airflow in Copilot Chat can trigger DAGs, check failed runs, and analyse logs directly.

3. Remote Development Pack (Microsoft) ms-vscode-remote.vscode-remote-extensionpack — This is the critical one for debugging. It lets VS Code attach directly to your running scheduler container so you can use full breakpoints inside DAG code.

4. Docker (Microsoft) ms-azuretools.vscode-docker — Visualise running containers, view logs per container, and open terminals inside them without leaving VS Code.

5. Ruff charliermarsh.ruff — The fastest Python linter/formatter available. Essential for keeping DAG code clean. Airflow 3 itself uses ruff for its own migration tooling, so this is an obvious pairing.

Setting Up In-Container Debugging

This is the part most tutorials skip. Linting a DAG is not the same as debugging it. Here is how to set breakpoints inside actual task execution.

Step 1: Open the Command Palette (Ctrl+Shift+P) and run Dev Containers: Attach to Running Container. Select the scheduler container.

Step 2: A new VS Code window opens inside the container. Install the Python extension inside the container when prompted.

Step 3: Create .vscode/launch.json in your project root:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Airflow Task",
      "type": "python",
      "request": "launch",
      "module": "airflow",
      "console": "integratedTerminal",
      "args": [
        "tasks",
        "test",
        "${input:dagId}",
        "${input:taskId}",
        "${input:executionDate}"
      ],
      "env": {
        "AIRFLOW_HOME": "/usr/local/airflow"
      },
      "justMyCode": false
    }
  ],
  "inputs": [
    {
      "id": "dagId",
      "type": "promptString",
      "description": "DAG ID",
      "default": "risk_scoring"
    },
    {
      "id": "taskId",
      "type": "promptString",
      "description": "Task ID",
      "default": "score_applications"
    },
    {
      "id": "executionDate",
      "type": "promptString",
      "description": "Execution date (YYYY-MM-DD)",
      "default": "2026-02-18"
    }
  ]
}

Step 4: Place a breakpoint on any line in your DAG file. Hit F5, enter your DAG and task IDs when prompted, and VS Code will pause execution at your breakpoint. You can inspect context, kwargs, XCom values — everything.

Important: airflow tasks test runs the task in isolation and does not write to the metadata database. This is intentional — it gives you a clean, repeatable debug loop without polluting your local DB with test runs.

Recommended Workspace Settings

Create .vscode/settings.json in your project root:

{
  "python.defaultInterpreterPath": "/usr/local/bin/python",
  "editor.formatOnSave": true,
  "[python]": {
    "editor.defaultFormatter": "charliermarsh.ruff"
  },
  "ruff.lint.args": ["--select=E,W,F,I"],
  "files.exclude": {
    "**/__pycache__": true,
    "**/.pytest_cache": true,
    "airflow.db": true,
    "airflow.cfg": true
  }
}

Pre-Commit Hooks: The Last Line of Defence

Before any of this reaches your team’s repository, add a pre-commit hook to catch broken DAGs at commit time.

pip install pre-commit

Create .pre-commit-config.yaml:

repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.3.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  - repo: local
    hooks:
      - id: airflow-dag-parse
        name: Airflow DAG Parse Check
        entry: astro dev parse
        language: system
        pass_filenames: false
        files: ^dags/

pre-commit install

Now every git commit that touches the dags/ directory will run astro dev parse automatically. Broken imports never make it to the remote.

Summary: The Decision Tree

In practice, Astro CLI is the right default for 90% of setups. You can always eject to vanilla Compose if you outgrow it.

Next up in 30 Days of Airflow: Writing your first production-grade DAG in Airflow 3 — TaskFlow API, dynamic task mapping, and why we are never writing PythonOperator explicitly again.