Data Engineering
Pipelines, orchestration, Spark, SQL, and the data platforms that carry production workloads.
2026
AWS security for data engineers: seventeen recipes, worked through
Most data engineers meet AWS security as a wall of AccessDenied errors and a bucket someone made public. This post is the walkthrough I wish I had been given early: seventeen short recipes in the order I would apply them to a new account. Account-wide guardrails first (public access block, CloudTrail, default encryption), then identity (roles instead of keys, MFA, stale-key hunting), then permissions you generate and simulate instead of guess, permissions boundaries and SCPs so a team can self-serve safely, Session Manager and Secrets Manager for the machines, and finally the bucket policies, cross-account roles, and CloudFront setup that keep data buckets private. Every step is an AWS CLI command you can run in a sandbox account and clean up afterwards.
37 min read
How to read the Spark UI
The Spark UI is the most underused tool in Spark. Most engineers learn to write DataFrame code long before they learn to read what Spark did with it. This is the walkthrough I give people joining my team: the five screens that matter — Jobs, the DAG, SQL, Stages, Storage — what each one is telling you, and the order I check them in when a job misbehaves. Every screenshot comes from a small retail join you can reproduce on a laptop.
15 min read
How granular should a pipeline task be?
A pipeline with one 300-line task is undebuggable; a pipeline with a task per pandas call is unreadable. The rule I use to cut between them: put a task boundary where you would stop and inspect the data, or where you would want a restart to begin — and nowhere else.
11 min read
TRY…CATCH, ROLLBACK, THROW: the case for explicit error handling in T-SQL
Plenty of warehouse procs rely on SET XACT_ABORT ON to clean up after failures — a one-liner most of the team has cargo-culted without knowing what it does. My position: make the error path explicit. TRY…CATCH, a visible ROLLBACK, and a THROW form a contract any reviewer can read — with XACT_ABORT kept underneath as the safety net it actually is.
8 min read
Metadata-driven ETL: how far should configuration go?
Spec-driven pipelines are the right instinct — until the spec language grows conditionals and becomes a worse programming language. Where the line sits between metadata and code, why load patterns should be a closed set, and the smells that say your platform has crossed it.
7 min read
One DAG, every environment: the environment-aware DAG factory
How a small decorator around Airflow's @dag plus a tiered config object lets the same DAG definition run in dev, sit, uat, and prod — with no per-environment copies, no config drift, and promotion as a one-line gate.
5 min read
The scheduler inside the scheduler: an Airflow anti-pattern and its paved-road fix
Why the controller-DAG-plus-generic-executor pattern hurts more than it helps, and how spec-driven DAG factories keep jobs as config without giving up per-job scheduling, backfills, and observability.
7 min read
Migrating from enterprise schedulers to Apache Airflow: patterns that hold up
A pattern guide for moving batch workloads off Control-M or Autosys onto Airflow: inventory first, standardize the DAG template, and reconcile before every cutover.
5 min read