Data Engineering

The Spark UI is the most underused tool in Spark. Most engineers learn to write DataFrame code long before they learn to read what Spark did with it. This is the walkthrough I give people joining my team: the five screens that matter — Jobs, the DAG, SQL, Stages, Storage — what each one is telling you, and the order I check them in when a job misbehaves. Every screenshot comes from a small retail join you can reproduce on a laptop.

A pipeline with one 300-line task is undebuggable; a pipeline with a task per pandas call is unreadable. The rule I use to cut between them: put a task boundary where you would stop and inspect the data, or where you would want a restart to begin — and nowhere else.

Plenty of warehouse procs rely on SET XACT_ABORT ON to clean up after failures — a one-liner most of the team has cargo-culted without knowing what it does. My position: make the error path explicit. TRY…CATCH, a visible ROLLBACK, and a THROW form a contract any reviewer can read — with XACT_ABORT kept underneath as the safety net it actually is.

Spec-driven pipelines are the right instinct — until the spec language grows conditionals and becomes a worse programming language. Where the line sits between metadata and code, why load patterns should be a closed set, and the smells that say your platform has crossed it.

How a small decorator around Airflow's @dag plus a tiered config object lets the same DAG definition run in dev, sit, uat, and prod — with no per-environment copies, no config drift, and promotion as a one-line gate.

Why the controller-DAG-plus-generic-executor pattern hurts more than it helps, and how spec-driven DAG factories keep jobs as config without giving up per-job scheduling, backfills, and observability.

A pattern guide for moving batch workloads off Control-M or Autosys onto Airflow: inventory first, standardize the DAG template, and reconcile before every cutover.