Writing
Notes from production: data engineering, generative AI, machine learning, and engineering in regulated finance. Subscribe via RSS.
2026
An Hour on LinkedIn Learning: Build Your First Agentic AI System
I already build agentic systems, so I did not take this course to learn what an agent is. I took it to see what LinkedIn Learning is like. The verdict: a well-structured one-hour refresher whose best idea is that an agent is never finished, only calibrated.
genai5 min read
AI in Financial Risk, Part 2: Machine Learning versus Statistical Models
The second part of the series puts one risk question, will the S&P 500 lose more than 1% tomorrow, through both ways of working. As a statistical model, a logistic regression finds one significant coefficient, and block-bootstrap standard errors double the uncertainty on another. As a prediction, the same model beats the base rate by 0.7% on held-out days, and a nearest-neighbour model that is perfect on its training data is 81% worse. A simulated market with a known truth splits the error into bias and variance, and a 10 to 1 cost on missed losses shows a rule with 89% accuracy to be the most expensive one on the table.
machine-learning20 min read
Data Engineering Design Patterns: Initial Load versus Daily Load
The third post in the data engineering design patterns series loads a month of history before a daily pipeline takes over. A one-off script that drifts from the pipeline in four ways, the pipeline run once per day and once with a month-wide interval, a wide run that deletes the history when one day is rerun, and a run handed a day that has not ended, each run on Postgres and on Spark with Delta Lake in the companion code.
data-engineering21 min read
Data Engineering Design Patterns: Incremental Loads and Watermarks
The second post in the data engineering design patterns series loads only the new rows from a source table, three hourly runs in a row. A watermark moved before the write commits, a write committed before the watermark, both in one transaction, a row that commits after a later one has been read, and the scheduler's data interval as the alternative, each run on Postgres and on Spark with Delta Lake in the companion code.
data-engineering15 min read
Data Engineering Design Patterns: Idempotent Loads
The first post in the data engineering design patterns series runs the same daily load twice and asks whether the table changed. A plain append, delete-then-insert keyed on the business date and on the batch date, a crash between the delete and the insert, and the Delta Lake form of the same pattern, each run on Postgres and on Spark with Delta Lake in the companion code.
data-engineering13 min read
AI in Financial Risk, Part 1: Financial Data and Its Distributions
The first part of a series on where machine learning and LLMs change risk work. Before any model, ten years of daily S&P 500 closes go through one inspectable step at a time: prices become returns, the returns are compared with a normal distribution of the same mean and volatility, and the gaps are counted. There are 36 days beyond three standard deviations where the normal expects about 7, a one-day 99% cut-off breached 51 times instead of 25, and volatility that clusters in a way the normal says it cannot. Removing 2020 makes the tails thinner, not normal.
machine-learning16 min read
Financial machine learning from zero, part 4: purged walk-forward validation
A date split is not enough when each mortgage label looks twelve months ahead. Part 4 builds a purged walk-forward schedule, separates label maturity from reporting delay and embargo, and gives calibration and policy selection their own place in time. A runnable calendar audit checks the boundaries, including the ones that fail without changing a model's score.
machine-learning19 min read
Financial machine learning from zero, part 3: a probability-of-default baseline you can trust
The third post in the financial machine learning series turns part 2's tensors into a complete probability-of-default baseline. Two simple benchmarks, logistic regression in PyTorch, the probability trap inside class-weighted loss, calibration curves, and a review threshold frozen before the test period is scored. Every result comes from the runnable companion code, on the same synthetic mortgage panel.
machine-learning20 min read
Financial machine learning from zero, part 2: tensors, time-ordered batching, and why leakage hides in .shift()
The second post in the financial machine learning series, and the first with code. It turns a Freddie Mac-style mortgage panel into PyTorch tensors one named step at a time, then breaks the pipeline on purpose with eight one-line mistakes and measures each one. Some push validation PR-AUC from 0.31 to 0.96. Others barely move it, which is worse, because nothing looks wrong. It ends with the checks that catch every one of them before a reviewer does.
machine-learning27 min read
AWS security for data engineers: seventeen recipes, worked through
Most data engineers meet AWS security as a wall of AccessDenied errors and a bucket someone made public. This post is the walkthrough I wish I had been given early: seventeen short recipes in the order I would apply them to a new account. Account-wide guardrails first (public access block, CloudTrail, default encryption), then identity (roles instead of keys, MFA, stale-key hunting), then permissions you generate and simulate instead of guess, permissions boundaries and SCPs so a team can self-serve safely, Session Manager and Secrets Manager for the machines, and finally the bucket policies, cross-account roles, and CloudFront setup that keep data buckets private. Every step is an AWS CLI command you can run in a sandbox account and clean up afterwards.
data-engineering37 min read
How to read the Spark UI
The Spark UI is the most underused tool in Spark. Most engineers learn to write DataFrame code long before they learn to read what Spark did with it. This is the walkthrough I give people joining my team: the five screens that matter — Jobs, the DAG, SQL, Stages, Storage — what each one is telling you, and the order I check them in when a job misbehaves. Every screenshot comes from a small retail join you can reproduce on a laptop.
data-engineering15 min read
Financial machine learning from zero, part 1: the vocabulary, the traps, and the metrics that matter
The first post in a beginner-friendly series on machine learning in finance, built around the problems banks actually run models on: credit scoring, fraud detection, and anti-money-laundering analytics, with market prediction as a supporting track. No code yet. Instead, the things I wish every newcomer knew before training a model on financial data: why it breaks the textbook assumptions, the vocabulary in every model notebook, the mistakes that make models look brilliant in development and fail in production, and a reference table of the metrics that matter, with which direction is good for each.
machine-learning22 min read
How granular should a pipeline task be?
A pipeline with one 300-line task is undebuggable; a pipeline with a task per pandas call is unreadable. The rule I use to cut between them: put a task boundary where you would stop and inspect the data, or where you would want a restart to begin — and nowhere else.
data-engineering11 min read
Adaptive AI agents: three layers of memory, from a Neo4j graph to Qwen's weights
An agent that solves a problem on Monday and rediscovers the same fix on Tuesday is burning money to stand still. This post works through what adaptation actually means — the agent loop, the four kinds of agent memory, and the three places an agent can improve — and a companion mini-project that builds all of it with Python, Neo4j, and a QLoRA fine-tune of Qwen.
genai10 min read
The Verhoeff algorithm: validating Aadhaar numbers at the ingestion edge
Every Aadhaar number carries its own tamper-evidence: the 12th digit is a Verhoeff check digit, computed from group theory rather than modular arithmetic. That makes it something a data engineer can exploit — a free, offline data-quality gate that catches typos, OCR misreads, and truncated numbers before they poison a customer dimension.
regulated-finance5 min read
TRY…CATCH, ROLLBACK, THROW: the case for explicit error handling in T-SQL
Plenty of warehouse procs rely on SET XACT_ABORT ON to clean up after failures — a one-liner most of the team has cargo-culted without knowing what it does. My position: make the error path explicit. TRY…CATCH, a visible ROLLBACK, and a THROW form a contract any reviewer can read — with XACT_ABORT kept underneath as the safety net it actually is.
data-engineering8 min read
Metadata-driven ETL: how far should configuration go?
Spec-driven pipelines are the right instinct — until the spec language grows conditionals and becomes a worse programming language. Where the line sits between metadata and code, why load patterns should be a closed set, and the smells that say your platform has crossed it.
data-engineering7 min read
Explaining the extra 2%: performance attribution as an Airflow pipeline
Fund managers live and die by whether they beat their benchmark — and by being able to say why. The math behind that answer is a clean three-factor decomposition, and it turns out to be a near-perfect fit for a pandas + Airflow pipeline with dynamic task mapping across dimensions.
regulated-finance7 min read
One DAG, every environment: the environment-aware DAG factory
How a small decorator around Airflow's @dag plus a tiered config object lets the same DAG definition run in dev, sit, uat, and prod — with no per-environment copies, no config drift, and promotion as a one-line gate.
data-engineering5 min read
The scheduler inside the scheduler: an Airflow anti-pattern and its paved-road fix
Why the controller-DAG-plus-generic-executor pattern hurts more than it helps, and how spec-driven DAG factories keep jobs as config without giving up per-job scheduling, backfills, and observability.
data-engineering7 min read
Migrating from enterprise schedulers to Apache Airflow: patterns that hold up
A pattern guide for moving batch workloads off Control-M or Autosys onto Airflow: inventory first, standardize the DAG template, and reconcile before every cutover.
data-engineering5 min read