series

Data engineering design patterns

My own map of the patterns data pipelines keep needing, with Bartosz Konieczny's Data Engineering Design Patterns (O'Reilly, 2025) as the reference I read alongside. Each part opens on a concrete failure, such as a day loaded twice or late rows lost without trace, then shows the pattern that prevents it and the variant of it that still breaks. The companion code runs on Spark with Delta Lake, Kafka, Airflow and Postgres in Docker, written step by step so every intermediate can be inspected.

for
Data engineers who have shipped a few pipelines and want names, and failure modes, for the things they keep rebuilding.
published
nothing yet
planned
21 more

parts

  1. 01

    Idempotent loadsplanned

    Running the same day twice and getting the same table, comparing overwrite, delete-then-insert and keyed merges against a plain append that doubles the rows.

  2. 02

    Incremental loading and watermarksplanned

    High-water-mark tables against orchestrator data intervals, and the rows lost when the watermark moves before the write commits.

  3. 03

    Initial load versus daily loadplanned

    Loading years of history through the same pipeline that runs every day, and what drifts when the history load is a separate one-off script.

  4. 04

    Upserts with MERGEplanned

    MERGE on Delta Lake against separate update and insert steps, and the batch with duplicate source keys that makes it fail or pick a row at random.

  5. 05

    Partition overwriteplanned

    Replacing only the partitions a run produced, and the static overwrite setting that wipes every partition it did not touch.

  6. 06

    Change data captureplanned

    Log-based capture from Postgres through Kafka into Delta against snapshot comparison, and what happens to deletes and out-of-order events.

  7. 07

    Slowly changing dimensions, type 2planned

    Keeping history with effective dates and MERGE, and the batch that carries two changes for the same key.

  8. 08

    Late-arriving dataplanned

    Late events and late dimension rows, how to detect them, and a fixed reprocessing window against one that follows the data.

  9. 09

    Deduplicationplanned

    Windowed deduplication on a stream and keyed idempotency in batch, and the redelivery that arrives after the window has closed.

  10. 10

    Backfills and reprocessingplanned

    Clear-and-reload against versioned loads, and what downstream consumers see while a past day is being rebuilt.

  11. 11

    Write, audit, publishplanned

    Staging a result, checking it and only then exposing it, and the publish step that lets readers see a half-written table.

  12. 12

    Data contracts and quality gatesplanned

    Constraints and schema compatibility checks as pipeline steps, and the change that passes every check while altering what a column means.

  13. 13

    Dead-letter queues and quarantineplanned

    Setting bad records aside without stopping the pipeline, and the quarantine table nobody ever replays.

  14. 14

    Reconciliationplanned

    Comparing source and target counts and totals as a pipeline of its own, and the matching totals that hide errors cancelling each other out.

  15. 15

    Knowing when data is readyplanned

    Readiness markers, sensors and external triggers, and the job that reads a partition while it is still being written.

  16. 16

    Sequencing, fan-in and fan-outplanned

    Ordering tasks, splitting work and joining it back in Airflow, and the concurrent runs of one pipeline that overwrite each other.

  17. 17

    Storage layoutplanned

    Partitioning, bucketing, sorting and compaction on Delta Lake, and the streaming job that buries a table in small files.

  18. 18

    Enriching and aggregating dataplanned

    Static and streaming joins, sessions and aggregates, and the join to today's dimension row that rewrites the past.

  19. 19

    Observabilityplanned

    Detecting lag, skew, missed deadlines and stalled flows, and the pipeline that reports success while writing an empty table.

  20. 20

    Personal data and secretsplanned

    Encryption, anonymisation, pseudonymisation, column-level access and secret references, and the pseudonymised key that a join turns back into a person.

  21. 21

    Layers and where transformation livesplanned

    Medallion layers, ETL against ELT and metadata-driven pipelines, and what breaks when the promise each layer makes is never written down.

Follow new parts via RSS.