Financial machine learning from zero, part 4: purged walk-forward validation

This is part 4 of a series on financial machine learning for people who know some Python and have never trained a model on financial data. Part 2 built the mortgage panel and found the ways tomorrow could get into today's features. Part 3 trained a probability-of-default model and froze it before opening the test year.

The test result was uncomfortable. The model still ranked risk better than the simple benchmarks, but its probabilities were too high. A single training period and a single test period told us that something had changed. They did not tell us how consistently the model would behave as we moved through time.

This post changes the experiment, not the model. We will move the training and evaluation windows forward, check exactly which labels were available at each boundary, and give calibration a separate job. The PyTorch logistic regression from part 3 can stay as it is.

The question I want every split to answer is: could every fitted number have been computed on the day this model was supposed to run?

#A chronological split can still see the future

Suppose we are about to start monitoring loans in January 2011. We train on every decision month before 2011 and evaluate on the months after it. The dates are sorted. There is no shuffle. It looks sensible.

Now take one training row: a loan observed on 31 December 2010. Its label asks whether that loan reaches our default event in the next twelve months. To finish that label we need reports through December 2011. We have just used a year's worth of outcomes that the model would not have had when it started work.

Sorting the rows did not help. The row's date described the decision, not the period needed to discover its answer.

That is the reason for purging: remove candidate training examples whose outcome intervals cross the boundary into the period we are trying to evaluate. In a walk-forward experiment, the stronger practical rule is to require the entire training label to be available before the model is fitted. A non-overlapping label that has not arrived yet is still unavailable.

For a model frozen immediately before 1 January 2011:

The first and last rows have the same decision date and horizon. Their availability dates make different decisions. A splitter that stores only a row timestamp cannot make that distinction.

#The companion code

The companion script for this part is served with the post. Save it as walk_forward.py. It needs Python 3.10 or later and no additional packages:

python walk_forward.py
python walk_forward.py --check
python walk_forward.py --out fold_membership.csv

The first command prints three folds. The second runs the boundary checks. The third writes the exact row membership to a new CSV; it refuses to overwrite an existing file.

This is a calendar experiment, not another model-performance run. There is one illustrative row per month, no borrower features and no default outcomes. The counts below come from that script. They are counts of calendar rows, not mortgages, and there are no new AP or calibration results to interpret.

That distinction matters here. Part 3's experiment observed reports through December 2012. The longer schedule below illustrates what we would need for later evaluations; it does not manufacture those later reports. Parts 2 and 3 still provide the panel builder and model. This part supplies the timing contract to put around them.

#Every example needs three dates

Keep the population and target from part 3. We monitor active mortgages currently less than 90 days past due. A positive label means reaching 90 or more days past due, or ending in a coded default, during the following twelve months. Each observation is a loan-month.

Part 2 distinguished the decision time from the outcome time. We now make the operational delay explicit:

For this example, the outcome interval is open at the decision time and closed at its end: (decision_time, label_end]. January's month-end decision looks at February through the following January, not at January's current delinquency again.

I use a conservative, fixed maturity rule: wait for the full twelve-month horizon for every label, including positives that became apparent earlier. Taking early known positives while the corresponding negatives are still unresolved changes the population available for fitting. That needs its own treatment, not an accidental exception inside a splitter.

In the calendar example, label_available_at equals label_end. In a real panel it must also account for delivery, ingestion and any required finalisation of the source reports. A later corrected file does not prove the corrected value was known at the original date. Features need their own availability checks too; an eligible label cannot make a retrospectively updated feature safe.

Missing observations remain missing. An unfinished horizon or an unexplained stop in reporting is not a zero. The splitter cannot solve censoring by putting a better date on it. Keep the completeness and closure checks from the earlier posts.

#The purge, written as a small function

The companion stores the timing metadata in a plain Python record:

@dataclass(frozen=True)
class Observation:
    key: str
    decision_time: date
    label_end: date
    label_available_at: date | None

The script includes the imports. key identifies an observation; in the mortgage panel it would combine the loan ID and decision month. None means that the outcome is unresolved.

Here is the complete eligibility rule:

def eligible_before(rows, boundary):
    return tuple(
        row for row in rows
        if row.decision_time < boundary
        and row.label_end < boundary
        and row.label_available_at is not None
        and row.label_available_at < boundary
    )

The inequalities are deliberately strict. boundary is the instant before which the fit must be finished. A report arriving exactly at that boundary is not assumed to have been processed already. For the monthly example the boundary is 1 January and the last usable report is dated 31 December. A system with intraday arrivals should use timestamps and a documented ordering of arrival, fit and prediction.

The label_end check removes the overlapping horizons. The availability check removes labels delayed beyond that. If a source has variable horizons, these checks work row by row; they do not assume that removing twelve rows means removing twelve months.

The script validates duplicate keys and impossible intervals before splitting. It also reports how many candidate training rows were excluded. Do not quietly drop unresolved labels and then describe the remaining population as if nobody had gone missing. On a real panel, break the exclusions down by reason and investigate unexpected loss.

#Why gap=12 is not twelve months of mortgages

Scikit-learn's TimeSeriesSplit has a useful gap parameter, but its unit is samples. On one row per month, twelve samples can represent twelve months. On a panel with thousands of loans in each month, twelve samples are twelve loan-month rows, possibly all from the same day.

Split on calendar boundaries shared by the whole panel, then apply the row's horizon and availability checks. Every loan observed in an evaluation month belongs on the same side of that date boundary. Passing a sorted panel to a row-count splitter does not establish that.

Nor does a rolling feature automatically require another twelve-month gap. A trailing repayment-history feature at prediction time may use history that was already observed. The question is when its inputs were available, not whether its lookback has the same length as the label horizon.

#Walk forward through several evaluation periods

Walk-forward validation repeats the historical experiment: fit using eligible past information, freeze the fitted objects, predict a later block, then move the schedule forward.

An expanding window keeps all eligible history from a fixed start. A rolling window also sets a lower date bound, discarding older observations. Expanding gives the model more examples; rolling asks it to rely on a more recent history. Neither is automatically better when risk changes over time.

We use expanding training here. Choosing a rolling width after seeing which one wins on the final evaluation years would turn those years into development data. Window length, retraining frequency and the label-availability rule are part of the experiment to specify first.

The extra complication is calibration. It learns from labelled data, so it needs its own place in this chronology.

#Four jobs, four windows

Part 3 used one validation year to choose a model and a review threshold. It deliberately did not fit a calibrator on that year and then present the same year's calibration curve as independent evidence.

For this schedule, give each block one job:

1. Training: fit preprocessing, the PyTorch model and the simple benchmarks.
2. Calibration: use the frozen model's scores and mature labels to fit a probability mapping.
3. Policy selection: choose the review rule using that frozen model and mapping. Any candidate selection must be specified here or in an earlier development procedure.
4. Evaluation: apply the saved pipeline and policy, then wait for outcomes before measuring performance.

Here is one deliberately spacious fold. Assume reports are available at month end and every label needs twelve months:

The intervening years are doing work. They allow outcomes from the previous decision block to mature. They are not additional labelled training blocks in this fold.

That does not make all their raw observations forbidden. A December 2012 repayment report may legitimately contribute to the features of a January 2013 prediction once it has arrived. It cannot contribute the still-unobserved twelve-month outcome of a December 2012 decision to a fit made before January 2013.

The base model stays frozen from before the calibration block through the evaluation block. Refitting it after calibration would change the score distribution the calibrator learned to translate. A more frequent refit needs a matching calibration design, such as an explicitly specified sequence of historical predictions from successive models. It cannot be slipped into the last step as an improvement.

This simple schedule makes the cost visible: with a long outcome horizon, well-separated fitting stages can make a model quite old by the time it reaches evaluation. More years of data, a different retraining design or a genuinely different target may be needed. Shortening the purge because the remaining sample looks inconvenient does not create information that was available earlier.

#Move the whole schedule, not just the test year

The companion repeats that layout one year later each time:

There is a fresh base-model fit for each fold. Within that fold the base model stays fixed while its later stages are fitted and evaluated. The calendar audit prints:

eval_year  train  excluded  calibration  policy  evaluation  report_before
2013       31     12        12           12      12          2015-01-01
2014       43     12        12           12      12          2016-01-01
2015       55     12        12           12      12          2017-01-01

excluded counts candidate training months removed before the calibration boundary. report_before is the cutoff used to check that every evaluation label is available strictly before the report is produced. There are twelve rows per later block because this toy calendar has one row per month.

The 2013 decision rows appear in fold 1's evaluation and fold 3's policy window. That is permitted in a predeclared sequential procedure: by January 2015 their outcomes have matured. A later fit may learn from what has become history. Fold 1's saved predictions must remain unchanged, and information learned in 2015 must not be used to rewrite what the procedure supposedly did in 2013.

There is a separate research limit. We have already inspected 2011 in part 3, and this schedule is designed with that knowledge. Reusing 2011 can help development; it cannot restore its status as an unseen test. A new final claim needs an untouched period and a protocol frozen before its outcomes are inspected.

#Where the embargo belongs

Purging and embargo are often drawn as two coloured gaps, which makes them easy to confuse.

In the usual purged K-fold setup, training observations can sit on both sides of a held-out block. Purging removes overlapping information intervals. An embargo additionally excludes a buffer of candidate training observations after that block, where dependence or overlapping feature histories may still be a concern. The MLFinLab cross-validation documentation describes that setup and its connection to López de Prado's method.

Our walk-forward folds only train on the past. There are no training rows after the evaluation block in the same fold, so a post-evaluation embargo has nothing to remove there. The twelve-month waits in our table pay for label maturity. They are not evidence that we have implemented a second, post-test embargo.

If the source arrives a month late, record that in label_available_at. If there is an additional operational buffer, state what it represents and move the usable cutoff accordingly. Neither should be hidden behind an unexplained percentage of the dataset called embargo.

An embargo also does not turn neighbouring observations into independent samples. A recession can affect many borrowers for years. Removing a few weeks cannot certify that dependence has disappeared. Use gaps to enforce a stated information boundary, and treat remaining dependence honestly when reporting uncertainty.

#Calibration is another fitted model

The base model returns a logit, z. One simple calibration family learns two numbers on the calibration window:

calibrated_probability = sigmoid(a * z + b)

b adjusts the overall level of the log-odds. a changes their spread. The fitting objective can be ordinary, unweighted binary cross-entropy on the calibration labels. The base model's logits are fixed inputs; gradients must not update its weights while the calibrator is fitted. If a class-weight correction is part of the base score, freeze that choice too.

The scikit-learn calibration guide describes sigmoid calibration and why its fitting observations must be separate from the base model's fitting observations. Temporal data adds the requirement that the outcomes have matured before that mapping is used.

The equation does not guarantee improvement. A small calibration sample can give unstable coefficients; a changed population can make the mapping stale before deployment. Decide in advance what happens if a calibration window has too few events, only one class or a fit that fails to converge. Do not let the script silently skip that fold and average the easier ones.

Fit the mapping on calibration, choose any permitted configuration on policy data, and judge the result on later evaluation data. A curve on the window used to fit the mapping shows its fit, not an independent test of its reliability.

For the review rule, carry forward part 3's distinction. A fixed probability threshold may produce a different queue size next year. A monthly top-10% queue enforces a different kind of capacity rule, with its tie handling specified in advance. Choose which experiment you are running before looking at evaluation outcomes.

#Put the dates around the entire pipeline

The splitter returns row membership. It does not make later calls to fit safe by itself.

For each fold, start with new preprocessing and model objects. Fit imputation, scaling, class weights, benchmark probabilities and model coefficients on that fold's eligible training rows. Use those fitted objects to score calibration rows; do not fit a fresh scaler there.

After the calibration labels mature, fit the mapping. Use the frozen base model and mapping on the policy block. After any required policy outcomes mature, freeze the selection and threshold. Then generate evaluation predictions using features available at each decision time. The model never receives the evaluation labels as inputs.

For the strict schedule here, all policy labels must be mature even if a particular capacity-only rule uses scores without labels. That keeps the boundary conservative and supports policies that do use outcomes. Relaxing it is a different protocol to document, not a silent shortcut.

The companion selects evaluation rows by decision date without requiring their labels to exist. It checks evaluation maturity in a separate reporting step. This is intentional: otherwise a report can quietly select only the borrowers whose outcomes happened to arrive first.

Save the same evidence as part 3, now with a fold ID: row keys, timing boundaries, source versions, feature order, fitted preprocessing, model coefficients, calibration parameters, policy and predictions. When an outcome arrives, join it to the prediction already made. Rebuilding an old prediction from today's corrected feature table answers a different question.

#Read the folds before the average

With real outcomes, each evaluation block should have its own report:

Keep both part 3 benchmarks: the training event rate and the delinquency-only estimate. Refit their values from the permitted training data in each fold. A benchmark made from the evaluation year's event rate already knows the answer.

For additive losses such as Brier score and log loss, a row-weighted average of fold losses equals the loss on the combined predictions. An equal-weight average instead gives a small year the same influence as a large one. Both can be useful if they are labelled.

AP and ROC-AUC are different: pooling predictions from successive models introduces comparisons between scores from different periods. The pooled value need not match an average of the within-period values. Show the individual periods first, including their base rates. If a fold has no positive events, mark the affected ranking measures as not estimable rather than hiding the fold.

Several folds are not several independent experiments. Their training histories overlap. The same loan can recur, and adjacent twelve-month labels can refer to the same default. A borrower-level resampling scheme addresses a different dependence problem from a calendar-block scheme; common economic shocks can remain in either case. Do not put a narrow interval around millions of loan-months as if they were millions of independent borrowers.

The calendar audit establishes which information each fit may use. Whether recalibration improves the mortgage model requires the later data and the frozen protocol, with these reports kept for every fold.

#Checks that should fail before a model is trained

The companion's checks exercise the boundaries directly. They verify that a label ending exactly at the cutoff is excluded, that a late report is excluded even when its horizon has ended, and that an unresolved outcome cannot pass as a completed label.

They also repeat each calendar month for three pretend loans. The number of excluded rows triples; the excluded months stay the same. That is the failure a twelve-row gap would miss.

Another check removes all evaluation-label availability dates. The training, calibration and policy memberships stay unchanged, as do the keys of the evaluation rows. Only the later reporting check fails. A model must be able to make predictions before it knows how they turn out.

The script rejects empty windows, duplicate observation keys, impossible timing metadata and incomplete calibration or policy outcomes. For the monthly example it refuses the full 2013 evaluation report at the strict cutoff of 31 December 2014, then accepts it at 1 January 2015, after the final label has become available.

Those are split checks. Keep part 2's future-perturbation checks on features and part 3's checks on frozen fitting state as well. Correct row membership cannot detect a feature built from tomorrow's file or a scaler accidentally refitted during evaluation.

#What is next

The next post is fraud detection with late-arriving labels. The transaction occurs today; the customer may dispute it weeks later; an investigation can take longer still. A transaction with no fraud label yet is not necessarily a legitimate transaction.

The three dates from this post will matter again, but a fixed twelve-month wait will no longer describe the whole problem. We will have to distinguish when an event happened, when someone reported it and when the system was allowed to learn from it.

#Summary to save

• A chronological row split can still leak when training labels look beyond the date of the fit.
• Store the decision time, label horizon end and label availability separately. Check both the interval and the arrival boundary.
• Purge by dates across the whole panel. A gap of twelve observations is not a gap of twelve months when each month contains many loans.
• Calibration and policy selection are fitted stages too. Give each permitted information and enough time for its required labels to mature.
• Keep the base model fixed while calibrating and evaluating that model. Refitting it requires a matching calibration design.
• A post-test embargo has nothing to remove in a fold whose training data is entirely in the past. Label maturity and reporting delay still have to be enforced.
• Select evaluation membership at prediction time; assess outcome completeness when reporting. Missing labels are not negatives.
• Later folds may learn from outcomes that have become history. They may not rewrite earlier predictions or turn an inspected period back into an unseen test.
• Report each period, its base rate and its benchmarks before the average. Purging does not make repeated loan-months independent.