Financial machine learning from zero, part 2: tensors, time-ordered batching, and why leakage hides in .shift()

This is part 2 of a series on financial machine learning for people who know some Python and have never trained a model on financial data. Part 1 had no code. It set out the vocabulary, the traps, and the metrics. This post is where the code starts, and it starts with the trap part 1 called common: leakage, the outcome or the future finding its way into today's features.

The plan is simple. Build one small pipeline that turns a table of mortgages into PyTorch training examples, correctly, in named steps. Then break it, one line at a time, and measure what each break does to the validation score. By the end you will have a dataset you can trust, a set of leakage checks you can reuse, and a feel for what the most common mistakes look like from the inside.

#A model that was too good

Here is a probability-of-default model built in this post. It predicts whether a mortgage that is paying on time today will be 90 days or more behind at some point in the next twelve months. On the validation year its PR-AUC (the area under the precision-recall curve, the honest metric for rare events from part 1) is 0.31, against a base rate of 5.3%. That is a modest, believable model.

Now add one feature: whether the loan's label was positive last month. It is built with groupby("loan_id").shift(1), the same positive, backward-looking shift used for every other history feature. It looks harmless.

PR-AUC goes to 0.96.

Nothing in the training loop complains. No error, no warning. The feature is a lag of a column that already looks twelve months forward, so it carries eleven of those months into the present. A model built on it would be approved on a validation report and would fail on its first day in production, because in production last month's label does not exist yet.

Every mistake in this post is like that: one line, no error, and a number that looks better than it should, or, worse, a number that looks exactly as it should while the data underneath is wrong.

#The companion code

Everything below comes from a small companion project: financial-ml-from-zero, part 2. It is plain pandas and PyTorch, runs on a laptop CPU in a few minutes, and has three scripts you run in order:

python make_synthetic_freddie.py --out data/synthetic --loans 12000
python run_clean.py --data data/synthetic
python break_it.py --data data/synthetic

The first writes a synthetic dataset. The second runs the clean pipeline and prints every intermediate table. The third breaks the pipeline and prints the table of results you will see further down.

#The one question every row must answer

Every training example in a finance model has to answer one question: what did we know, and when did we know it?

That needs two points in time for every row. The decision time is when the model would have produced its score: here, the end of a reporting month t, when that month's payment status is known. The label window is the period in which the outcome is observed: here, months t+1 to t+12.

The rule that follows is short. Every feature must be computable from data known at or before the decision time. Only the label may look forward. Every step of the pipeline below is an application of that rule, and every mistake further down is a violation of it.

#The data: Freddie Mac, and why the numbers here are synthetic

The dataset for this post and the next is Freddie Mac's Single-Family Loan-Level Dataset. Freddie Mac buys mortgages from US lenders and publishes loan-level data on them, including a free sample of 50,000 loans per origination year. It comes as two files per vintage, the year the loans were made:

• An origination file: one row per loan, with what was known when the loan was made. Credit score, loan-to-value ratio (LTV, the loan as a share of the property's value), debt-to-income ratio (DTI, monthly debt payments as a share of income), interest rate, and original balance.
• A monthly performance file: one row per loan per month, with the balance, the loan's age, the delinquency status (how many months behind on payments), and a zero balance code when the loan ends, saying how it ended: prepaid, sold, foreclosed.

The performance file is a panel: the same loans observed repeatedly over time. Panels are where .shift() does its damage, because "the previous row" and "the previous month for this loan" are the same thing only if you make them so.

Freddie Mac asks you to register (free) before downloading, and its terms do not allow redistribution. So the companion project includes a generator that writes a synthetic panel in exactly the same file layout: same pipe-delimited format, no header row, same column positions, same codes. Twelve thousand loans originated between 2005 and 2010, simulated month by month through 2012 with a delinquency process that depends on credit score, LTV, DTI, and a shock centred on early 2009. The loader cannot tell the difference, and the README explains how to point the same code at the real files.

Every number in this post comes from the synthetic panel. That is a feature for this particular post: the data-generating process is known, so when a score jumps, the jump is the leak and not some quirk of the real world. It also means the sizes of the jumps say nothing about real mortgages. The directions will hold on real data. The magnitudes will not.

For two of the mistakes I also use real column names from the Lending Club loan dataset, because it contains live examples of them.

#The clean pipeline, one step at a time

The pipeline has eight steps. Each one is a small function that returns a new table you can print, inspect, and assert on before moving to the next. Nothing is chained. When a leak exists, it lives in exactly one of these functions, and you can find it by looking at one intermediate table at a time.

#Step 1: load, and nothing else

The files have no header row, so the column names come from Freddie Mac's user guide and live in one place:

PERFORMANCE_COLUMNS: dict[int, str] = {
    0: "loan_id",
    1: "period",
    2: "current_upb",
    3: "dq_status_raw",
    4: "loan_age",
    8: "zero_balance_code",
    9: "zero_balance_month",
}

current_upb is the unpaid principal balance, what the borrower still owes. The loader reads only these positions and converts types. One conversion needs a decision: delinquency status is a count of months behind ("0", "1", "2", ...) except for RA, which means the lender has repossessed the property. I map RA to 6 and cap every status at 6, because past six months behind the loan is deep in default and the exact count adds nothing:

status_text = raw["dq_status_raw"].str.strip()
status_text = status_text.replace(DQ_STATUS_REO, str(DQ_STATUS_CAP))
performance["dq_status"] = pd.to_numeric(status_text, errors="coerce").clip(upper=DQ_STATUS_CAP)

There is also a guard against reading the wrong column: every loan ID starts with F, so the loader asserts it. If Freddie Mac adds a column in front and the positions shift, the pipeline stops here, not three steps later with plausible-looking numbers.

#Step 2: sort the panel, then prove it is sorted

def sort_panel(performance: pd.DataFrame) -> pd.DataFrame:
    return performance.sort_values(["loan_id", "period"], ignore_index=True)

Every history feature below relies on this order, so it gets an assertion, not just a comment:

def assert_sorted_panel(panel: pd.DataFrame) -> None:
    assert panel["loan_id"].is_monotonic_increasing, "panel is not sorted by loan_id"
    same_loan = panel["loan_id"].eq(panel["loan_id"].shift())
    months = month_number(panel["period"])
    later_month = months > months.shift()
    assert (~same_loan | later_month).all(), "periods are not strictly increasing within a loan"

#Step 3: complete the monthly calendar

Sorted is not enough. If a loan is missing its March report, then shift(1) on its April row returns February, and a "last 12 rows" window covers thirteen months. So the next step builds a full calendar, one row per loan per month from its first report to its last, and merges the reports onto it. Missing months become rows with empty values and is_reported = False:

completed = calendar.merge(panel, on=["loan_id", "period"], how="left", indicator=True)
completed["is_reported"] = completed.pop("_merge") == "both"

After this step, and only after it, shift(1) means "one month earlier". The synthetic panel has no gaps, and the real sample files rarely do, so the step usually changes nothing. It costs one merge, and an assertion (assert_no_calendar_gaps) keeps it honest. Mistake 4 below shows what happens without it.

#Step 4: build the label, the only place allowed to look forward

The label is "90 or more days past due, or ended in default, at any point in the next twelve months". It is the only thing in the project built with negative shifts:

by_loan = panel.groupby("loan_id", sort=False)
future_status = pd.concat(
    [by_loan["dq_status"].shift(-k) for k in range(1, LABEL_HORIZON_MONTHS + 1)], axis=1
)
worst_future_status = future_status.max(axis=1)

Twelve explicit shifts instead of a clever reversed rolling window. It is slower, and you can read it.

The label also needs to know when it is mature. For a decision in November 2012 the window runs to November 2013, beyond the end of the data, so the label is not known yet. Rows like that are not negatives. They are unknowns, and they are dropped:

window_end = month_number(panel["period"]) + LABEL_HORIZON_MONTHS
labelled["label_is_mature"] = (window_end <= data_end.ordinal) | (
    loan_closed & (loan_last_month <= window_end)
)

The second condition keeps loans that closed inside the window. A loan that prepaid in month t+4 without falling behind has a known label of 0.

One more decision belongs here: which rows the model scores. A probability-of-default model scores loans that have not defaulted yet, so the scoring population is reported, active loans less than 90 days behind with a mature label. Scoring loans that are already in default would add easy positives and inflate every metric, a mild leak of its own.

#Step 5: features, known at the end of month t

features["dq_status_now"] = panel["dq_status"]
features["dq_status_prev"] = by_loan["dq_status"].shift(1)
 
is_late = (panel["dq_status"] > 0).astype(float).where(panel["dq_status"].notna())
late_by_loan = is_late.groupby(panel["loan_id"], sort=False)
features["months_late_last_12"] = (
    late_by_loan.rolling(12, min_periods=1).sum().reset_index(level=0, drop=True)
)

Ten features in all. Four come from the origination file (credit score, LTV, DTI, rate) and are known at every later month. Three describe month t itself (loan age, the share of the balance still owed, the current delinquency status), which the decision time allows because month t's report is in by the end of month t. Three describe history (last month's status, months late in the last twelve, the worst status in the last twelve). Every history feature uses a positive shift or a trailing window, and every one of them goes through groupby.

A note on rolling windows, because "the window includes the current row" is often listed as a leak. With decision time defined as the end of month t, including month t is correct. If your decision happens at the start of month t, before its report arrives, it is not, and you would need closed="left" or an extra shift(1). The decision time you write down determines which version is correct.

#Step 6: split by time, with gaps

SPLIT_WINDOWS = {
    "train": ("2005-06", "2007-12"),
    "valid": ("2009-01", "2009-12"),
    "test": ("2011-01", "2011-12"),
}

The gap between splits must be longer than the label window. A training row from December 2007 has a label that looks through December 2008. If validation started in, say, June 2008, the model would have been trained on outcomes that happen during the validation period. The check is an assertion, not a convention:

gap = later["period"].min().ordinal - earlier["period"].max().ordinal
assert gap > horizon, f"{earlier_name} ends {gap} months before {later_name} starts"

On the synthetic panel:

The default rate nearly doubles from train to validation, because validation sits in the simulated crisis. That is non-stationarity from part 1, arriving on schedule. The test split is not touched in this post. It is scored once, at the end of part 3.

#Step 7: fit preprocessing on train only

def fit_preprocessing(train: pd.DataFrame, columns: list[str]) -> Preprocessing:
    medians = train[columns].median()
    filled = train[columns].fillna(medians)
    return Preprocessing(columns, medians, filled.mean(), filled.std().replace(0, 1.0))

Missing values are filled with training medians, then every feature is standardised with the training mean and standard deviation. The same fitted object is applied, unchanged, to validation and test. The function takes one argument, and that argument is always the training split. That is the whole defence against mistake 8 below.

#Step 8: tensors, with a way back to the source

def to_tensors(frame, prep):
    x = torch.from_numpy(apply_preprocessing(frame, prep))
    y = torch.from_numpy(frame[LABEL].to_numpy(dtype=np.float32, copy=True))
    keys = frame[["loan_id", "period"]].reset_index(drop=True)
    return x, y, keys

The third return value matters as much as the first two. A tensor row is just ten numbers. The keys table says which loan and which month they came from, so any prediction, and any suspicious one in particular, can be traced back to its source rows:

x_train (100018, 10) torch.float32, y_train (100018,)
train_set[0] traces back to ('F05Q10000013', Period('2005-06', 'M'))

The dataset class is as small as PyTorch allows:

class LoanMonthDataset(Dataset):
    def __init__(self, x, y, keys):
        assert len(x) == len(y) == len(keys)
        self.x, self.y, self.keys = x, y, keys
 
    def __len__(self):
        return len(self.y)
 
    def __getitem__(self, i):
        return self.x[i], self.y[i]

#Time-ordered batching: what it does and does not mean

"Split by time, never at random" from part 1 leads many people to a second rule that is not actually a rule: never shuffle. So I will be direct about it. Shuffling mini-batches inside the training window is fine, and usually helps optimisation.

The reason is the rule from earlier. Each row already carries only what was known at its own decision time, and its label is fixed. The order in which the optimiser sees those rows changes how it gets to the answer, not what information it has. Leakage comes from how features and labels are built and from where the splits fall. It does not come from batch order. The training loop in the companion code uses DataLoader(train, batch_size=2048, shuffle=True) without apology.

Two things are true alongside that. First, rows from the same loan are correlated, so 100,000 loan-months are worth much less than 100,000 independent examples. That does not leak anything, but it makes validation scores noisier than the row count suggests. Second, order does matter when a single example is a sequence, which is the other way to feed a panel to a model:

class LoanHistoryDataset(Dataset):
    """One example per decision month: the loan's last `window` months as a sequence."""
 
    def __getitem__(self, i):
        end = self.decision_rows[i] + 1  # inclusive of the decision month
        start = max(self.loan_start[self.decision_rows[i]], end - self.window)
        history = self.panel_x[start:end]
        sequence = torch.zeros(self.window, n_features)
        mask = torch.zeros(self.window, dtype=torch.bool)
        sequence[self.window - len(history):] = history
        mask[self.window - len(history):] = True
        return sequence, mask, self.labels[i]

Each example is the slice of the sorted, calendar-complete panel that ends at the decision row. It cannot reach past it, and it cannot reach into another loan because it starts no earlier than the loan's first month. Loans with shorter histories are padded on the left, and the mask says which months are real:

history[0]: sequence (6, 10), real months 6, label 0.0
one batch: (256, 6, 10) = [batch, months, features]

The batches of sequences can be shuffled too, for the same reason. The time order lives inside each example, where the model needs it.

What neither dataset handles is retraining over time: a model refitted every quarter, validated on the quarter after. That is walk-forward validation and it is part 4.

#Eight ways to break it

Now the breaking. Each experiment changes one thing in the feature or label construction, then fits the same logistic regression on the same training window and scores it on the same validation window.

One thing about the measuring instrument first, because I got it wrong on the first run. The models in this section are fitted to convergence with full-batch L-BFGS, not with a few epochs of mini-batch Adam. The first version of the experiments used four epochs of Adam, and scaling features with statistics from all splits instead of just train (mistake 8) appeared to cut PR-AUC from 0.32 to 0.20. It was not leakage. The off-centre features made Adam converge more slowly, and four epochs stopped short. Fitted to convergence, both versions score 0.309. Before you read anything into a score difference, make sure both models finished training. Differences of about 0.01 in the table below are noise.

Validation base rate is 5.32%, so a model with no skill scores a PR-AUC of about 0.05. The "caught by" column refers to the checks in the next section.

Read the table in two halves. Mistakes 1, 3, 5, 6 and 7 make the model look better. They are dangerous, but they are also the ones a sceptical reviewer might notice, because the numbers are too good. Mistakes 2, 4 and 8 leave the score where it was. Those are the ones that reach production.

#1. Shifting the wrong way

# wrong
features["dq_status_prev"] = by_loan["dq_status"].shift(-1)
# right
features["dq_status_prev"] = by_loan["dq_status"].shift(1)

shift(1) moves values down a row, so each row sees the one before it. shift(-1) moves them up, so each row sees the one after it: next month's status, which is the first month of the label window. PR-AUC goes from 0.309 to 0.447.

The sign is easy to get wrong because the label step is full of shift(-k) and copying a line from there is natural. That is one reason the label and the features live in separate functions here. In build_features, a negative number after shift( is a bug by definition, which makes it easy to search for.

#2. Forgetting groupby

# wrong
features["dq_status_prev"] = panel["dq_status"].shift(1)
features["months_late_last_12"] = is_late.rolling(12, min_periods=1).sum()
# right
features["dq_status_prev"] = by_loan["dq_status"].shift(1)
features["months_late_last_12"] = (
    late_by_loan.rolling(12, min_periods=1).sum().reset_index(level=0, drop=True)
)

On a sorted panel, the first month of each loan now gets the previous loan's last month as its "previous status", and the first eleven months of each loan get a twelve-month window that reaches into another borrower's history. PR-AUC: 0.311. No change.

That is what makes this one dangerous. A third of the rows in the two rolling features carry another borrower's history, and the score does not notice, partly because the contamination is concentrated in young loans and partly because the neighbouring loan's history is mostly noise to the model. It will not stay harmless. Sort the panel differently, or add a feature where the neighbour's value correlates with the outcome, such as a borrower's second loan sorted next to their first, and it becomes a leak. You find this one by testing the data, not by watching the score.

#3. The right groupby, on the wrong row order

# wrong: rows arrived in file order, or a merge reordered them
features = build_features(panel_in_arbitrary_order, origination)

groupby("loan_id").shift(1) does not sort anything. It takes "the previous row for this loan" in whatever order the rows happen to be. Here the panel was shuffled before building features, which is an exaggeration of what a merge, a concat of monthly files, or a parallel read does quietly. "Previous month" becomes "some other month", often a future one, and the trailing twelve-month windows become random samples of the loan's whole life, including its end.

PR-AUC goes to 0.722. The code of build_features is identical to the clean version. Only the input order changed, which is why step 2 has an assertion and not a comment.

#4. Shifting by rows when months are missing

Real panels have holes: a servicer transfer, a late file, a month that failed to load. To simulate that, this experiment deletes 10% of the monthly reports at random and skips the calendar step.

The label is where this bites. shift(-12) now means "twelve rows ahead", which on a loan with missing months is more than twelve months ahead. The label quietly becomes "90 days past due within about thirteen months" for some rows and "within twelve" for others. On the scoring population across all three splits, 8.5% of the positive labels are loans that did not reach 90 days past due within twelve months. The default rate rises from 3.09% to 3.37%.

The score barely moves: 0.311, evaluated against the correct labels. The model is fine. The target is wrong. It is a different problem from the one you think you are solving, and every report built on it, including the default rate you would quote to a credit committee, is off by roughly 9%.

#5. A lag of a column that already looks forward

# wrong
features["label_last_month"] = labelled.groupby("loan_id")[LABEL].shift(1)

This is the one from the opening. It is the most instructive mistake in the post because it breaks the heuristic most people rely on: "positive shift, so it only looks backward". The shift is positive. The column being shifted is not a history. It is the label, which at month t−1 covers months t to t+11. Lagging it by one still leaves eleven months of the future in the feature.

PR-AUC: 0.959. On its own, with no other features, it scores 0.939.

The same thing happens in forms that are harder to spot: a "segment default rate" feature computed from recent labels, a target encoding fitted on the full training set, a "previous application outcome" that was only recorded after the observation window. The test is always the same: when did the source column become known? A label becomes known at the end of its window, not at its decision time.

#6. A field that only exists because of how the loan ended

# wrong
ever_exits_in_default = by_loan["exits_in_default"].transform("max")
features["loan_exited_in_default"] = ever_exits_in_default.astype(float)

The zero balance code records how a loan ended. Aggregated per loan and joined back onto every month, it tells each row how its story ends. PR-AUC: 0.758.

Nobody writes this line on purpose. It arrives as a loan-level attribute in a table that looks static, a "loan summary" or a "final status" dimension, and gets joined in with the origination fields. The Lending Club dataset is the classic real example. Alongside the application fields it ships total_pymnt, total_rec_prncp, recoveries, collection_recovery_fee, last_pymnt_d, and out_prncp. Every one of these describes what happened after the loan was granted, and recoveries is only non-zero because the loan was charged off. A default model trained on all the numeric columns of that file reports remarkable accuracy for exactly this reason.

#7. Today's snapshot instead of the value at the time

# wrong: a credit bureau refresh taken at the end of the data
features["credit_score"] = panel["loan_id"].map(snapshot_by_loan["latest_credit_score"])

Freddie Mac only publishes the credit score at origination. Many internal tables, though, hold a current score, refreshed from the bureau every month and overwritten each time. Join that onto historical rows and a 2006 decision is made with a 2012 score, and a borrower who defaulted in 2009 has a 2012 score that shows it.

The synthetic panel includes a simulated bureau refresh for this experiment. PR-AUC: 0.722. The size of that jump is set by how I simulated the refresh, so do not read much into it. The direction is not an artefact. The Lending Club dataset has the real version: last_fico_range_high and last_fico_range_low are the borrower's most recent credit score range, pulled long after the application, and they predict default remarkably well for exactly that reason.

The fix is not a smarter feature. It is a rule for joins: any table joined from outside the panel must carry the date its values were true, and that date must be on or before the row's decision month.

#8. Scaler statistics from every split

# wrong
prep = fit_preprocessing(pd.concat([train, valid, test]), FEATURE_COLUMNS)

Part 1 warned about this one. Measured here, at convergence, it does nothing: 0.309 either way. That is expected for this model. Logistic regression can undo any shift and rescaling of its inputs, so the only leak is through the medians used to fill missing values, and there are very few missing values.

I still keep the rule, for three reasons. It costs nothing. It stops being harmless as soon as preprocessing learns more from the data: binning, target encoding, or dropping features by their correlation with the label. And as the note at the top of this section showed, it changes how an under-trained model behaves, which is a good way to waste a day on a difference that means nothing.

#Checks that catch leakage before a reviewer does

Reading code catches some of these. Checks catch more, and they keep working after you have stopped paying attention. The companion project collects them in leakage_checks.py, which later posts import.

#Structural assertions

Four cheap assertions run on every pipeline run:

• assert_sorted_panel: loan IDs ascending, months strictly increasing within each loan. Catches mistake 3 at its source.
• assert_no_calendar_gaps: consecutive rows of a loan are exactly one month apart. Catches mistake 4.
• assert_split_gaps: each split starts more than one label window after the previous one ends.
• assert_joined_table_is_point_in_time: every value joined from outside the panel is dated no later than the row's decision month. On mistake 7 it fails with the message "100,018 rows use a value dated after their decision month", which is every training row.

#The future check

This is the most useful check in the post, and it is short. Pick a cutoff month. Build the features. Replace everything reported after the cutoff with random noise, then build the features again. Any feature whose value changes on a row at or before the cutoff was reading the future.

def features_that_see_the_future(build_features, panel, feature_columns, cutoff):
    original = build_features(panel)
    rebuilt = build_features(perturb_future(panel, cutoff))
    past = panel["period"] <= cutoff
 
    leaks = {}
    for column in feature_columns:
        before = original.loc[past, column]
        after = rebuilt.loc[past, column]
        changed = ~((before == after) | (before.isna() & after.isna()))
        if changed.any():
            leaks[column] = float(changed.mean())
    return leaks

It does not need to understand how a feature is built. It treats the feature code as a black box and tests the one property that matters. On the clean pipeline it returns nothing. On the broken ones:

Two details in that table matter. The share for mistake 1 is small because only the last row before the cutoff can see across it. The check does not need a large share, only a non-zero one. And mistake 2 is caught even though it did not move the score. A loan's first rows borrow from the previous loan in the file, and loan IDs are not in time order, so "the previous loan" can be one whose history runs past the cutoff. That makes this check the only thing in this post that would have caught mistake 2.

What it cannot catch: anything that does not flow through the panel, such as mistake 7's external snapshot (the point-in-time assertion handles that), and anything wrong with the label itself, such as mistake 4 (the calendar assertion handles that). No single check covers everything, which is why there are four of them.

#Single-feature models

Fit the same model once per feature, one feature at a time, and rank the features by validation PR-AUC. On the clean pipeline the best single feature is the current delinquency status at 0.216, then the two twelve-month history features at 0.130 and 0.120, then credit score at 0.115. That is sensible: no one feature gets anywhere near the full model's 0.309.

Add last month's label and it scores 0.939 alone.

So here is the threshold I use, and I would rather state a rule than a magic number. If one new feature, on its own, outscores the whole model you had before adding it, stop and find out why before doing anything else. A genuine new signal in credit risk adds a few points of PR-AUC to a model that already has a borrower's payment history. It does not replace the model. The absolute numbers vary with the portfolio and the base rate, so no fixed PR-AUC or Gini is suspicious everywhere. But "better alone than everything else together" is suspicious everywhere.

#What is next

Part 3 is the first probability-of-default baseline, built on exactly these tensors: logistic regression in PyTorch, a naive benchmark to beat (the current delinquency status is a strong one, as the single-feature table shows), a loss that handles the class imbalance, and an honest report using the metric tables from part 1, including the calibration that this post ignored. It is also where the test split finally gets used, once.

Part 4 is purged walk-forward validation, the splitting scheme every later post uses.

Before then, a question for you: which of these eight have you shipped? I would guess mistake 7 is the most common in banks, because the "current" version of every customer table is always the easiest one to join.

#Summary to save

• Every row needs two times: when the decision was made, and when the outcome was observed. Features may use data up to the first. Only the label may look past it.
• Keep the label in its own function. It is the only place a negative shift is allowed, so anywhere else a negative shift is a bug you can search for.
• Sort the panel and assert it. Complete the monthly calendar and assert it. groupby().shift() means "previous month" only after both.
• A positive shift is not automatically safe. Lagging a column that looks forward, such as a label, still leaks.
• Loan-level "summary" fields and "current" snapshots are the two most common ways the outcome gets in through a join. Every joined table needs an as-of date.
• Leave a gap longer than the label window between splits. Fit preprocessing on train only.
• Shuffling batches inside the training window is fine. Leakage comes from how rows are built, not the order they are fed in.
• Fit to convergence before comparing scores, and treat small differences as noise.
• Some mistakes raise the score and some leave it untouched. Test the data, not just the metric: the future check, the calendar and point-in-time assertions, and single-feature models.