AI in Financial Risk, Part 2: Machine Learning versus Statistical Models

This is part 2 of a series on AI in financial risk. Part 1 showed that daily returns have fat tails and that large moves cluster. Clustering means tomorrow's risk is partly knowable today. That makes it a good place to ask the question this part is about: what is the difference between a statistical model and a machine learning model, when both can be the same few lines of code?

The usual answer is a list of algorithms. I think the more useful answer is that they are two different questions asked of the same data:

• Inference asks: which inputs are related to the outcome, by how much, and how sure am I? The output is a coefficient and its uncertainty. The whole sample is used to estimate it.
• Prediction asks: how wrong will this be on days it has not seen? The output is a score on held-out data. Nothing else counts.

This post runs one risk question through both, then covers the two ideas that follow from taking prediction seriously: the trade-off between bias and variance, and the confusion matrix read with a cost on each kind of error.

#The Companion Code

The companion script for this part is served with the post. Save it as prediction_vs_inference.py. It needs Python 3.10 or later, pandas and numpy. Both models are written out in numpy, so there is no library call hiding a step:

python prediction_vs_inference.py --start 2016-11-14 --end 2026-11-13
python prediction_vs_inference.py --check

It uses the same SP500 series from FRED as part 1. FRED publishes a rolling ten years, so the window has moved two weeks since part 1 and the counts differ slightly: 2,514 closes and 2,513 daily log returns, from 14 November 2016 to 13 November 2026.

#Step 1: One Question per Row

Every row has to answer the same question: given what is known at today's close, will tomorrow's log return be below −1%?

The −1% line is a fixed round number, not a quantile of the data, so nothing about the label is estimated from days the model will later be tested on. The four inputs are all things part 1 measured:

def add_features(returns):
    data = returns[["date", "log_return"]].copy()
    data["return_1d"] = data["log_return"]
    data["abs_return_1d"] = data["log_return"].abs()
    # Rolling windows end on today's row, so they use today and earlier only.
    data["mean_abs_return_5d"] = data["abs_return_1d"].rolling(5).mean()
    data["mean_abs_return_21d"] = data["abs_return_1d"].rolling(21).mean()
    return data
 
 
def add_label(data):
    labelled = data.copy()
    labelled["next_date"] = labelled["date"].shift(-1)
    labelled["next_log_return"] = labelled["log_return"].shift(-1)
    labelled["large_loss_next_day"] = (labelled["next_log_return"] < LOSS_THRESHOLD).astype(int)
    complete = labelled.dropna(subset=FEATURES + ["next_log_return"])
    return complete.reset_index(drop=True)

Today's return, its size, and the average size of the last 5 and 21 days. The label is the only column built with shift(-1), and next_date sits beside it in the file so each row shows which day it is being scored against. One of the self-checks crashes the price on the last day and asserts that no feature on any earlier row changes.

The first 20 returns have no 21-day window and the last one has no tomorrow, which leaves 2,492 rows. 280 of them, 11.2%, are followed by a large loss.

#Step 2: Split by Time

Part 1 showed the days are not independent, so the split is by date and never shuffled: the first 60% of rows to fit, the next 20% to make choices, the last 20% to be scored once.

The numbers that matter in this table are 174, 51 and 55. A model of a rare event learns from the events, not from the rows. Everything below is decided by a few dozen days.

#Step 3: The Inference View

A logistic regression models the log-odds of a large loss as a weighted sum of the inputs. The script fits it with Newton's method, in full:

def fit_logistic(x, y, max_steps=50):
    beta = np.zeros(x.shape[1])
    for _ in range(max_steps):
        p = sigmoid(x @ beta)
        gradient = x.T @ (y - p)
        hessian = x.T @ (x * (p * (1 - p))[:, None])
        step = np.linalg.solve(hessian, gradient)
        beta = beta + step
        if np.abs(step).max() < 1e-10:
            break
    p = sigmoid(x @ beta)
    hessian = x.T @ (x * (p * (1 - p))[:, None])
    return beta, np.linalg.inv(hessian)

The second return value is what makes this a statistical model. The inverse of the Hessian is the covariance of the coefficients, and the square roots of its diagonal are the standard errors. A coefficient divided by its standard error is a z-score, and from that comes a p-value: how often a value this far from zero would turn up if the true coefficient were zero.

The inputs are standardised with the training mean and standard deviation, so each coefficient is the effect of a one standard deviation change. Fitted on the 1,495 training days:

This is the kind of answer inference gives, and it is a useful one. One input carries the relationship: when the last five days have been one standard deviation rougher than usual, the odds of a large loss tomorrow are 1.61 times higher, and that is unlikely to be chance. Today's move on its own adds nothing once the five-day average is in the model. Whether today was up or down adds nothing either.

#The Standard Errors Assume Something Part 1 Disproved

Those standard errors come from a formula that treats each row as an independent observation. These rows are not. The 21-day average on Tuesday shares 20 of its 21 days with Monday's, and large-loss days arrive in clusters.

The script checks how much that matters with a block bootstrap. It rebuilds the training set 500 times by drawing whole 21-day blocks with replacement, refits each time, and takes the spread of the coefficients across the refits as the standard error. Drawing blocks keeps neighbouring days together, so the dependence survives the resampling.

The five-day result survives. The uncertainty on the 21-day input doubles, which is what you would expect from the input that overlaps itself the most. Nothing changed sides of the 5% line here, but a p-value of 0.14 that is really 0.49 is the kind of gap that does change a conclusion on another dataset.

So the inference view has its own discipline. The question is whether the assumptions behind the standard error hold. It is not a question about held-out data at all, and the table above says nothing about how well this model predicts.

#Step 4: The Prediction View

Now the same fitted model is treated as a forecaster. It outputs a probability for each day, and the probability is scored against what happened with the Brier score: the mean of (probability − outcome)², where the outcome is 1 or 0. Lower is better.

A score needs something to beat. The benchmark is the base rate: say 11.6%, the training frequency, every single day. Skill is the percentage improvement in Brier score over that.

The comparison model is deliberately not a statistical one. k nearest neighbours has no coefficients and no assumptions about shape. To score a day it finds the k most similar days in the training set and reports the share of them that were followed by a large loss:

def predict_knn(train_x, train_y, x, k_values):
    squared_distance = (
        (x**2).sum(axis=1)[:, None] - 2 * x @ train_x.T + (train_x**2).sum(axis=1)[None, :]
    )
    largest_k = min(max(k_values), len(train_x))
    nearest = np.argpartition(squared_distance, largest_k - 1, axis=1)[:, :largest_k]
    nearest_distance = np.take_along_axis(squared_distance, nearest, axis=1)
    order = np.argsort(nearest_distance, axis=1)
    nearest_labels = train_y[np.take_along_axis(nearest, order, axis=1)]
    running_share = np.cumsum(nearest_labels, axis=1) / np.arange(1, largest_k + 1)
    return {k: running_share[:, min(k, largest_k) - 1] for k in k_values}

k is a dial. At k = 1 the model copies the single most similar day. At k = 1,495 it averages every training day and becomes the base rate.

Read the two skill columns against each other.

On the training data the most flexible model is perfect. With k = 1, each training day's nearest neighbour is itself, so it reproduces every label. On days it has not seen it is 81% worse than quoting the base rate. The training score rises steadily as the model gets more flexible and the validation score falls. A training score measures flexibility, not skill.

The logistic regression, with a coefficient significant at 0.002, improves on the base rate by 0.7%. That is the other half of the lesson. A real, statistically solid relationship can be worth almost nothing as a forecast. Significance says the effect is not zero. It does not say the effect is large enough to use.

The model with the best validation score is the logistic regression, so that is the one carried forward. The choice was made on validation data, and the test split has still not been touched.

#Step 5: Bias and Variance, with a Known Truth

Why does the flexible model lose? The standard explanation is that a model's expected error on new data splits into three parts:

• Noise: the outcome is uncertain even if you know the true probability. No model removes this.
• Bias: the model's shape cannot represent the truth, so it is wrong on average no matter how much data it gets.
• Variance: the model changes when the training sample changes, so any one fit is off by a random amount.

On real data this cannot be measured directly, because bias is the gap to a true probability that nobody knows. So, as in part 1, the script builds a control: a simulated market where the truth is written down.

def true_probability(features, calm_cut, stress_cut):
    vol = features["mean_abs_return_21d"].to_numpy()
    fell_today = features["return_1d"].to_numpy() < LOSS_THRESHOLD
    p = np.full(len(features), 0.10)
    p[vol < calm_cut] = 0.03
    p[vol >= stress_cut] = 0.30
    p[(vol >= stress_cut) & fell_today] = 0.50
    return p

Three volatility regimes, cut at the median and the 90th percentile of the 21-day average, and a rule that a fall today matters only in the stressed regime. The feature rows are the real ones. Only the labels are simulated, by drawing each one from its true probability.

This truth is chosen to be awkward for a logistic regression. It has steps and an interaction, and a weighted sum cannot draw either. It is tilted towards the flexible model on purpose, to see what the flexible model needs in order to collect.

For each training size the script draws 100 separate training sets, fits every model on each, and predicts the same 500 evaluation days. Across the 100 fits, the average prediction's squared distance from the truth is the bias, and the spread of the predictions is the variance. Noise is 0.0688 for every model. The table shows what each model adds on top, multiplied by 1,000 to keep the digits readable:

(With 100 fits, each bias figure carries sampling noise of about one hundredth of the variance beside it. For k = 1 that is about 0.7, so its bias is not distinguishable from zero.)

Four things are in this table.

The dial trades one error for the other. Going down the nearest-neighbour rows at 1,500 training days, bias rises from 0.9 to 4.2 and variance falls from 67 to 0.2. The total is lowest in the middle, at k = 100. This is the trade-off, in numbers rather than as a U-shaped sketch.

The logistic regression's error is almost all bias. 3.97 of its 4.47. It is the wrong shape for this truth and it is very stable about being the wrong shape.

More data does not fix bias. With ten times the training data the logistic regression's variance falls by nine tenths and its total barely moves, from 4.47 to 4.05. The k = 100 model's total halves, from 3.51 to 1.77, because its neighbours are now closer and so more alike. A statistical model stops improving once its shape is the limit. A flexible model keeps converting data into accuracy.

At the size of the real dataset it is close to a tie. With 1,500 training days, on a truth built to favour the flexible model, the best nearest-neighbour setting beats the logistic regression by about a fifth of the avoidable error, and only if k is chosen well. Two of the five settings are far worse.

And 15,000 daily rows is sixty years of one index. That is the practical constraint in market risk: the number of rows grows by 252 a year, and the number of crises grows far more slowly. Where data is plentiful, as with millions of loans or card transactions, the right half of the table is the relevant one. That is one reason the series treats market and credit risk separately.

On the real data in step 4 the logistic regression won outright. The simulation suggests why that is not surprising: the real signal is much weaker than the simulated one, so there is less structure for a flexible model to find and the same variance to pay for looking.

#Step 6: The Confusion Matrix, with Costs

A probability is not a decision. To act on it you pick a threshold and raise an alarm above it, which might mean cutting a position or buying protection for the day. Each day then lands in one of four cells: a large loss that was flagged, a false alarm, a missed loss, and a quiet day correctly left alone. Part 1 of my other series covers the vocabulary. What matters here is that the two kinds of error do not cost the same.

Suppose a false alarm costs 1 unit, the price of protection that was not needed, and a missed large loss costs 10. These costs are assumptions. In practice they come from the desk, not from the data, and the script takes the ratio as an input. Given a ratio, the arithmetic gives a threshold before any data is looked at:

def break_even_threshold(cost_ratio):
    # Alarm when p * cost_of_miss > (1 - p) * cost_of_false_alarm.
    return 1 / (1 + cost_ratio)

At 10 to 1 an alarm is worth raising whenever the probability is above 1/11, about 9.1%. That has an immediate consequence. The base rate is about 11%, which is already above 9.1%. With no model at all, the cheapest policy is to raise the alarm every day.

The script also searches thresholds from 0.01 to 0.60 on the validation split and keeps the cheapest, which is 0.10. Both thresholds were fixed before the test split was scored. On the 499 test days, 55 of them large losses:

The most accurate rule is the most expensive. Never raising the alarm is right on 89% of days and misses every loss. The default threshold of 0.50 does exactly the same thing, because this model almost never outputs a probability above one half for an event with an 11% base rate. Accuracy rewards agreeing with the majority, and in risk the majority of days are not the ones you are paid to worry about.

The model barely beats having no model. The best rule costs 419 against 444 for alarming every day, a saving of under 6%, and at the break-even threshold the model is slightly worse than alarming every day. That is the 0.7% Brier skill from step 4 seen from the decision side. A forecast this weak can only move the cost a little.

The threshold is fragile. On validation, the cost was 427 at a threshold of 0.09, 401 at 0.10 and 458 at 0.11. The model's median probability is about 0.094, so the threshold sits in the thickest part of the distribution and moving it by one hundredth switches more than a hundred days from alarm to no alarm. A choice made on 51 validation events is not one to trust to two decimal places.

The cost ratio decides more than the model does.

At 5 to 1 the cheapest policy on validation is to never raise the alarm. At 20 to 1 it is to always raise it. Only in between does the model's ranking of days matter. Before comparing algorithms, it is worth finding out which of these three rows the problem is in, and that is a question about costs, not about models.

#Scoring the Test Split

The test Brier scores, computed once at the end:

The order from validation holds. The logistic regression is slightly ahead of the base rate and every nearest-neighbour setting is behind it. The nearest-neighbour rows are shown for completeness; only the logistic regression was selected, and a result on 55 events is weak evidence either way.

#What This Does Not Show

• It does not show that statistical models beat machine learning. Nearest neighbours is the simplest flexible model there is, chosen because every line of it fits on the page. Part 3 maps the stronger ones, and bagging and boosting exist largely to cut the variance that sank it here.
• It is one index, four inputs built from one series, and one decade. A real market risk model has more to work with.
• It does not show that tomorrow's loss cannot be forecast. It shows that a yes or no forecast from these inputs is weak. Forecasting the size of tomorrow's move, which is what a volatility model does, is a different and easier problem, and it is where the series goes next on the market side.

#What Is Next

Three things carry forward:

• Say which question is being asked. A coefficient table and a held-out score are answers to different questions. A model can be good at one and useless at the other.
• Flexibility is paid for in variance, and data is the currency. How much flexibility a problem can afford depends on how many events it has, not how many rows.
• Cost the errors before choosing the model. The cost ratio set the policy at two of the three ratios tried here.

Part 3 is a map of the algorithms: supervised and unsupervised learning, bagging and boosting, sampling for rare events, and clustering, each with the risk question it answers.

#Summary to Save

• Inference asks what is related to the outcome and how sure you are. Prediction asks how wrong you will be on unseen data. Decide which one the work is for.
• A standard error is only as good as its assumptions. With overlapping windows and clustered events, check it with a block bootstrap. Here one standard error doubled.
• Significant is not the same as useful. A coefficient with a p-value of 0.002 gave a forecast 0.7% better than the base rate.
• Always score against the base rate, on days the model has not seen.
• A training score measures flexibility. The model that was perfect in training was 81% worse than the base rate on validation.
• Error is noise plus bias plus variance. More data shrinks variance and leaves bias where it was.
• Count events, not rows. This dataset has 2,492 rows and 280 events.
• Accuracy is the wrong measure for a rare event. The rule with 89% accuracy had the highest cost.
• With calibrated probabilities, the alarm threshold is 1 / (1 + cost ratio). Work it out before looking at the data, and compare it with the base rate.
• Check "always alarm" and "never alarm" as benchmarks. A model has to beat both.
• Choose the model and the threshold on validation, then score the test split once.