AI in financial risk, part 1: financial data and its distributions

This is part 1 of a series on AI in financial risk, written alongside a thirteen-week course I am taking. The first half compares the way market and credit risk were measured before machine learning with bagging and boosting models, and ends in a backtest of both. The second half builds up how large language models work and where they fit in portfolio management and surveillance.

None of that starts with a model. Every risk number sits on top of an assumption about how the data is distributed, and the most common one is that returns are roughly normal: a bell curve, symmetric, with thin tails. Parametric value at risk assumes it. So do many textbook volatility formulas and a lot of the intuition people bring to "a three-sigma day".

So the first job is to check it. This post takes ten years of daily S&P 500 closes, turns them into returns, and asks three questions:

1. Are the tails as thin as a normal distribution says?
2. Is a big move today unrelated to a big move yesterday?
3. Where exactly does the normal assumption go wrong for a risk number?

The answers are no, no, and "in the place you care about most".

#The companion code

The companion script for this part is served with the post. Save it as distributions.py. It needs Python 3.10 or later, pandas and numpy:

python distributions.py --start 2016-10-31 --end 2026-10-29
python distributions.py --check

The first command downloads the SP500 series from FRED, the St. Louis Fed's economic data service, and writes every step to a CSV file in part01_output/. The second runs a few self-checks.

FRED only publishes the most recent ten years of this series, and the window moves every day. The two dates pin the window used in this post. If you run it later, the earliest dates will no longer be available and your numbers will differ slightly. The script also takes --prices saved_file.csv so you can keep a copy and rerun against it.

Each step writes its own file, numbered in the order it runs:

If a number in this post looks wrong to you, open the file it came from.

What this data is and is not. The SP500 series is the price index: closing levels, no dividends, one market. Dividends add a small, steady amount to the return of the S&P 500, which matters for long-run performance and very little for the size of one day's move. One US equity index over one decade is enough to show the shape of the problem. It is not a survey of all assets, and credit data, which part 6 deals with, looks different again.

#Step 1: prices are not the thing to model

The download has 2,609 rows. 96 of them are blank, because FRED keeps a row for weekdays the market was closed. After dropping those there are 2,513 closes, from 31 October 2016 to 29 October 2026.

def load_prices(path, start, end):
    raw = pd.read_csv(path)
    raw.columns = ["date", "close"]
    raw["date"] = pd.to_datetime(raw["date"])
    raw["close"] = pd.to_numeric(raw["close"], errors="coerce")
    ...
    prices = raw.dropna(subset=["close"]).reset_index(drop=True)

errors="coerce" turns the blank holiday cells into missing values instead of failing. The function then refuses duplicate or unsorted dates and non-positive prices, and prints how many blank rows it dropped. Silently dropping 96 rows is fine. Not knowing that you dropped them is not.

Over the decade the index went from 2,126 to 7,720. A price level like that has no stable average or spread: 50 points was a 2.4% move in 2016 and a 0.6% move in 2026. A risk model wants something whose scale means the same thing across the whole sample. That is the return.

#Step 2: simple returns and log returns

There are two ways to measure a day's change, and the script keeps both so they can be compared row by row:

def add_returns(prices):
    returns = prices.copy()
    returns["prev_close"] = returns["close"].shift(1)
    returns["simple_return"] = returns["close"] / returns["prev_close"] - 1
    returns["log_return"] = np.log(returns["close"] / returns["prev_close"])
    returns["calendar_gap_days"] = returns["date"].diff().dt.days
    return returns.dropna(subset=["prev_close"]).reset_index(drop=True)

The simple return is the percentage change: what happened to the money. The log return is the natural log of the price ratio. For small moves they are almost equal. For large ones they are not. On 16 March 2020 the index fell 11.98% as a simple return and 12.77% as a log return.

The difference that matters is how they add up over time:

Simple returns do not add up across days. Log returns do, exactly. That is why most of the statistics below use log returns: a sum or an average over days means something. Simple returns are the right choice when you add across positions on the same day, because a portfolio's return is the weighted sum of its holdings' simple returns. Pick one per calculation and say which.

The last column is a reminder that "daily" is not one fixed interval. Of the 2,512 returns, 1,964 cover one calendar day, 452 cover a weekend, and 96 span a holiday as well. A Monday return after a long weekend has had more time to move. I keep these as ordinary daily returns, which is the usual convention, but the column is there so that choice can be checked rather than assumed.

#Step 3: a normal control

Comparing data with a formula is easy to get subtly wrong. The script also builds a control: 2,512 draws from a normal distribution with the same mean and standard deviation as the observed returns, using a fixed seed. Every table puts the two side by side, so "what would a normal decade have looked like?" has a concrete answer instead of a textbook one.

def simulate_normal(observed, seed=SEED):
    rng = np.random.default_rng(seed)
    draws = rng.normal(observed.mean(), observed.std(ddof=1), len(observed))
    return pd.Series(draws, name="simulated_normal")

The four summary moments, written out instead of taken from a library so each formula is on the page:

def moments(r):
    centred = r - r.mean()
    std = r.std(ddof=1)
    return {
        "days": len(r),
        "mean": r.mean(),
        "std": std,
        "annualised_vol": std * math.sqrt(TRADING_DAYS),
        "skew": (centred**3).mean() / std**3,
        "excess_kurtosis": (centred**4).mean() / std**4 - 3,
        "min": r.min(),
        "max": r.max(),
    }

Skew measures asymmetry: negative means the long tail is on the loss side. Excess kurtosis measures how much of the variance comes from rare large moves. A normal distribution has an excess kurtosis of zero.

The first three rows match, as they should: the control was built to match them. (The control's mean differs by sampling noise: the standard error of a 2,512-day mean at this volatility is about 0.023%.) The volatility number, the one most risk reports lead with, gives no hint that anything is unusual.

The last four rows are where the two series part company. The real returns lean to the loss side, and their kurtosis is 16.7 where a normal has none. The worst real day is three times the size of the worst day in a normal decade with the same volatility.

#Step 4: counting the tails

Kurtosis is one number. Counting days is easier to reason about. The script standardises each return, z = (r − mean) / std, and counts how many land beyond 2, 3, 4 and 5 standard deviations, either side:

At two standard deviations the normal is about right: 112 observed against 114 expected. That is worth noticing, because it is the zone where most people check a model and come away reassured.

Past three, it is wrong by multiples. There are five times more three-sigma days than the normal allows. Over 2,512 days a normal expects a four-sigma day about once in six decades; the index had 18. A five-sigma day should be about a one-in-700-decades event; the index had 11 in one.

The worst days make the scale of the mismatch concrete. The last column is how often a normal distribution with this decade's mean and volatility says a day this bad should happen:

The universe is about 1.4 × 10¹⁰ years old. When a model says an event that happened three times in one week should take longer than that, the model is what has failed, not the market.

It is tempting to wave this away as one pandemic. The 4 April 2025 drop is in the list too, and step 7 removes 2020 entirely and tests what is left.

#Step 5: what a normal cut-off does to a risk number

The quantity that matters in risk work is not kurtosis. It is a cut-off: "on 99% of days, we should not lose more than X". This is the core of value at risk, which part 4 builds properly. Here is the shortest version of the question, with the normal cut-off computed as mean + std × the normal quantile, and the empirical cut-off read straight from the sorted data:

This table has the most important result in the post, and it goes in two directions.

At 95% the normal cut-off is too cautious. Real returns breach it less often than promised: 105 times instead of 126. A fat-tailed distribution is not just wider at the edges. It is also more peaked in the middle, with more quiet days than a normal of the same volatility.

At 99% the normal cut-off is breached twice as often as it should be. At 99.9% it is breached eight times as often, and the real loss at that level is twice the normal estimate.

So a normal model does not fail everywhere. It fails in the tail, and regulatory and internal risk numbers are set in the tail. That is also why a model can pass a casual check at 95% and still be wrong where it matters.

This is a description, not a backtest. Every number in this table uses the mean and standard deviation of the whole decade, including days that come after the ones being tested. A real risk model estimates from the past and is scored on the future. Part 4 computes value at risk that way, and part 5 counts breaches over time. The point here is narrower: even with perfect hindsight about the volatility, a normal shape puts the 99% line in the wrong place.

#Step 6: big days come in clusters

The normal model also makes a claim about time: each day is a fresh, independent draw. If that were true, a big move today would say nothing about tomorrow.

The script measures autocorrelation, the correlation of a series with itself shifted by some number of days, written out in full:

def autocorrelation(r, lag):
    centred = r - r.mean()
    return float((centred * centred.shift(lag)).sum() / (centred**2).sum())

It is applied twice: to the returns, which asks "does today's direction predict tomorrow's?", and to the absolute returns, which asks "does today's size predict tomorrow's?"

For independent data, about 95% of these values should fall within ±0.039 of zero. The normal control does. The absolute returns are seven to ten times that bound for the first two weeks, and still above it a full month later. A large move, in either direction, makes another large move more likely for weeks. That is volatility clustering.

The first column needs more care. A lag-1 value of −0.143 looks like a reversal effect: a fall today predicts a rise tomorrow. But the ±0.039 band assumes independent days, and the second column just showed the days are not independent; under clustering the honest band is wider. Step 7 also shows where most of it comes from.

Clustering shows up just as clearly as a calendar. The 21-day rolling volatility in 06b_rolling_vol.csv ranges from an annualised 3.5% in October 2017 to 97.6% in April 2020, with a median of 12.5%. By calendar year:

(2016 has only 42 trading days in the window and is left out of the table; it is in the file.)

One volatility number for the whole decade, 18.1%, describes almost none of these years. 2017 was a third of it. 2020 was double it.

These two findings are connected. If calm periods and stormy periods each looked roughly normal, mixing them together would still give fat tails: most days come from the calm regime, and the extremes all come from the stormy one. Fat tails might partly be clustering seen from a distance. That is the idea behind GARCH, the classical volatility model in part 5, and it is a claim to test there, not one to assume here.

#Step 7: is it just 2020?

March 2020 contributes the three most extreme days of the decade. A fair objection is that one shock is doing all the work. The last step reruns the main measures with every 2020 date removed:

Without 2020, the standard deviation is smaller, so the three-sigma line moves in too, and the count barely changes: 33 days against 6 expected. Kurtosis falls from 16.7 to 6.7, which is still far from zero. Volatility clustering weakens but stays several times the noise band.

Two things do mostly go away. The negative skew drops to −0.14, so much of the "losses are bigger than gains" shape in this sample is the 2020 crash. And the lag-1 reversal falls to −0.024, inside the noise band. That apparent pattern in daily direction was the whipsaw of March and April 2020, not a durable feature of the market.

This is why the script makes the robustness step a file of its own. Some findings survive removing the worst year. Some turn out to be that year. A risk model should be built on the first kind.

#What this means for the rest of the series

Three facts carry forward:

• Tails are fat. A risk number that assumes normality will be too small exactly where risk numbers are set. Any method, classical or machine learning, has to be judged on how it behaves in the tail, not on an average error.
• Volatility is not constant. The right risk number for a quiet month and a crisis month is different. A model with one fixed volatility is already wrong before it is fitted.
• Days are not independent. Random shuffles and k-fold cross-validation assume they are. Every evaluation in this series will respect time order, which the purged walk-forward post in my other series covers in depth.

These are what the classical baselines in parts 4 and 5 are designed around, and the bar any boosted model has to clear. Part 2 steps back first: what changes when the goal is prediction rather than inference, and how the confusion matrix reads once each kind of error has a cost.

#Summary to save

• Model returns, not prices. A price level has no stable scale.
• Log returns add up over time; simple returns add up across positions on the same day. Choose per calculation and say which.
• Compare against a simulated normal control with the same mean, volatility and length, not only against a formula.
• Volatility alone hides the problem. Report skew and excess kurtosis, and count the days beyond 3, 4 and 5 standard deviations.
• A normal model can look right at 95% and be wrong at 99%. On this data the normal 99% cut-off was breached 51 times against 25 expected.
• Check autocorrelation of absolute returns, not just returns. Large moves cluster for weeks.
• The ±1.96/√n noise band assumes independent days. Under clustering, read small return autocorrelations with suspicion.
• Rerun with the worst period removed. Keep the findings that survive, and be honest about the ones that were one event.
• Using the whole sample's mean and volatility is description, not a backtest. Risk models are estimated on the past and scored on the future.