This is post one of a series on financial machine learning for people who know some Python, have used a banking app or applied for a loan or glanced at a stock chart, and have never trained a model on financial data. Every code example in the posts that follow will be in PyTorch. This post has no code at all. It is the ground the rest of the series stands on.
My background is the banking side of this field: building credit-scoring models that decide who gets a loan, fraud models that decide which card transactions to block, and the anti-money laundering (AML) analytics pipelines that decide which accounts an investigator looks at next. Those three problems are the spine of this series. Market prediction, the thing most "ML for finance" content is about, appears as a supporting track, because it teaches a few lessons about time and noise that banking models need too.
In every one of these settings I have watched models that looked excellent in development quietly fail in production. The cause was almost never the algorithm. It was something in this post.
Why this series
Most machine learning tutorials assume the data is i.i.d., short for independent and identically distributed: each example is drawn from the same unchanging process and does not depend on the ones before it. Financial data violates both halves. A customer's transactions this month depend on last month's. The borrowers who applied for loans in 2019 were not facing the economy of 2021, and the fraudsters of 2021 had learned from every model deployed against them in 2019.
Three more realities follow. Labels are noisy: whether a borrower defaults depends partly on things no feature captures, and whether a transaction is labelled fraud depends on whether the customer noticed. Leakage is common: it is astonishingly easy to let the outcome, or the future, into today's features without noticing. And accuracy is often the wrong score: a fraud model that flags nothing is 99.9 percent accurate.
This series starts from those realities instead of bolting them on at the end. The code will be PyTorch because research-style finance work needs three things it does well: models you can shape freely rather than pick from a menu, custom loss functions that encode what you actually care about (a loss is the quantity a model minimises during training), such as the asymmetric cost of a missed default versus a declined good customer, and a short path from a research idea to a running prototype. You will not need a GPU for anything in this series.
What machine learning is, in a bank
A traditional decision rule is written by hand. "If a card is used in two countries within an hour, block it." "If debt payments exceed 50 percent of income, decline." A human chose the signal, the threshold, and the action. The rule does exactly what it says, forever, whether or not customers and fraudsters still behave that way.
Machine learning replaces the hand-written rule with a procedure that finds patterns in historical data. You supply examples of inputs and outcomes, past applications and whether they defaulted, past transactions and whether they were disputed, and the algorithm adjusts a model until its outputs match the outcomes as well as it can. The model may discover that the two-country rule matters only for new cards, or not at all. The human's job shifts from writing rules to choosing the data, the target, the model family, and the way success is measured.
Finance is harder than classic machine learning for four reasons that recur throughout the series. Non-stationarity: the relationships change over time. The economy moves, products change, and fraud is adversarial, meaning the people generating the positive class adapt to whatever you deploy. Low signal-to-noise: defaults, fraud, and laundering are rare events buried in millions of ordinary records, and in markets the predictable part of a daily return is a sliver of the random part. Asymmetric costs: a missed fraud and a wrongly blocked card are both errors, and they cost different amounts to different people. Market models have their own version in transaction costs. Look-ahead bias: using information that was not available at the time of the decision, which inflates every development result and vanishes in production.
The three types of machine learning, mapped to finance
Supervised learning
You have inputs and the correct answer for each, and the model learns the mapping. This is most of applied finance ML.
Examples: credit scoring, which estimates a borrower's probability of default, the chance they fail to repay, from income, repayment history, and existing debt; fraud detection, which scores each card transaction on how likely it is to be unauthorised, using the amount, the merchant, and how far and how fast the card has moved since its last use; and, on the market track, forecasting next-month volatility, the size of price swings.
What good looks like: predictions better than a naive baseline on data the model never saw, by a margin large enough to matter after costs. For default and fraud, good means catching most of the rare bad cases without flagging too many good ones, with probabilities a lender can price from. A credit model that ranks borrowers correctly but says 2 percent when the true default rate is 6 percent loses money on every loan it approves.
Unsupervised learning
You have inputs and no answers. The model finds structure.
AML is where unsupervised methods earn their living. Money laundering has very few confirmed labels, because a suspicious activity report filed by the bank rarely comes back with a verdict. So an AML analytics pipeline typically combines hand-written rules (cash deposits just under the reporting threshold, rapid movement of funds through a new account) with anomaly scores that rank accounts by how unusual their behaviour is relative to a peer group, and hands the top of that list to human investigators. The same anomaly-detection idea is how fraud detection begins on a new product before any disputes exist. Other uses: segmenting customers by behaviour, and on the market track, clustering trading days into regimes such as calm, volatile, and crisis.
What good looks like: groups a human recognises that stay stable when you re-run on a different period. Metrics such as the silhouette score or the Davies-Bouldin index measure how cleanly separated clusters are, but the real test is operational. In AML, good means the investigators' queue contains more genuine cases per hundred alerts than it did before, and the same tool can retire alerts nobody would ever act on.
Reinforcement learning
An agent takes actions, receives rewards, and learns a policy that maximises reward over time.
Examples in banking are still emerging: deciding which delinquent customers to contact first in collections, or how to adjust credit limits over a customer's life. In markets it is used for trade execution, breaking a large order into pieces to minimise market impact.
What good looks like: lower cost or higher recovery than a simple policy, measured on customers or orders the agent did not train on. I mention reinforcement learning for completeness. It is powerful, data-hungry, and easy to fool with a badly designed simulator. It will appear late in this series, if at all.
Vocabulary you will see in every model notebook
- Features, also called signals: the inputs. A borrower's credit utilisation ratio, the number of transactions on a card in the last hour, the share of an account's inflows that leave within a day, or on the market track, the 20-day volatility of a stock.
- Labels or targets: the thing to predict. A default flag of 1 if the loan went bad within twelve months. A fraud flag set when the customer disputed the charge. A five-day forward return for a stock. Notice that the first two arrive late: a default takes a year to observe and a fraud dispute can take two months. The label for a loan approved today does not exist yet.
- Model: the function that maps features to a prediction. A logistic regression is a model. So is a neural network.
- Training versus inference: training adjusts the model using historical examples. Inference runs the trained model on new data. You train on loans booked between 2015 and 2022 and run inference on this morning's applications.
- Parameters versus hyperparameters: parameters are the numbers the model learns, such as the weight on each feature. Hyperparameters are settings you choose before training, such as how many months of history to use or how strongly to penalise complexity.
- Loss versus metric: the loss is what training minimises, for example log loss. The metric is what you judge the model by, for example the expected credit loss at the approval cut-off you would actually use. They are often different, and the gap between them is where a lot of finance ML goes wrong.
- Train / validation / test: three separate slices of history. Train fits the parameters. Validation chooses hyperparameters. Test is touched once, at the end, to estimate real performance. In finance these slices must be in time order, with a gap between them. A purged split removes training examples whose labels overlap the validation period. An embargoed split adds a further buffer after the validation window. Both exist because a twelve-month default label for a loan booked in June already contains information about the following year.
- Overfitting versus underfitting: an overfit model has memorised the noise in its training data and fails on new data. An underfit model is too simple to capture the real pattern. In finance, overfitting is the default failure mode.
- Leakage: any way that information from the future, or from the label itself, reaches the features. Using "number of missed payments" computed over the whole life of the loan as a feature at application time leaks. Normalising features using the mean of the whole dataset leaks.
- Point-in-time data: data as it was known on each historical date, not as it was later corrected. A bureau score gets refreshed. A merchant's category code gets recoded. A model trained on the corrected version has seen the future.
Ideas that destroy beginner finance models
A random 80/20 split is dangerous on time-ordered data
The textbook approach shuffles all rows and holds out 20 percent. On financial data this puts a customer's March transactions in training and their April ones in test, or loans from the same month of the same economy on both sides. Adjacent records share most of their information, so the model appears to generalise when it is really interpolating between neighbours it has already seen. Always split by time, always leave a gap.
Late labels and the base rate
Because a default label takes a year to mature, the most recent year of loans has no labels yet, and a model trained on the loans that do have labels has learned from an older economy. Retraining schedules have to respect that lag. Fraud labels depend on customers noticing and disputing, so undisputed fraud is silently labelled legitimate, which teaches the model that some fraud patterns are fine.
Then there is the base rate, the share of positives in the data. Defaults might run at 2 percent, card fraud at a tenth of a percent, and confirmed money laundering far below that. At those rates a model that flags nothing looks superb on accuracy, and a fraud model that catches every fraud while blocking 5 percent of good transactions will be switched off within a week by the people who answer the customer calls. Every metric choice later in this post is shaped by that imbalance.
Look-ahead bias and survivorship bias
Look-ahead bias is using information before it was available: scoring a January application with a bureau report pulled in March. Survivorship bias is subtler in banking than in markets. You only observe defaults on the loans you approved. The applicants you declined have no label, so a model trained on approved loans has never seen the riskiest part of the population it will be asked to score. Market datasets have the same problem when they contain only companies that still exist today. Both make development results look better than anything achievable live.
High accuracy can still lose money
A fraud model that is right 99.9 percent of the time may be right on every legitimate transaction and wrong on every fraud. A credit model that is right on 97 percent of applicants may be wrong on exactly the 3 percent whose losses exceed the margin on all the rest, because a single default costs many times the profit on a good loan. On the market track the same thing appears as a model that calls direction correctly 70 percent of the time but is wrong on the few large moves that dominate the total. Accuracy sees neither cost nor magnitude.
Costs of acting on the prediction
Every prediction that becomes an action has a cost the model never saw. A blocked card means a call centre conversation and a customer who may leave. A declined application means lost interest income. An AML alert means an investigator's hour. Market strategies pay transaction costs and slippage, the gap between the price expected and the price obtained, and have limited capacity before their own trading destroys the edge. A development result that ignores the cost of acting is a fiction.
In-sample, out-of-sample, and walk-forward
In-sample results come from the data the model was fitted on. They are almost meaningless in finance. Out-of-sample results come from data the model never saw. Walk-forward validation repeats the process the way you would actually deploy: train on loans booked before a date, test on the next quarter's, roll forward, repeat. It is slower and it is the only honest picture of how a model would have behaved.
Metrics that actually matter in financial machine learning
This section is the reference you will come back to. Each table lists the metric, where it is used, what it measures, whether higher or lower is better, and the trap to watch.
Classification: default, fraud, suspicious activity
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| Accuracy | Approve/decline, fraud/not | Share of predictions that are correct | Context | If 1% of loans default, predicting "no default" for everyone scores 99%. Accuracy lies on imbalanced data |
| Precision | Fraud and AML alerts | Of the cases flagged positive, the share that were truly positive | Higher | High precision means few false alarms; the cost of a wrong fraud block is an angry customer |
| Recall | Default and fraud screening | Of the true positives, the share the model caught | Higher | High recall means few missed frauds; usually traded off against precision |
| F1 | Summarising imbalanced classification | Harmonic mean of precision and recall | Higher | One number for the trade-off; hides which side is weak |
| ROC-AUC | Ranking risk across borrowers | Probability a random positive is scored above a random negative | Higher | Looks healthy even when positives are rare, because it rewards ranking the vast negative class correctly |
| Gini coefficient | Credit scorecards | Ranking power, equal to 2 × ROC-AUC − 1 | Higher | The number credit risk teams quote. A Gini of 0.5 is a decent retail scorecard; it inherits ROC-AUC's blindness to calibration |
| KS statistic | Credit scorecards | Largest gap between the cumulative distributions of good and bad borrowers across the score | Higher | Points to the single cut-off that best separates the groups, which is how approval thresholds get set |
| PR-AUC | Fraud, default, any rare event | Area under the precision-recall curve | Higher | The honest one for imbalanced data. A no-skill PR-AUC equals the positive rate, so know that baseline |
| Log loss | Probability outputs | Penalises confident wrong predictions | Lower | Rewards calibrated probabilities, not just correct ordering |
| Population stability index (PSI) | Monitoring deployed models | How far the distribution of scores, or of a feature, has drifted from the training population | Lower | Not a quality metric but a drift alarm. A PSI above 0.25 on a scorecard usually triggers a review |
Risk and calibration
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| Brier score | Default probabilities, event forecasts | Mean squared difference between predicted probability and the 0/1 outcome | Lower | A model that says 80% should be right about 80% of the time. Brier checks that |
| Reliability (calibration curve) | Any probability used for pricing or capital | Whether predicted probabilities match observed frequencies in each bucket | Closer to the diagonal | A model can rank borrowers perfectly and still say 5% when the truth is 15%. For pricing a loan, that gap is the whole problem |
Business outcomes: what the bank feels
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| Bad rate at cut-off | Credit approval policy | Share of approved loans that default at the chosen score threshold | Lower | The number the business owns. Move the cut-off and the approval rate moves the other way |
| Approval rate | Credit approval policy | Share of applicants accepted | Context | Higher grows the book and the losses together. Always report next to bad rate |
| Expected credit loss | Pricing and provisioning | Probability of default × exposure × loss given default, summed over the book | Lower | Only as good as the calibration behind it |
| False positives per fraud caught | Fraud thresholds | Good transactions blocked for each fraud stopped | Lower | The ratio the call centre feels. Ten to one is uncomfortable; one hundred to one is unusable |
| Alerts per confirmed case | AML tuning | Investigator alerts generated for each case that becomes a filed report | Lower | Rules-only systems commonly run above 95% false positives. This ratio is what the analytics are there to improve |
Prediction and regression: amounts, volatility, returns
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| MAE (mean absolute error) | Loss-given-default, exposure and volatility forecasts | Average size of the error, in the units of the target | Lower | Robust to outliers; treats a 1% miss and ten 0.1% misses the same |
| MSE (mean squared error) | Default training loss for regression | Average squared error | Lower | Punishes large misses heavily; one crisis month can dominate |
| RMSE (root mean squared error) | Reporting regression error | Square root of MSE, back in target units | Lower | Same ranking as MSE, easier to read |
| MAPE (mean absolute percentage error) | Balance and volume forecasts | Average error as a percentage of the actual | Lower | Explodes when the actual is near zero, so never use it on returns |
| R² | Explanatory power | Share of the target's variance the model explains | Higher | On daily returns an R² near zero is normal, and 0.3 is a leakage alarm. On loss amounts, modest values are also the norm |
Ranking and selection: which accounts to review, which names to buy
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| Precision at k | AML and fraud review queues | Of the top k alerts, the share that were genuine | Higher | Matches how investigators work: they get through k cases a day, not the whole list |
| MAP and nDCG | Review queues; top-k stock selection | Ranking quality with more credit for true cases placed higher | Higher | Borrowed from search engines; useful whenever position in the list matters |
| Information coefficient (IC) and Rank IC | Cross-sectional stock signals | Correlation (Pearson, or Spearman on ranks) between predicted and realised returns across assets on a date | Higher | An IC of 0.05 is a real edge in equities. Report the mean and its stability; Rank IC resists one huge winner |
Strategy quality, market track only
| Metric | Typical use in finance | What it measures | Better when | One-line intuition / trap |
|---|---|---|---|---|
| Sharpe ratio | Universal strategy score | Average excess return divided by its standard deviation, annualised | Higher | Easy to inflate by overfitting the development window |
| Sortino and Calmar ratios | Asymmetric or drawdown-sensitive strategies | Excess return over downside deviation; annual return over maximum drawdown | Higher | Variants that stop penalising good volatility, or that let one bad month set the denominator |
| Maximum drawdown | Risk of ruin | Largest peak-to-trough loss over the period | Lower | The number that gets a strategy shut down. Always report next to Sharpe |
| Hit rate | Trade-level diagnostics | Share of trades that made money | Context | A 40% hit rate with large winners beats 70% with large losers |
| Turnover | Cost and capacity | How much of the portfolio is traded per period | Context | Higher means higher costs and lower capacity; zero means the model is doing nothing |
The cheat box
Lower is better: MAE, MSE, RMSE, MAPE, log loss, Brier score, PSI, bad rate, expected credit loss, false positives per fraud caught, alerts per confirmed case, maximum drawdown, Davies-Bouldin index, inertia, perplexity
Higher is better: precision, recall, F1, ROC-AUC, Gini, KS, PR-AUC, precision at k, MAP, nDCG, R², IC and Rank IC, Sharpe, Sortino, Calmar, silhouette score
Context-dependent: accuracy, approval rate, alert volume, hit rate, turnover, number of trades
A warning that deserves its own paragraph. Never approve a credit or fraud model on ROC-AUC or Gini alone, computed on the development sample. A ranking metric on the data you fitted measures how well you fit the past, and any sufficiently flexible model fits the past beautifully. Pair the ranking metric with a calibration check such as the Brier score, and with the operational number the business will feel, such as bad rate at the chosen cut-off or good customers declined per fraud caught. Compute all of them on a clean out-of-time window that neither the model nor you have touched. If the ranking is strong and the calibration is poor, or the numbers collapse out of time, suspect leakage before you suspect genius.
The market-track version of the same rule: never pick a strategy on Sharpe ratio alone computed on the training period. Pair a statistical metric such as Rank IC or log loss with an economic one such as Sharpe and maximum drawdown, on a clean out-of-sample window.
What this post does not do
There are no derivations here, no explanation of how gradient descent finds the parameters, no PyTorch, and no scorecard or backtest engine. Each of those is a post of its own, and each will be built in the open with the metrics above applied honestly. Treating them properly matters more than getting to them quickly.
What is next
The next three posts, all in PyTorch:
- Tensors, time-ordered batching, and why leakage hides in
.shift(). How to turn a table of loans or transactions into training examples without letting the outcome into the features, and the one-line mistakes that do exactly that. - A first probability-of-default baseline. Logistic regression in PyTorch on a public credit dataset, a naive benchmark to beat, a custom loss for class imbalance, and an honest report using the tables above. The result will be humble. That is the point.
- Purged walk-forward validation. The splitting scheme every later post uses, with the purge and embargo explained properly.
After that, fraud detection with late-arriving labels, the anomaly layer of an AML pipeline and how to measure it when almost nothing is labelled, and then a short market track: a return-prediction baseline and the ranking and strategy metrics applied honestly.
Before then, a question for you. Which finance metric do you trust least, and why? Mine is ROC-AUC quoted for a fraud model without the base rate next to it.
Summary to save
- Financial data is not independent, not stationary, and mostly noise. Every method in this series is chosen with that in mind.
- Machine learning learns rules from data instead of hard-coding them. In finance the hard part is not the model but the data, the target, and the evaluation.
- Split by time, never at random. Leave a gap. Use point-in-time data.
- Credit, fraud, and AML labels are rare and arrive late. Accuracy lies on them. Prefer PR-AUC and log loss, and check calibration with Brier.
- Measure what the business feels: bad rate at the cut-off, good customers declined per fraud caught, alerts per confirmed case.
- You only see defaults on loans you approved. Your training data is missing the riskiest people it will be asked to score.
- Pair one statistical metric with one business or economic metric, out of time. Trust neither in isolation.
- All future code in this series is PyTorch, because finance research needs flexible models and custom losses more than it needs speed.