Imagine you studied for a spelling test using the exact same list of words that would appear on the test. You would probably score 100% — but that score would not tell anyone whether you are actually good at spelling. It would only tell them you are good at memorising one specific list. Cross-validation is the technique that stops machine learning models from making this same mistake.

In machine learning, a model “learns” by studying a set of examples — this is called training data. Once it has learned, we need to check whether it actually understood the underlying patterns, or whether it simply memorised the answers. Cross-validation is a family of techniques for splitting up the available data into different training and testing groups, again and again, so the model gets evaluated fairly and honestly.

The basic idea is simple: instead of testing a model just once on one slice of data, you test it several times on several different slices, and then average the results. This gives a much more trustworthy estimate of how the model will behave on completely new data it has never encountered — which is the entire point of building a model in the first place.

Cross-validation is used almost everywhere in machine learning: choosing which algorithm to use, tuning settings called hyperparameters, comparing different feature sets, and producing the final performance number that gets reported in a research paper, a dashboard, or a business decision. Without it, almost any claim about how “accurate” a model is would be unreliable.

A model that looks brilliant during testing but fails badly once it meets real-world data is not just disappointing — in fields like medicine, banking, and self-driving cars, it can be dangerous. Cross-validation exists to catch this problem before it ever reaches the real world.

Two problems sit at the heart of why cross-validation is so important: overfitting and underfitting. An overfit model has essentially memorised the training data, including its noise and quirks, so it performs brilliantly on data it has already seen but poorly on anything new. An underfit model never learned the patterns well enough in the first place, so it performs poorly everywhere. Cross-validation helps reveal both problems by repeatedly testing a model on data that was deliberately withheld from training.

Cross-validation also matters because it gives data scientists a fair way to compare different models against each other. If Model A scores 91% on one random slice of data and Model B scores 89% on a different slice, it is impossible to say which model is genuinely better — the scores might just reflect which slice happened to be easier. Cross-validation removes this unfairness by testing every model under the exact same repeated conditions.

Before cross-validation strategies became standard, the simplest approach was the “hold-out” method — and it is still a useful starting point for understanding why more advanced strategies were invented.

The hold-out method works by splitting the dataset just once: a chunk (commonly around 70–80%) becomes the training set, and the rest (commonly 20–30%) becomes the testing set. The model learns only from the training portion and is then scored once on the testing portion it never saw.

The hold-out method is fast and easy to understand, which is why it is often used for very large datasets or for quick first experiments. But it has a serious weakness: the test result depends heavily on which particular examples happened to end up in the test slice. Run the split again with a different random seed, and the reported accuracy can shift noticeably — sometimes enough to change which model looks “best.”

This single weakness — an unstable score caused by an unlucky or lucky split — is the exact problem that the rest of this guide is dedicated to solving.

K-Fold is the technique most people mean when they say “cross-validation.” It is the workhorse of machine learning evaluation, and almost every other strategy in this guide is a variation built on top of its core idea.

In K-Fold cross-validation, the dataset is divided into K roughly equal-sized chunks, called folds (a common choice is K = 5 or K = 10). The model is then trained and tested K separate times. In each round, one fold is set aside as the test set, while the remaining K−1 folds are combined and used for training. After K rounds, every single example in the dataset has been used for testing exactly once — and for training in all the other rounds.

How the Final Score Is Calculated

Each of the K rounds produces its own accuracy (or other metric) score. The final reported number is usually the simple average of all K scores, often paired with the standard deviation, which tells you how much the scores varied from round to round. A small standard deviation suggests the model is consistently reliable; a large one suggests its performance is unstable and sensitive to which data it happens to see.

Choosing the Value of K

There is no single “correct” K, but some values are used far more often than others in practice:

• K = 5: A common, balanced default — fast to compute, and usually stable enough for medium-sized datasets.
• K = 10: Slightly more thorough and slightly slower; widely recommended in research literature as a strong general-purpose choice.
• Smaller K (e.g. 3): Faster, but each training fold has noticeably less data, which can make the model perform slightly worse than it would with more data.
• Larger K (closer to the dataset size): More thorough and less biased, but dramatically more expensive to compute, since the model must be retrained K separate times.

Plain K-Fold has a hidden weakness: if your data is unevenly grouped, an ordinary random split can accidentally create folds that do not represent the dataset fairly. Stratified K-Fold was designed specifically to fix that.

Imagine a dataset for detecting a rare disease, where only 5% of patients actually have the condition. With plain random K-Fold, it is entirely possible that one fold ends up with almost no examples of the disease at all, while another fold ends up with an unusually high concentration of them. That imbalance would make the test scores wildly inconsistent and misleading from round to round.

Stratified K-Fold solves this by making sure that each fold contains roughly the same proportion of each class (or category) as the full dataset. If the overall dataset is 5% positive cases and 95% negative cases, every single fold will also be approximately 5% positive and 95% negative.

Stratified K-Fold is especially important for classification problems with imbalanced classes — fraud detection, rare disease diagnosis, spam filtering, and manufacturing defect detection are all classic examples where the “interesting” category is a small minority of the data. Using plain (non-stratified) K-Fold on such datasets can produce evaluation scores that swing wildly between rounds purely due to chance, making it hard to trust any single number.

What happens if you push K-Fold to its absolute extreme — making K equal to the total number of examples in the dataset? That extreme case has its own name: Leave-One-Out Cross-Validation.

Leave-One-Out Cross-Validation (LOOCV) trains the model on every single example except one, tests it on that one leftover example, and repeats this process once for every example in the dataset. If a dataset has 200 rows, LOOCV trains and evaluates the model 200 separate times, each time leaving out a different single example.

LOOCV is useful mainly when a dataset is very small — for example, in some medical or scientific studies where collecting more data is expensive or impossible, and every data point is precious. In those situations, “wasting” a large chunk of data on testing (as a normal train/test split would) is simply not affordable, so LOOCV’s “use almost everything for training” approach becomes attractive despite its computational cost.

Leave-P-Out Cross-Validation

A related but more general technique is Leave-P-Out Cross-Validation (LPOCV), where instead of leaving out just one example per round, you leave out every possible combination of P examples. When P = 1, this is identical to LOOCV. As P grows, the number of possible combinations explodes extremely quickly — even a small dataset with P = 3 can require thousands or millions of training rounds — which is why LPOCV is rarely used in practice beyond very small values of P and very small datasets.

Ordinary K-Fold makes a dangerous assumption: that it is fine to shuffle data randomly. But for anything involving time — stock prices, weather, website traffic, sensor readings — shuffling the data before splitting is a serious mistake that can make a model look far better than it really is.

Imagine training a model to predict tomorrow’s stock price using data from next month — that would obviously be cheating, since in real life you could never know next month’s prices in advance. Yet plain random K-Fold can accidentally do exactly this kind of cheating with time-based data, because it shuffles rows without any regard for which came first.

Time Series Cross-Validation (sometimes called “rolling-window” or “walk-forward” validation) fixes this by always keeping the training data earlier in time than the test data. A common version, sometimes called expanding window validation, looks like this:

This approach is essential for forecasting tasks: predicting sales, demand, weather, traffic, energy usage, or any sequence where the order of events genuinely matters. Skipping it and using ordinary K-Fold instead is one of the most common — and most damaging — mistakes made by newcomers working with time-based data.

Two more advanced strategies solve two very different problems: one stops a model from “cheating” using related records, and the other stops a model’s evaluation score from being secretly inflated during tuning.

Group K-Fold

Sometimes a dataset contains multiple rows that belong to the same underlying real-world entity — for example, ten different photos of the same patient’s X-ray, or fifty different purchase records from the same customer. If ordinary K-Fold randomly scatters these related rows across both the training and test folds, the model can effectively “recognise” that entity from training and unfairly ace the test — not because it learned a real pattern, but because it secretly saw a near-duplicate of the test example during training.

Group K-Fold fixes this by making sure that all rows belonging to the same group (the same patient, the same customer, the same user) always stay together, either entirely in the training set or entirely in the test set, never split between the two.

Nested Cross-Validation

When tuning a model’s hyperparameters (its internal settings), it is tempting to use cross-validation to both pick the best settings and report the final score — but doing both with the same data tends to produce an overly optimistic, slightly dishonest final number, because the settings were chosen specifically to perform well on that exact data.

Nested Cross-Validation solves this with two loops running inside each other: an inner loop that searches for the best hyperparameters using only the inner training data, and an outer loop that evaluates the resulting model on data the inner loop never touched at all. This produces a far more honest, unbiased final performance estimate, at the cost of significantly more computation.

Nested cross-validation is considered best practice whenever a model’s hyperparameters are being tuned and the resulting score needs to be reported as a trustworthy, publishable, or business-critical number — for example, in academic research papers or regulated industries like healthcare and finance.

Even K-Fold itself depends on a single random way of dividing the data into folds. Two further strategies exist for squeezing out even more reliability by repeating the whole process with different random divisions.

These repeated approaches cost more computing time than a single round of K-Fold, but they produce a more stable, trustworthy estimate of model performance — which matters most when a dataset is small, when results from a single K-Fold run seem to vary a lot, or when the final number will be used to make an important decision.

With so many strategies available, picking the right one comes down to asking a few simple questions about your data and your goals.

Cross-validation is enormously valuable, but it is not free — it trades extra computing time for extra trustworthiness. Knowing both sides helps you use it wisely.

Cross-validation is simple in concept but surprisingly easy to get wrong in practice. Most of these mistakes have the same underlying theme: information from the test set quietly “leaking” into training.

1. Preprocessing before splitting: Scaling, normalising, or selecting features using the whole dataset before splitting into folds lets information from the test fold influence the training fold — always fit preprocessing steps only on the training portion of each fold.
2. Using plain K-Fold on time-based data: As covered in Section 7, this allows the model to unintentionally peek into the future, producing scores that look great in testing but collapse in real deployment.
3. Ignoring grouped or duplicated records: Letting rows from the same person, device, or session appear in both train and test folds (Section 8) inflates scores artificially.
4. Tuning and reporting on the same folds: Selecting the best hyperparameters using cross-validation and then reporting that same score as the final result, without an outer nested loop, tends to be overly optimistic.
5. Forgetting to shuffle when appropriate: For ordinary, non-time-based data, forgetting to shuffle before splitting into folds can accidentally create folds biased by how the data happened to be sorted or collected.
6. Choosing K purely by habit: Defaulting to K = 5 or K = 10 without considering dataset size or class balance can lead to unnecessarily noisy or slow evaluation.

As datasets and models keep growing larger and more complex, the way researchers evaluate models is evolving too.

Sources & Further Reading