Backtesting: Methods, Pitfalls and Validation

Backtesting is the cheapest way to reject a trading idea and the easiest way to fall in love with a bad one. Traders on public forums say it bluntly: most backtests they have seen were useless. This course is about the methods that make a backtest worth reading.

What are the most common backtesting mistakes?

Using information that was not available at the time, such as revised data or a closing price to trade at the close. Testing only on companies that still exist today. Assuming orders fill at prices that would never have been offered. And trying hundreds of variations, then reporting the best as if it were the only one. Each has a fix, and the course works through all of them.

How do you know if a backtest is overfit?

There is no single test, but there are strong signs: performance that collapses out of sample, parameters that only work in a narrow range, results that depend on a few trades. The course adds a statistical answer, adjusting the Sharpe ratio for the number of trials, so you can say how likely a result is to be luck.

Who is this course for?

Anyone who tests trading strategies, from individuals to research teams.

What you will be able to do

• Find and remove look-ahead, survivorship and selection bias in a backtest.
• Model fills, costs and slippage appropriate to the strategy's horizon.
• Design walk-forward and purged cross-validation tests.
• Adjust performance statistics for the number of strategies tried.
• Write a validation report that a reviewer can check.

Syllabus, module by module

1. Why backtests mislead

   The gap between published backtests and live results, and its main causes.

2. Data biases

   Look-ahead from revised and timestamped data, survivorship, and point-in-time universes.

3. Execution realism

   Fill assumptions on bars, limit-order fills, slippage models and market impact.

4. Overfitting

   Parameter search, degrees of freedom, and why a smooth equity curve is a warning.

5. Validation designs

   Holdout sets, walk-forward analysis, and purged and embargoed cross-validation.

What every course contains

Every paid course comes with the first seven and the certificate, whichever level it sits at. The proctored exam and the credential come when the course is taken as part of a Certificate, Diploma or Fellowship. Everything is self-paced: the material is yours to study in any order, at any hour, for the whole of the license.

1. Workbooks. Written to train the thinking patterns and the important concepts of the course, with worked problems and the reasoning behind each answer, so that you learn to think the way the subject is thought.
2. Audio edition. Focused, well-presented recordings with explainers and examples of the material, made to be interesting, relevant and entertaining, so that the general subject can be learned away from the desk.
3. Notes. Pre-produced notes with every fact the course relies on, written out in full, so that nothing has to be transcribed from the films or the audio.
4. Cheat sheet. One sheet with every important learning concept of the course on it, to keep beside you during the work and to revise from before the exam.
5. Named sources. Every fact in the notes names its source, the paper, the book or the dataset, for anyone who wants to dig in further.
6. Explainer films. Focused films, in the style of the sample films on the catalog page, that walk through the most difficult concepts of the course with visual explanations of a high standard.
7. Transcript. A full transcript of every film and every audio recording, searchable and quotable.
8. Proctored exam. Taken as part of a Certificate, Diploma or Fellowship, the course ends in a proctored multiple-choice questionnaire, with a pass mark of 60% and three retakes included. A course taken on its own is assessed by its coursework and has no exam.
9. Verifiable certificate. A certificate of completion that a third party can verify: an employer sends the code on it to the credentials office and receives written confirmation of the course, the date and the result.
10. Credential of the program. The Certificate, Diploma or Fellowship of the program, track or path the course belongs to, awarded when every course in it is passed. Which one depends on the program you choose.

How the certificate is earned

An assessed project: you receive a flawed backtest, find and correct its biases, re-run it under a walk-forward design, and submit the corrected code with a validation report. On a pass you receive the Course certificate in Backtesting, issued by the Institute with a verification code. A course taken as part of a Certificate, Diploma or Fellowship ends in a proctored online exam with a pass mark of 60%. The fee includes three retakes; each attempt after that carries a reduced proctored exam fee. A course taken on its own is assessed by its coursework and has no exam. Read how certification works.

Before you start

This course assumes the material of the courses below, or the same knowledge gained elsewhere.

• Algorithmic Trading Foundations: From Idea to Paper to Live
• Probability for Quantitative Finance, with Statistics