Researcher course in machine learning
Reinforcement Learning for Trading
Reinforcement learning applied to sequential trading decisions: Markov decision processes, dynamic programming, Q-learning and policy gradient methods, and their use in optimal execution, market making and portfolio allocation, with the simulation and evaluation problems that limit them.
Reinforcement learning trains an agent to make a sequence of decisions by trial and error, guided by rewards. Trading has many sequential decisions: how fast to sell a large position, where to quote, when to rebalance. That makes reinforcement learning for trading a natural fit in principle. In practice, its success depends on the quality of the simulator the agent learns in.
Where is reinforcement learning used in trading?
Its most credible applications are in execution and market making. In both, the problem is well defined, the reward is clear, and analytical models give a benchmark to beat. Applying it to choose which assets to buy, from historical prices alone, is much harder, because the agent can easily learn the quirks of a short history.
Why is evaluating an RL trading agent hard?
The agent learns in a simulator, and a simulator built from historical data does not react to the agent’s own trades. An agent can learn to exploit that, producing results that disappear in a real market. The course compares replay simulators with agent-based ones and shows how to test whether results depend on the simulator.
Who is this course for?
Quant researchers and machine learning practitioners working on execution, market making or allocation. It counts toward the Certificate in Machine Learning for Trading.
What you will be able to do
- Formulate a trading problem as a Markov decision process.
- Implement tabular and function-approximation Q-learning.
- Implement a policy gradient method and explain its variance problems.
- Train an agent for optimal execution and compare it with Almgren-Chriss.
- Evaluate an RL trading agent honestly against simulation bias.
Syllabus, module by module
Sequential decisions
States, actions, rewards and why trading problems are sequential.
Markov decision processes
Bellman equations, value iteration and policy iteration.
Q-learning
Tabular methods, exploration, and function approximation with neural networks.
Policy gradients
REINFORCE, actor-critic methods and variance reduction.
Optimal execution
An RL agent for liquidating a position, benchmarked against Almgren-Chriss.
Market making
Learning quotes with inventory penalties, compared with Avellaneda-Stoikov.
Simulators and evaluation
Market replay versus agent-based simulators, market impact, and overfitting to the simulator.
Assessment preparation
Supervised work on the written assessment.
How the certificate is earned
A written paper-style assessment with a computational component: you train an agent for an execution or market making problem in a simulator, compare it with a model-based benchmark, and write up the design and results. On a pass you receive the Course certificate in Reinforcement Learning for Trading, issued by the Institute with a verification code. Read how certification works.
Before you start
The material of Machine Learning for Trading and Execution Algorithms, probability, and fluent Python. Some familiarity with neural networks is assumed.
This course counts toward
Questions applicants ask
Does reinforcement learning work for trading?
Its strongest results are in problems with a clear structure and a good simulator, such as optimal execution and market making. For predicting prices to pick trades, it faces the same data limits as other machine learning and adds problems of its own. The course covers both.
Why compare the agent with a model-based benchmark?
Because an agent that does not beat a known analytical solution, such as Almgren-Chriss for execution, has not learned anything useful. The comparison is the minimum test of whether reinforcement learning added value.