Module Directory

Machine learning has become a prominent tool in data analytics. One major category of it: the reinforcement learning, has been widely used in industry to maximise the notion of cumulative reward. This module is concerned with the conceptual background of reinforcement learning, i.e. Markov decision process (MDP) and dynamic programming.

Reinforcement learning, has been covered under Dynamic Programming for decades, designed on the divide-and-conquer basis, provide the structure to all the methods that have been developed in recent years. Describing the problem status by stages, states and transition matrices, but allowing decisions in the whole process.

The aims of this module are:

• To teach the basic mathematical concepts behind dynamic programming and reinforcement learning.


• To link the solution approach with statistics, computing, simulation, calculus and standard optimization techniques.

By the end of this module, students will be expected to have obtained:

1. The ability to formulate sequential decision problems, write Bellman’s equations and convert to the desired form of policy and/or value iteration.


2. A basic understanding of the advantages and limitations of different formulations and algorithms.


3. Awareness of concepts such as “curse of dimensionality” and “exploration/exploitation trade-off”. As well as understanding the difference between exact and approximate methods, and when to use each one of them.


4. An understanding of techniques such as backward induction and be able to write pseudo codes to implement them on small scale problems.


5. An ability to evaluate different  methodologies/approaches/software packages to solve typical applications.

Dynamic programming is both a mathematical optimisation method and algorithmic paradigm based on the concept of breaking down a problem into several sub-problems, aiming to find an overall optimal solution. In this module, said solution will be either a ‘value function’ that represents how good/bad are different states and actions; or a ’policy’ that recommends the best action to take, given the current information.

To use dynamic programming we require to have full knowledge of the dynamics of the problem (for example, knowing the probability that an autonomous car will not crash if it turns right at any point in time). These assumptions rarely hold in real-life problems, and to tackle those issues we will explore other methods that do not need such information, and just consider agents that learn by experience (trial & error).

We will also learn about modern reinforcement learning approaches and typical applications, that are designed for large scale problems and have shown excellent results in recent years. A very good example is the algorithm developed by DeepMind based on Deep Reinforcement Learning and Monte Carlo Tree Search that beat professional Go players. Currently, most of the Large Language Models (LLMs such as ChatGPT) use some version of Reinforcement Learning (from human feedback: RLHF).

The theory behind these problems and all the required concepts will be discussed in the Lectures, and complemented with ad-hoc experiments in the Laboratory Sessions that will build upon libraries such as OpenAI Gym and the use of Python.

Indicative Syllabus

• Basics of sequential decision process: stage, state, action, objective, policy, etc. Applications to Multi-armed bandits.


• Markov Decision Process (MDP), Dynamic Programming and the Bellman Equation.


• Algorithmic concepts: Value iteration, (general) Policy iteration.


• Reinforcement learning techniques: Monte Carlo Methods, Temporal difference methods (Q-Learning and SARSA).


• Approximate methods: Deep Q-Networks, Policy Gradient methods.


• Convergence/Divergence and the Deadly Triad.


• Applications: Revenue Management, Inventory Optimisation, Energy Auctions, Large Language Models, etc.

Teaching in the School will be delivered using a range of face to face lectures, classes and lab sessions as appropriate for each module. Modules may also include online only sessions where it is advantageous, for example for pedagogical reasons, to do so.