Module 7: Reinforcement Learning


Executive Summary

  • Topics
    • We will do a very broad overview of reinforcement learning in this module
    • If you love the topic, the ML class does a lot more detail on RL.  This module is designed to get you excited about the topics and familiar with the ideas.  
  • Length: This module will take one week to complete
  • Assigned chapters: Chapter 22
    • Optional reading: If you really like reinforcement learning, the full book on reinforcement learning is online for free here.  Highly recommended!

Reinforcement Learning

Reinforcement learning (RL) is a different kind of machine learning than what we have been studying so far.  In the introduction to ML module, we talked about the different types of ML and we mostly have been studying supervised learning methods with a handful of unsupervised methods.  RL is very different because the external teacher provides evaluative feedback but does not provide the right answer.  This is perfect for situations such as games (where the goal is usually to win but the path to winning is not known) and robotics (where the goal is usually able to be specified but the exact sequence of motors to turn on is difficult to pre-specify).  

This module will give a quick overview of the main idea behind RL along with an overview of some of the popular RL learning methods.  If you are interested in learning more, I linked to both the RL book (available online for free) and/or you could take the ML class which covers RL in more depth.

Deep RL is a relatively new area where deep learning methods are combined with RL, typically for playing complicated games.  Deep RL is used in Alpha Zero and Alpha Star.  It has also been used successfully for robotics.  Although the full extent of deep RL is beyond the scope of the class, we will have a brief topic below explaining how Alpha Zero ties together adversarial search (minimax and Monte Carlo tree search) with RL and deep learning.

An RL Agent Learning to Ride a Bicycle 

RL bicycle path (and book cover)

The cover of the RL book shows the tracks of an RL agent learning to ride a bicycle