Unsupervised and Reinforcement Learning

• Weekly hours
• Competences
• Objectives
• Contents
• Activities
• Teaching methodology
• Evaluation methodology
• Bibliography
• Previous capacities

Teachers

Person in charge

• Javier Béjar Alonso ( )
• Mario Martín Muñoz ( )

Weekly hours

Competences

Transversal Competences

Transversals

• CT6 [Avaluable] - Autonomous Learning. Detect deficiencies in one's own knowledge and overcome them through critical reflection and the choice of the best action to extend this knowledge.

Basic

• CB5 - That the students have developed those learning skills necessary to undertake later studies with a high degree of autonomy

Technical Competences

Especifics

• CE18 - To acquire and develop computational learning techniques and to design and implement applications and systems that use them, including those dedicated to the automatic extraction of information and knowledge from large volumes of data.

Generic Technical Competences

Generic

• CG2 - To use the fundamental knowledge and solid work methodologies acquired during the studies to adapt to the new technological scenarios of the future.
• CG4 - Reasoning, analyzing reality and designing algorithms and formulations that model it. To identify problems and construct valid algorithmic or mathematical solutions, eventually new, integrating the necessary multidisciplinary knowledge, evaluating different alternatives with a critical spirit, justifying the decisions taken, interpreting and synthesizing the results in the context of the application domain and establishing methodological generalizations based on specific applications.

Objectives

1. To distinguish the kind of problems can be modeled as a reinforcement learning problem and identify the techniques that can be applied to solve them.
   Related competences: CG2, CT6, CE18,
2. To understand the need, fundamentals, and particularities of behavior learning and the differences it has from supervised and unsupervised machine learning.
   Related competences: CG2, CE18,
3. To understand the most important algorithms and state of the art in the area of learning by reinforcement
   Related competences: CG4, CE18,
4. To know how to computationally formalize a real world problem as learning by reinforcement and know how to implement in the most current environments the learning algorithms that solve them
   Related competences: CG2, CG4, CT6, CE18,
5. Know the problems that can be modeled with deep unsupervised algorithms
   Related competences: CG2, CT6, CE18,
6. Understand the particularities of deep unsupervised algorithms
   Related competences: CG4, CT6, CE18,
7. Know the most important algorithms and the state of the art of deep unsupervised learning
   Related competences: CG2, CT6, CB5, CE18,
8. Knowing how to implement and apply deep learning algorithms to a problem using the most current environment
   Related competences: CG2, CT6, CB5, CE18,

Contents

1. Introduction: Behavior Learning in Agents and description of main elements in Reinforcement Learning
   Intuition, motivation and definition of the reinforcement learning (RL) framework. Key elements in RL.
2. Finding optimal policies using Dynamic Programming
   How to learn the optimal policy with full knowledge of the world model: algebraic solution, policy iteration and value iteration.
3. Introduction to Model-Free approaches.
   Basic algorithms for reinforcement learning: Monte-Carlo, Q-learning, Sarsa, TD(lambda). The need for Exploration. Differences between On-policy and Off-policy methods.
4. Function approximation in Reinforcement Learning
   Need for function approximation and Incremental methods in RL. The Gradient Descent approach. RL with Linear function approximation. The deadly triad for function approximation in RL. Batch methods and Neural Networks for function Approximation.
5. Deep Reinforcement Learning (DRL)
   Introducing Deep Learning in RL. Dealing with the deadly triad with the DQN algorithm. Application to the Atari games case. Evolutions of the DQN algorithm: Double DQN, Prioritized Experience Replay, multi-step learning and Distributional value functions. Rainbow: the state-of-the-art algorithm in discrete action space.
6. Policy gradient methods
   What to do in continuous action spaces. How probabilistic policies allow to apply the gradient method directly in the policy network. The REINFORCE algorithm. The Actor-Critic algorithms. State-of-the-art algorithms in continuous action spaces: DDPG, TD3 and SAC.
7. Advanced Topics: How to deal with sparse rewards
   The problem of the sparse reward. Introduction to advanced exploration techniques: curiosity and empowerment in RL. Introduction to curriculum learning to easy the learning of the goal. Hierarchical RL to learn complex tasks. The learning of Universal Value Functions and Hindsight Experience Replay (HER).
8. Reinforcement Learning in the multi-agent framework
   Learning of behaviors in environment where several agents act. Learning of cooperative behaviors, Learning of competitive behaviors, and mixed cases. State-of-the art algorithms. The special case of games: The AlfaGo case and the extension to Alfa-Zero.
9. Introduction: Deep unsupervised learning
   Introduction to the need for deep unsupervised learning and its applications
10. Autoregressive models
    Introduction to learning probability distributions defined as autoregressive distributions and main models
11. Normalizing flows
    Introduction to normalized flows for learning probability distributions
12. Latent variables models
    Introduction to models based on latent variables and variational autoencoders
13. Generative Adversarial Networks
    Introduction to generative adversarial networks, conditional and unconditional generation, attribute disentanglement
14. Denoising Diffusion netwoks
    Introduction to models based on noise diffusion, denoising networks, conditioning, multimodal generation
15. Self supervised learning
    Introduction to self-supervised learning for feature-generating networks and embeddings, contrastive and non contrastive methods, masking

Activities

Activity Evaluation act

Teaching methodology

Evaluation methodology

Bibliography

Basic:

• Reinforcement learning : an introduction - Sutton, Richard S; Barto, Andrew G, The MIT Press, [2020]. ISBN: 9780262039246
  https://discovery.upc.edu/discovery/fulldisplay?docid=alma991004166329706711&context=L&vid=34CSUC_UPC:VU1&lang=ca
• Grokking deep reinforcement learning - Morales, Miguel, Manning Publications, 2020. ISBN: 9781617295454
  https://discovery.upc.edu/discovery/fulldisplay?docid=alma991004208939706711&context=L&vid=34CSUC_UPC:VU1&lang=ca
• Generative deep learning: teaching machines to paint, write, compose, and play - Foster, D, O'Reilly Media, Incorporated, 2023. ISBN: 9781098134143
  
• Hands-on image generation with TensorFlow: a practical guide to generating images and videos using deep learning - Cheong, S.Y, Packt Publishing, 2020. ISBN: 9781838821104
  

Complementary:

• Deep reinforcement learning in action - Zai, Alexander; Brown, Brandon, Manning Publications Co , 2020. ISBN: 9781617295430
  https://discovery.upc.edu/discovery/fulldisplay?docid=alma991004203829706711&context=L&vid=34CSUC_UPC:VU1&lang=ca
• Generative AI with Python and TensorFlow 2: harness the power of generative models to create images, text, and music - Babcock, J.; Bali, R, Packt Publishing , 2021. ISBN: 9781800208506
  

Previous capacities