Login
OneManager
Librarys
Share
Home
/
Librarys
/
量化学习笔记
/
Book-Mathematical-Foundation-of-Reinforcement-Learning-main
File
CopyAllDownloadUrl
EditTime
Size
Code for grid world
2025-05-27 13:36:58
292.7 KB
Lecture slides
2025-05-27 13:36:58
14.38 MB
1 - Table of contents.pdf
2025-05-16 07:15:12
102.32 KB
2 - Overview of this book.pdf
2025-05-16 07:15:12
278.58 KB
2 - Preface.pdf
2025-05-16 07:15:12
99.63 KB
3 - Chapter 1 Basic Concepts.pdf
2025-05-16 07:15:12
6.48 MB
3 - Chapter 10 Actor-Critic Methods.pdf
2025-05-16 07:15:12
658.19 KB
3 - Chapter 2 State Values and Bellman Equation.pdf
2025-05-16 07:15:12
811.99 KB
3 - Chapter 3 Optimal State Values and Bellman Optimality Equation.pdf
2025-05-16 07:15:12
943.43 KB
3 - Chapter 4 Value Iteration and Policy Iteration.pdf
2025-05-16 07:15:12
1.54 MB
3 - Chapter 5 Monte Carlo Methods.pdf
2025-05-16 07:15:12
2.93 MB
3 - Chapter 6 Stochastic Approximation.pdf
2025-05-16 07:15:12
736.36 KB
3 - Chapter 7 Temporal-Difference Methods.pdf
2025-05-16 07:15:12
1.69 MB
3 - Chapter 8 Value Function Methods.pdf
2025-05-16 07:15:12
2.14 MB
3 - Chapter 9 Policy Gradient Methods.pdf
2025-05-16 07:15:12
563.59 KB
4 - Appendix.pdf
2025-05-16 07:15:12
276.28 KB
a.md
2025-05-29 14:13:19
289 B
Book-all-in-one.pdf
2025-05-16 07:15:12
18.98 MB
Figure_chapterMap.png
2025-05-16 07:15:12
394.32 KB
Figure_ChineseBookCover.png
2025-05-16 07:15:12
1004.6 KB
Figure_EnglishLectureVideo.png
2025-05-16 07:15:12
69.17 KB
Readme.md
2025-05-16 07:15:12
19.33 KB
springerBookCover.png
2025-05-16 07:15:12
131.28 KB
# (May 2025) 9,000+ stars! This textbook has received 9,000+ stars! Glad that it is helpful to many readers. # (Mar 2025) English lecture videos completed! [](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&si=B6mRR7vxBAjRAm_F) **My English open course is online now.** You can click the above figure or the [link here](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&si=D1T4pcyHsMxj6CzB) to jump to our YouTube channel. You can also click the following links to be directed to specific lecture videos. You are warmly welcome to check out the English videos to help your learning. - [Overview of Reinforcement Learning in 30 Minutes](https://www.youtube.com/watch?v=ZHMWHr9811U&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=1) - [L1: Basic Concepts (P1-State, action, policy, ...)](https://www.youtube.com/watch?v=zJHtM5dN69g&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=2) - [L1: Basic Concepts (P2-Reward,return, Markov decision process)](https://www.youtube.com/watch?v=repVl3_GYCI&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=3) - [L2: Bellman Equation (P1-Motivating examples)](https://www.youtube.com/watch?v=XCzWrlgZCwc&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=4) - [L2: Bellman Equation (P2-State value)](https://www.youtube.com/watch?v=DSvi3xEN13I&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=5) - [L2: Bellman Equation (P3-Bellman equation-Derivation)](https://www.youtube.com/watch?v=eNtId8yPWkA&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=6) - [L2: Bellman Equation (P4-Matrix-vector form and solution)](https://www.youtube.com/watch?v=EtCfBG_eP2w&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=7) - [L2: Bellman Equation (P5-Action value)](https://www.youtube.com/watch?v=zJo2sLDzfcU&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=8) - [L3: Bellman Optimality Equation (P1-Motivating example)](https://www.youtube.com/watch?v=lXKY_Hyg4SQ&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=9) - [L3: Bellman Optimality Equation (P2-Optimal policy)](https://www.youtube.com/watch?v=BxyjdHhK8a8&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=10) - [L3: Bellman Optimality Equation (P3-More on BOE)](https://www.youtube.com/watch?v=FXftTCKotC8&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=11) - [L3: Bellman Optimality Equation (P4-Interesting properties)](https://www.youtube.com/watch?v=a--bck2ow9s&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=12) - [L4: Value Iteration and Policy Iteration (P1-Value iteration)](https://www.youtube.com/watch?v=wMAVmLDIvQU&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=13) - [L4: Value Iteration and Policy Iteration (P2-Policy iteration)](https://www.youtube.com/watch?v=Pka6Om0nYQ8&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=14) - [L4: Value Iteration and Policy Iteration (P3-Truncated policy iteration)](https://www.youtube.com/watch?v=tUjPFPD3Vc8&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=15) - [L5: Monte Carlo Learning (P1-Motivating examples)](https://www.youtube.com/watch?v=DO1yXinAV_Q&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=16) - [L5: Monte Carlo Learning (P2-MC Basic-introduction)](https://www.youtube.com/watch?v=6ShisunU0zs&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=17) - [L5: Monte Carlo Learning (P3-MC Basic-examples)](https://www.youtube.com/watch?v=axA0yns9FxU&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=18) - [L5: Monte Carlo Learning (P4-MC Exploring Starts)](https://www.youtube.com/watch?v=Qt8OMHPkLqg&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=19) - [L5: Monte Carlo Learning (P5-MC Epsilon-Greedy-introduction)](https://www.youtube.com/watch?v=dM3fYE630pY&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=20) - [L5: Monte Carlo Learning (P6-MC Epsilon-Greedy-examples)](https://www.youtube.com/watch?v=x6X_5ePT9gQ&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=21) - [L6: Stochastic Approximation and SGD (P1-Motivating example)](https://www.youtube.com/watch?v=1bMgejvWoAo&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=22) - [L6: Stochastic Approximation and SGD (P2-RM algorithm: introduction)](https://www.youtube.com/watch?v=1FTGcNUUnCE&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=23) - [L6: Stochastic Approximation and SGD (P3-RM algorithm: convergence)](https://www.youtube.com/watch?v=juNDoAFEre4&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=24) - [L6: Stochastic Approximation and SGD (P4-SGD algorithm: introduction)](https://www.youtube.com/watch?v=EZO7Iadp5m4&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=25) - [L6: Stochastic Approximation and SGD (P5-SGD algorithm: examples)](https://www.youtube.com/watch?v=BsxU_4qvvNA&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=26) - [L6: Stochastic Approximation and SGD (P6-SGD algorithm: properties)](https://www.youtube.com/watch?v=fWxX9YuEHjE&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=27) - [L6: Stochastic Approximation and SGD (P7-SGD algorithm: comparison)](https://www.youtube.com/watch?v=yNEV2cLKuzU&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=28) - [L7: Temporal-Difference Learning (P1-Motivating example)](https://www.youtube.com/watch?v=u1X-7XX3dtI&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=29) - [L7: Temporal-Difference Learning (P2-TD algorithm: introduction)](https://www.youtube.com/watch?v=XiCUsc7CCE0&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=30) - [L7: Temporal-Difference Learning (P3-TD algorithm: convergence)](https://www.youtube.com/watch?v=faWg8M91-Oo&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=31) - [L7: Temporal-Difference Learning (P4-Sarsa)](https://www.youtube.com/watch?v=jYwQufkBUPo&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=32) - [L7: Temporal-Difference Learning (P5-Expected Sarsa & n-step Sarsa)](https://www.youtube.com/watch?v=0kKzQbWZOlk&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=33) - [L7: Temporal-Difference Learning (P6-Q-learning: introduction)](https://www.youtube.com/watch?v=4BvYR2hm730&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=34) - [L7: Temporal-Difference Learning (P7-Q-learning: pseudo code)](https://www.youtube.com/watch?v=I0YhlOIFF4s&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=35) - [L7: Temporal-Difference Learning (P8-Unified viewpoint and summary)](https://www.youtube.com/watch?v=3t74lvk1GBM&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=36) - [L8: Value Function Approximation (P1-Motivating example–curve fitting)](https://www.youtube.com/watch?v=uJXcI8fcdWc&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=37) - [L8: Value Function Approximation (P2-Objective function)](https://www.youtube.com/watch?v=Z3HI1TfpJP0&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=38) - [L8: Value Function Approximation (P3-Optimization algorithm)](https://www.youtube.com/watch?v=piBDwrKt0uU&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=39) - [L8: Value Function Approximation (P4-illustrative examples and analysis)](https://www.youtube.com/watch?v=VFyBNEZxMMs&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=40) - [L8: Value Function Approximation (P5-Sarsa and Q-learning)](https://www.youtube.com/watch?v=C-HtY4-W_zw&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=41) - [L8: Value Function Approximation (P6-DQN–basic idea)](https://www.youtube.com/watch?v=lZCcbZbqVSQ&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=42) - [L8: Value Function Approximation (P7-DQN–experience replay)](https://www.youtube.com/watch?v=rynEdAdebi0&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=43) - [L8: Value Function Approximation (P8-DQN–implementation and example)](https://www.youtube.com/watch?v=vQHuCHjd6hA&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=44) - [L9: Policy Gradient Methods (P1-Basic idea)](https://www.youtube.com/watch?v=mtFHOj83QSo&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=45) - [L9: Policy Gradient Methods (P2-Metric 1–Average value)](https://www.youtube.com/watch?v=la8jQc3hX1M&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=46) - [L9: Policy Gradient Methods (P3-Metric 2–Average reward)](https://www.youtube.com/watch?v=8RZ_rQFe69E&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=47) - [L9: Policy Gradient Methods (P4-Gradients of the metrics)](https://www.youtube.com/watch?v=MvmtPXur3Ls&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=48) - [L9: Policy Gradient Methods (P5-Gradient-based algorithms & REINFORCE)](https://www.youtube.com/watch?v=1DQnnUC8ng8&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=49) - [L10: Actor-Critic Methods (P1-The simplest Actor-Critic)](https://www.youtube.com/watch?v=kjCZAT5Wh80&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=50) - [L10: Actor-Critic Methods (P2-Advantage Actor-Critic)](https://www.youtube.com/watch?v=vZVXJJcZNEM&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=51) - [L10: Actor-Critic Methods (P3-Importance sampling & off-policy Actor-Critic)](https://www.youtube.com/watch?v=TfO5mnsiGKc&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=52) - [L10: Actor-Critic Methods (P4-Deterministic Actor-Critic)](https://www.youtube.com/watch?v=dTjz1RNtic4&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=53) - [L10: Actor-Critic Methods (P5-Summary and goodbye!)](https://www.youtube.com/watch?v=npvnnKcXoBs&list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8&index=54) *** *** # Why a new book on reinforcement learning? This book aims to provide a **mathematical but friendly** introduction to the fundamental concepts, basic problems, and classic algorithms in reinforcement learning. Some essential features of this book are highlighted as follows. - The book introduces reinforcement learning from a mathematical point of view. Hopefully, readers will not only know the procedure of an algorithm but also understand why it was designed in the first place and why it works effectively. - The depth of the mathematics is carefully controlled to an adequate level. The mathematics is also presented in a carefully designed manner to ensure that the book is friendly to read. Readers can selectively read the materials presented in gray boxes according to their interests. - Many illustrative examples are given to help readers better understand the topics. All the examples in this book are based on a grid world task, which is easy to understand and helpful for illustrating concepts and algorithms. - When introducing an algorithm, the book aims to separate its core idea from complications that may be distracting. In this way, readers can better grasp the core idea of an algorithm. - The contents of the book are coherently organized. Each chapter is built based on the preceding chapter and lays a necessary foundation for the subsequent one. [](https://link.springer.com/book/9789819739431)  # Contents The topics addressed in the book are shown in the figure below. This book contains ten chapters, which can be classified into two parts: the first part is about basic tools, and the second part is about algorithms. The ten chapters are highly correlated. In general, it is necessary to study the earlier chapters first before the later ones.  # Lecture videos By combining the book with my lecture videos, I believe you can study better. - **Chinese lecture videos:** You can check the [Bilibili channel](https://space.bilibili.com/2044042934) or the [Youtube channel](https://www.youtube.com/channel/UCztGtS5YYiNv8x3pj9hLVgg/playlists). - **English lecture videos:** The English lecture videos have been uploaded to YouTube. Please see the links and details in another part of this document. The lecture videos have received **1,500,000+ views** over the Internet and received very good feedback! # Readership This book is designed for senior undergraduate students, graduate students, researchers, and practitioners interested in reinforcement learning. It does not require readers to have any background in reinforcement learning because it starts by introducing the most basic concepts. If the reader already has some background in reinforcement learning, I believe the book can help them understand some topics more deeply or provide different perspectives. This book, however, requires the reader to have some knowledge of probability theory and linear algebra. Some basics of the required mathematics are also included in the appendix of this book. # About the author You can find my info on my homepage https://www.shiyuzhao.net (GoogleSite) and my research group website https://shiyuzhao.westlake.edu.cn I have been teaching a graduate-level course on reinforcement learning since 2019. Along with teaching, I have been preparing this book as the lecture notes for my students. I sincerely hope this book can help readers smoothly enter the exciting field of reinforcement learning. # Citation ``` @book{zhao2025RLBook, title={Mathematical Foundations of Reinforcement Learning}, author={S. Zhao}, year={2025}, publisher={Springer Nature Press and Tsinghua University Press} } ``` # Third-party code and materials Many enthusiastic readers sent me the source code or notes that they developed when they studied this book. If you create any materials based on course, you are welcome to write an email. I am happy to share the links here and hope they may be helpful to other readers. I must emphasize that I have not verified the code. If you have any questions, you can directly contact the developers. **Code** *Python:* - https://github.com/zhoubay/Code-for-Mathematical-Foundations-of-Reinforcement-Learning (Mar 2025, by Xibin ZHOU) - https://github.com/10-OASIS-01/minrl (Feb 2025) - https://github.com/SupermanCaozh/The_Coding_Foundation_in_Reinforcement_Learning (by Zehong Cao, Aug 2024) - https://github.com/ziwenhahaha/Code-of-RL-Beginning by RLGamer (Mar 2024) - Videos for code explanation: https://www.bilibili.com/video/BV1fW421w7NH - https://github.com/jwk1rose/RL_Learning by Wenkang Ji (Feb 2024) *R:* - https://github.com/NewbieToEverything/Code-Mathmatical-Foundation-of-Reinforcement-Learning *C++:* - https://github.com/purundong/test_rl **Study notes** *English:* - https://lyk-love.cn/tags/reinforcement-learning/ by a graduate student from UC Davis *Chinese:* - https://zhuanlan.zhihu.com/p/692207843 - https://blog.csdn.net/qq_64671439/category_12540921.html - http://t.csdnimg.cn/EH4rj - https://blog.csdn.net/LvGreat/article/details/135454738 - https://xinzhe.blog.csdn.net/article/details/129452000 - https://blog.csdn.net/v20000727/article/details/136870879?spm=1001.2014.3001.5502 - https://blog.csdn.net/m0_64952374/category_12883361.html There are also many others notes made by many other readers on the Internet. I am not able to put them all here. You are welcome to recommend to me if you find a good one. **Bilibili videos made based on my course** - https://www.bilibili.com/video/BV1fW421w7NH - https://www.bilibili.com/video/BV1Ne411m7GX - https://www.bilibili.com/video/BV1HX4y1H7uR - https://www.bilibili.com/video/BV1TgzsYDEnP - https://www.bilibili.com/video/BV1CQ4y1J7zu # Update history **(Apr 2025) 8,000+ stars!** This textbook has received 8,000+ stars! Glad that it is helpful to many readers. **(Mar 2025) 7,000+ stars!** This textbook has received 7,000+ stars! Glad that it is helpful to many readers. **(Feb 2025) 5,000+ stars** This textbook has received 5,000+ stars! Glad that it is helpful to many readers. **(Dec 2024) 4,000+ stars** This textbook has received 4,000+ stars! Glad that it is helpful to many readers. **(Oct 2024) Book cover** The design of the book cover is finished. The book will be officially published by Springer early next year. It has been published by Tsinghua University Press. **(Sep 2024) Minor update before printing by Springer** I revised some very minor places that readers may hardly notice. It is supposed to be the final version before printing by Springer. **(Aug 2024) 3000 Stars and more code** The book has received 3000+ stars, which is a great achievement to me. Thanks to everyone. Hope it really helped you. I also received more code implementation from enthusiastic readers. For example, this [GitHub page](https://github.com/SupermanCaozh/The_Coding_Foundation_in_Reinforcement_Learning) provided Python implementation of almost all examples in my book. On the one hand, I am very glad to see that. On the other hand, I am a little worried that my students in my offline class may use the code to do their homework:-). Overall, I am happy because it indicates that the book and open course are really helpful to the readers; Otherwise, they would not bother to develop the code by themselves:-) **(Jun 2024) Minor update before printing** This is the fourth version of the book draft. It is supposed to be the final one before the book is officially published. Specifically, when proofreading the book manuscript, I detected some very minor issues. Together with some reported by enthusiastic readers, they have been revised in this version. **(Apr 2024) Code for the Grid-World Environment** We added the code for the grid-world environment in my book. Interested readers can develop and test their own algorithms in this environment. Both Python and MATLAB versions are provided. Please note that we do not provide the code of all the algorithms involved in the book. That is because they are the homework for the students in offline teaching: the students need to develop their own algorithms using the provided environment. Nevertheless, there are third-party implementations of some algorithms. Interested readers can check the links on the home page of the book. I need to thank my PhD students, Yize Mi and Jianan Li, who are also the Teaching Assistants of my offline teaching. They contributed greatly to the code. You are welcome to provide any feedback about the code such as bugs if detected. **(Mar 2024) 2K stars** The book has received 2K stars. I also received many positive evaluations of the book from many readers. Very glad that it can be helpful. **(Mar 2024) Minor update** The third version of the draft of the book is online now. Compared to the second version, the third version is improved in the sense that some minor typos have been corrected. Here, I would like to thank the readers who sent me their feedback. **(Sep 2023) 1000+ stars** The book received 1000+ stars! Thank everybody! **(Aug 2023) Major update - second version** *The second version of the draft of the book is online now!!* Compared to the first version, which was online one year ago, the second version has been improved in various ways. For example, we replotted most of the figures, reorganized some contents to make them clearer, corrected some typos, and added Chapter 10, which was not included in the first version. I put the first draft of this book online in August 2022. Up to now, I have received valuable feedback from many readers worldwide. I want to express my gratitude to these readers. **(Nov 2022) Will be jointly published** This book will be published *jointly by Springer Nature and Tsinghua University Press*. It will probably be printed in the second half of 2023. I have received some comments and suggestions about this book from some readers. Thanks a lot, and I appreciate it. I am still collecting feedback and will probably revise the draft in several months. Your feedback can make this book more helpful for other readers! **(Oct 2022) Lecture notes and vidoes** The *lecture slides* have been uploaded in the folder "Lecture slides." The *lecture videos* (in Chinese) are online. Please check our Bilibili channel https://space.bilibili.com/2044042934 or the Youtube channel https://www.youtube.com/channel/UCztGtS5YYiNv8x3pj9hLVgg/playlists **(Aug 2022) First draft** The first draft of the book is online.
Close
2025-06-07 01:00:57 Saturday 216.73.216.22 Runningtime:0.007s Mem:495.21 KB
