Reinforcement Learning

For ownersHow a model learns from feedback and rewards. Mostly for engineers.

Agents that learn by doing. The foundation of RLHF.

12 lessons

Start with lesson 1

Lesson 1 of 12, ~45 minutesMDPs, States, Actions & Rewards

The lessons

  1. 01MDPs, States, Actions & RewardsA Markov Decision Process describes a decision problem as states, actions, results of actions, rewards and how much future rewards count. Most reinforcement learning is built on this frame.Learn · ~45 minutes
  2. 02Dynamic Programming — Policy Iteration & Value IterationDynamic programming solves a decision problem exactly when you already know all the rules and rewards, by repeatedly updating value estimates until they settle.Build · ~75 minutes
  3. 03Monte Carlo Methods — Learning from Complete EpisodesMonte Carlo methods learn how good actions are by running whole episodes, recording the total reward, and averaging the results. No model of the world is needed.Build · ~75 minutes
  4. 04Temporal Difference — Q-Learning & SARSAQ-learning and SARSA learn the value of each action after every step, using the next estimate as a stand-in for the future. Q-learning assumes the best next move, SARSA uses the move actually taken.Build · ~75 minutes
  5. 05Deep Q-Networks (DQN)A Deep Q-Network uses a neural network to estimate action values from raw inputs like screen pixels, with tricks such as replaying past experience and a slow-moving target network to keep training steady.Build · ~75 minutes
  6. 06Policy Gradient — REINFORCE from ScratchPolicy gradient methods skip estimating values and directly adjust the model's action choices to make high-reward outcomes more likely. REINFORCE is the simplest version.Build · ~75 minutes
  7. 07Actor-Critic — A2C and A3CActor-critic methods pair a policy that picks actions with a critic that estimates how good situations are. The critic's estimate reduces noise in the policy updates, so learning is steadier.Build · ~75 minutes
  8. 08Proximal Policy Optimization (PPO)PPO is a policy gradient method that limits how far each update can move the policy, so the same batch of experience can be reused several times without training going off the rails.Build · ~75 minutes
  9. 09Reward Modeling & RLHFRLHF trains a reward model on human choices between two responses, then uses reinforcement learning to push a language model toward responses that reward model scores highly.Build · ~45 minutes
  10. 10Multi-Agent RLMulti-agent reinforcement learning studies several learning agents in one environment. Each agent's choices change the world the others face, so the learning target keeps moving.Build · ~45 minutes
  11. 11Sim-to-Real TransferSim-to-real is the work of taking a policy trained in a simulator and making it work on real hardware, by randomizing the simulator, adapting to real data or measuring the real system closely.Learn · ~45 minutes
  12. 12RL for Games — AlphaZero, MuZero, and the LLM-Reasoning EraHow game-playing systems like AlphaZero and MuZero combine search with self-play learning, and how a similar reward-driven recipe now trains language models to reason.Build · ~120 minutes

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes