A Practical Introduction to Reinforcement Learning

Reinforcement learning (RL) is fundamentally different from the supervised learning most data scientists encounter first. There’s no labelled dataset, no ground-truth output to regress toward. Instead, an agent takes actions inside an environment, receives rewards (or penalties), and gradually learns a policy that maximises cumulative reward.

The Core Framework

RL problems are formulated as Markov Decision Processes (MDPs):

• State (s) — a snapshot of the environment at a given moment
• Action (a) — a choice the agent can make from state s
• Reward (r) — a scalar signal the environment returns after each action
• Policy (π) — the agent’s strategy: a mapping from states to actions
• Value function (V) — the expected cumulative reward from a given state under policy π

The agent’s goal is to find the optimal policy π* that maximises the expected discounted return:

G_t = r_t + γ·r_{t+1} + γ²·r_{t+2} + …

where γ ∈ [0, 1) is the discount factor, controlling how much future rewards are valued.

Why Reinforcement Learning?

Supervised learning needs labels. Unsupervised learning finds structure. RL, uniquely, handles sequential decision-making problems where:

1. Actions affect future states (delayed consequences)
2. There’s no teacher telling you the right move — only environmental feedback
3. Exploration is required; you can’t learn only from what you already know

Classic use cases include game-playing (AlphaGo, OpenAI Five), robotics control, recommendation systems, and — increasingly — LLM alignment via RLHF.

Deep Q-Networks (DQN)

The breakthrough that brought RL into the deep learning era was DQN (Mnih et al., 2015). Instead of a tabular Q-table, a neural network approximates the Q-function:

import torch
import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim),
        )

    def forward(self, x):
        return self.net(x)

Key stabilisation tricks DQN introduced:

• Experience replay — store transitions in a buffer; sample mini-batches randomly to break correlation
• Target network — a periodically-updated copy of the Q-network used to compute TD targets, preventing oscillation

Proximal Policy Optimisation (PPO)

DQN is value-based. PPO (Schulman et al., 2017) is a policy-gradient method — it directly optimises the policy. It’s become the workhorse algorithm for continuous-action problems and RLHF because of its simplicity and robustness.

The key innovation is a clipped surrogate objective that prevents destructively large policy updates:

L_CLIP(θ) = E[min(r_t(θ)·Â_t,  clip(r_t(θ), 1-ε, 1+ε)·Â_t)]

where r_t(θ) is the probability ratio between the new and old policy, and Â_t is the advantage estimate.

Practical Tips

1. Start with a simple environment — CartPole or MountainCar before trading bots or robotics
2. Reward shaping matters enormously — sparse rewards make learning extremely slow; add intermediate signals carefully
3. Hyperparameters are sensitive — learning rate, entropy coefficient, and discount factor all interact
4. Log everything — episode reward, policy entropy, value loss; RL training is noisy and hard to debug blind

Getting Started

The fastest path to hands-on RL in Python:

pip install gymnasium stable-baselines3

import gymnasium as gym
from stable_baselines3 import PPO

env = gym.make("CartPole-v1")
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=50_000)

obs, _ = env.reset()
for _ in range(1000):
    action, _ = model.predict(obs)
    obs, reward, done, truncated, info = env.step(action)
    if done or truncated:
        obs, _ = env.reset()

Stable-Baselines3 is production-quality and a great place to experiment before writing algorithms from scratch.

What’s Next

In a follow-up post I’ll go deeper into applying RL to financial trading — one of the most challenging domains due to non-stationarity, transaction costs, and the temptation to overfit. If you have questions or want to compare notes, reach out via the contact form.