UK What reinforcement learning is
Reinforcement learning (RL) is a branch of machine learning in which an agent learns what to do by acting and seeing what happens. Each action earns a reward or a penalty, and over many attempts the agent adjusts its behaviour to collect as much reward as possible over time. Nobody tells it the right answer; it has to find out.
The problem is usually framed as a Markov decision process. The agent observes the state of its environment, picks an action, and the environment moves to a new state and returns a reward. The rule the agent uses to choose actions is called its policy, and the purpose of training is to find a good policy. Richard Sutton and Andrew Barto’s textbook, Reinforcement Learning: An Introduction, set out most of the field’s basic language.
Robotics is one of the most natural fits for RL, because walking, balancing and grasping are easy to score but very hard to program by hand.
How reinforcement learning works
Training repeats the same short cycle millions of times. The agent also has to balance exploring new actions against exploiting the ones it already knows work.
-
//
Observe. The agent reads the current state: joint angles, body orientation, camera or depth data, and the target.
-
//
Act. The policy chooses an action, such as a set of joint torques or target positions.
-
//
Score. The reward function scores the result, for example rewarding forward speed and penalising falls, jerky motion or wasted energy.
-
//
Update. An algorithm adjusts the policy so actions that led to higher reward become more likely.
-
//
Repeat. The cycle runs again from the new state. Over millions of steps, behaviour that started out random becomes deliberate. Designing the reward is the hard part: a poorly chosen reward lets the agent find a shortcut that scores well without doing the task, which is known as reward hacking.
Reinforcement learning vs supervised and unsupervised learning
These are the three main kinds of machine learning, and they differ mainly in the feedback the model gets.
| Aspect | Supervised learning | Unsupervised learning | Reinforcement learning |
|---|---|---|---|
| Learns from | Labelled examples | Unlabelled data | Its own interaction with an environment |
| Feedback | The correct answer for each example | None | A reward, often delayed |
| Produces | Predictions or classes | Clusters and patterns | A policy that chooses actions |
| Robotics example | Detecting parts in a camera image | Spotting unusual vibration in a motor | Learning to walk or grasp |
Real robot software usually mixes all three. A supervised vision model finds the object, and an RL-trained policy decides how to move.
How reinforcement learning is used in robotics
Training a real robot by trial and error would be slow and would break hardware, so almost all robot RL happens in simulation. GPU-based simulators such as NVIDIA Isaac Lab, and physics engines such as MuJoCo, can run thousands of copies of a robot in parallel on one GPU, compressing years of practice into hours. Proximal Policy Optimisation (PPO), published by OpenAI in 2017, is the most widely used algorithm for this work, with Soft Actor-Critic (SAC) common where sample efficiency matters.
The simulated robot never matches the real one exactly. To make the policy robust, engineers randomise masses, friction, motor strength, sensor noise and delays during training, so the real robot looks like just another variation. This is called domain randomisation, and it is the main reason RL-trained policies now transfer to real legged robots, robot hands and drones. Results include quadruped robots that walk over rough terrain, humanoids that recover from pushes, and mobile robots that learn to navigate crowded spaces.
Deep reinforcement learning simply means the policy is a deep neural network, which is now the norm. It lets the policy work from raw sensor data such as depth images instead of hand-picked features.
From the simulator to the real robot
Reinforcement learning splits the work between two very different computers. Training needs heavy parallel compute, usually a server or workstation fitted with workstation GPUs or data centre GPUs. The trained policy, by contrast, is often a small network that has to run at a fixed rate, typically tens to hundreds of times a second, right next to the motors. This is the same split between training and inference found across AI, with tighter timing.
On the robot, the policy usually runs on an embedded GPU computer such as one from our Jetson Orin NX range, alongside the perception models that feed it. The policy depends on clean, low-latency state data, so the IMU, joint encoders and cameras have to be read with consistent timing. Safety does not rest on the learned policy: hard limits and emergency stops stay in a separate certified controller.
Quick answers on reinforcement learning
Reinforcement learning: common questions
Is reinforcement learning supervised or unsupervised?
Neither. It is a third category. It has no labelled answers like supervised learning, but unlike unsupervised learning it does get feedback, in the form of a reward.
What is deep reinforcement learning?
RL with a deep neural network as the policy. It lets an agent learn straight from high-dimensional inputs such as images, and it underpins most modern robot RL.
What is a policy in reinforcement learning?
The agent’s decision rule. It maps what the agent observes to the action it takes. Training aims to find the policy that collects the most reward over time.
What is PPO in reinforcement learning?
Proximal Policy Optimisation. A policy-gradient algorithm that limits how far the policy changes in each update, which makes training stable. It is the default choice in many robot simulators.
What is reinforcement learning from human feedback?
RL where people supply the reward signal. Known as RLHF, it trains a reward model from human preferences and is best known for tuning large language models.
We build the on-robot computers that run reinforcement learning policies, on NVIDIA Jetson Orin and Jetson Thor, and the GPU systems used to train them, with the software image fitted and validated as standard. Tell us about the robot and its sensors, and we will specify both to match.