Bipedal Locomotion

A reinforcement learning framework for training bipedal walking policies with PPO.

PyTorchMuJoCoFastAPIPPOReinforcement Learning

The Problem

Training a robust bipedal walking policy from scratch is a classic hard problem in RL — balancing stability, sample efficiency, and generalization, usually with a reward function that's easy to accidentally game (e.g. a policy that maximizes forward velocity by falling forward instead of walking).

The Approach

Built a from-scratch PPO framework in MuJoCo, and — rather than relying purely on RL to discover a good gait — also implemented a hand-engineered baseline controller (a symmetric two-phase gait generator driving inverse kinematics and PD control) as both a sanity check and a comparison point for the learned policy.

Try It Live

The same 2-link inverse kinematics the hand-engineered reference controller uses to convert a desired foot position into hip and knee joint angles. Drag the foot target.

hip: 77.6°knee: 167.9°

The physical 3D-printed robot, its CAD model, and its MuJoCo simulation reconstruction side by side — The build, end to end: a physical 3D-printed biped, its CAD model, and the matching MuJoCo reconstruction used for simulation.

Joint position time series for 6 joints over a 100-second rollout — Joint positions over a 100s rollout — clean periodicity confirms a stable, repeating gait rather than a fall or a reward-hacking shortcut.

Animated simulated robot stuck in a degenerate collapsed pose — Unexpected local optima found during training — reward hacking can converge on a stable-looking pose that isn't actually walking.

Animated simulated robot stuck in a different degenerate collapsed pose — A second local optimum found in a separate training run.

Animated simulated robot walking successfully across the checkerboard floor — A successful walking policy in motion — the counterpart to the local optima above, actually making forward progress.

Animated simulated robot walking successfully across the checkerboard floor, a different run — A second successful run, from a separately trained policy.

▹PPO: standalone GAE-λ computation (δₜ = rₜ + γV(sₜ₊₁) − V(sₜ), backward accumulation), ratio clipping to [1−ε, 1+ε] (ε=0.2), Adam at lr=3e-4, GAE λ=0.95, γ=0.99, 80 update epochs per batch
▹Actor-Critic: shared two-layer (64-unit, Tanh) MLP trunk feeding separate policy and value heads; actions sampled from a Gaussian with a learnable, state-independent log-std

Highlights

▹PPO implemented from scratch: GAE-λ advantage estimation, clipped surrogate objective, combined policy/value/entropy loss
▹A genuinely hand-engineered baseline controller — sinusoidal swing trajectories, 2R planar inverse kinematics, PD tracking — not just a random-policy sanity check
▹Reward function is deliberately decomposed into 4 independently-weighted terms (forward velocity, alive bonus, control cost, lateral-drift penalty) to avoid common RL locomotion failure modes
▹Built a live FastAPI/uvicorn MJPEG streaming server to watch rollouts in-browser in real time, plus multi-process parallel rollout collection
▹Supports both a simplified 2D Walker2D and a full 3D biped morphology

Results

Diagnostic joint-position and phase-portrait plots confirm the resulting gait is genuinely periodic — not a policy that's fallen over or found a degenerate shortcut. The joint trajectories repeat cleanly across a 100-second rollout, and the phase portraits (position vs. velocity) trace closed, cyclical loops consistent with a stable, repeating walking gait.