Experiment 03
Bipedal avatar
A jointed body and a neural network that starts out knowing nothing learn, together and unsupervised, to stand and then to move forward — one reward signal at a time.
What is reinforcement learning?
Here, no gait is written anywhere. LifeLabs drops a jointed body — a torso, two thighs, two shins, two feet, seven physical segments simulated by the MuJoCo engine — and an empty neural network into a world with gravity. That network starts out knowing nothing at all: not how to stand, not what "forward" means. At every step it picks forces to apply to the joints, observes what happens, and gets back a single number — a reward — combining staying alive, moving forward, and spending as little effort as possible.
The learning algorithm, PPO (Proximal Policy Optimization), then adjusts the network's weights to make the actions that led to a higher reward more likely, and the ones that preceded a fall less likely. Repeated tens of thousands of times, that single signal is enough to give rise to a control policy: first incoherent spasms, then a balance that holds for longer and longer, then — given enough training time — a gait. None of this is written in the code: the code only records a score and corrects weights.
Progress captured at checkpoints
At regular intervals during training, a video of the agent's current behavior is recorded. Here are the checkpoints of the latest published run, in order — the same agent, growing more experienced:
Individual learning, not collective emergence
In the evolutionary agents experiment (milestone 3), an entire population changes from generation to generation: no single agent learns over its own lifetime — selection is what moves the group forward. Here it's the opposite: one agent, one policy, improving over the course of a single simulated lifetime — reinforcement learning directly adjusts a network's weights from its own experience, with no notion of generations or reproduction.
In Lenia, emergence produces a shape: one local rule, identical everywhere and never changing, gives rise to a coherent, mobile creature. Here it's the opposite: the rule itself — the network's weights — keeps changing, because that rule is what is being learned. So this isn't, strictly speaking, emergence, but learning — and for once the word is used in its exact technical sense: the network adjusts its own parameters to improve a score. That says nothing about the avatar's awareness: a set of weights minimizing a cost function has no known subjective experience.
Pilot
3D replay of a gait
The replay below reconstructs, frame by frame, the real recorded positions and orientations of each of the body's seven segments during a rollout — the same physical data used to produce the checkpoint videos above, recomputed in 3D in your browser instead of pre-rendered to video.
Loading trajectory…
Drag to orbit the camera, scroll to zoom.
The 3D replay couldn't be loaded
This run's 3D trajectory couldn't be fetched (or WebGL isn't available in this browser). In the meantime, here's a video capture of the latest trained checkpoint, when available.
The vital signs we measure
Each run records three time series, computed at every training rollout, which serve as "vital signs" for the learning process:
- Mean episode reward
- the mean reward earned per episode — it combines staying alive, moving forward, and limiting effort; its rise is the most direct signal that the agent is learning, well before that translates into distance covered.
- Episode length
- the mean number of simulation steps before an episode ends (typically a fall) — a rising episode length signals an agent learning to stay upright longer, the very first skill it acquires.
- Distance travelled (x)
- the distance the torso travels along the x axis — the last signal to improve: an avatar learns to survive first, and only later to move forward.
