LifeLabs
← Back to the Bipedal avatar experiment

First steps: learning to walk by trial and reward

Completed
Seed
42
Steps
516096 / 500000
Created
25 July 2026 at 10:53
Finished
25 July 2026 at 11:23

Hypothesis

With no programmed gait, a PPO agent can learn — from reward signal alone — to balance a bipedal body and then move it forward.

Vital signs

Mean episode reward587
-49.2300649step 8192step 516096 step 8192 : -1.04 step 90112 : 236 step 172032 : 331 step 253952 : 386 step 335872 : 435 step 417792 : 512 step 499712 : 601 step 516096 : 587
View data
Mean episode reward by step
StepMean episode reward
8192-1.04
163842.15
2457610.3
3276820.1
4096032.6
4915259.8
5734480.6
6553697.1
73728144
81920195
90112236
98304260
106496256
114688272
122880271
131072297
139264304
147456312
155648326
163840342
172032331
180224348
188416357
196608379
204800377
212992373
221184385
229376384
237568384
245760385
253952386
262144389
270336392
278528405
286720411
294912418
303104411
311296409
319488419
327680430
335872435
344064448
352256457
360448467
368640476
376832476
385024478
393216497
401408500
409600511
417792512
425984524
434176533
442368537
450560540
458752542
466944553
475136566
483328583
491520599
499712601
507904596
516096587
Episode length240 steps
-0.52137274step 8192step 516096 step 8192 : 18.4 steps step 90112 : 208 steps step 172032 : 211 steps step 253952 : 218 steps step 335872 : 208 steps step 417792 : 213 steps step 499712 : 255 steps step 516096 : 240 steps
View data
Episode length by step
StepEpisode length steps
819218.4
1638421.3
2457635.5
3276848.2
4096072.3
49152102
57344119
65536134
73728165
81920196
90112208
98304218
106496217
114688222
122880224
131072214
139264209
147456217
155648223
163840226
172032211
180224217
188416223
196608230
204800224
212992211
221184215
229376216
237568216
245760218
253952218
262144213
270336214
278528219
286720222
294912223
303104216
311296211
319488210
327680210
335872208
344064213
352256214
360448216
368640217
376832213
385024210
393216214
401408212
409600215
417792213
425984217
434176219
442368222
450560222
458752221
466944226
475136233
483328242
491520254
499712255
507904249
516096240
Distance travelled (x)2.79 u
-0.571.233.04step 8192step 516096 step 8192 : -0.15 u step 90112 : 0.23 u step 172032 : 0.97 u step 253952 : 1.36 u step 335872 : 1.83 u step 417792 : 2.40 u step 499712 : 2.78 u step 516096 : 2.79 u
View data
Distance travelled (x) by step
StepDistance travelled (x) u
8192-0.15
16384-0.14
24576-0.19
32768-0.22
40960-0.31
49152-0.32
57344-0.30
65536-0.28
73728-0.16
819200.01
901120.23
983040.35
1064960.32
1146880.41
1228800.39
1310720.67
1392640.78
1474560.77
1556480.83
1638400.95
1720320.97
1802241.06
1884161.09
1966081.20
2048001.24
2129921.30
2211841.38
2293761.36
2375681.36
2457601.35
2539521.36
2621441.42
2703361.43
2785281.50
2867201.52
2949121.57
3031041.58
3112961.60
3194881.69
3276801.78
3358721.83
3440641.89
3522561.96
3604482.02
3686402.09
3768322.11
3850242.16
3932162.27
4014082.31
4096002.38
4177922.40
4259842.47
4341762.53
4423682.53
4505602.55
4587522.58
4669442.63
4751362.68
4833282.74
4915202.78
4997122.78
5079042.79
5160962.79

Video captures

Configuration

ParameterValue
env_idWalker2d-v5
algoPPO
n_envs4
n_checkpoints7
devicecuda
ppo{"n_steps":2048,"batch_size":64,"n_epochs":10,"gamma":0.99,"gae_lambda":0.95,"clip_range":0.2,"ent_coef":0,"learning_rate":0.0003}

Journal notes