Training¶

Training runs rl/train.py. It uses PPO with 32 games in parallel and saves its progress as it goes, so long runs can be split into blocks and resumed.

Train the final agent¶

This is the setup of the final agent: the fixed view with arrow keys, and 8 of the 32 parallel games reserved for failure replay.

python rl/train.py --tag child --view fixed --replay-envs 8

A run of 5 million steps takes a few hours on a mid-range GPU. The final agent trained for 30 million steps in six such blocks, each resumed from the end of the previous one:

python rl/train.py --tag child --view fixed --replay-envs 8 --resume snake_ppo_child.pt --steps 5000000

The first 15 million steps used a death penalty of -10. For the last 15 million, DEATH_REWARD in include/snake/engine.hpp was set to -50 and the module rebuilt. See Rules and rewards.

Files¶

Every checkpoint stores the settings it was trained with, such as the view and the extra channels. The other scripts read them from the file, so you never have to pass them again.

What the log shows¶

Every 8,192 steps, training prints a line like this:

Step:  4505600 | Avg Score: 129.47 | Entropy: 0.361 | Value Loss: 0.0579 | KL: 0.0012 | Clip: 0.010 | Replay Pool: 2417 | Replays: 7350 | Replay Share: 24.6%

Every 100,000 steps, an evaluation line follows. It plays 50 games without randomness:

[Eval] Step:  4505600 | Greedy Mean: 188.90 | Max: 225 | Timeouts: 0/50 *

The star marks a new best, which is saved as snake_ppo_<tag>_best.pt. Evaluation games are only for measurement and do not train the agent.

Options¶