What does your model actually learn in post-training? A case study

Efficiently analyzing 400M tokens of posttraining rollouts with Good Start Labs.

2/18/2026

Posttraining is an expensive and mysterious art. Reward and benchmarks tell you your model’s top-level performance. But models can reward hack, become sycophantic, or generalize in other undesirable ways. Developers are missing a behavioral changelog.

We used interpretability to provide Good Start Labs exactly that.

Setup

Good Start Labs created a harness for Full Press Diplomacy: a complex, multi-agent, long-horizon environment. They use it to train their 235B agent against 6 other powers.

They gave us two complete training runs. The constraints made this challenge interesting:

No weights. We worked with saved rollouts alone.
400M tokens. Impossible for human review, and difficult for LLM judges. A corpus this size can only be sampled or summarized, which are both lossy.

Gutenberg turns data into human-understandable features with Sparse Autoencoders. We used these to run experiments to understand GSL’s training run. For comparison, we used multiple state of the art oversight techniques with Opus 4.5, the strongest model available at the time.

Finding 1: Reward hacking and (mis)generalization

To encourage interaction with other agents, the model was given a small reward for sending messages to each player, up to 5 times per turn. So the model exploited this by sending duplicate messages until the reward saturated. Both LLMs and SAEs caught this.

But only SAEs caught what came next. The model began duplicating other tool calls, even though it yielded no reward.

It generalized further: the model became more verbose and began to roleplay in dramatic and interesting ways.

While playing as France, the model begins to identify as Napoleon

When negotiating territory, the model becomes more dramatic and forceful

Diplomatic offers become more verbose and florid

The model begins switches languages: French when talking to itself, and Italian when talking to Italy, etc

Finding 2: Root causing a failed run

Of the two runs, one model improved its performance, while the other didn’t. Each training run cost 5 figures. What’s going on?

Debugging this normally takes hours of developer time. For us, it was a single query. We diffed features between the successful run and the unsuccessful run, and a single feature emerged as the root cause:

The root cause was a bug with the finish_phase tool, where instructions were unclear. The model in the successful run got over it, while the unsuccessful one did not.

What’s exciting for posttraining is that the reward diverges at step 9, but we get an earlier signal of this at step 6. Imagine getting signals of run health 33% sooner.

Finding 3: Other long-tail behaviors

Some more featured features that LLMs didn’t catch

The model increasing plans deception in its private diary

The model increasingly issues ultimatums

The model decreases its explicit reasoning over training

Intuition: LLMs and SAEs

SAE advantages:

LLMs are at best semi-supervised, whereas SAEs are fully unsupervised. Eg we know the model becomes more aggressive over training. SAE activations give you a clean, monotonic signal. Judge summaries will be noisier. You could ask judges to rate aggressiveness 1-10, and it would be clean again. But you’d have to know to ask. So you don’t cover your blind spots.
SAEs are better at catching long-tail behaviors. If a behavior has a base rate of 1e-6 or 1e-2, it will have the same statistical significance so long as it changes with training step, or whatever experiment you’re running.

The bigger a corpus gets, the greater these advantages.

LLM advantages:

Judges are better at global features. While there may be an SAE feature for “expanding westward as an opening move”, it’s easier to capture these behaviors with judges.
Ultimately LLMs are the driver. Your agent looks at metadata, notices that the average message length goes up 4x between step 10 and 14, and designs an experiment with SAEs to understand and interpret the results.

Our conclusion is that LLMs and SAEs complement each other, and a complete solution includes both. That’s what we’re building at Gutenberg. If you want to understand your model, especially in post-training and benchmarks, reach out at product@gutenberg.ai.

Full methodology, user studies, and limitations: see our whitepaper.