OPD–EOS GitHub

ON-POLICY DISTILLATION · TERMINATION ALIGNMENT

When EOS Tokens Disagree

Understanding Length Inflation in On-Policy Distillation

Yuxiao Yang1,*Tianrun Yu2,*Shangzhe Li1,*Kaixiang Zhao2,*Xuchao Zhang3Chetan Bansal3Huaxiu Yao1Taylor W. Killian2,†Weitong Zhang1,‡

1 University of North Carolina at Chapel Hill2 Brigham Young University3 Microsoft

* Equal contribution · † tkillian@cs.byu.edu · ‡ weitongz@unc.edu

Read the paper Code Explore the findings

THE CENTRAL FINDING

The answer is right.
The stopping signal is wrong.

A student can reach the correct answer, then spend the rest of its budget repeating it. Teacher and student may mean “stop” with different tokens—and token-level supervision can penalize the student for stopping.

See why this happens
01

Decoding alone is insufficient. A shared stop set leaves the distillation signal unchanged.

02

Align termination probabilities. A semantic STOP action reduces mismatch-driven inflation.

03

Length has more than one cause. Training stages and evaluation protocols still matter.

01 / THE PHENOMENON

Correct early. Repetitive until the budget runs out.

These are examples from the paper’s Qwen3 rollouts. Every response reaches the 8,192-token evaluation budget without emitting EOS.

1,094tokens to first correct answer
7,098subsequent redundant tokens
86.6%of the response after the answer
Through the first correct answer Redundant continuation8,192-token limit

FIRST CORRECT ANSWER

The units digit of 2^2024 − 1 is 6 − 1 = 5.

Final Answer: \boxed{5}

WHAT FOLLOWS

### Final Answer:
\boxed{5}

### Final Answer:
\boxed{5}

[Repeated continuation; truncated without EOS]
Abbreviated excerpts; omitted text is marked.Read supplied continuation ↗
View the original training curves Response length & length-clipping rateQwen3 vanilla OPD: response lengths approach the 7,168-token training budget and clipping approaches 100 percent under both TTRL and DAPO templates.

Training uses a 7,168-token response budget; the examples above use the separate 8,192-token evaluation budget.

02 / THE MECHANISM

Same decision. Different surface tokens.

Sampled-token OPD supervises the token the student actually generates. If the student emits its native EOS, a teacher preference for a different EOS token can produce a negative update—even when both models assign the same total probability to stopping.

ILLUSTRATIVE EXAMPLE

Move the teacher’s stopping mass.

Both models keep an 80% total stopping probability. Only the teacher’s preference between the two EOS tokens changes.

99%

Toy probabilities for explanation; these are not experimental measurements.

At the same prefixStudentTeacher
<|endoftext|>0.7920.008
<|im_end|>0.0080.792
Other tokens0.2000.200
Total STOP probability0.8000.800
INDIVIDUAL EOS SIGNAL

log pT(e) − log pS(e)

−4.595Suppresses the student’s sampled EOS.
SEMANTIC STOP SIGNAL

log qT(STOP) − log qS(STOP)

0.000The two models agree on stopping.
SEMANTIC EOS AGGREGATION

qπ(STOP | s) = ∑e ∈ EEOS π(e | s)

Aggregate termination probabilities for both teacher and student. Use the semantic probability in both the detached distillation coefficient and the student log-probability gradient. Non-EOS actions retain their original probabilities.

EEOS contains only tokens that are equivalent stopping events under the rollout protocol. The paper studies single-turn generation; multi-turn end-of-message and end-of-conversation actions need separate treatment.

03 / EXPERIMENTAL EVIDENCE

Correct the signal, and termination recovers.

The Qwen3 ablation isolates four interventions. Cross-family experiments test the semantic correction on Llama 3.2 and Gemma 3, including Gemma’s shared declared EOS set.

A shared decoding stop set does not solve the failure.

Teacher-side mapping, semantic aggregation, and canonical single-EOS mapping behave similarly in Qwen3. All three align termination in the distillation signal.

Four Qwen3 corrections: two-stop decoding approaches the length budget and 100 percent clipping; probability-level corrections retain lower lengths and high native EOS probability.
Qwen3-4B teacher → Qwen3-1.7B-Base student. TTRL template. Gray lines show teacher reference statistics. Open full-resolution figure ↗
What each termination correction changes
InterventionDecodingDistillation signalQwen3 finding
1 Shared-set decodingStop on either EOSIndividual tokens, unchangedInflation persists
2 Teacher-side mappingCanonical student EOSTeacher EOS mass → canonical tokenSubstantially mitigated
3 Semantic EOS class Cross-family defaultStop on either EOSAggregate STOP mass for both modelsSubstantially mitigated
4 Canonical single-EOSCanonical EOS; other EOS removedTeacher mapping + student renormalizationSubstantially mitigated
DAPO-Math-17KTraining data
7,168 tokensMain training response budget
AMC23 · AIME24 · AIME25Math benchmarks · Avg@16

“Clipping” on this page means reaching the response-length budget, not PPO importance-ratio clipping. Avg@16 averages correctness over 16 sampled responses per problem, then averages the three benchmarks equally.

04 / BEYOND EOS MISMATCH

An important mechanism.
Not a complete explanation.

Aligning termination does not eliminate every change in response length or guarantee a downstream reasoning improvement.

TRAINING STAGES

K2-Horizon separates token preference from stopping behavior.

Midtraining and SFT checkpoints already support the teacher-preferred token and remain relatively stable. A pretrained student can learn that surface token, yet later lose its tendency to stop.

With semantic EOS alignment, the pretrained student still shows initial length growth, an elevated plateau, and late-stage re-inflation. The plateau is not recovery to the teacher’s response length, and surface-token identity alone cannot explain the renewed collapse.

Pretrain → Final: 400 steps. Midtrain/SFT → Final: 200 steps. Labels 1–3 mark initial growth, an intermediate plateau, and late-stage re-inflation; gray dashed lines indicate approximate phase boundaries.

K2-Horizon training stages: pretrained students show late length growth both with and without semantic correction; SFT and midtraining students are comparatively stable.
K2-Horizon stage comparison from the final manuscript. View full size ↗
PROMPTING & EVALUATION

Response length and measured accuracy can move differently.

DAPO-trained students tend to produce shorter responses under both evaluation formats. TTRL evaluation generally reports higher measured accuracy. Because prompting and grading both change, the result concerns the overall evaluation protocol.

With semantic EOS correction, TTRL, DAPO, and raw-question training templates produce different lengths and scores under TTRL and DAPO evaluation.
Semantic EOS correction is applied throughout this template comparison. View full size ↗
Scope of the findings

The experiments study sampled-token OPD on single-turn mathematical reasoning. Recovery is model-dependent, Llama retains a larger length gap, and better termination does not consistently translate into higher mathematical accuracy. Multi-turn and agentic settings remain open directions.

PAPER & CODE

Explore the work.

The manuscript contains the full diagnostic experiments, termination corrections, prompt templates, and rollout examples.

Research paperarXiv:2609.20511 · PDFUNCSciML/opd-eosCode repository

Citation

@article{yang2026eos,
  title = {When {EOS} Tokens Disagree: Understanding Length Inflation in On-Policy Distillation},
  author = {Yuxiao Yang and Tianrun Yu and Shangzhe Li and Kaixiang Zhao and Xuchao Zhang and Chetan Bansal and Huaxiu Yao and Taylor W. Killian and Weitong Zhang},
  journal = {arXiv preprint arXiv:2609.20511},
  year = {2026},
  eprint = {2609.20511},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2609.20511}
}