Self Distillation

Introduction

Policy gradient based post-training has become an important driver of rapidly improving LLM capabilities, particularly in verifiable domains such as math and code [1]. InstructGPT [2] was among the first major success stories which used the PPO algorithm [3] with a learned reward model to train a language model towards human preferences. While powerful under carefully tuned hyperparameters, PPO comes with its own challenges. PPO’s actor-critic formulation requires training a value function along with the model, which is usually another LLM initialized from a reward model. This adds significant computational overhead and training instability at frontier scale. Initializing the value function from a reward model has its own problems [5]. Shao et al. introduced GRPO to tackle the memory and training instabilities of PPO at scale.[4]. GRPO eliminates the value function entirely. To compute the advantage, GRPO instead samples a group of \(G\) responses to the same prompt and uses their relative rewards as a baseline. Assuming outcome rewards \(r_i\) the advantage becomes :

GRPO uses a PPO style clipped objective function with a KL anchor on the reference policy. More recent GRPO variants have done away with the KL term [6]. Ignoring the clipping and KL penalty, the policy-gradient term has the same well known policy gradient form :

where \(\rho_{i,t}(\theta)=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}\) is the importance sampling ratio (useful even in a purely on-policy setting).

GRPO has proved to be quite successful for post-training LLMs in verifiable domains [1] [4]. Over the last couple of years it has spawned numerous variants that tackle its different amendable shortcomings [6] [7] [8]. Halving the memory overhead and alleviating training instabilities of PPO, that become more and more acute at frontier scale, is naturally quite appealing. GRPO however is fundamentally incapable of a particularly desirable attribute of PPO : granular credit assignment per rollout.

Most state-of-the-art LLMs are now thinking models. They generate reasoning traces that can be thousands or even tens of thousands of tokens long before producing a user-facing answer or solution [1] [9]. GRPO’s group level advantage estimation assigns uniform credit to all such tokens. With enough rollouts and training steps (i.e. enough compute), models are able to successfully localize credit which explains the success of GRPO. One can also argue that if compute is not particularly a constraint, sampling task advantage directly is more desirable over PPO’s biased value function.

Compute however is a constraint outside of frontier labs, and perhaps it should be. We would like to be able to train reasoning models in as compute-efficient a manner as possible. Algorithms that are able to perform accurate granular credit assignment at a similar rollout budget should intuitively be much more sample and compute efficient than GRPO. If such an algorithm scales well, then that would also be superior to group level advantage estimation at frontier scale. Granular credit assignment is also something humans naturally attempt to do, often very successfully. Therefore we should aspire to mitigate the significant shortcomings of PPO like policy gradient methods or attempt to assign token level credit in other ways.

On-policy distillation

On-policy distillation (OPD) offers a convenient way of assigning granular credit to rollouts when we have a access to a much stronger model [10] [11] [12]. As the name suggests, it computes the teacher’s next-token distribution at every prefix of the student’s rollouts. An f-divergence between the student and teacher distributions then becomes the token level supervision or ‘credit’. This formulation also has the advantage of not needing a verifier or a reward model and can be readily applied to non-verifiable domains, assuming the teacher has enough domain expertise. This has proven to be extremely useful for distilling a large post-trained model into smaller model of the same family [10] [11] [13]. The goal is obviously to do such credit assignment without first training a large model with group level advantage estimation. However its still worth looking into OPD carefully as it offers a surrogate to study and analyze how our desirable algorithm should behave.

What is self-distillation

On-policy self-distillation (OPSD) builds upon OPD by attempting to construct the teacher from the student model itself [14] [15] [18] [19]. The key idea is that a student might be able to critique¹ its own rollouts if given access to privileged information (PI) in its context; PI that enables the self-teacher to exceed the performance of the student. In addition to token level feedback, OPSD also preserves OPD’s desirable property of not needing an explicit verifier. The objective can be written as :

where \(c\) is the PI. The teacher can be frozen or an exponential moving average of the student. Differentiating this w.r.t \(\theta\) again results in the familiar policy gradient expression :

Comparing equations (2) and (4) allows us to write out the OPSD advantage :

The OPSD objective is minimized whereas the GRPO objective is maximized (expected sum of rewards). Therefore the advantage sign needs to be flipped which is the same as flipping the ratio. Since the advantage is per token it also has a \(t\) subscript, unlike the GRPO advantage.

Why should some choice of PI enable this self-teacher construction? This is because the student in this case is an instruction tuned LLM with in-context learning abilities (ICL) [16] [17]. Having the PI in context, demonstrably increases task performance of the self-teacher beyond the student [19]. Its easy to be convinced by this. For example, an instruction tuned LLM should be able to trivially solve a problem when the entire solution (PI) is present in its context. It only needs to have the minimal intelligence to recognize the solution and copy it. This also leads to the interesting question of how should one choose the PI. The choice of PI is crucial as we’ll see but the formulation does not impose strong constraints on what the PI should be.

Exploring self-distillation

Does OPSD work? It seems to work well for a range of tasks and domains [14] [18] [19] [20]. However it seems to have a particularly important and interesting failure mode which will be the subject of our discussion here. OPSD can lead to training collapse in thinking models for reasoning tasks like math [21] [22] [23]. It seems that while the self-teacher can produce the correct solution (trivially, if the PI is the solution), the PI makes the self-teacher increasingly confident, causing it to penalize the student’s exploration, essential for thinking models [21] [22] [23].

Its hard to precisely define exploration of an LLM’s chain-of-thought (COT) but in our context it loosely refers to the model exploring different strategies or related concepts, expressing uncertainty, backtracking and correcting itself before committing to a final solution. Can we try to quantify these behaviors in the PI conditioned self-teachers? Gandhi et. al. [30] attempt to measure them using an LLM-as-a-judge. They define four metrics :

• Average Backtracking Count
• Average Verification Count
• Average Backward-Chaining Count
• Average Subgoal-Setting Count

and use GPT-4o [31] to approximately estimate these quantities. Lets utilize these metrics as well. Before that we need to decide on the choice of PI. Here are a few reasonable choices :

• Full : A full correct solution to the problem generated by a different, strong LLM
• Answer : The correct answer to the problem
• Hint : A self generated hint from a correct demonstration (the Full PI)
• Rollout : A self-generated rollout for the same problem which may or may not be the correct solution

We choose two small reasoning models for our experiments : Qwen3-1.7B and the Qwen3-4B [25]. We generate from these models under the different PIs on the Deepmath dataset [24] ². We then prompt the Qwen3.8-27B model [32] with behavioral examples extracted from Deepmath and annotated by Claude Opus 5 [33] :

Figure 2 shows numbers of occurrences of the four behaviors if we generate from the self-teacher under the different PIs we’ve chosen. We see a significant drop in verification under the PIs that are the most informative about the reasoning process (full and rollout). Subgoal setting on the other hand increases slightly under the same PIs, and backward chaining is essentially flat. This is perhaps not too surprising if we think about what the PI does to the self-teacher’s task. With a correct demonstration in context there is little reason to verify or backtrack, but the self-teacher still has to lay the solution out as a sequence of steps. So it is specifically the error-correcting behaviors that are degraded by the informative PIs. Backward chaining seems mostly unaffected but this metric also showed the least inter-rater agreement levels in Gandhi et. al. [30]. Interestingly, the hint conditioned self-teacher is the most similar to the student on all four behaviors. We’ll see shortly that the hint PI also has other desirable qualities.

Kim et. al [23] report that the self-teacher generates fewer expressions of uncertainty (‘wait’, ‘hmm’, ‘perhaps’, etc) as the PI becomes more informative. If we consider uncertainty verbalization to be also associated with reasoning and exploration, this finding should also align with our previous result. To replicate this we again take the Qwen3-1.7B and the Qwen3-4B models and evaluate their uncertainty verbalization and pass@k metrics under different PIs on the Deepmath :

Figure 3 suggests that as we make the PI progressively more informative about the problem, uncertainty verbalization decreases. Its somewhat interesting what happens under the rollout PI. Recall that its simply self-generated rollout for the same problem which may or may not be the correct. Lets measure how it affects pass@k at different token budgets when the rollout is correct vs when it is incorrect :

Figure 4 shows that correct rollouts predictably increase pass@k across token budgets. With 8k tokens, where many incorrect rollouts are max length truncations, even incorrect rollouts as PI boost pass@k. At 16k, incorrect rollouts are strongly detrimental as PI for both models. This suggests that models can extract useful ⁴ information in-context from even incorrect but truncated rollouts. Completed incorrect solutions appear to be extremely harmful.

What happens when we distill from these self-teachers?

Figure 5 shows evaluation results (best checkpoint) after training on Deepmath for 200 steps and 8k token budget, with a frozen teacher and sampled-token distillation (see TRL SDFTConfig). Its clear that higher self-teacher pass@k does not directly translate to higher student pass@k. Only hint PI consistently improves the model beyond its baseline accuracy on AIME24 [26]; the rest generally degrade the model, often substantially. This also matches our intution about distillation in general : A more accurate teacher is not necessarily a better teacher [28].

The rollout PI ⁵ again has an interesting behavior if we contrast it with the full PI. The full PI are correct (final answer) reasoning traces from the Deepmath dataset, produced by Deepseek-R1 [1]. It is an older model but significantly larger and have higher pass@1 accuracies than both the Qwen models we are evaluating. In contrast, the rollout PI is often demonstrably wrong. The rollout PI conditioned self-teacher also has significantly lower pass@k than the full PI. However, the rollout PI conditioned self-teacher degrades the student less than the full PI. A plausible explanation for this could be that the full PI is perhaps distributionally quite different from the rollout PI, since it comes from a different model. We will investigate this but before that lets look at the relationship between self-teacher uncertainty verbalization and student accuracy :

Figure 6 shows the opposite trend of Figure 3 (note that the x-axis is inverted in Figure 3). Rising self-teacher uncertainty verbalization seems to be related to higher student pass@k. Hint PI leads to the highest student pass@k with the hint conditioned self-teacher being closest to the student in terms of uncertainty verbalization. Distilling from highly confident (in terms of verbalization) teachers could also lead to more (perhaps incorrectly) confident students. We could measure uncertainty verbalization in the student as well but this can also be seen in the student response lengths on training data :

As expected Figure 7 shows response length collapse for full and rollout PI trained models.

What have we learned so far?

Our experiments so far suggest a few things which we list below :

1. The PI needs to be correct and minimally informative about the reasoning process :

• An incorrect PI strongly degrades the self-teacher accuracy
• Only the hint conditioned self-teacher consistently lifts student pass@k above baseline, outperforming both full and answer PIs
• Even an incorrect PI in context can boost self-teacher accuracy if the PI is truncated before reaching the (incorrect) final answer

2. The self-teacher needs to preserve verification, backtracking and uncertainty verbalization :

• A more informative PI tends to reduce verification, backtracking and uncertainty verbalization in the self-teacher
• Distilling from highly confident teachers is detrimental

3. A self-generated PI is better than an off-policy PI :

• Even unverified but self-generated rollouts as PI is less destructive to the student compared to correct full solution PIs from a stronger but different model
• The best performing PI is the hint PI which is student generated

What do we want?

Let’s step back a little and think about what is it that we want to achieve through our analysis here. We are looking for quantifiable properties of the self-teacher that predicts how “good” the self-teacher is for the student. This will give us actionable insights on how to generate PIs or optimize the self-teacher. So far we only have some qualitative indicators of how the self-teacher needs to behave. But these are not exactly objectives we can meaningfully optimize towards ⁶.

Our current observations suggests that a self-generated PI might be preferable to one from a different distribution. Nicolicioiu et al. [27] argue in favor of this by suggesting that self-distillation might be suboptimal when the PI and the student rollout trajectories share few common patterns. However our hint PI results show that the PI can be structurally very different from the student rollout while still resulting in an effective self-teacher. Instead of the PI, perhaps it makes sense to analyze the structural and distributional similarities between the student and the PI conditioned self-teacher. After all, its the self-teacher properties that directly influence student learning and not the PI by itself. As the choice of PI seems crucial for self-distillation, we’d like to have a better understanding of the self-teacher distribution under different PIs relative to the student distribution.

Self-teacher vs student distributions

The OPSD objective is itself an f-divergence between the self-teacher and the student. In our setting its the reverse KL. A simple quantity we could measure is the average reverse KL across a rollout. In our setup we train with the KL evaluated at the sampled token, not the true reverse KL over the vocabulary. Therefore it would be more faithful to average over this log ratio instead :

If we take the negative, this is simply the OPSD advantage in equation (5) averaged over the rollout :

Let’s measure \(\overline{A}(o)\) across training checkpoints. This will give us an indication of how strong the self-teacher signal is across checkpoints. Lets also measure :

\(A^{none}(o)\) roughly tells us how much the trained model is shifting away from the base model in terms of the average next-token distribution.

Figure 8 suggests that training under hint and answer conditioned self-teachers shifts the model the least from its initial per-token distributions. Their self-teacher signals are also weaker than the full and rollout PIs. Intuitively this should be desirable for thinking models. They’ve already undergone RLVR post-training and perhaps their distributions do not need to be rapidly altered. This is reflected in the AIME24 results in Figure 5.

References

1. DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
2. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems, 35.
3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
4. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
5. Yuan, Y., Yue, Y., Zhu, R., Fan, T., & Yan, L. (2025). What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret. arXiv:2503.01491.
6. Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
7. Liu, Z., Chen, C., Li, W., et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783.
8. Zheng, C., Liu, S., Li, M., et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071.
9. Kimi Team et al. (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599.
10. Gu, Y., Dong, L., Wei, F., & Huang, M. (2024). MiniLLM: On-Policy Distillation of Large Language Models. International Conference on Learning Representations.
11. Agarwal, R., Vieillard, N., Zhou, Y., et al. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. International Conference on Learning Representations.
12. Ko, J., Kim, S., Chen, T., & Yun, S. (2024). DistiLLM: Towards Streamlined Distillation for Large Language Models. International Conference on Machine Learning.
13. Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., & Lin, Y. (2026). Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125.
14. Zhao, S., Xie, Z., Liu, M., et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734.
15. Penaloza, E., Vattikonda, D., Gontier, N., et al. (2026). Privileged Information Distillation for Language Models. arXiv:2602.04942.
16. Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33.
17. Min, S., Lyu, X., Holtzman, A., et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Conference on Empirical Methods in Natural Language Processing.
18. Hübotter, J., Lübeck, F., Behric, L., et al. (2026). Reinforcement Learning via Self-Distillation. arXiv:2601.20802.
19. Shenfeld, I., Damani, M., Hübotter, J., & Agrawal, P. (2026). Self-Distillation Enables Continual Learning. arXiv:2601.19897.
20. Rezaei, M., Mahmoud, A., Wang, Z., et al. (2026). Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers. arXiv:2606.12507.
21. Kaur, S., Ri, N., He, Y., Fowl, L., & Arora, S. (2026). Rethinking On-Policy Self-Distillation for Thinking Models. arXiv:2607.05184.
22. Peng, K., Li, C., Ouyang, Y., Yuan, Y., & Ding, L. (2026). Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. arXiv:2607.10805.
23. Kim, J., Luo, X., Kim, M., et al. (2026). Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?. arXiv:2603.24472.
24. He, Z., Liang, T., Xu, J., et al. (2025). DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning. arXiv:2504.11456.
25. Yang, A., Li, A., Yang, B., et al. (2025). Qwen3 Technical Report. arXiv:2505.09388.
26. Hugging Face H4 (2024). AIME 2024. Hugging Face Datasets. The 30 problems of AIME 2024 I and II, derived from AI-MO/aimo-validation-aime.
27. Nicolicioiu, A. L., Pezeshki, M., & Courville, A. (2026). On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity. arXiv:2606.26091.
28. Cho, J. H., & Hariharan, B. (2019). On the Efficacy of Knowledge Distillation. IEEE/CVF International Conference on Computer Vision.
29. Feng, Y., Feng, Z., & Chen, J. (2026). PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation. arXiv:2608.08726.
30. Gandhi, K., Chakravarthy, A., Singh, A., Lile, N., & Goodman, N. D. (2025). Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. arXiv:2503.01307.
31. OpenAI et al. (2024). GPT-4o System Card. arXiv:2410.21276.
32. Qwen Team (2026). Qwen3.8-27B. Hugging Face. See also the Qwen3.8 release blog.
33. Anthropic (2026). Introducing Claude Opus 5.