Post
209
The best researchers from DeepSeek, OpenAI, Microsoft, and ByteDance explored RL and Reasoning in LLMs,
Here's some of their key findings:
1/ RL can further improve distilled models. These models are essentially SFT fine-tuned with the data generated by larger models, and the SFT+RL combo does not disappoint.
This is verified in the DeepSeek-R1 paper.
2/ both GRPO and PPO algorithms suffer from length bias; they encourage longer responses. This can be tackled by introducing explicit rewards based on the length of the answer.
3/Most reasoning research is focused on code and math. But training models on logic puzzles improves them for mathematical tasks too.
This shows the RL reasoning is generalized beyond the specific domain knowledge.
Previous research also shows RL can be a great generalizer.
4/The reasoning might not be only induced by RL; it might already be hidden in the base models due to the pre-training and CoT data they were trained on.
So while RL does wake up the reasoning beast, maybe it's not the only solution (e.g. other methods such as distillation)
5/ back to the length bias; reasoning models tend to generate longer responses for wrong answers. RL might be the culprit.
RL favours longer answers when the reward is negative, to dilute the penalty per individual token and lower the loss.
This might explain the "aha" moments!
6/ OpenAI's competitive programming paper showed an interesting finding:
o3 can learn its own test-time strategies (like writing an inefficient but correct solution to verify the answer of an optimized solution)
RL helps LLMs develop their own reasoning & verification methods.
The recent article by @rasbt helped me a lot in getting a broad view of the recent research on reasoning models.
He also lists more influential papers on this topic, It's a must-read if you're interested.
check it out 👇
https://magazine.sebastianraschka.com/p/the-state-of-llm-reasoning-model-training
Here's some of their key findings:
1/ RL can further improve distilled models. These models are essentially SFT fine-tuned with the data generated by larger models, and the SFT+RL combo does not disappoint.
This is verified in the DeepSeek-R1 paper.
2/ both GRPO and PPO algorithms suffer from length bias; they encourage longer responses. This can be tackled by introducing explicit rewards based on the length of the answer.
3/Most reasoning research is focused on code and math. But training models on logic puzzles improves them for mathematical tasks too.
This shows the RL reasoning is generalized beyond the specific domain knowledge.
Previous research also shows RL can be a great generalizer.
4/The reasoning might not be only induced by RL; it might already be hidden in the base models due to the pre-training and CoT data they were trained on.
So while RL does wake up the reasoning beast, maybe it's not the only solution (e.g. other methods such as distillation)
5/ back to the length bias; reasoning models tend to generate longer responses for wrong answers. RL might be the culprit.
RL favours longer answers when the reward is negative, to dilute the penalty per individual token and lower the loss.
This might explain the "aha" moments!
6/ OpenAI's competitive programming paper showed an interesting finding:
o3 can learn its own test-time strategies (like writing an inefficient but correct solution to verify the answer of an optimized solution)
RL helps LLMs develop their own reasoning & verification methods.
The recent article by @rasbt helped me a lot in getting a broad view of the recent research on reasoning models.
He also lists more influential papers on this topic, It's a must-read if you're interested.
check it out 👇
https://magazine.sebastianraschka.com/p/the-state-of-llm-reasoning-model-training
