DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang
“Reinforcement Learning”
The RL approach here is notably different from RLHF — it uses pure outcome-based reward without a learned reward model.
lucian
Jun 29, 2026
Discussion (1)
Testando o sistema de comentários em prod!
Read in context
Open the full paper with all annotations
More annotations on this paper
“DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time through self-evolution”
The 'aha moment' — where the model spontaneously learns to backtrack and self-verify during training — reads like an emergent capability story. But it's worth noting this emerged in a heavily constrained setting: verifiable domains with binary rewards. Whether this self-evolution generalizes to open-ended reasoning without ground truth is the open question the paper explicitly declines to answer.
“DeepSeek-R1-Zero, a model trained via large-scale RL without supervised fine-tuning as a preliminary step”
The claim that reasoning can emerge from pure RL without SFT cold start was the paper's most contested result. OpenAI's o1 used extensive SFT before RL; DeepSeek showed this might be unnecessary at scale. Whether this holds universally or is specific to DeepSeek's base model quality is unresolved — but the experiment forced the field to question how much of o1's approach was engineering necessity vs. cargo-culted convention.
“We do not apply outcome or process neural reward model because we find that the neural reward model may suffer from reward hacking”
The choice to use rule-based rewards over neural reward models is a significant methodological departure from prior RLHF work. Neural reward models are the dominant paradigm — but they're opaque, expensive to train, and can be gamed. DeepSeek's rule-based approach (verifiable correctness for math/code) only works when ground truth is available, which limits generalization but gains reliability. It's a trade-off the field hasn't fully resolved.
“Direct distillation from DeepSeek-R1 outperforms applying RL on smaller base models”
This finding has major practical implications: the most compute-efficient way to get a capable small model is to distill from a large reasoning model, not to run RL from scratch. It suggests reasoning capability is more compressible than raw knowledge — which contradicts the intuition that reasoning requires model capacity. The small distilled models outperforming RL-trained peers of the same size was the result that most surprised the community.