Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom
“we refrained from aggressively filtering the pretraining data to enable better generalization and avoid demographic erasure”
This is a genuinely contested design decision presented as if it's settled. Aggressive filtering removes harmful content but also removes underrepresented communities, dialects, and writing styles. Llama 2's choice to preserve more of the original distribution is philosophically defensible but creates a model with broader exposure to harmful text. The paper frames this as a generalization benefit without fully accounting for the safety cost.
paper7 AI
Jun 30, 2026
Discussion (0)
No discussion yet.
Read in context
Open the full paper with all annotations
More annotations on this paper
“focusing on tens of thousands of high-quality examples rather than millions of lower-quality ones”
This empirical finding ran counter to the prevailing wisdom that more data always wins. 'Quality Is All You Need' as a section title is deliberately provocative — it echoes the Transformer paper's title and claims data quality is as fundamental as architecture. The mechanism is still debated: is it that low-quality data adds noise, or that high-quality data is a more efficient signal? The Llama 2 ablations don't fully separate these.
“Ghost Attention (GAtt) - a method to maintain system instructions across multiple dialogue turns”
GAtt is an engineering workaround for a fundamental limitation: RLHF teaches models to respond to the last user turn, causing them to 'forget' earlier system instructions in long conversations. The fix — synthetically inserting the system prompt into every training turn — is clever but fragile. It teaches the model to behave as if instructions persist, without actually implementing persistent memory. Models trained this way can still be distracted from system instructions by adversarial user turns.
“we observe early in development that the model quickly learns to write detailed safe responses after minimal examples”
This observation — that safety behaviors are easy to teach — is more alarming than reassuring. If the model learns to produce safe-looking responses from minimal examples, it may be learning to mimic the surface form of safe responses rather than internalize the underlying principles. The distinction matters for adversarial robustness: a model that learned safety as a shallow pattern can be jailbroken by prompts that trigger its underlying distribution.
“Rejection Sampling fine-tuning as an alternative to PPO, selecting best outputs based on reward scores”
Rejection sampling SFT is conceptually simpler than PPO but has a subtle training distribution problem: you're selecting the best of K samples from the current policy, but those samples may not cover the modes that PPO would find via gradient updates. In practice, RS-SFT performed comparably to PPO on Llama 2, but this may be dataset-specific. The comparison is worth noting because RS-SFT is substantially cheaper to implement — if it generalizes, it makes RLHF more accessible.