Read papers. Together.

"DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time through self-evolution"

The 'aha moment' — where the model spontaneously learns to backtrack and self-verify during training — reads like an emergent capability story. But it's worth noting this emerged in a heavily constrained setting: verifiable domains with binary rewards. Whether this self-evolution generalizes to open-ended reasoning without ground truth is the open question the paper explicitly declines to answer.

"Humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do."

This framing positioned GPT-3 as closing the gap with human generalization, but 'struggling' carries more freight than acknowledged. GPT-3 still fails on tasks a child handles trivially — counting objects, spatial reasoning, tracking referents across long contexts. The paper conflates benchmark performance with cognitive generalization in a way that influenced how the field measured 'intelligence' for years.

"A random square crop from resized images is the only data augmentation used during training."

The minimalism here is striking given how much augmentation lore existed by 2021 — color jitter, cutout, RandAugment, mixup. The authors essentially proved that at sufficient scale, augmentation tricks become second-order. This is a recurring lesson of the scaling era that many practitioners resist accepting because it devalues years of engineering investment.

"smaller models trained longer will ultimately be cheaper at inference"

This single sentence reframed the entire scaling debate. Kaplan et al.'s original scaling laws optimized for training compute — LLaMA argued inference cost should be the optimization target. Given that a model runs inference millions of times but trains once, this is obviously correct in hindsight. The field spent two years optimizing the wrong objective before LLaMA made the switch explicit.

"50% of the time B is the actual next sentence that follows A (labeled as IsNext), and 50% of the time it is a random sentence from the corpus (labeled as NotNext)."

Next Sentence Prediction turned out to be largely useless. RoBERTa (2019) dropped it entirely and improved on BERT across the board. The authors' intuition that cross-sentence relationships need explicit supervision was wrong — MLM on long documents already captures it. NSP is now a cautionary tale about adding pre-training objectives without rigorous ablation first.

"dispensing with recurrence and convolutions entirely"

This was the bold bet that paid off. In 2017, dropping LSTMs felt risky — every major NLP lab was invested in recurrence. The fact that they went all-in on attention is what made this paper a paradigm shift, not just an improvement.