“GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding”

Grouped-query attention is the kind of engineering improvement that doesn't appear in loss curves but dominates practical deployment economics. Halving KV cache memory means doubling batch size at the same GPU budget, which directly cuts serving cost per token. The fact that this made it into a 7B academic release — rather than staying a datacenter trade secret — is the real contribution.

“language models may compress knowledge more than what was previously thought”

This throwaway line in the conclusion is actually a testable hypothesis, and it hasn't been tested rigorously. 'Knowledge compression' is doing a lot of work — it conflates parameter efficiency, training data quality, and architectural inductive biases. Mistral's result could mean any of these. Papers that claim compression improvements without separating these factors are common; this one is honest that the mechanism is unknown.

“SWA is designed to handle longer sequences more effectively at a reduced computational cost”

Sliding window attention doesn't attend to the full context — it restricts each token to a local window. This is a fundamental trade-off against global coherence, not a free lunch. Tasks requiring cross-document reasoning or long-range dependencies are where SWA degrades. The paper benchmarks on tasks that don't stress this limitation, which makes the efficiency claim look better than it generalizes.

“information can move forward by up to k×W tokens after k attention layers”

This recurrence-like property of stacked sliding window attention is the paper's most underappreciated theoretical result. It shows that local attention can approximate global attention given sufficient depth — but the required depth scales linearly with sequence length, which means the compute savings disappear for very long documents. The caveat is buried in the architecture section and never surfaced in the evaluation.