Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Introduction

Hyperparameter (HP) tuning is critical to deep learning. Poorly chosen HPs result in subpar performance and training instability. Many published baselines are hard to compare to one another due to varying degrees of HP tuning. These issues are exacerbated when training extremely large deep learning models, since state-of-the-art networks with billions of parameters become prohibitively expensive to tune.

Recently, showed that different neural network parametrizations induce different infinite-width limits and proposed the Maximal Update Parametrization (abbreviated $\mu$P) (summarized in Table 3) that enables “maximal” feature learning in the limit. Intuitively, it ensures that each layer is updated on the same order during training regardless of width.111i.e., the updates’ effect on activations becomes roughly independent of width in the large width limit. In contrast, while the standard parametrization (SP) ensures activations are of unit order at initialization, it actually causes them to blow up in wide models during training essentially due to an imbalance of per-layer learning rate (also see Fig. 5). We leverage $\mu$P to zero-shot transfer HPs from small models to large models in this work – that is, we obtain near optimal HPs on a large model without directly tuning it at all! While practitioners have always guessed HPs of large models from those of small models, the results are hit-or-miss at best because of incorrect parametrization. For example, as shown in Fig. 1, in a Transformer, the optimal learning rate is stable with width in $\mu$P (right) but far from so in standard parametrization (left). In addition to width, we empirically verify that, with a few caveats, HPs can also be transferred across depth (in Section 6.1) as well as batch size, language model sequence length, and training time (in Section G.2.1). This reduces the tuning problem of an (arbitrarily) large model to that of a (fixed-sized) small model. Our overall procedure, which we call $\mu$Transfer, is summarized in Algorithm 1 and Fig. 2, and the HPs we cover are summarized in Tables 1 and 2.

There are several benefits to our approach: 1. Better Performance: $\mu$Transfer is not just about predicting how the optimal learning rate scales in SP. In general, we expect the $\mu$Transferred model to outperform its SP counterpart with learning rate optimally tuned. For example, this is the case in Fig. 1 with the width-8192 Transformer. We discuss the reason for this in Sections 5 and C. 2. Speedup: It provides massive speedup to the tuning of large models. For example, we are able to outperform published numbers of (350M) BERT-large purely by zero-shot HP transfer, with tuning cost approximately equal to 1 BERT-large pretraining. Likewise, we outperform the published numbers of the 6.7B GPT-3 model with tuning cost being only 7% of total pretraining cost. For models on this scale, HP tuning is not feasible at all without our approach. 3. Tune Once for Whole Family: For any fixed family of models with varying width and depth (such as the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse its HPs for all models in the family.222but possibly not for different data and/or tasks. For example, we will use this technique to tune BERT-base (110M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M model. 4. Better Compute Utilization: While large model training needs to be distributed across many GPUs, the small model tuning can happen on individual GPUs, greatly increasing the level of parallelism for tuning (and in the context of organizational compute clusters, better scheduling and utilization ratio). 5. Painless Transition from Exploration to Scaling Up: Often, researchers explore new ideas on small models but, when scaling up, find their HPs optimized during exploration work poorly on large models. $\mu$Transfer would solve this problem.

In addition to the HP stability property, we find that wider is better throughout training in $\mu$P, in contrast to SP (Section 8). This increases the reliability of model scaling in deep learning.

In this work, we primarily focus on hyperparameter transfer with respect to training loss. In settings where regularization is not the bottleneck to test performance, as in all of our experiments here, this also translates to efficacy in terms of test loss. In other settings, such as finetuning of models on small datasets, $\mu$Transfer may not be sufficient, as we discuss in Section 6.1.

We demonstrate it is possible to zero-shot transfer near optimal HPs to a large model from a small version via the Maximal Update Parametrization ($\mu$P) from .

While only covered SGD, here we derive $\mu$P for Adam as well (Table 3).

We propose a new HP tuning technique, $\mu$Transfer, for large neural networks based on this observation that provides massive speedup over conventional methods and covers both SGD and Adam training;

We thoroughly verify our method on machine translation and large language model pretraining (in Section 7.3) as well as image classification (in Section G.1);

We release a PyTorch package for implementing $\mu$Transfer painlessly. A sketch of this package is given in Appendix H.

Sometimes, to be less ambiguous, we often refer to the “large model” as the target model, as it is the model we wish to ultimately tune, while we refer to the “small model” as the proxy model, as it proxies the HP tuning process. We follow standard notation $d_{model},d_{head}=d_{k},d_{v},n_{head},d_{ffn}$ regarding dimensions in a Transformer; one can see Fig. 11 for a refresher.

This paper is the 5th installment of the Tensor Programs series. While it is self-contained with the target audience being practitioners and empirical researchers, this paper presents the first major practical payoff of the theoretical foundation built in previous works .

Parametrization Matters: A Primer

In this section, we give a very basic primer on why the correct parametrization can allow HP transfer across width, but see Sections J.1, J.2 and J.3 for more (mathematical) details.

The Central Limit Theorem (CLT) says that, if $x_{1},\ldots,x_{n}$ are iid samples from a zero-mean, unit-variance distribution, then $\frac{1}{\sqrt{n}}(x_{1}+\cdots+x_{n})$ converges to a standard Gaussian $\mathcal{N}(0,1)$ as $n\to\infty$. Therefore, we can say that $\frac{1}{\sqrt{n}}$ is the right order of scaling factor $c_{n}$ such that $c_{n}(x_{1}+\cdots+x_{n})$ converges to something nontrivial. In contrast, if we set $c_{n}=1/n$, then $c_{n}(x_{1}+\cdots+x_{n})\to 0$; or if $c_{n}=1$, then $c_{n}(x_{1}+\cdots+x_{n})$ blows up in variance as $n\to\infty$.

Now suppose we would like to minimize the function

subscript𝑥1⋯subscript𝑥𝑛F_{n}(c)\mathbin{\overset{\mathrm{def}}{=}}\operatorname*{\mathbb{E}}_{x_{1},\ldots,x_{n}}f(c(x_{1}+\cdots+x_{n})) (1) over $c\in\mathbb{R}$, for some bounded continuous function $f:\mathbb{R}\to\mathbb{R}$. If we reparametrize $c=\alpha/\sqrt{n}$ for $\alpha\in\mathbb{R}$, then by CLT, $G_{n}(\alpha)\mathbin{\overset{\mathrm{def}}{=}}F_{n}(c)\to\operatorname*{\mathbb{E}}f(\mathcal{N}(0,\alpha^{2}))$ stabilizes into a function of $\alpha$ as $n\to\infty$. Then for sufficiently large $n$, the optimal $\alpha_{n}^{*}\mathbin{\overset{\mathrm{def}}{=}}\operatorname*{arg\;min}_{\alpha}G_{n}(\alpha)$ should be close to $\alpha_{N}^{*}$ for any $N>n$, and indeed, for $N=\infty$ — this precisely means we can transfer the optimal $c_{n}^{*}$ or $\alpha_{n}^{*}$ for a smaller problem (say $F_{n}$) to a larger problem (say $F_{N}$): $G_{N}$ is approximately minimized by $\alpha_{n}^{*}$ and $F_{N}$ is approximately minimized by $c_{n}^{*}\sqrt{n/N}$. Because the transfer algorithm is simply copying $\alpha$, we say the parametrization $c=\alpha/\sqrt{n}$ is the correct parametrization for this problem.

In the scenario studied in this paper, $x_{1},\ldots,x_{n}$ are akin to randomly initialized parameters of a width-$n$ neural network, $c$ is akin to a HP such as learning rate, and $f$ is the test-set performance of the network after training, so that $F_{n}$ gives its expectation over random initializations. Just as in this example, if we parametrize the learning rate and other HPs correctly, then we can directly copy the optimal HPs for a narrower network into a wide network and expect approximately optimal performance — this is the (zero-shot) hyperparameter transfer we propose here. It turns out the Maximal Update Parametrization ($\mu$P) introduced in is correct (akin to the parametrization in $\alpha$ above), while the standard parametrization (SP) is incorrect (akin to the parametrization in $c$). We will review both parametrizations shortly. Theoretically, a $\mu$P network has a well-defined infinite-width limit — akin to $(x_{1}+\cdots+x_{n})/\sqrt{n}$ having a $\mathcal{N}(0,1)$ limit by CLT — while a SP network does not (the limit will blow up) .333The more theoretically astute reader may observe that SP with a $\Theta(1/width)$ learning rate induces a well-defined infinite-width limit exists as well. Nevertheless, this does not allow HP transfer because this limit is in kernel regime as shown in . See Section J.3 for more discussions. In fact, based on the theoretical foundation laid in , we argue in LABEL:{sec:otherparamdontwork} that $\mu$P should also be the unique parametrization that allows HP transfer across width. For a more formal discussion of the terminologies parametrization and transfer, see Appendix A

We emphasize that, to ensure transferability of any hyperparameter (such as learning rate), it’s not sufficient to reparametrize only that hyperparameter, but rather, we need to identify and correctly reparametrize all hyperparameters in Table 2. For example, in Fig. 1, the wide models in SP still underperform their counterparts in $\mu$P, even with learning rate tuned optimally. This is precisely because SP does not scale parameter multipliers and input/output layer learning rates correctly in contrast to $\mu$P (see Table 3). See Appendix C for more intuition via a continuation of our example here. We shall also explain this more concretely in the context of neural networks in Section 5.

Hyperparameters Don’t Transfer Conventionally

In the community there seem to be conflicting assumptions about HP stability. A priori, models of different sizes don’t have any reason to share the optimal HPs. Indeed, papers aiming for state-of-the-art results often tune them separately. On the other hand, a nontrivial fraction of papers in deep learning fixes all HPs when comparing against baselines, which reflects an assumption that the optimal HPs should be stable — not only among the same model of different sizes but also among models of different designs — therefore, such comparisons are fair. Here, we demonstrate HP instability across width explicitly in MLP and Transformers in the standard parametrization. We will only look at training loss to exclude the effect of regularization.

We start with a 2-hidden-layer MLP with activation function $\phi$, using the standard parametrization444i.e. the default parametrization offered by common deep learning frameworks. See Table 3 for a review. with LeCun initialization555The key here is that the init. variance $\propto 1/\mathrm{fan\_in}$, so the same insights here apply with e.g. He initialization. akin to the default in PyTorch:

superscript𝑊limit-from2topitalic-ϕsuperscript𝑊limit-from1top𝜉superscript𝑏1superscript𝑏2with init.formulae-sequencesimilar-tosuperscript𝑊1𝒩01subscript𝑑𝑖𝑛formulae-sequencesimilar-tosuperscript𝑊23𝒩01𝑛superscript𝑏120\begin{gathered}f(\xi)=W^{3\top}\phi(W^{2\top}\phi(W^{1\top}\xi+b^{1})+b^{2})\\ \text{with init.}\quad W^{1}\sim\mathcal{N}(0,\nicefrac{{1}}{{d_{in}}}),\ W^{\{2,3\}}\sim\mathcal{N}(0,\nicefrac{{1}}{{n}}),\ b^{\{1,2\}}=0,\end{gathered} (2) Figure 3: MLP width different hidden sizes trained for 20 epoch on CIFAR-10 using SGD. Left uses standard parametrization (SP); right uses maximal update parametrization ($\mu$P). $\mu$P networks exhibit better learning rate stability than their SP counterparts. where $W^{1}\in\mathbb{R}^{d_{in}\times n},b^{1}\in\mathbb{R}^{n}$, $W^{2}\in\mathbb{R}^{n\times n},b^{2}\in\mathbb{R}^{n}$, $W^{3}\in\mathbb{R}^{n\times d_{out}}$ and $d_{in}$, $n$, and $d_{out}$ are the input, hidden, and output dimensions. The particular MLP we use has $\phi=ReLU$ and a cross-entropy (xent) loss function. We define the width of MLP as the hidden size $n$, which is varied from 256 to 8192. The models are trained on CIFAR-10 for 20 epochs, which is more than enough to ensure convergence.

As shown on the left in Fig. 3, the optimal learning rate shifts by roughly an order of magnitude as the width increases from 256 to 8192; using the optimal learning of the smallest model on the largest model gives very bad performance, if not divergence.

This perhaps unsurprising observation holds for more complex architectures such as Transformer as well, as shown in Fig. 1 (left). We define width as $d_{model}$, with $d_{k}=d_{q}=d_{v}=\nicefrac{{d_{model}}}{{n_{head}}}$ and $d_{ffn}=4d_{model}$. The models are trained on wikitext-2 for 5 epochs. In Fig. 18 in the appendix we also show the instability of initialization scale and other HPs.

Unlocking Zero-Shot Hyperparameter Transfer with μ𝜇\muP

We show that $\mu$P solves the problems we see in Section 3.

For the MLP in Section 3, to switch to $\mu$P, we just need to modify Eq. 2’s initialization of the last layer and its learning rates of the first and last layer as well as of the biases. The basic form is666 While superficially different, this parametrization is equivalent to the $\mu$P defined in .

Here, $\eta$ specifies the “master” learning rate, and we highlighted in purple the differences in the two parametrizations. This basic form makes clear the scaling with width $n$ of the parametrization, but in practice we will often insert (possibly tune-able) multiplicative constants in front of each appearance of $n$. For example, this is useful when we would like to be consistent with a SP MLP at a base width $n_{0}$. Then we may insert constants as follows: For $\tilde{n}\mathbin{\overset{\mathrm{def}}{=}}n/n_{0}$,

Then at width $n=n_{0}$, all purple factors above are 1, and the parametrization is identical to SP (Eq. 2) at width $n_{0}$. Of course, as $n$ increases from $n_{0}$, then Eq. 4 quickly deviates from Eq. 2. In other words, for a particular $n$, $\mu$P and SP can be identical up to the choice of some constants (in this case $n_{0}$), but $\mu$P determines a different “set" of networks and optimization trajectory than SP as one varies $n$. As we will see empirically in the next section, this deviation is crucial for HP transfer.

Indeed, in Fig. 3(right), we plot the CIFAR10 performances, over various learning rates and widths, of $\mu$P MLPs with $n_{0}=128$. In contrast to SP, the optimal learning rate under $\mu$P is stable. This means that, the best learning rate for a width-128 network is also best for a width-8192 network in $\mu$P — i.e. HP transfer works — but not for SP. In addition, we observe performance for a fixed learning rate always weakly improves with width in $\mu$P , but not in SP.

This MLP $\mu$P example can be generalized easily to general neural networks trained under SGD or Adam, as summarized in Table 3, which is derived in Appendix J.

We repeat the experiments with base width $n_{0}=128$ for Transformers:

The Maximal Update Parametrization ($\mu$P) for a Transformer is given by Table 3 and $1/d$ attention instead of $1/\sqrt{d}$, i.e. the attention logit is calculated as $q^{\top}k/d$ instead of $q^{\top}k/\sqrt{d}$ where query $q$ and key $k$ have dimension $d$.777This is roughly because during training, $q$ and $k$ will be correlated so $q^{\top}k$ actually scales like $d$ due to Law of Large Numbers, in contrast to the original motivation that $q$, $k$ are uncorrelated at initialization so Central Limit applies instead. See LABEL:{sec:MUPDerivation} for a more in-depth discussion.

The results are shown on the right in Fig. 1, where the optimal learning rate is stable, and the performance improves monotonically with width. See Appendix B for further explanation of $\mu$P.

The Defects of SP and How μ𝜇\muP Fixes Them

The question of SP vs $\mu$P has already been studied at length in . Here we aim to recapitulate the key insights, with more explanations given in Section J.3.

As shown in and Section J.3, in SP, the network output will blow up with width after 1 step of SGD. It’s instructive to consider a 1-hidden-layer linear perceptron $f(x)=V^{\top}Ux$ with scalar inputs and outputs, as well as weights $V,U\in\mathbb{R}^{n\times 1}$. In SP, $V_{\alpha}\sim\mathcal{N}(0,1/n)$ ad $U_{\alpha}\sim\mathcal{N}(0,1)$ for each $\alpha\in[n]$. This sampling ensures that $f(x)=\Theta(|x|)$ at initialization. After 1 step of SGD with learning rate 1, the new weights are $V^{\prime}\leftarrow V+\theta U,U^{\prime}\leftarrow U+\theta V$, where $\theta$ is some scalar of size $\Theta(1)$ depending on the inputs, labels, and loss function. But now

superscript𝑉top𝑈𝜃superscript𝑈top𝑈𝜃superscript𝑉top𝑉superscript𝜃2superscript𝑈top𝑉𝑥f(x)=V^{\prime}{}^{\top}U^{\prime}x=(V^{\top}U+\theta U^{\top}U+\theta V^{\top}V+\theta^{2}U^{\top}V)x (5) blows up with width $n$ because $U^{\top}U=\Theta(n)$ by Law of Large Numbers.

Now consider the same network in $\mu$P. According to Table 3, we now have $V_{\alpha}\sim\mathcal{N}(0,1/n^{2})$ in contrast to SP, but $U_{\alpha}\sim\mathcal{N}(0,1)$ as before, with learning rates $\eta_{V}=1/n,\eta_{U}=n$. After 1 step of SGD, we now have

superscript𝑉top𝑈𝜃superscript𝑛1superscript𝑈top𝑈𝜃𝑛superscript𝑉top𝑉superscript𝜃2superscript𝑈top𝑉𝑥f(x)=(V^{\top}U+\theta n^{-1}U^{\top}U+\theta nV^{\top}V+\theta^{2}U^{\top}V)x, and one can verify this is $\Theta(1)$ and thus does not blow up with width.888 Note in this example, Glorot initialization (i.e. with variance $1/(\mathrm{fan\_in}{}+\mathrm{fan\_out}{})$) would scale asymptotically the same as $\mu$P and thus is similarly well-behaved. However, if one adds layernorm or batchnorm, then Glorot will cause logit blowup like SP, but $\mu$P still will not.

One can observe the same behavior in more advanced architectures like Transformers and optimizers like Adam; in fact, in SP, other hidden quantities like attention logits will also blow up with width after 1 step, but in $\mu$P still remain bounded, as shown in LABEL:{fig:blowupafter1step}(middle).

One might think scaling down the learning rate with width can solve this problem in SP. However, other hidden activations like the word embedding (Fig. 5(right)) in a Transformer update by a width-independent amount for each step of training, so scaling down the learning rate will effectively mean the word embeddings are not learned in large width models. Similar conclusions apply to other models like ResNet (in fact, one can observe in the SP linear MLP example above, the input layer is updated much more slowly than the output layer). On the other hand, $\mu$P is designed so that all hidden activations update with the same speed in terms of width (see Section J.2 for why).

This is why a wide model tuned with $\mu$Transfer should in general outperform its SP counterpart with (global) learning rate tuned. For example, this is the case for the width-8192 Transformer in Fig. 1, where, in SP, the optimal learning rate needs to mollify the blow-up in quantities like logits and attention logits, but this implies others like word embeddings do not learn appreciably. This performance advantage means $\mu$Transfer does more than just predicting the optimal learning rate of wide SP models. Relatedly, we observe, for any fixed HP combination, training performance never decreases with width in $\mu$P, in contrast to SP (e.g., the $\mu$P curves in Figs. 1, LABEL:{fig:_MLP_relu_xent} and 16 do not cross, but the SP curves do; see also LABEL:{sec:widerbetter}).

Which Hyperparameters Can Be μ𝜇\muTransferred?

In this section, we explore how common HPs fit into our framework. In general, they can be divided into three kinds, summarized in Table 1:

those that can transfer from the small to the large model, such as learning rate (Table 2);

those that primarily control regularization and don’t work well with our technique; and

those that define training scale, such as width as discussed above as well as others like depth and batch size, across which we transfer other HPs.

Those in the first category transfer across width, as theoretically justified above in Section 2. To push the practicality and generality of our technique, we empirically explore the transfer across the other dimensions in the third category. Note that $\mu$Transfer across width is quite general, e.g. it allows varying width ratio of different layers or number of attention heads in a Transformer; see Section E.2. This will be very useful in practice. For the second category, the amount of regularization (for the purpose of controlling overfitting) naturally depends on both the model size and data size, so we should not expect transfer to work if the parametrization only depends on model size. We discuss these HPs in more detail in Section E.1.

Our empirical investigations focus on Transformers (here) and ResNet (in Section G.1.1), the most popular backbones of deep learning models today. We train a 2-layer pre-layernorm $\mu$P999“2 layers” means the model has 2 self-attention blocks. To compare with SP Transformer, see LABEL:{fig:_wikitext2_SP_MUP}. Transformer with 4 attention heads on Wikitext-2. We sweep one of four HPs (learning rate, output weight multiplier, initialization standard deviation, and learning rate schedule) while fixing the others and sweeping along width and depth (with additional results in Fig. 19 on transfer across batch size, sequence length, and training time). Fig. 4 shows the results averaged over 5 random seeds.

Empirically, we find that for language modeling on Transformers, HPs generally transfer across scale dimensions if some minimum width (e.g. 256), depth (e.g., 4), batch size (e.g., 32), sequence length (e.g., 128), and training steps (e.g., 5000) are met, and the target scale is within the “reasonable range” as in our experiments. Now, there are some caveats. While the exact optimum can shift slightly with increasing scale, this shift usually has very small impact on the loss, compared to SP (Figs. 1 and 3(left)). However, there are some caveats. For example, the best initialization standard deviation does not seem to transfer well across depth (2nd row, 3rd column), despite having a stabler optimum across width. In addition, while our results on width, batch size, sequence length, and training time still hold for post-layernorm (Fig. 17),101010in fact, post-layernorm Transformers are much more sensitive to HPs than pre-layernorm, so our technique is more crucial for them, especially for transfer across width. Fig. 1 uses post-layernorm. the transfer across depth only works for pre-layernorm Transformer. Nevertheless, in practice (e.g. our results in Section 7.3) we find that fixing initialization standard deviation while tuning other HPs works well when transferring across depth.

Efficiency and Performance of μ𝜇\muTransfer

Now that the plausibility of $\mu$Transfer has been established in toy settings, we turn to more realistic scenarios to see if one can achieve tangible gains. Specifically, we perform HP tuning only on a smaller proxy model, test the obtained HPs on the large target model directly, and compare against baselines tuned using the target model. We seek to answer the question: Can $\mu$Transfer make HP tuning more efficient while achieving performance on par with traditional tuning? As we shall see by the end of the section, the answer is positive. We focus on Transformers here, while experiments on ResNets on CIFAR10 and Imagenet can be found as well in Section G.1. All of our experiments are run on V100 GPUs.

IWSLT14 De-En is a well-known machine translation benchmark. We use the default IWSLT (post-layernorm) Transformer implemented in fairseq with 40M parameters, which we denote as the 1x model.111111https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md. For $\mu$Transfer, we tune on a 0.25x model with $1/4$ of the width, amounting to 4M parameters. For this experiment, we tune via random search the learning rate $\eta$, the output layer parameter multiplier $\alpha_{output}$, and the attention key-projection weight multiplier $\alpha_{attn}$. See the grid and other experimental details in Section F.1.

We compare transferring from the 0.25x model with tuning the 1x model while controlling the total tuning budget in FLOPs.121212Ideally we would like to measure the wall clock time used for tuning. However, smaller models such as the proxy Transformer used for IWSLT are not efficient on GPUs, so wall clock time would not reflect the speedup for larger models like GPT-3. Thus, we measure in FLOPs, which is less dependent on hardware optimization. To improve the reproducibility of our result: 1) we repeat the entire HP search process (a trial) 25 times for each setup, with number of samples as indicated in Table 4, and report the 25th, 50th, 75th, and 100th percentiles in BLEU score; 2) we evaluate each selected HP combination using 5 random initializations and report the mean performance.131313We do not report the standard deviation over random initializations to avoid confusion.

We pick the HP combination that achieves the lowest validation loss141414We find this provides more reliable result than selecting for the best BLEU score. for each trial. The reported best outcome is chosen according to the validation loss during tuning. We compare against the default in fairseq, which is presumably heavily tuned. The result is shown in Table 4.

The result above only describes a particular compute budget. Is $\mu$Transfer still preferable when we have a lot more (or less) compute? To answer this question, we produce the compute-performance Pareto frontier in Fig. 6(left), where we repeat the above experiment with different compute budgets. Evidently, our approach completely dominates conventional tuning.

The Pareto frontier in Fig. 6(right) suggests that, given a fixed number of random samples from the HP space, 1) tuning the target model directly yields slightly better results than tuning the proxy model (while taking much more compute of course), but 2) this performance gap seems to vanish as more samples are taken. This can be explained by the intuition that the narrower proxy model is a “noisy estimator” of the wide target model .With few samples, this noise can distort the random HP search, but with more samples, this noise is suppressed.

2 Transformer on WMT14 En-De

We scale up to WMT14 En-De using the large (post-layernorm) Transformer from with 211M parameters. We tune on a proxy model with 15M parameters by shrinking $d_{model}$, $d_{ffn}$, and $n_{head}$. For this experiment, we tune via random search the learning rate $\eta$, the output layer parameter multiplier $\alpha_{output}$, and the attention key-projection weight multiplier $\alpha_{attn}$ following the grid in Section F.2. The result is shown in Table 5: While random search with 3 HP samples far underperforms the fairseq default, we are able to match it via transfer using the same tuning budget.

3 BERT

Finally, we consider large-scale language model pretraining where HP tuning is known to be challenging. Using Megatron (pre-layernorm) BERT as a baseline, we hope to recover the performance of the published HPs by only tuning a proxy model that has roughly 13M parameters, which we call BERT-prototype. While previous experiments scaled only width, here we will also scale depth, as discussed in Section 6 and validated in Fig. 4. We use a batch size of 256 for all runs and follow the standard finetuning procedures. For more details on BERT-prototype, what HPs we tune, and how we finetune the trained models, see Section F.3.

During HP tuning, we sample 256 combinations from the search space and train each combination on BERT-prototype for $10^{5}$ steps. The total tuning cost measured in FLOPs is roughly the same as training 1 BERT-large for the full $10^{6}$ steps; the exact calculation is shown in Section F.3. The results are shown in Table 6. Notice that on BERT-large, we obtain sizeable improvement over the well-tuned Megatron BERT-large baseline.

4 GPT-3

In order to further verify $\mu$Transfer at scale, we applied it to GPT-3 6.7B with relative attention. This target model consists of 32 residual blocks with width 4096. We form the small proxy model by shrinking width to 256, resulting in roughly 40 million trainable parameters, 168 times smaller than the target model. HPs were then determined by a random search on the proxy model. The total tuning cost was only 7% of total pretraining cost. Details of the HP sweep can be found in Section F.4.

In order to exclude code difference as a possible confounder, we also re-trained GPT-3 6.7B from scratch using the original HPs from . Unfortunately, after we have finished all experiments, we found this baseline mistakenly used absolute attention (like models in ) when it was supposed to use relative attention like the target model. In addition, during training of the $\mu$Transfer model we encountered numerical issues that lead to frequent divergences. In order to avoid them, the model was trained using FP32 precision, even though the original 6.7B model and our re-run were trained using FP16.151515 While we are mainly focused on the efficacy of $\mu$Transfer regardless of precision, it would be interesting to ablate the effect of precision in our results, but we did not have enough resources to rerun the baseline in FP32 161616 It is quite interesting that $\mu$Transfer identified a useful region of hyperparameters leading to much improved performance, which probably would be difficult to discover normally because 1) researchers usually change hyperparameters to accomodate precision and 2) there was no precise enough justification to go against this judgment until $\mu$Transfer. The resulting $\mu$Transfer model outperforms the 6.7B from , and is in fact comparable to the twice-as-large 13B model across our evaluation suite (see LABEL:{tab:gpt3-evals-13b-comparison}). Selected evaluation results can be found in Table 7 and further details are given in Table 10 and Section F.4.

Wider is Better in μ𝜇\muP Throughout Training

In earlier plots like LABEL:{fig:_figure1} and LABEL:{fig:_MLP_relu_xent}, we saw that at the end of training, wider is always better in $\mu$P but not in SP. In fact, we find this to be true throughout training, as seen in Fig. 7, modulo noise from random initialization and/or data ordering, and assuming the output layer is zero-initialized (which has no impact on performance as discussed in Section D.2). We then stress-tested this on a $\mu$P GPT-3 Transformer (on the GPT-3 training data) by scaling width from 256 to 32,768 using a fixed set of HPs (Fig. 8). Wider models consistently match or outperform narrower models at each point in training (except a brief period around 1e8 training tokens, likely due to noise because we ran only 1 seed due to computational cost). Our observation suggests that wider models are strictly more data-efficient if scaled appropriately. By checking “wider-is-better” early in training, one can also cheaply debug a $\mu$P implementation.

Useful Hyperparameter Transfer: A Theoretical Puzzle

We want to tune HPs on a small model with width $N$ such that its HP landscape looks like that of a large model with width $\gg N$. Our intuition in LABEL:{sec:primer}, LABEL:{sec:primermulti} and LABEL:{sec:intuitiveintro2muP} leads us to $\mu$P. However, for this to be useful, we do not want the small model (as a function) after training to be close to that of the large model — otherwise there is no point in training the large model to begin with. So $N$ 1) must be large enough so that the HP optimum converges, but 2) cannot be so large that the functional dynamics (and the loss) converges. The fact that such $N$ exists, as demonstrated by our experiments, shows that: In some sense, the HP optimum is a “macroscopic” or “coarse” variable which converges quickly with width, while the neural network function (and its loss) is a very “microscopic” or “fine” detail that converges much more slowly with width. However, theoretically, it is unclear why this should happen, and where else we should expect such useful HP transfer. We leave an explanation to future work.

Related Works

Many have sought to speedup HP tuning beyond the simple grid or random search. Snoek et al. treated HP tuning as an optimization process and used Bayesian optimization by treating the performance of each HP combination as a sample from a Gaussian process (GP). Snoek et al. further improved the runtime by swapping the GP with a neural network. Another thread of work investigated how massively parallel infrasture can be used for efficient tuning under the multi-arm bandit problem . There are also dedicated tools such as Optuna and Talos which integrate with existing deep learning frameworks and provide an easy way to apply more advanced tuning techniques.

Our approach is distinct from all of the above in that it does not work on the HP optimization process itself. Instead, it decouples the size of the target model from the tuning cost, which was not feasible prior to this work. This means that no matter how large the target model is, we can always use a fixed-sized proxy model to probe its HP landscape. Nevertheless, our method is complementary, as the above approaches can naturally be applied to the tuning of the proxy model; it is only for scientific reasons that we use either grid search or random search throughout this work.

2 Hyperparameter Transfer

Many previous works explored transfer learning of HP tuning (e.g. ). However, to the best of our knowledge, our work is the first to explore zero-shot HP transfer. In addition, we focus on transferring across model scale rather than between different tasks or datasets. Some algorithms like Hyperband can leverage cheap estimates of HP evaluations (like using a small model to proxy a large model) but they are not zero-shot algorithms, so would still be very expensive to apply to large model training. Nevertheless, all of the above methods are complementary to ours as they can be applied to the tuning of our proxy model.

3 Previously Proposed Scaling Rules of Hyperparameters

proposed to scale learning rate with batch size while fixing the total epochs of training; proposed to scale learning rate as $\sqrt{batchsize}$ while fixing the total number of steps of training. However, showed that there’s no consistent (learning rate, batch size) scaling law across a range of dataset and models. Later, studied the trade-off of training steps vs computation as a result of changing batch size. They proposed an equation of $a/(1+b/batchsize)$, where $a$ and $b$ are task- and model-specific constants, for the optimal learning rate (see their fig 3 and fig 5). This law suggests that for sufficiently large batch size, the optimal learning rate is roughly constant.171717while the optimal learning is roughly linear in batch size when the latter is small This supports our results here as well as the empirical results in [41, fig 8].

Assuming that the optimal learning rate should scale with batch size following , empirically investigated how the optimal “noise ratio” $LR/batchsize$ scales with width for MLP and CNNs in NTK parametrization (NTP) or standard parametrization (SP) trained with SGD. They in particular focus on test loss in the regime of small batch size and training to convergence. In this regime, they claimed that in networks without batch normalization, the optimal noise ratio is constant in SP but scales like $1/width$ for NTP. However, they found this law breaks down for networks with normalization.

In contrast, here we focus on training loss, without training to convergence and with a range of batch sizes from small to very large (as is typical in large scale pretraining). Additionally, our work applies universally to 1) networks with normalization, along with 2) Adam and other adaptive optimizers; furthermore 3) we empirically validate transfer across depth and sequence length, and 4) explicitly validate tuning via $\mu$Transfer on large models like BERT-large and GPT-3.

Finally, as argued in and LABEL:{sec:otherparamdontwork}, SP and NTP lead to bad infinite-width limits in contrast to $\mu$P and hence are suboptimal for wide neural networks. For example, sufficiently wide neural networks in SP and NTP would lose the ability to learn features, as concretely demonstrated on word2vec in .

The original formulation of $\mu$P in (see Table 9, which is equivalent to Table 3) uses a fan-out initialization for the input layer. This is atypical in vision models, but in language models where the input and output layers are shared (corresponding to word embeddings), it can actually be more natural to use a fan-out initialization (corresponding to fan-in initialization of the output layer). In fact, we found that fairseq by default actually implements both the fan-out initialization and the $\sqrt{\mathrm{fan\_out}}$ multiplier.

Many previous works proposed different initialization or parametrizations with favorable properties, such as better stability for training deep neural networks . Our work differs from these in that we focus on the transferability of optimal HPs from small models to large models in the same parametrization.

4 Infinite-Width Neural Networks: From Theory to Practice and Back

introduced $\mu$P as the unique parametrization that enables all layers of a neural network to learn features in the infinite-width limit, especially in contrast to the NTK parametrization (which gives rise to the NTK limit) that does not learn features in the limit. Based on this theoretical insight, in LABEL:{sec:otherparamdontwork}, we argue that $\mu$P should also be the unique parametrization (in the sense of ) that allows HP transfer across width; in short this is because it both 1) preserves feature learning, so that performance on feature learning tasks (such as language model pretraining) does not become trivial in the limit, and 2) ensures each parameter tensor is not stuck at initialization in the large width limit, so that its learning rate does not become meaningless. At the same time, our results here suggest that $\mu$P is indeed the correct parametrization for wide neural networks and thus provide empirical motivation for the theoretical study of the infinite-width $\mu$P limit. Note, parametrization here refers to a rule to scale hyperparameters with width (“how should my initialization and learning rate change when my width doubles?”), which is coarser than a prescription for setting hyperparameters at any particular width (“how should I set my initialization and learning rate at width 1024?”).

Conclusion

Leveraging the discovery of a feature learning neural network infinite-width limit, we hypothesized and verified that the HP landscape across NNs of different width is reasonably stable if parametrized according to Maximal Update Parametrization ($\mu$P). We further empirically showed that it’s possible to transfer across depth, batch size, sequence length, and training time, with a few caveats. This allowed us to indirectly tune a very large network by tuning its smaller counterparts and transferring the HPs to the full model. Our results raise an interesting new theoretical question of how useful HP transfer is possible in neural networks in the first place.

Nevertheless, our method has plenty of room to improve. For example, initialization does not transfer well across depth, and depth transfer generally still does not work for post-layernorm Transformers. This begs the question whether a more principled parametrization in depth could solve these problems. Additionally, Fig. 4 shows that the optimal HP still shifts slightly for smaller models. Perhaps by considering finite-width corrections to $\mu$P one can fix this shift. Finally, it will be interesting to study if there’s a way to transfer regularization HPs as a function of both the model size and data size, especially in the context of finetuning of pretrained models.

In alphabetical order, we thank Arthur Jacot, Arturs Backurs, Colin Raffel, Denny Wu, Di He, Huishuai Zhang, Ilya Sutskever, James Martens, Janardhan Kulkarni, Jascha Sohl-Dickstein, Jeremy Bernstein, Lenaic Chizat, Luke Metz, Mark Chen, Michael Santacroce, Muhammad ElNokrashy, Pengchuan Zhang, Sam Schoenholz, Sanjeev Arora, Taco Cohen, Yiping Lu, Yisong Yue, and Yoshua Bengio for discussion and help during our research.

References

Appendix A Parametrization Terminologies

This section seeks to make formal and clarify some of the notions regarding parametrization discussed informally in the main text.

In a neural network, one may insert a “multiply by $c$” operation anywhere, where $c$ is a non-learnable scalar hyperparameter. If $c=1$, then this operation is a no-op. This $c$ is called a multiplier.

Relatedly, for any parameter tensor $W$ in a neural network, we may replace $W$ with $cW$ for some non-learnable scalar hyperparameter $c$. When $c=1$, we recover the original formulation. This $c$ is referred to as a parameter multiplier.

For example, in the attention logit calculation $\langle k,q\rangle/\sqrt{d_{head}}$ where $q=Wx$, the $1/\sqrt{d_{head}}$ factor is a multiplier. It may also be thought of as the parameter multiplier of $W$ if we rewrite the attention logit as $\langle k,(W/\sqrt{d_{head}})x\rangle$.

Note parameter multipliers cannot be absorbed into the initialization in general, since they affect backpropagation. Nevertheless, after training is done, parameter multipliers can always be absorbed into the weight.

In this work, a parametrization is a rule for how to change hyperparameters when the widths of a neural network change, but note that it does not necessarily prescribes how to set the hyperparameters for any specific width. In particular, for any neural network, an abc-parametrization is a rule for how to scale a) the parameter multiplier, b) the initialization, and c) the learning rate individually for each parameter tensor as the widths of the network change, as well as any other multiplier in the network; all other hyperparameters are kept fixed with width.

For example, SP and $\mu$P are both abc-parametrizations. Again, we note that, in this sense, a parametrization does not prescribe, for example, that the initialization variance be $1/\mathrm{fan\_in}$, but rather that it be halved when $\mathrm{fan\_in}$ doubles.

In this work, we say a parametrization admits zero-shot transfer of a set of hyperparameters $\mathcal{H}$ w.r.t. a metric $\mathcal{L}$ if the optimal combination of values of $\mathcal{H}$ w.r.t. $\mathcal{L}$ converges as width goes to infinity, i.e. it stays approximately optimal w.r.t. $\mathcal{L}$ under this parametrization as width increases.

Throughout this paper, we take $\mathcal{L}$ to be the training loss, but because regularization is not the bottleneck in our experiments (especially large scale pretraining with BERT and GPT-3), we nevertheless see high quality test performance in all of our results. We also remark that empirically, using training loss as the metric can be more robust to random seed compared to validation loss and especially BLEU score. See Table 1(left) for $\mathcal{H}$. By our arguments in LABEL:{sec:otherparamdontwork} and our empirical results, $\mu$P is the unique abc-parametrization admitting zero-shot transfer for such $\mathcal{H}$ and $\mathcal{L}$ in this sense.

More generally, one may define a $K$-shot transfer algorithm of a set of hyperparameters $\mathcal{H}$ w.r.t. a metric $\mathcal{L}$ as one that 1) takes width values $n$ and $n^{\prime}$ and an approximately optimal combination of values of $\mathcal{H}$ w.r.t. $\mathcal{L}$ at a width $n$ and 2) returns an approximately optimal combination of values of $\mathcal{H}$ w.r.t. $\mathcal{L}$ at width $n^{\prime}$, given 3) a budget of $K$ evaluations of candidate hyperparameter combinations on models of width $n^{\prime}$. However, we will have no use for this definition in this paper.

Appendix B Further Explanations of the μ𝜇\muP Tables

In addition to Table 3, we provide Table 8 as an equivalent $\mu$P formulation that is easier to implement, as well as Table 9 for those more familiar with the original $\mu$P formulation in . Below, we provide some commentary on corner cases not well specified by the tables. Ultimately, by understanding Appendix J, one can derive $\mu$P for any architecture, new or old.

We can classify any dimension in a neural network as “infinite” if it scales with width, or “finite” otherwise. For example, in a Transformer, $d_{model},d_{ffn},d_{head},n_{head}$ are all infinite, but vocab size and context size are finite. Then we can categorize parameter tensors by how many infinite dimensions they have. If there are two such dimensions, then we say the parameter is matrix-like; if there is only one, then we say it is vector-like; if there is none, we say it is scalar-like. Then in Tables 3, 8 and 9, “input weights & all biases” and “output weights” are all vector-like parameters, while hidden weights are matrix-like parameters. An advantage of Table 8 is that it gives a uniform scaling rule of initialization and learning rate for all vector-like parameters. The multiplier rule in Table 8 can be more interpreted more generally as the following: a multiplier of order $1/\mathrm{fan\_in}$ should accompany any weight that maps an infinite dimension to a finite one. This interpretation then nicely covers both the output logits and the attention logits (i.e. $1/d$ attention).

Scalar-like parameters are not as common as matrix-like and vector-like ones, but we will mention a few examples in Section B.2. The scaling rule for their initialization, learning rate (for both SGD and Adam), and multiplier is very simple: hold them constant with width.

We did not specify the initialization mean in the tables, since most commonly the mean is just set to 0, but it can be nonzero for vector-like parameters (e.g., layernorm weights) and scalar-like parameters but must be 0 for matrix-like parameters.

The initialization scaling rules in our tables can all be trivially satisfied if the initialization variance is set to 0. This can be useful in some settings (e.g., Section D.2) but detrimental in other settings (e.g., hidden weights).

Here, input weights very specifically refer to weights that map from an infinite dimension to a finite dimension. As a counterexample, in some architectures, the first layer can actually map from a finite dimension to another finite dimension, e.g., a PCA layer. Then this is not an “input weight”; if the next layer maps into an infinite dimension, then that’s the input weight. A similar, symmetric discussion applies to output weights.

For our tables, a model is specifically a function that maps a finite dimension to another finite dimension, consistent with the discussion above. For example, for an image model on CIFAR10, it maps from $3\times 32\times 32=3072$ dimensions to 10 dimensions, and these numbers are fixed regardless of the width of the model. Likewise, for an autoregressive Transformer model, the input and output dimension are both the vocab size, which is independent of the width. In contrast, an MLP inside a Transformer is not a “model” in this sense because its input and output dimension are both equal to the width of the Transformer.

B.1 Walkthrough of μ𝜇\muP Implementation in a Transformer

To ground the abstract description in Tables 3, 8 and 9, we walk through the parameters of a typical Transformer and discuss concretely how to parametrize each.

We assume that the user wants to replicate SP when the model widths are equal to some base widths, for example, when $d_{model}=d_{model,0}=128,d_{ffn}=d_{ffn,0}=512$, etc, as in the MLP example in Section 4. For this purpose, it’s useful to define $\tilde{d}_{model}=d_{model}/d_{model,0},\ \tilde{d}_{ffn}=d_{ffn}/d_{ffn,0}$, and so on. One can always take $d_{model,0}=d_{ffn,0}=\cdots=1$ for a “pure” $\mu$P.

Below, we introduce hyperparameters $\sigma_{\bullet},\eta_{\bullet}$ for each parameter tensor, as well as a few multipliers $\alpha_{\bullet}$. One may always tie $\sigma_{\bullet}$ (resp. $\eta_{\bullet}$) across all parameter tensors, but in our experiments, we found it beneficial to at least distinguish the input and output layer initialization and learning rates.

The input word embedding matrix $W^{wordemb}$ has size $d_{model}\times vocabsize$, where $vocabsize$ is the fan-in and $d_{model}$ is the fan-out. Follow the “input weight & all biases” column in Tables 3, 8 and 9. For example, for Tables 3 and 8,

Note here, because fan-in ($vocabsize$) here is independent of width ($d_{model}$), the “$1/\mathrm{fan\_in}$” for the initialization variance in these tables is equivalent to “$1$”, i.e. the initialization variance can be anything fixed with width. In this case of the word embedding, setting the variance to 1, for example, is more natural than setting the variance to $1/\mathrm{fan\_in}$, because the embedding is one-hot ($1/\mathrm{fan\_in}$ would be more natural for image inputs).

The (absolute or relative) positional embedding matrix $W^{posemb}$ has size $d_{model}\times contextsize$, where $contextsize$ is the fan-in and $d_{model}$ is the fan-out. With the same discussion as above for input word embeddings, follow the “input weight & all biases” column in Tables 3, 8 and 9. For example, for Tables 3 and 8,

Layernorm weights $w^{LN}$ and biases $b^{LN}$ both have shape $d_{model}$ and can be thought of “input weights” to the scalar input of 1. Hence one should follow the “input weight & all biases” column in Tables 3, 8 and 9. In particular, the usual initialization of layernorm weights as all 1s and biases as all 0s suffice (where the initialization variance is 0). For example, for Tables 3 and 8,

There are 4 matrices, $W^{q},W^{k}\in\mathbb{R}^{(d_{k}n_{head})\times d_{model}}$, $W^{v}\in\mathbb{R}^{(d_{v}n_{head})\times d_{model}}$, and $W^{o}\in\mathbb{R}^{d_{model}\times(d_{v}n_{head})}$ (where the shapes are $\mathbb{R}^{\mathrm{fan\_out}\times\mathrm{fan\_in}}$). Since $d_{model}$, $(d_{k}n_{head})$, and $(d_{v}n_{head})$ all scale with width (where the latter two are commonly just set to $d_{model}$), all 4 matrices should be parametrized according to the “hidden weights” column in Tables 3, 8 and 9. For example, for Tables 3 and 8,

We use $1/d$ instead of $1/\sqrt{d}$ attention. To be compatible with $1/\sqrt{d}$ attention when at a particular base $d_{head}=d_{head,0}$, we set

where $\alpha_{attn}$ is a tunable multiplier.

There are 2 matrices, $W^{1}\in\mathbb{R}^{d_{ffn}\times d_{model}},W^{2}\in\mathbb{R}^{d_{model}\times d_{ffn}}$ (where the shapes are $\mathbb{R}^{\mathrm{fan\_out}\times\mathrm{fan\_in}}$), where $d_{ffn}$ is commonly set to $4d_{model}$. Since both $d_{model},d_{ffn}$ scale with width, both matrices are considered “hidden weights.” For example, for Tables 3 and 8,

Symmetric to the discussion on input word embeddings, the output word unembeddings should be parametrized according to the “output weights” column of Tables 3, 8 and 9. Often, the unembeddings are tied with the embeddings, and Tables 8 and 9 allow for this as their initialization schemes are symmetric between input and output weights.

(note $d_{model,0}$ here is the base width and therefore is a constant) and the output is computed as

where $z$ is the final layer embedding of a token, and $\alpha_{output}$ is a tunable multiplier.

B.2 Other Parameters

For learnable scalar multipliers (e.g., softmax inverse temperature), one can initialize them to 1 and use a constant (in width) learning rate for both SGD and Adam. This is compatible with Tables 3, 8 and 9.

Some Transformers use positional bias (of size $contextsize\times contextsize$, which are added to the attention logits). They are considered “scalar-like” in that it has no width dimension. One can initialize them to 0 and use a constant (in width) learning rate for both SGD and Adam. This is compatible with Tables 3, 8 and 9.

Recent works on MLP-only architectures in NLP and CV replace the self-attention layer in Transformers with MLPs across tokens or spatial locations. In our language here, such MLPs have finite input and output dimensions (the context size) and infinite hidden dimensions, so their input, output, and hidden weights should be parametrized via the corresponding columns in Tables 3, 8 and 9.

B.3 Optimizer Variants and Hyperparameters

Exactly the same as Adam in all of our tables, with the added benefit that weight decay is automatically scaled correctly in AdamW (but is incompatible with $\mu$P Adam). For this reason, we recommend using AdamW when weight decay is desired (which is consistent with current standard practice).

LARS , Adafactor , Lamb , Layca , Fromage , Nero all involve a normalization step in which the update $g$ (which may be obtained from SGD, Adam, or other optimzers) is normalized to have Frobenius norm equal to that of the parameter $w$: $g\leftarrow\frac{\|w\|_{F}}{\|g\|_{F}}g$. They can be made compatible with $\mu$P in Table 8 by scaling their learning rate for hidden weights like $1/\sqrt{\mathrm{fan\_in}}$ (for Table 3, the output weight learning rate should be likewise scaled). The intuitive reasoning (which can be formalized straightforwardly using Tensor Programs) is as follows.

This normalization implicitly encodes a width scaling: If one initializes a weight matrix with variance $1/\mathrm{fan\_in}{}$, then an $n\times n$ matrix (e.g., a hidden weight matrix) has Frobenius norm $\sqrt{n}$ at initialization. Thus, in the first step and, by induction, in any step $t$, the normalized update to this $n\times n$ weight also has Frobenius norm $\Theta(\sqrt{n})$ (for any fixed $t$, as $n\to\infty$). Heuristically, this means each entry of $g$ is approximately of size $\Theta(1/\sqrt{n})$. But, by the derivation of LABEL:{sec:intuitiveintro2muP}, we want $\Theta(1/n)$ and this is $\Theta(\sqrt{n})$ too large! Thus, in wide enough networks, one should see a network blowup after one update, like demonstrated in Fig. 5.

However, note that the $\Theta(1/\sqrt{n})$ coordinate size induced by the normalization here is closer to the right size $\Theta(1/n)$ than Adam, whose update have coordinate size $\Theta(1)$. This may partially explain the apparent benefit of these optimizers. In particular, this may explain the observation that T5 , using Adafactor, was able to train its entire range of models from 220 million to 11 billion parameters with a fixed set of hyperparameters, while GPT-3 , using Adam, needed to decrease its learning rate with model size.

RAdam is a variant of Adam that uses SGD with momentum in an initial stage with learning rate warmup, followed by a second stage of Adam with a particular setting of learning rate with time. Thus, one can adapt RAdam to $\mu$P by individually scaling the learning rates of the initial SGD stage and the final Adam stage according to Table 3, Table 8, or Table 9.

Exactly the same as Adam in all of our tables.

All of our derivations here assume $\epsilon$ is negligible in Adam. If it is set to a non-negligible number, then it needs to be scaled, for all parameters, like $1/\mathrm{fan\_in}^{2}$ if it is added before the square root, or like $1/\mathrm{fan\_in}$ if it is added after the square root.

Gradient ($\ell_{2}$-norm-wise) clipping is compatible with Table 3 (as well as Tables 8 and 9), for either SGD or Adam, if the clip value is held constant with respect to width.

Weight decay should be scaled independently of width in SGD and AdamW, for all of our tables. However, note it’s not compatible with $\mu$P Adam.

Momentum should be scaled independently of width for all of our tables.

Appendix C Parametrization Matters: A Primer for Multiple Hyperparameters

Here we give more intuition why we need to reparametrize all hyperparameters. In practice, neural networks have multitudes of hyperparameters all interacting together. In our example of Section 2, hyperparameter optimization would be akin to minimizing the function181818 Here, for simplicity of the example, we model the interaction between “hyperparameters” $c^{1},\ldots,c^{k}$ as additive, but in real neural networks such interactions are usually much more complicated.

superscript𝑐1⋯superscript𝑐𝑘subscript𝑥1⋯subscript𝑥𝑛F_{n}(c^{1},\ldots,c^{k})\mathbin{\overset{\mathrm{def}}{=}}\operatorname*{\mathbb{E}}_{x_{1},\ldots,x_{n}}f((c^{1}+\cdots+c^{k})(x_{1}+\cdots+x_{n})). where $x_{1},\ldots,x_{n}$ are as in Eq. 1 and $c^{1},\ldots,c^{k}$ are analogous to $k$ hyperparameters. For the same reasoning in Section 2, the correct parametrization is in $(\alpha^{1},\ldots,\alpha^{k})$ where $\alpha^{i}=c^{i}\sqrt{n}$.

While this is straightforward, in practice, researchers often fix some hyperparameters (e.g., they tune only learning rate but neglects to scale parameter multipliers or initialization correctly). For example, if we only partially reparametrize and optimize in $\alpha^{1}$ while fixing $c^{2},\ldots,c^{k}$, then the optimal $\alpha^{1}$ is $(\alpha^{1})^{*}=\alpha^{*}-(c^{1}+\ldots+c^{k})\sqrt{n}$ where $\alpha^{*}$ is the optimal $\alpha$ for Eq. 1. Thus, as $n\to\infty$, $(\alpha^{1})^{*}$ still blows up even though we parametrized $\alpha^{1}$ correctly. More generally, the incorrect parametrization of some hyperparameters forces other hyperparameters to increasingly compensate for it as width grows, distorting their optima, even if the latter are correctly parametrized.

Appendix D Practical Considerations

In this section, we outline several useful tips and tricks that can improve the quality of hyperparameter transfer in practice.

Even though $\mu$P is neatly encapsulated by Table 3, implementing it correctly can in practice be error-prone, just like how implementing autograd by hand can be error-prone even though the math behind is just chain-rule. In the case of autograd, gradient checking is a simple way of verifying implementation correctness; similarly, we propose coordinate checking to verify the correctness of $\mu$P implementation: Exemplified by Fig. 5, one calculates the average coordinate size of every (pre)activation vector in the network over a few steps of training, as width is varied over a large range. An incorrect implementation will see some activation vector blow up or shrink to zero with width (like in the top row of Fig. 5). In the mup package we release with this paper, we include an easy-to-use method for coordinate checking.

D.2 Zero Initialization for Output Layers and Query Layers in Attention

We find that the optimal hyperparameters of small and large width models match more closely when we initialize output layers at 0 (i.e. with variance $\sigma^{2}/\mathrm{fan\_in}{}$ where $\sigma=0$ instead of positive $\sigma$). This is because the neural network in $\mu$P is approximately a Gaussian process (GP) at initialization with variance on the order $\Theta(\sigma^{2}/width)$ (contrast this with SP networks, which approximates a GP with $\Theta(\sigma^{2})$ variance) . Of course, when width is large, this variance vanishes, but this can be far from so in the small proxy model. This discrepancy in the initial GP can cause the training trajectory of the proxy model to be very different from the trajectory of the large target model, causing a mismatch in the optimal hyperparameters. By initializing the output layer at 0, we remove this mismatch in the initial GP. Empirically we do not find this modification to be detrimental to performance.

A similar consideration applies to the query layer in self-attention: At initialization, the attention logit $q^{\top}k/{d_{head}}$ looks like a Gaussian with variance $\Theta(1/d_{head})$ because $q$ and $k$ are almost independent and zero-mean. In the limit $d_{head}\to\infty$, the logit is exactly 0, which can be a large discrepancy compared to when $d_{head}$ is small in the small proxy model we want to tune. By initializing the query projection matrix $W^{q}$ to 0, $q$ will also be 0, and hence the attention logit is always 0 at initialization regardless of width (but will generally become nonzero after a gradient step), resolving this discrepancy.

More generally, any layer or computation that goes from an “infinite” dimension (i.e. width) to a “finite” dimension (e.g. output dimension or sequence length) can exhibit this kind of discrepancy due to the initial GP. When $d_{head}\to\infty$ and $n_{head}$ is fixed, attention logit calculation can be viewed in the same vein as a function $\mathbb{R}^{seqlen\times d_{model}}\to\mathbb{R}^{n_{head}\times seqlen\times seqlen}$, which “reduces to” $\mathbb{R}^{\infty}\to\mathbb{R}^{1}$.

D.3 Activation Functions

When the network is narrow, its approximation to the infinite-width behavior becomes crude, which is manifested as large fluctuations in preactivation coordinates. When using a squashing activation functions like softmax or tanh, this causes narrower networks to saturate the activation more than wider ones, which results in a systematic bias toward small gradients and therefore distorting the hyperparameter landscape. This can be seen in Fig. 9, where we use tanh as the network activation function.

Therefore, we recommend replacing non-essential squashing activation functions with ReLU, whose derivative depends only on the sign of the pre-activation. A similar reasoning can be applied to superlinear activation functions, where the distribution of activation values can have heavy tails, leading to slow convergence to the infinite-width limit. However, such activations are rarely used in practice.

D.4 Enlarge dksubscript𝑑𝑘d_{k}

We find that small $d_{head}=d_{k}$ can lead to a highly noisy HP landscape, as shown in LABEL:{fig:Widen_QK}. This can significiantly decrease the quality of random HP search on the small proxy model. To solve this, we find it useful to decouple $d_{k}$ from $d_{model}$ (so that $d_{model}\neq d_{k}\cdot n_{head}$) and maintain a relatively large $d_{k}$ even as $d_{model}$ is shrunk in the proxy model. For example, pegging $d_{k}=32$ is generally effective. Training or inference speed are not usually affected much by the larger $d_{k}$ because of CUDA optimizations. By Section E.2, this decoupling of $d_{k}$ from $d_{model}$ is theoretically justified, and as shown in LABEL:{fig:Widen_QK}, it significantly denoises the HP landscape.

D.5 Non-Gaussian vs Gaussian Initialization

We find non-Gaussian (e.g. uniform) initialization can sometimes cause wider models to perform worse than narrower models, whereas we do not find this behavior for Gaussian initialization. This is consistent with theory, since in the large width limit, one should expect non-Gaussian initialization to behave like Gaussian initializations anyway (essentially due to Central Limit Theorem, or more precisely, universality), but the non-Gaussianity slows down the convergence to this limit.

D.6 Using a Larger Sequence Length

For Transformers, we empirically find that we can better transfer initialization standard deviation from a narrower model (to a wide model) if we use a larger sequence length. It is not clear why this is the case. We leave an explanation to future work.

D.7 Tuning Per-Layer Hyperparameters

The techniques in this paper allow the transfer across width of (learning rate, initialization, multipliers) simultaneously for all parameter tensors. Thus, to get the best results, one should ideally tune all such hyperparameters. In practice, we find that just tuning the global learning rate and initialization, along with input, output, and attention multipliers, yield good results.

Appendix E Which Hyperparameters Can Be Transferred? (Continued)

Below, we discuss the reasoning behind each kind, which are supported by our empirical evidence collected in Fig. 4 on Transformers as well as those in Section G.1 on ResNet.

In Table 2, we summarize which HPs can be transferred across training scale. The transfer across width, as explained in Section 2, is theoretically justified, while we present the transfer across the other dimensions as empirical results.

These cover most of the well-known and important HPs when the need for regularization is not paramount, e.g., during large scale language model pretraining. Parameter Multipliers are not well-known HPs, yet we include them here as they serve a bridge between SP and $\mu$P and can impact model performance in practice. Concretely, any SP and $\mu$P neural networks of the same width can have their Parameter Multipliers tuned so that their training dynamics become identical.

Not all HPs transfer well even if we use $\mu$P. In particular, those whose primary function is to regularize training to mitigate “overfitting" tend not to transfer well. Intuitively, regularization needs to be applied more heavily in larger models and when data is scarce, but $\mu$P does not know the data size so cannot adjust the regularization accordingly.

To the best of our knowledge, there is no strict separation between HPs that regularize and those that don’t. However, conventional wisdom tells us that there exists a spectrum of how much regularizing effect a HP has. For example, dropout probability and weight decay are among those whose primary function is to regularize, whereas batch size and learning rate might regularize training in some cases but affect the dynamics more so in other ways. Our empirical exploration tells us that the former do not transfer well, while the latter do. Our subsequent discussion will focus on the latter; we leave to future works the expansion to the former.

We have left out a category of HPs that defines the training scale, or in practical terms, training cost. This includes 1) those that define how many operations a model’s forward/backward pass takes, such as the model’s width, depth, and in the case of language modeling, sequence length; and 2) those that define how many such passes are performed, such as batch size and number of training steps.

As recent works have shown , improvements along any of these scale dimensions lead to apparently sustainable gain in performance; as a result, we are primarily interested in transferring other HPs across these dimensions that define scale, rather than finding the optimal scale.191919 In particular, we are not fixing the total training FLOPs when we scale, which requires understanding the tradeoff of different scale HPs. For example, when we transfer across batch size, we fix the number of steps of training (not the number of epochs), so that the total FLOPs scales linearly. This category of HPs is particularly crucial as one can speedup training by downsizing in one or multiple such dimensions. Indeed, it’s very common for practitioners to implicitly transfer HPs across the number of training samples by tuning on only a subset of the full training data.

Our insights from the infinite-width limit inspired us to explore HP tranfer across width, which does not work under SP as we have shown earlier. Building upon our success with width, which is well explained theoretically, we hope to push the limit of compute-saving by investigating the other dimensions empirically. To the best of our knowledge, the transferability of optimal HPs across depth, batch size, sequence length, and training time has not been rigorously investigated previously, with the main exception of the literature on (learning rate, batch size) scaling where our transferability result of learning rate across batch size recapitulates .202020 There’s also a literature on the proper initialization for training deep networks effectively (e.g. ), but they do not study the transferability per se. See Section 10.3 See Section 10.3 on how our results relate to prior works. We will primarily focus on the Transformer architecture in the main text with evidence for ResNet in Section G.1.

E.2 On the Definitions of Width

Our theory allows more general notions of width. This is especially relevant in Transformers, where $d_{model},d_{head}=d_{k},d_{v},n_{head},d_{ffn}$ (see Fig. 11) can all be construed as measures of width. We briefly discuss these here, with more theoretical justification in Section J.2.1 and empirical validation below.

So far we have assumed that every hidden layer is widened by the same factor. But in fact we can widen different hidden layers differently. This is useful, for example, in a Transformer where we may want to use a smaller $d_{ffn}$ during tuning. If we are using Adam, as long as the width of every layer still tends to infinity, we still obtain approximately the same limit212121 This also applies for SGD, but we need more involved scaling to keep the limit approximately the same. , so the $\mu$Transfer remains theoretically justified.

See Fig. 12 for an empirical validation on IWSLT-14 using a Transformer.

In attention-based models, one typically splits hidden size into multiple attention heads following $d_{model}=d_{head}\times n_{head}$. So far we have assumed $d_{head}$ and $d_{model}$ to be width, but it’s possible and potentially advantageous to fix $d_{head}$ and treat $n_{head}$ as the width, or increasing both simultaneously. This allows our technique to handle many popular models, including GPT-3 , which scale up by fixing $d_{head}$ and increasing $n_{head}$. See Fig. 13 for an empirical validation on Wikitext-2.

A specific useful instance of varying width ratio is decoupling the key and value dimensions $d_{k}$ and $d_{v}$ and scaling $d_{k}$ differently from (typically larger than) $d_{model}/n_{head}$. This works as long as we use $1/d$ scaled-attention as in 4.1 (instead of $1/\sqrt{d}$ as is done commonly). When tuning on the small proxy model, if $d_{k}$ is too small, the HP landscape can be quite noisy. Keeping $d_{k}$ relatively large while shrinking all other dimensions solves this problem, while still obtaining significant speedup.

Appendix F Experimental Details

IWSLT14 De-En is a well-known machine translation benchmark. We use a Transformer implemented in fairseq with a default $d_{model}=\nicefrac{{1}}{{4}}d_{ffn}=512$ and $d_{k}=d_{q}=d_{v}=\nicefrac{{d_{model}}}{{n_{head}}}=128$ (amounting to 40M parameters), which we denote as the 1x model. For transfer, we tune on a proxy model with the same $n_{head}$ but with $d_{model}$ and other dimensions 4 times smaller; we will call this the 0.25x model (but it has 4M parameters). All models are trained with Adam for 100 epochs and validated at the end of every epoch. We tune via random search the learning rate $\eta$, the output layer parameter multiplier $\alpha_{output}$, and the attention key-projection weight multiplier $\alpha_{attn}$ following the grid

$\eta$: $5\times 10^{-4}\times 2^{z},\text{where }z\in\{-1.5,-1.25,-1,...,1.25\}$

$\alpha_{output}$: $2^{z},\text{where }z\in\{-8,-7,-6,...,7\}$

$\alpha_{attn}$: $2^{z},\text{where }z\in\{-3,-2,-1,...,8\}$

F.2 WMT

We scale up to WMT14 En-De using the large Transformer from , with a $d_{model}=\nicefrac{{1}}{{4}}d_{ffn}=1024$ and $d_{q}=d_{k}=d_{v}=\nicefrac{{d_{model}}}{{n_{head}}}=64$. We use the exact same setup and reproduce their result as our baseline. Then, we build the proxy model by shrinking the target model’s $d_{model}$ from the original $1024$ to $256$, $d_{ffn}$ from $4096$ to $256$ and $n_{head}$ from $16$ to $4$. This reduces the total parameter count from 211M to 15M. We then perform the HP search on the proxy model and take the best according to validation loss, before testing on the target model. We tune via random search the learning rate $\eta$, the output layer parameter multiplier $\alpha_{output}$, and the attention key-projection weight multiplier $\alpha_{attn}$ following the grid

$\eta$: $6\times 10^{-4}\times 2^{z},\text{where }z\in\{-1.5,-1.25,-1,...,1.25\}$

$\alpha_{output}$: $2^{z},\text{where }z\in\{-8,-7,-6,...,7\}$

$\alpha_{attn}$: $2^{z},\text{where }z\in\{-3,-2,-1,...,8\}$

F.3 BERT

Our proxy model has 10 Transformer layers with $d_{model}=d_{ffn}=256$. We also reduce the number of attention heads to 8 with a $d_{head}$ of 32. We call it BERT Prototype since we can increase its width and depth according to our definitions to recover both BERT Base and BERT Large, which enables us to sweep HPs once and use for both models. Overall, BERT Prototype has 13M trainable parameters, a fraction of the 110M in BERT Base and the 350M in BERT Large.

We tune the following HPs for pretraining: Adam learning rate $\eta$, embedding learning rate $\eta_{emb}$, output weight multiplier $\alpha_{output}$, attention logits multiplier $\alpha_{attn}$, layernorm gain multiplier $\alpha_{LN_{gain}}$, and bias multiplier $\alpha_{bias}$.

We sample 256 combinations from the follow grid:

$\eta$: $1\times 10^{-4}\times 2^{z},\text{where }z\in\{1.5,2,2.5,3,3.5\}$

$\eta_{emb}$: $1\times 10^{-4}\times 2^{z},\text{where }z\in\{-1,-0.5,0,0.5,1\}$

$\alpha_{output}$: $2^{z},\text{where }z\in\{2,4,6\}$

$\alpha_{attn}$: $2^{z},\text{where }z\in\{3,3.5,4,...,7\}$

$\alpha_{LN_{gain}}$: $2^{z},\text{where }z\in\{8.5,9,9.5,10,10.5\}$

$\alpha_{bias}$: $2^{z},\text{where }z\in\{8.5,9,9.5,10,10.5\}$

The ranges are chosen to include the implicit choices of these HPs in SP BERT Large.

We hand-pick the finetuning HPs after training the full-sized model. As regularization is an essential ingredient in successful finetuning, we do not transfer such HPs (at least via the suite of techniques presented in this work) (see Table 1). We focus on MNLI and QQP, which are two representative tasks from GLUE . Following , we used Adam with a learning rate of $5\times 10^{-5}$ and a batch size of 64. The maximum number of epochs was set to $5$. A linear learning rate decay schedule with warm-up of $0.1$ was used. All the texts were tokenized using wordpieces and were chopped to spans no longer than $128$ tokens.

F.4 GPT-3

As the GPT-3 codebase has evolved since the publication of , we re-trained the 6.7B model from scratch to remove changes in our codebase as a possible confounder. The main differences to are 1) a modified learning rate decay schedule, where the learning rate is decayed to zero at the end of training rather than being decayed to 0.1 of the initial value, and 2) use of relative attention in place of absolute attention. Unfortunately, after all experiments were finished, we found this re-run baseline used absolute attention instead of relative attention, while the $\mu$Transfer model still used relative attention.

In order to find a good set of hyperparameters for the $\mu$Transfer version of the 6.7B model, we performed a hyperparameter search over a reduced version of the model (i.e., the proxy model), where the width is set to 256 hidden units. This proxy model inherits changes from the evolved GPT-3 codebase: it uses relative (instead of absolute) position encoding. Early on, we noted that on the proxy model, linear learning rate decay outperformed the default cosine schedule, so all subsequent experiments for the proxy models use a linear decay schedule. By Fig. 4, $\mu$Transferring this linear decay schedule to the full model should maintain such a performance advantage over the cosine schedule.

The hyperparameter search space consists of the following hyperparameters:

learning rate: Sampled from $10^{\text{Uniform}(-4,-1)}$

initialization scale: All the parameters are multiplied - sampled from $10^{\text{Uniform}(-1,1)}$

attention temperature: Reciprocal of the multiplier applied to the input to attention softmax. Sampled from $4^{\text{Uniform}(-1,1)}$.

output temperature: Reciprocal of the multiplier applied to the input to softmax that produces the distribution over output tokens. Sampled from $4^{\text{Uniform}(-1,1)}$.

embedding multiplier: Scalar by which we multiply the output of the embedding layer. Sampled from $10^{\text{Uniform}(-1,1)}$.

relative position embedding multiplier: Scalar by which we multiply vectors representing relative position. Sampled from $10^{\text{Uniform}(-1,1)}$.

In order to make the search more efficient we reduced the total number of training tokens. We hypothesized that tuning hyperparameters on a reduced total number of tokens does not significantly affect optimal hyperparameters. To verify, we trained two different horizons and compared the results. While the target model was to be trained on 300 billion tokens, we tuned the proxy model on only subsets consisting of 4 billion and 16 billion tokens. This impacts both the total training time and the length of the linear learning rate decay schedule. Other than hyperparameters explicitly listed above and the training horizon, the rest was the same as what we intended to use for the full width 6.7B training run.

We performed 467 training runs of the proxy model, out of which 350 were for 4 billion tokens (286 completed without diverging) and 117 for 16b tokens (80 completed without diverging). See Fig. 14 for summary of the results.

As suspected, we observed that the results are well-aligned for both 4 and 16 billion tokens versions. We observe learning rate and initialization scale impact the results the most. Based on the results we chose 0.006 for the former and 2.5 for the latter. Since most other hyperparameters appear to have negligible effect on performance, they were kept at their default values of 1, the only exception being the embedding scale, where higher values seem to perform better and it was therefore set to 10.

We encountered frequent divergences in our initial attempt to train the $\mu$Transfer model. We traced the issue back to underflow of FP16 tensors in the backwards pass and therefore switched to training the model in FP32. This allowed us to finish the training run without divergences. We hypothesize that the divergence issue is related to $\mu$Transfer picking more aggressive hyperparameters, for example a higher learning rate on linear weight tensors compared to the original model. In order to exclude code differences as a possible confounder, we re-trained GPT-3 6.7B from scratch using the original hyperparameters. The only difference compared to the version published in is that the learning rate was decayed fully, whereas the learning rate of the model from was only decayed to 10% of its starting value. The retrained model performs slightly worse than the original published in . We suspect that this is because it made less progress during the last phase of training where the learning rate is close to zero. The training curves of the $\mu$Transfer model and the re-run of the original 6.7B can be seen in Fig. 15. Detailed evaluation results can be found in Table 10 and Table 11.

subscript𝑡1subscript𝑁1subscript𝑡2subscript𝑁2𝑆𝑇0.07\frac{s(t_{1}N_{1}+t_{2}N_{2})}{ST}\approx 0.07 where

$s=40$ Million is number of parameters of the proxy model

$S=6.7$ Billion is number of parameters of the target model

$t_{1}=4$ Billion is the number of training tokens for the short horizon HP search, and $N_{1}=350$ is the corresponding number of random HP search trials.

$t_{2}=16$ Billion is the number of training tokens for the longer horizon HP search, and $N_{1}=117$ is the corresponding number of random HP search trials.

$T=300$ Billion is the number of training tokens for the 6.7B target model.

Here we are using the fact that the training FLOPs of a Transformer per token is roughly proportional to its number of parameters.

Appendix G Additional Experiments

For this case we use Davidnet , a ResNet variant that trains quickly on CIFAR-10, so as to efficiently investigate its HP landscape. We train with SGD on CIFAR-10 for 10 epochs; all results are averaged over 15 random seeds. We use a width multiplier to identify models of different width, and a multiplier of 1 corresponds to the original model in . We look at validation accuracy here as the model barely overfits, and our observations will hold for the training accuracy as well. We first conduct a learning rate sweep for models of different widths using SP; the result is shown in Fig. 16, on the left.

Note that the best model with a width multiplier of 8 under-performs that with a multiplier of 4. We run the same sweep with $\mu$P, along with a sweep of the output multiplier ($\alpha_{output}$); the result is shown in Fig. 16, on the right. We notice that wider models always perform better under $\mu$P and that the optimal learning rate $\eta$ and $\alpha_{output}$ are stable across width.

Next, we perform a grid search for learning rate ($\eta$) and $\alpha_{output}$ on the 0.5x model for both SP and $\mu$P.222222Here we tune the 0.5x model instead of the 1x model to simulate the situation that one does “exploratory work” on the 1x model but, when scaling up, would like to tune faster by using a smaller proxy model. Then, we take the best combination and test on the 8x model, simulating how a practitioner might use $\mu$Transfer. The result is shown in Table 12, where $\mu$P outperforms SP by $0.43\%\pm.001\%$.

G.1.2 Wide ResNet on ImageNet

For this case we use Wide-Resnet, or WRN , a ResNet variant with more channels per layer, to further showcase $\mu$Transfer across width, i.e., number of channels. We train with SGD on ImageNet for 50 epochs following standard data augmentation procedures. We use a width multiplier to identify models of different width, and a multiplier of 1 corresponds to the original WRN-50-2-bottleneck in .

We start with a proxy model with a width multiplier of 0.125 and tune several HPs using the following grid:

$\eta$: $1\times 2.048\times 2^{z},\text{where }z\in\{-5,-4,-3,...,4\}$

$\alpha_{output}$: $10\times 2^{z},\text{where }z\in\{-5,-4,-3,...,4\}$

weight decay co-efficient $\gamma$: $3.05\times 10^{-5}\times 2^{z},\text{where }z\in\{-2,-1.5,-1,...,1.5\}$

SGD momentum $\beta$: $0.875\times 2^{z},\text{where }z\in\{-2,-1.5,-1,...,1.5\}$

The grid is centered around the default HPs used by for ResNet-50; while not expected to be competitive for WRN, they represent a reasonable starting point for our experiment.

We randomly sample 64 HP combinations from the grid and train for 50 epochs, before selecting the one with the highest top-1 validation accuracy. Then, we scale up the model following both $\mu$P and SP and run with the same HPs we just selected. The result is shown in Table 13, where $\mu$P outperforms SP by $0.41\%$ in terms of top-1 validation accuracy.

G.2 Experiments on Transformers

G.2.2 Post-Layernorm Transformers

Fig. 17 shows the transferability of learning rate, $\alpha_{output}$, initialization standard deviation, and Adam $\beta_{2}$ across width, batch size, sequence length, and training steps for post-layernorm Transformers. However, in general, we find transfer across depth to be fragile.

G.2.3 Hyperparameter Instability of SP Transformers

Fig. 18 and Fig. 20 show the HP instability inherent in SP Transformers.

Appendix H Implementing μ𝜇\muTransfer in a Jiffy

As we have shown, one can enable $\mu$Transfer by just reparametrizing the desired model in Maximal Update Parametrization ($\mu$P). While conceptually simple, switching from Standard Parametrization (SP) to $\mu$P can be error-prone, as popular deep learning frameworks are built around SP. We strive to build a tool that fulfills two goals:

Minimize code changes when switching to $\mu$P;

Keep model behavior invariant, under this switch, at a given base model shape.

By model shape, we mean the collection of dimensions of all parameters of the model. The latter goal, which we call parametrization backward compatibility, ensures that any code base works exactly as before at the base model shape, similar to Eq. 4, e.g. the loss at any time step remains exactly the same before and after the switch to $\mu$P. Of course, when widths start to differ from the base model shape, the model behavior necessarily changes so that HPs can be transferred.

There are two common approaches to setting the base model shape: 1) If one intends to tune a large target model, then the user can set the base model shape to be the shape of the target model (e.g. BERT-large or T5-large), so that the target model itself is in standard parametrization. Then one can tune a proxy model with e.g. $width=124$ to obtain the optimal HPs for the target model. In addition, if one wishes to scale up further e.g. $width=1024$, then these HPs remain optimal. 2) If one has done exploration on a new idea with a small model and now wishes to scale up, reusing the HP found during this exploration, then one can set the base model shape to be the shape of the exploratory small model. Of course, in both scenarios, depth, batch size, and sequence lengths can be scaled up and down as well according to Fig. 19 (though note that currently we require users to recreate the base model shape at new depths, since the number of parameters now change with depth).

We provide our tool as a Python package called mup designed to work with PyTorch. The following example illustrates the usage of our package.

Under the hood, mup implements the $\mu$P formulation in Table 8. By invoking set_base_shape(model, base_model), each parameter tensor p of model gets a p.infshape attribute that stores, for each of its dimensions, the corresponding base dimension and whether that dimension should be considered “infinite” (i.e. will be scaled up/down, e.g., $d_{model}$ of a Transformer) or “finite” (i.e. will be fixed, e.g., vocabulary size). This information is used in the initializers and optimizers to automatically scale the parameters or learning rates to be compliant with $\mu$P. For example, by Table 8, the Adam learning rate of hidden weights p is calculated as $\eta/\texttt{p.infshape.width\_mult()}$, where p.infshape.width_mult() essentially calculates $\frac{\texttt{fan\_in}}{\texttt{base\_fan\_in}}.$

Appendix I Reverse-μ𝜇\muTransfer for Diagnosing Training Instability in Large Models

Large Transformers are famously fickle to train . We note that a possible source of this instability for larger transformers is the failure of naive hyperparameter transfer via the standard parametrization. This is certainly consistent with Fig. 1, which shows that the optimal learning rate for small Transformers can lead to trivial performance in large Transformers. We support this hypothesis further by reverse-$\mu$Transferring the instability-inducing HPs from a large Transformer to a small one and replicating the training instability. This is shown in Footnote 23.

Practically, this reverse-$\mu$Transfer technique can be used to diagnose or debug training instability problems of large models. We offer two case studies toward this claim.

1) When training transformers of width 8192 on Wikitext-2, we found certain HP combinations caused divergence in the middle of training. We reverse-$\mu$Transferred one such HP combination to a model of width 256 and replicated this divergence. By analyzing this small model’s activations right before this divergence, we found that the cause is due to attention logits blowing up. Note this debugging session proceeded much more quickly than if we directly worked with the large model. Later we confirmed this is indeed the same cause of the width-8192 model’s divergence.

2) A 6B-parameter language model (in standard parametrization) in a separate project experienced repeated blow-up in the middle of training. We reverse-$\mu$Transferred its hyperparameters to a smaller, 100M-parameter model and replicated the training instability. This was solved by a retuning of the small model via random search.

Appendix J An Intuitive Introduction to the Theory of Maximal Update Parametrization

In what follows, we seek to describe useful intuitions and rules of thumb that would be helpful to practitioners and empirical researchers alike in figuring out what is the right neural network parametrization. The intuitions we shall describe regarding SGD can be made rigorous as in ; those regarding Adam are new, and their formalization will be done in an upcoming paper.

First, we write down the most basic intuition regarding sums of many random elements, which will underlie all of the calculations that follow.

If $x_{1},\ldots,x_{n},\ldots$ “look like” random independent samples of a random variable $X$, then

where $\sigma(X)$ is the standard deviation of the random variable $X$.

Of course, there are many subtleties one must resolve to make the statements above truly rigorous (e.g., what is the meaning of “look like”?), but as rules of thumb, they typically give the correct prediction.

In particular, here we want to note the following basic intuition regarding the size of a sum of $x_{i}$:

Here, “typical size” can be taken to mean the size $99\%$ of time. Again, we stress that this is a good rule of thumb that yields the correct prediction in the cases we are concerned with here; the rigorous versions of this will come from the Tensor Programs framework (e.g., ).

J.1 Behaviors of Gaussian Matrices vs Tensor Product Matrices

Central to the derivation of $\mu$P for any architecture are key insights on the behaviors of two kinds of random matrices: 1) iid Gaussian random matrix and 2) tensor product matrix (by which we mean a sum of outer products) and more generally what we call nonlinear tensor product matrix (see Eq. 7). For example, a neural network, randomly initialized in the typical way, will have each weight matrix look like the former. However, every step of training by gradient descent adds a sum of outer products to this initial matrix, so that the change in weights constitute a tensor product matrix. For Adam, the change in weights is not a tensor product but a more general nonlinear tensor product matrix (see Eq. 7). In this section, we will particularly focus on the right scaling for the entries of such matrices, leading to a discussion of the right neural network parametrization in the next section. We concentrate on the key heuristics but eschew burdensome rigor.

Consider a random vector $v\in\mathbb{R}^{n}$ with approximately iid entries and a random matrix $A$ of either size $n\times n$ or $1\times n$, both having entries of size $\Theta(1)$.242424in the sense that the the variance of the entries are $\Theta(1)$ In the context of deep learning, $v$ for example can be an activation vector in an MLP, a Gaussian $A$ the hidden weights at initialization, a (nonlinear) tensor product $A$ the change in hidden weights due to training, and a vector $A$ the readout layer weights. Then $Av$ corresponds to a part of the next layer preactivation or the network output. To make sure the preactivations and the output don’t blow up, we thus need to understand the scale of $Av$, especially in the general case where $A$ is correlated with $v$.252525Here “correlated” formally means $v$ depends on $W^{\top}$ in a Tensor Program. This essentially captures all scenarios of “$v$ correlated with $W$” that occurs in deep learning. This is summarized in Table 14, with the derivations below. Intuitively, a (nonlinear) tensor product or vector $A$ will interact with a correlated $v$ via Law of Large Numbers, hence the $n$-scaling, while a Gaussian $A$ interacts with $v$ via Central Limit Theorem, hence the $\sqrt{n}$-scaling.

In the derivations below, we answer a slightly different but equivalent question of “how to scale $A$ such that $Av$ has entry size $\Theta(1)$?”

J.1.1 Preparation for the Derivations

By the results of , each (pre-)activation vector and its gradient vector in a multi-layer perceptron, at any time during training, have approximately iid coordinates in the large width limit,262626Our intuition here is derived from the assumption that width is much larger than training time; of course, as illustrated by our myriad experiments, these intuition are very useful even when this is not the case, such as when training to convergence. and something similar can be said for more advanced networks such as ResNet and Transformers 272727E.g. in a convnet, the (pre-)activations are iid across channels, but correlated across pixels.

We say any such vector $v\in\mathbb{R}^{n}$ has $\Theta(n^{a})$-sized coordinates, or just $\Theta(n^{a})$-coordinates for short, if $\|v\|^{2}/n=\Theta(n^{2a})$ as $n\to\infty$. Because, by the above discussion, the coordinates are roughly iid when $n$ is large, this intuitively means that each entry of $v$ has “typical size” $\Theta(n^{a})$. We make similar definitions with $\Theta$ replaced by $O$ and $\Omega$.

Furthermore, to each such vector $v$ with $\Theta(1)$-sized coordinates, we can associate a random variable $Z^{v}$, independent of $n$, that represents the coordinate distribution of $v$, in such a way that: If vector $u$ is correlated with $v$, then $Z^{u}$ will also be correlated with $Z^{v}$, and $\lim_{n\to\infty}v^{\top}u/n=\operatorname*{\mathbb{E}}Z^{u}Z^{v}$.

J.1.2 Linear Tensor Product Matrix (e.g. SGD Updates)

The case of (linear) tensor product matrix can be reduced to the outer product case by linearity. Given $u,v,x\in\mathbb{R}^{n}$ having approximately iid coordinates (of size $\Theta(1)$) like discussed above, we can form the outer product

which is the form of a single (batch size 1) gradient update to a weight matrix. Then, by Law of Large Numbers,

So $Ax$ also has approximately iid coordinates, distributed like $Z^{Ax}\mathbin{\overset{\mathrm{def}}{=}}Z^{u}\operatorname*{\mathbb{E}}Z^{v}Z^{x}$. Likewise, if $A$ is a sum of outer products $A=\sum_{i=1}^{k}u^{i}\otimes v^{i}/n$, then

Notice that each coordinate of $A$ has size $\Theta(1/n)$. The above reasoning shows that, in order for $Ax$ to have coordinate size $\Theta(1)$ (assuming $x$ does), then $\Theta(1/n)$ is the right coordinate size for $A$, in the general case that $v^{i}$ and $x$ are correlated (as is generically the case during gradient descent, with $A=\Delta W$ for some weights $W$ and $x$ being the previous activations).282828In some corner cases when $x$ is uncorrelated with $v$, then $v^{\top}x=\Theta(\sqrt{n})$ by Central Limit, so actually $Ax$ has $\Theta(1/\sqrt{n})$-coordinates. However, this case does not come up much in the context of training neural networks.

J.1.3 Nonlinear Tensor Product Matrix (e.g. Adam Updates)

When using Adam or another adaptive optimizer that normalizes the gradient coordinatewise before applying them, we need to modify our argument slightly to obtain the right coordinate size scaling of the matrix. The gradient update $A$, after such normalization, will take the form of

We say a matrix of this form is a nonlinear tensor product matrix.

First, note the tensor product matrices (e.g. the form of SGD update) discussed previously (Eq. 6) already takes this form, with $\psi(u_{\alpha}^{1},\ldots,u_{\alpha}^{k},v_{\beta}^{1},\ldots,v_{\beta}^{k})=n^{-1}(u_{\alpha}^{1}v_{\beta}^{1}+\cdots+u_{\alpha}^{k}v_{\beta}^{k})$, so Eq. 7 is a strict generalization of linear tensor products. Next, for the example of Adam, each gradient update is $\mu/\sigma$ where $\mu$ (resp. $\sigma^{2}$) is the moving average of previous (unnormalized) gradients (resp. the coordinatewise square of the same).292929Adam also has bias correction for the moving averages which can be accomodated easily, but for simplicity we omit them here. If these unnormalized gradients are the outer products $u^{1}\otimes v^{1},\ldots,u^{k}\otimes v^{k}$, then the update has coordinates

where $\gamma_{i}$ and $\omega_{i}$ are the weights involved in the moving averages.

Now suppose we have some $A\in\mathbb{R}^{n\times n}$ of the form Eq. 7, where $u^{i},v^{i}\in\mathbb{R}^{n}$ have approximately iid coordinates (of size $\Theta(1)$), and $\psi=n^{-1}\bar{\psi}$ where $\bar{\psi}$ doesn’t depend on $n$ (in terms of Adam where $\bar{\psi}$ corresponds to the $\psi$ of Eq. 8, this corresponds to using a learning rate of $1/n$). Then for $x\in\mathbb{R}^{n}$ having approximately iid coordinates of size $\Theta(1)$, by Law of Large Numbers,

Thus $Ax$ also has approximately iid coordinates (of size $\Theta(1)$),

For example, in the SGD example with $A=u\otimes v/n$ and $\bar{\psi}(u_{\alpha},v_{\beta})=u_{\alpha}v_{\beta}$, this formula gives $Z^{Ax}=\Psi(Z^{u})$ where $\Psi(z)=z\operatorname*{\mathbb{E}}Z^{v}Z^{x}$, recovering the earlier derivation.

In any case, the point here is that $A$ has coordinate size $\Theta(1/n)$, and this is the unique scaling that leads to $Ax$ having coordinate size $\Theta(1)$.

J.1.4 Vector Case (e.g. Readout Layer)

The vector $A$ case is similar to the tensor product cases above.

J.1.5 Gaussian Matrix (e.g. Hidden Weights Initialization)

Now consider the case where $A\in\mathbb{R}^{n\times n}$ is random Gaussian matrix with $A_{\alpha\beta}\sim\mathcal{N}(0,1/n)$ and $x\in\mathbb{R}^{n}$ has approximately iid coordinates distributed like $Z^{x}$. In the context of neural network training, $A$ should be thought of as a randomly initialized weight matrix, and $x$ for example can be taken to be an activation vector in the first forward pass.

By standard random matrix theory, $A$ has $\Theta(1)$ operator norm with high probability. Thus, with high probability, for any “typical” vector $x$, we expect $\|Ax\|=\Theta(\|x\|)$, even if $x$ is correlated with $A$. If $Ax$’s coordinates are “evenly distributed”, then this would imply $Ax$ has $\Theta(1)$-coordinates if $x$ does. However, this is not so clear. Below we provide intuitions for why this would be the case.

If $x$ is independent from $A$ (or sufficiently uncorrelated), then each coordinate $(Ax)_{\alpha}$ has variance $\operatorname*{\mathbb{E}}(Z^{x})^{2}=\Theta(1)$ (so by definition has size $\Theta(1)$). Thus, here $A$ having $\Theta(1/\sqrt{n})$-coordinates leads to $Ax$ having $\Theta(1)$-coordinates, in contrast to the tensor product case above.

When $x$ is correlated with $A$, it turns out the same scaling applies ($\Theta(1/\sqrt{n})$ is the unique scaling for $A$’s entries such so that $Ax$ has $\Theta(1)$ entries), but the reasoning is much more subtle: In the context of neural network training, it turns out all scenario where $x$ is correlated with $A$ can be reduced to the case where $x=\phi(A^{\top}y,\ldots)$ for some coordinatewise nonlinearity $\phi$ and some other vector $\mathbb{R}^{n}$.303030This is because every “reasonable” deep learning computation can be expressed in a Tensor Program. Let’s consider a very simple example with $x=A^{\top}\mathbf{1}$ for the all 1s vector $\mathbf{1}\in\mathbb{R}^{n}$ (which has coordinate size $\Theta(1)$ as can be checked easily). Then, for each index $\alpha\in[n]$, we can calculate

subscript𝛽superscriptsubscript𝐴𝛼𝛽2subscript𝛽subscript𝛾𝛼subscript𝐴𝛼𝛽subscript𝐴𝛾𝛽(AA^{\top}\mathbf{1})_{\alpha}=\sum_{\beta,\gamma}A_{\alpha\beta}A_{\gamma\beta}=\sum_{\beta}A_{\alpha\beta}^{2}+\sum_{\beta}\sum_{\gamma\neq\alpha}A_{\alpha\beta}A_{\gamma\beta}. Since $\operatorname*{\mathbb{E}}A_{\alpha\beta}^{2}=1/n$, by the Law of Large Number, the first sum $\sum_{\beta}A_{\alpha\beta}^{2}\approx 1$. On the other hand, there are $n$ summands of the form $\sum_{\gamma\neq\alpha}A_{\alpha\beta}A_{\gamma\beta}$, all iid with variance $\frac{n-1}{n^{2}}=\Theta(1/n)$. Thus by the Central Limit Theorem, we expect $\sum_{\beta}\sum_{\gamma\neq\alpha}A_{\alpha\beta}A_{\gamma\beta}\approx\mathcal{N}(0,1)$. Therefore, each coordinate of $(AA^{\top}\mathbf{1})_{\alpha}$ looks like $1+\mathcal{N}(0,1)=\mathcal{N}(1,1)$ and thus has size $\Theta(1)$; again this is caused by $A$ having $\Theta(1/\sqrt{n})$-coordinates.