Efficient Neural Architecture Search via Parameter Sharing

Introduction

Neural architecture search (NAS) has been successfully applied to design model architectures for image classification and language models (Zoph & Le, 2017; Zoph et al., 2018; Cai et al., 2018; Liu et al., 2017, 2018). In NAS, an RNN controller is trained in a loop: the controller first samples a candidate architecture, i.e. a child model, and then trains it to convergence to measure its performance on the task of desire. The controller then uses the performance as a guiding signal to find more promising architectures. This process is repeated for many iterations. Despite its impressive empirical performance, NAS is computationally expensive and time consuming, e.g. Zoph et al. (2018) use $450$ GPUs for $3$-$4$ days (i.e. 32,400-43,200 GPU hours). Meanwhile, using less resources tends to produce less compelling results (Negrinho & Gordon, 2017; Baker et al., 2017a). We observe that the computational bottleneck of NAS is the training of each child model to convergence, only to measure its accuracy whilst throwing away all the trained weights.

The main contribution of this work is to improve the efficiency of NAS by forcing all child models to share weights to eschew training each child model from scratch to convergence. The idea has apparent complications, as different child models might utilize their weights differently, but was encouraged by previous work on transfer learning and multitask learning, which established that parameters learned for a particular model on a particular task can be used for other models on other tasks, with little to no modifications (Razavian et al., 2014; Zoph et al., 2016; Luong et al., 2016).

We empirically show that not only is sharing parameters among child models possible, but it also allows for very strong performance. Specifically, on CIFAR-10, our method achieves a test error of $2.89\%$, compared to $2.65\%$ by NAS. On Penn Treebank, our method achieves a test perplexity of $55.8$, which significantly outperforms NAS’s test perplexity of $62.4$ (Zoph & Le, 2017) and which is a new state-of-the-art among Penn Treebank’s approaches that do not utilize post-training processing. Importantly, in all of our experiments, for which we use a single Nvidia GTX 1080Ti GPU, the search for architectures takes less than $16$ hours. Compared to NAS, this is a reduction of GPU-hours by more than 1000x. Due to its efficiency, we name our method Efficient Neural Architecture Search (ENAS).

Methods

Central to the idea of ENAS is the observation that all of the graphs which NAS ends up iterating over can be viewed as sub-graphs of a larger graph. In other words, we can represent NAS’s search space using a single directed acyclic graph (DAG). Figure 2 illustrates a generic example DAG, where an architecture can be realized by taking a subgraph of the DAG. Intuitively, ENAS’s DAG is the superposition of all possible child models in a search space of NAS, where the nodes represent the local computations and the edges represent the flow of information. The local computations at each node have their own parameters, which are used only when the particular computation is activated. Therefore, ENAS’s design allows parameters to be shared among all child models, i.e. architectures, in the search space.

In the following, we facilitate the discussion of ENAS with an example that illustrates how to design a cell for recurrent neural networks from a specified DAG and a controller (Section 2.1). We will then explain how to train ENAS and how to derive architectures from ENAS’s controller (Section 2.2). Finally, we will explain our search space for designing convolutional architectures (Sections 2.3 and 2.4).

To design recurrent cells, we employ a DAG with $N$ nodes, where the nodes represent local computations, and the edges represent the flow of information between the $N$ nodes. ENAS’s controller is an RNN that decides: 1) which edges are activated and 2) which computations are performed at each node in the DAG. This design of our search space for RNN cells is different from the search space for RNN cells in Zoph & Le (2017), where the authors fix the topology of their architectures as a binary tree and only learn the operations at each node of the tree. In contrast, our search space allows ENAS to design both the topology and the operations in RNN cells, and hence is more flexible.

To create a recurrent cell, the controller RNN samples $N$ blocks of decisions. Here we illustrate the ENAS mechanism via a simple example recurrent cell with $N=4$ computational nodes (visualized in Figure 1). Let $\mathbf{x}_{t}$ be the input signal for a recurrent cell (e.g. word embedding), and $\mathbf{h}_{t-1}$ be the output from the previous time step. We sample as follows.

At node 1: The controller first samples an activation function. In our example, the controller chooses the $\tanh$ activation function, which means that node $1$ of the recurrent cell should compute $h_{1}=\tanh{(\mathbf{x}_{t}\cdot\mathbf{W}^{(\mathbf{x})}+\mathbf{h}_{t-1}\cdot\mathbf{W}^{(\mathbf{h})}_{1})}$.

At node 2: The controller then samples a previous index and an activation function. In our example, it chooses the previous index $1$ and the activation function ReLU. Thus, node $2$ of the cell computes $h_{2}=\text{ReLU}(h_{1}\cdot\mathbf{W}^{(\mathbf{h})}_{2,1})$.

At node 3: The controller again samples a previous index and an activation function. In our example, it chooses the previous index $2$ and the activation function ReLU. Therefore, $h_{3}=\text{ReLU}(h_{2}\cdot\mathbf{W}^{(\mathbf{h})}_{3,2})$.

At node 4: The controller again samples a previous index and an activation function. In our example, it chooses the previous index $1$ and the activation function $\tanh$, leading to $h_{4}=\tanh{(h_{1}\cdot\mathbf{W}^{(\mathbf{h})}_{4,1})}$.

For the output, we simply average all the loose ends, i.e. the nodes that are not selected as inputs to any other nodes. In our example, since the indices $3$ and $4$ were never sampled to be the input for any node, the recurrent cell uses their average $(h_{3}+h_{4})/2$ as its output. In other words, $\mathbf{h}_{t}=(h_{3}+h_{4})/2$.

Our search space includes an exponential number of configurations. Specifically, if the recurrent cell has $N$ nodes and we allow $4$ activation functions (namely $\tanh$, ReLU, identity, and sigmoid), then the search space has $4^{N}\times N!$ configurations. In our experiments, $N=12$, which means there are approximately $10^{15}$ models in our search space.

2 Training ENAS and Deriving Architectures

Our controller network is an LSTM with $100$ hidden units (Hochreiter & Schmidhuber, 1997). This LSTM samples decisions via softmax classifiers, in an autoregressive fashion: the decision in the previous step is fed as input embedding into the next step. At the first step, the controller network receives an empty embedding as input.

In ENAS, there are two sets of learnable parameters: the parameters of the controller LSTM, denoted by $\theta$, and the shared parameters of the child models, denoted by $\omega$. The training procedure of ENAS consists of two interleaving phases. The first phase trains $\omega$, the shared parameters of the child models, on a whole pass through the training data set. For our Penn Treebank experiments, $\omega$ is trained for about $400$ steps, each on a minibatch of $64$ examples, where the gradient $\nabla_{\omega}$ is computed using back-propagation through time, truncated at $35$ time steps. Meanwhile, for CIFAR-10, $\omega$ is trained on $45,000$ training images, separated into minibatches of size $128$, where $\nabla_{\omega}$ is computed using standard back-propagation. The second phase trains $\theta$, the parameters of the controller LSTM, for a fixed number of steps, typically set to $2000$ in our experiments. These two phases are alternated during the training of ENAS. More details are as follows.

Training the controller parameters θ𝜃\theta.

The reward $\mathcal{R}(\mathbf{m},\omega)$ is computed on the validation set, rather than on the training set, to encourage ENAS to select models that generalize well rather than models that overfit the training set well. In our language model experiment, the reward function is $c/\text{valid\_ppl}$, where the perplexity is computed on a minibatch of validation data. In our image classification experiments, the reward function is the accuracy on a minibatch of validation images.

Deriving Architectures.

We discuss how to derive novel architectures from a trained ENAS model. We first sample several models from the trained policy $\pi(\mathbf{m},\theta)$. For each sampled model, we compute its reward on a single minibatch sampled from the validation set. We then take only the model with the highest reward to re-train from scratch. It is possible to improve our experimental results by training all the sampled models from scratch and selecting the model with the highest performance on a separated validation set, as done by other works (Zoph & Le, 2017; Zoph et al., 2018; Liu et al., 2017, 2018). However, our method yields similar performance whilst being much more economical.

3 Designing Convolutional Networks

We now discuss the search space for convolutional architectures. Recall that in the search space of the recurrent cell, the controller RNN samples two decisions at each decision block: 1) what previous node to connect to and 2) what activation function to use. In the search space for convolutional models, the controller RNN also samples two sets of decisions at each decision block: 1) what previous nodes to connect to and 2) what computation operation to use. These decisions construct a layer in the convolutional model.

The decision of what previous nodes to connect to allows the model to form skip connections (He et al., 2016a; Zoph & Le, 2017). Specifically, at layer $k$, up to $k-1$ mutually distinct previous indices are sampled, leading to $2^{k-1}$ possible decisions at layer $k$. We provide an illustrative example of sampling a convolutional network in Figure 3. In this example, at layer $k=4$, the controller samples previous indices $\{1,3\}$, so the outputs of layers $1$ and $3$ are concatenated along their depth dimension and sent to layer $4$.

Meanwhile, the decision of what computation operation to use sets a particular layer into convolution or average pooling or max pooing. The $6$ operations available for the controller are: convolutions with filter sizes $3\times 3$ and $5\times 5$, depthwise-separable convolutions with filter sizes $3\times 3$ and $5\times 5$ (Chollet, 2017), and max pooling and average pooling of kernel size $3\times 3$. As for recurrent cells, each operation at each layer in our ENAS convolutional network has a distinct set of parameters.

Making the described set of decisions for a total of $L$ times, we can sample a network of $L$ layers. Since all decisions are independent, there are $6^{L}\times 2^{L(L-1)/2}$ networks in the search space. In our experiments, $L=12$, resulting in $1.6\times 10^{29}$ possible networks.

4 Designing Convolutional Cells

Rather than designing the entire convolutional network, one can design smaller modules and then connect them together to form a network (Zoph et al., 2018). Figure 4 illustrates this design, where the convolutional cell and reduction cell architectures are to be designed. We now discuss how to use ENAS to search for the architectures of these cells.

We utilize the ENAS computational DAG with $B$ nodes to represent the computations that happen locally in a cell. In this DAG, node $1$ and node $2$ are treated as the cell’s inputs, which are the outputs of the two previous cells in the final network (see Figure 4). For each of the remaining $B-2$ nodes, we ask the controller RNN to make two sets of decisions: 1) two previous nodes to be used as inputs to the current node and 2) two operations to apply to the two sampled nodes. The $5$ available operations are: identity, separable convolution with kernel size $3\times 3$ and $5\times 5$, and average pooling and max pooling with kernel size $3\times 3$. At each node, after the previous nodes and their corresponding operations are sampled, the operations are applied on the previous nodes, and their results are added.

As before, we illustrate the mechanism of our search space with an example, here with $B=4$ nodes (refer to Figure 5). Details are as follows.

Nodes $1$, $2$ are input nodes, so no decisions are needed for them. Let $h_{1}$, $h_{2}$ be the outputs of these nodes.

At node 3: the controller samples two previous nodes and two operations. In Figure 5 Top Left, it samples node 2, node 2, separable_conv_5x5, and identity. This means that $h_{3}=\text{sep\_conv\_5x5}(h_{2})+\text{id}(h_{2})$.

At node 4: the controller samples node 3, node 1, avg_pool_3x3, and sep_conv_3x3. This means that $h_{4}=\text{avg\_pool\_3x3}(h_{3})+\text{sep\_conv\_3x3}(h_{1})$.

Since all nodes but $h_{4}$ were used as inputs to at least another node, the only loose end, $h_{4}$, is treated as the cell’s output. If there are multiple loose ends, they will be concatenated along the depth dimension to form the cell’s output.

A reduction cell can also be realized from the search space we discussed, simply by: 1) sampling a computational graph from the search space, and 2) applying all operations with a stride of $2$. A reduction cell thus reduces the spatial dimensions of its input by a factor of $2$. Following Zoph et al. (2018), we sample the reduction cell conditioned on the convolutional cell, hence making the controller RNN run for a total of $2(B-2)$ blocks.

Finally, we estimate the complexity of this search space. At node $i$ ($3\leq i\leq B$), the controller can select any two nodes from the $i-1$ previous nodes, and any two operations from $5$ operations. As all decisions are independent, there are $(5\times(B-2)!)^{2}$ possible cells. Since we independently sample for a convolutional cell and a reduction cell, the final size of the search space is $(5\times(B-2)!)^{4}$. With $B=7$ as in our experiments, the search space can realize $1.3\times 10^{11}$ final networks, making it significantly smaller than the search space for entire convolutional networks (Section 2.3).

Experiments

We first present our experimental results from employing ENAS to design recurrent cells on the Penn Treebank dataset and convolutional architectures on the CIFAR-10 dataset. We then present an ablation study which asserts the role of ENAS in discovering novel architectures.

Penn Treebank (Marcus et al., 1994) is a well-studied benchmark for language model. We use the standard pre-processed version of the dataset, which is also used by previous works, e.g. Zaremba et al. (2014).

Since the goal of our work is to discover cell architectures, we only employ the standard training and test process on Penn Treebank, and do not utilize post-training techniques such as neural cache (Grave et al., 2017) and dynamic evaluation (Krause et al., 2017). Additionally, as Collins et al. (2017) have established that RNN models with more parameters can learn to store more information, we limit the size of our ENAS cell to $24M$ parameters. We also do not tune our hyper-parameters extensively like Melis et al. (2017), nor do we train multiple architectures and select the best one based on their validation perplexities like Zoph & Le (2017). Therefore, ENAS is not at any advantage, compared to Zoph & Le (2017); Yang et al. (2018); Melis et al. (2017), and its improved performance is only due to the cell’s architecture.

Training details.

Our controller is trained using Adam, with a learning rate of $0.00035$. To prevent premature convergence, we also use a tanh constant of $2.5$ and a temperature of $5.0$ for the sampling logits (Bello et al., 2017a, b), and add the controller’s sample entropy to the reward, weighted by $0.0001$. Additionally, we augment the simple transformations between nodes in the constructed recurrent cell with highway connections (Zilly et al., 2017). For instance, instead of having $h_{2}=\text{ReLU}(h_{1}\cdot\mathbf{W}^{(\mathbf{h})}_{2,1})$ as shown in the example from Section 2.1, we have $h_{2}=c_{2}\otimes\text{ReLU}(h_{1}\cdot\mathbf{W}^{(\mathbf{h})}_{2,1})+(1-c_{2})\otimes h_{1}$, where $c_{2}=\text{sigmoid}(h_{1}\cdot\mathbf{W}^{(\mathbf{c})}_{2,1})$ and $\otimes$ denotes elementwise multiplication.

Results.

Running on a single Nvidia GTX 1080Ti GPU, ENAS finds a recurrent cell in about $10$ hours. In Table 1, we present the performance of the ENAS cell as well as other baselines that do not employ post-training processing. The ENAS cell achieves a test perplexity of $55.8$, which is on par with the existing state-of-the-art of $56.0$ achieved by Mixture of Softmaxes (MoS) (Yang et al., 2018). Note that we do not apply MoS to the ENAS cell. Importantly, ENAS cell outperforms NAS (Zoph & Le, 2017) by more than $6$ perplexity points, whilst the search process of ENAS, in terms of GPU hours, is more than $1000$x faster.

Our ENAS cell, visualized in Figure 6, has a few interesting properties. First, all non-linearities in the cell are either ReLU or tanh, even though the search space also has two other functions: identity and sigmoid. Second, we suspect this cell is a local optimum, similar to the observations made by Zoph & Le (2017). When we randomly pick some nodes and switch the non-linearity into identity or sigmoid, the perplexity increases up to $8$ points. Similarly, when we randomly switch some ReLU nodes into tanh or vice versa, the perplexity also increases, but only up to $3$ points. Third, as shown in Figure 6, the output of our ENAS cell is an average of $6$ nodes. This behavior is similar to that of Mixture of Contexts (MoC) (Yang et al., 2018). Not only does ENAS independently discover MoC, but it also learns to balance between i) the number of contexts to mix, which increases the model’s expressiveness, and ii) the depth of the recurrent cell, which learns more complex transformations (Zilly et al., 2017).

2 Image Classification on CIFAR-10

The CIFAR-10 dataset (Krizhevsky, 2009) consists of $50,000$ training images and $10,000$ test images. We use the standard data pre-processing and augmentation techniques, i.e. subtracting the channel mean and dividing the channel standard deviation, centrally padding the training images to $40\times 40$ and randomly cropping them back to $32\times 32$, and randomly flipping them horizontally.

Search spaces.

We apply ENAS to two search spaces: 1) the macro search space over entire convolutional models (Section 2.3); and 2) the micro search space over convolutional cells (Section 2.4).

Training details.

The policy parameters $\theta$ are initialized uniformly in $[-0.1,0.1]$, and trained with Adam at a learning rate of $0.00035$. Similar to the procedure in Section 3.1, we apply a tanh constant of $2.5$ and a temperature of $5.0$ to the controller’s logits, and add the controller entropy to the reward, weighted by $0.1$. Additionally, in the macro search space, we enforce the sparsity in the skip connections by adding to the reward the KL divergence between: 1) the skip connection probability between any two layers and 2) our chosen probability $\rho=0.4$, which represents the prior belief of a skip connection being formed. This KL divergence term is weighted by $0.8$. More training details are in Appendix B.

Results.

Table 2 summarizes the test errors of ENAS and other approaches. In this table, the first block presents the results of DenseNet (Huang et al., 2016), one of the highest-performing architectures that are designed by human experts. When trained with a strong regularization technique, such as Shake-Shake (Gastaldi, 2016), and a data augmentation technique, such as CutOut (DeVries & Taylor, 2017), DenseNet impressively achieves the test error of $2.56\%$.

The second block of Table 2 presents the performances of approaches that attempt to design an entire convolutional network, along with the the number of GPUs and the time these methods take to discover their final models. As shown, ENAS finds a network architecture, which we visualize in Figure 7, and which achieves $4.23\%$ test error. This test error is better than the error of $4.47\%$, achieved by the second best NAS model (Zoph & Le, 2017). If we keep the architecture, but increase the number of filters in the network’s highest layer to $512$, then the test error decreases to $3.87\%$, which is not far away from NAS’s best model, whose test error is $3.65\%$. Impressively, ENAS takes about $7$ hours to find this architecture, reducing the number of GPU-hours by more than 50,000x compared to NAS.

The third block of Table 2 presents the performances of approaches that attempt to design one more more modules and then connect them together to form the final networks. ENAS takes $11.5$ hours to discover the convolution cell and the reduction cell, which are visualized in Figure 8. With the convolutional cell replicated for $N=6$ times (c.f. Figure 4), ENAS achieves $3.54\%$ test error, on par with the $3.41\%$ error of NASNet-A (Zoph et al., 2018). With CutOut (DeVries & Taylor, 2017), ENAS’s error decreases to $2.89\%$, compared to $2.65\%$ by NASNet-A.

In addition to ENAS’s strong performance, we also find that the models found by ENAS are, in a sense, the local minimums in their search spaces. In particular, in the model that ENAS finds from the marco search space, if we replace all separable convolutions with normal convolutions, and then adjust the model size so that the number of parameters stay the same, then the test error increases by $1.7\%$. Similarly, if we randomly change several connections in the cells that ENAS finds in the micro search space, the test error increases by $2.1\%$. This behavior is also observed when ENAS searches for recurrent cells (c.f. Section 3.1), as well as in Zoph & Le (2017). We thus believe that the controller RNN learned by ENAS is as good as the controller RNN learned by NAS, and that the performance gap between NAS and ENAS is due to the fact that we do not sample multiple architectures from our trained controller, train them, and then select the best architecture on the validation data. This extra step benefits NAS’s performance.

3 The Importance of ENAS

A question regarding ENAS’s importance is whether ENAS is actually capable of finding good architectures, or if it is the design of the search spaces that leads to ENAS’s strong empirical performance.

We uniformly sample a recurrent cell, an entire convolutional network, and a pair of convolutional and reduction cells from their search spaces and train them to convergence using the same settings as the architectures found by ENAS. For the macro space over entire networks, we sample the skip connections with an activation probability of $0.4$, effectively balancing ENAS’s advantage from the KL divergence term in its reward (see Section 3.2). Our random recurrent cell achieves the test perplexity of $81.2$ on Penn Treebank, which is far worse than ENAS’s perplexity of $55.8$. Our random convolutional network reaches $5.86\%$ test error, and our two random cells reache $6.77\%$ on CIFAR-10, while ENAS achieves $4.23\%$ and $3.54\%$, respectively.

Disabling ENAS Search.

In addition to random search, we attempt to train only the shared parameters $\omega$ without updating the controller. We conduct this study for our macro search space (Section 2.3), where the effect of an untrained random controller is similar to dropout with a rate of $0.5$ on the skip connections, and to drop-path on the operations (Zoph et al., 2018; Larsson et al., 2017). At convergence, the model has the error rate of $8.92\%$. On the validation set, an ensemble of $250$ Monte Carlo configurations of this trained model can only reach $5.49\%$ test error. We therefore conclude that the appropriate training of the ENAS controller is crucial for good performance.

Related Work and Discussions

There is a growing interest in improving the efficiency of NAS. Concurrent to our work are the promising ideas of using performance prediction (Baker et al., 2017b; Deng et al., 2017), using iterative search method for architectures of growing complexity (Liu et al., 2017), and using hierarchical representation of architectures (Liu et al., 2018). Table 2 shows that ENAS is significantly more efficient than these other methods, in GPU hours.

ENAS’s design of sharing weights between architectures is inspired by the concept of weight inheritance in neural model evolution (Real et al., 2017, 2018). Additionally, ENAS’s choice of representing computations using a DAG is inspired by the concept of stochastic computational graph (Schulman et al., 2015), which introduces nodes with stochastic outputs into a computational graph. ENAS’s utilizes such stochastic decisions in a network to make discrete architectural decisions that govern subsequent computations in the network, trains the decision maker, i.e. the controller, and finally harvests the decisions to derive architectures.

Closely related to ENAS is SMASH (Brock et al., 2018), which designs an architecture and then uses a hypernetwork (Ha et al., 2017) to generate its weight. Such usage of the hypernetwork in SMASH inherently restricts the weights of SMASH’s child architectures to a low-rank space. This is because the hypernetwork generates weights for SMASH’s child models via tensor products (Ha et al., 2017), which suffer from a low-rank restriction as for arbitrary matrices $\mathbf{A}$ and $\mathbf{B}$, one always has the inequality: $\text{rank}(\mathbf{A}\cdot\mathbf{B})\leq\min{\{\text{rank}(\mathbf{A}),\text{rank}(\mathbf{B})\}}$. Due to this limit, SMASH will find architectures that perform well in the restricted low-rank space of their weights, rather than architectures that perform well in the normal training setups, where the weights are no longer restricted. Meanwhile, ENAS allows the weights of its child models to be arbitrary, effectively avoiding such restriction. We suspect this is the reason behind ENAS’s superior empirical performance to SMASH. In addition, it can be seen from our experiments that ENAS can be flexibly applied to multiple search spaces and disparate domains, e.g. the space of RNN cells for the text domain, the macro search space of entire networks, and the micro search space of convolutional cells for the image domain.

Conclusion

NAS is an important advance that automatizes the designing process of neural networks. However, NAS’s computational expense prevents it from being widely adopted. In this paper, we presented ENAS, a novel method that speeds up NAS by more than 1000x, in terms of GPU hours. ENAS’s key contribution is the sharing of parameters across child models during the search for architectures. This insight is implemented by searching for a subgraph within a larger graph that incorporates architectures in a search space. We showed that ENAS works well on both CIFAR-10 and Penn Treebank datasets.

Acknowledgements

The authors want to thank Jaime Carbonell, Zihang Dai, Lukasz Kaiser, Azalia Mirhoseini, Ashwin Paranjape, Daniel Selsam, and Xinyi Wang for their suggestions on improving the paper.

References

Appendix A Details on Penn Treebank Experiments

We think of the cell at time step $t$ as a DAG with $N$ computational nodes, indexed by $\mathbf{h}^{(t)}_{1}$, $\mathbf{h}^{(t)}_{2}$, … $\mathbf{h}^{(t)}_{N}$. Node $\mathbf{h}^{(t)}_{1}$ receives two inputs: 1) the RNN signal $\mathbf{x}^{(t)}$ at its current time step; and 2) the output $\mathbf{h}^{(t-1)}_{D}$ from the cell at the previous time step. The following computations are performed:

Parameters Initialization.

Our controller’s parameters $\theta$ are initialized uniformly in $[-0.1,0.1]$. We find that for Penn Treebank, ENAS quite insensitive to its initialization than for CIFAR-10. Meanwhile, the shared parameters $\omega$ are initialized uniformly in $[-0.025,0.025]$ during architecture search, and $[-0.04,0.04]$ when we train a fixed architecture recommended by the controller.

Stabilizing the Updates of ω𝜔\omega.

To stabilize the updates of $\mathbf{\omega}$, during the architectures search phase, a layer of batch normalization (Ioffe & Szegedy, 2015) is added immediately after the average of these layers, before the average are sent out of the cell as its output. When a fixed cell is sampled by the controller, we find that we can remove the batch normalization layer without any loss in performance.

Appendix B Details on CIFAR-10 Experiments

We find the following tricks crucial for achieving good performance with ENAS. Standard NAS (Zoph & Le, 2017; Zoph et al., 2018) rely on these and other tricks as well.

Each convolution in our model is applied in the order of relu-conv-batchnorm (Ioffe & Szegedy, 2015; He et al., 2016b). Additionally, in our micro search space, each depthwise separable convolution is applied twice (Zoph et al., 2018).

Stabilizing Stochastic Skip Connections.

If a layer receives skip connections from multiple layers before it, then these layers’ outputs are concatenated in their depth dimension, and then a convolution of filter size $1\times 1$ (followed by a batch normalization layer and a ReLU layer) is performed to ensure that the number of output channels does not change between different architectures. When a fixed architecture is sampled, we find that one can remove these batch normalization layers to save computing time and parameters of the final model, without sacrificing significant performance.

Global Average Pooling.

After the final convolutional layer, we average all the activations of each channel and then pass them to the Softmax layer. This trick was introduced by (Lin et al., 2013), with the purpose of reducing the number of parameters in the dense connection to the Softmax layer to avoid overfitting.

The last two tricks are extremely important, since the gradient updates of the shared parameters $\omega$, as described in Eqn 1, have very high variance. In fact, we find that without these two tricks, the training of ENAS is very unstable.