Deconstructing the Ladder Network Architecture

Introduction

Labeling data sets is typically a costly task and in many settings there are far more unlabeled examples than labeled ones. Semi-supervised learning aims to improve the performance on some supervised learning problem by using information obtained from both labeled and unlabeled examples. Since the recent success of deep learning methods has mainly relied on supervised learning based on very large labeled datasets, it is interesting to explore semi-supervised deep learning approaches to extend the reach of deep learning to these settings.

Since unsupervised methods for pre-training layers of neural networks were an essential part of the first wave of deep learning methods (Hinton et al., 2006; Vincent et al., 2008; Bengio, 2009), a natural next step is to investigate how ideas inspired by Restricted Boltzmann Machine training and regularized autoencoders can be used for semi-supervised learning. Examples of approaches based on such ideas are the discriminative RBM (Larochelle & Bengio, 2008) and a deep architecture based on semi-supervised autoencoders that was used for document classification (Ranzato & Szummer, 2008). More recent examples of approaches for semi-supervised deep learning are the semi-supervised Variational Autoencoder (Kingma et al., 2014) and the Ladder Network (Rasmus et al., 2015) which obtained very impressive, state of the art results (1.13% error) on the MNIST handwritten digits classification benchmark using just 100 labeled training examples.

The Ladder Network adds an unsupervised component to the supervised learning objective of a deep feedforward network by treating this network as part of a deep stack of denoising autoencoders or DAEs (Vincent et al., 2010) that learns to reconstruct each layer (including the input) based on a corrupted version of it, using feedback from upper levels. The term ’ladder’ refers to how this architecture extends the stacked DAE in the way the feedback paths are formed.

This paper is focusing on the design choices that lead to the Ladder Network’s superior performance and tries to disentangle them empirically. We identify some general properties of the model that make it different from standard feedforward networks and compare various architectures to identify those properties and design choices that are the most essential for obtaining good performance with the Ladder Network. While the original authors of the Ladder Network paper explored some variants of their model already, we provide a thorough comparison of a large number of architectures controlling for both hyperparameter settings and data set selection. Finally, we also introduce a variant of the Ladder Network that yields new state-of-the-art results for the Permutation-Invariant MNIST classification task in both semi- and fully- supervised settings.

The Ladder Network Architecture

In this section, we describe the Ladder Network ArchitecturePlease refer to (Rasmus et al., 2015; Valpola, 2014) for more detailed explanation of the Ladder Network architecture.. Consider a dataset with $N$ labeled examples ${(x(1),y^{*}(1)),(x(2),y^{*}(2)),...,(x(N),y^{*}(N))}$ and $M$ unlabeled examples ${x(N+1),x(N+2),...,x(N+M)}$ where $M\gg N$. The objective is to learn a function that models $P(y|x)$ by using both the labeled examples and the large quantity of unlabeled examples. In the case of the Ladder Network, this function is a deep Denoising Auto Encoder (DAE) in which noise is injected into all hidden layers and the objective function is a weighted sum of the supervised Cross Entropy cost on the top of the encoder and the unsupervised denoising Square Error costs at each layer of the decoder. Since all layers are corrupted by noise, another encoder path with shared parameters is responsible for providing the clean reconstruction targets, i.e. the noiseless hidden activations (See Figure 1).

Through lateral skip connections, each layer of the noisy encoder is connected to its corresponding layer in the decoder. This enables the higher layer features to focus on more abstract and task-specific features. Hence, at each layer of the decoder, two signals, one from the layer above and the other from the corresponding layer in the encoder are combined.

Formally, the Ladder Network is defined as follows:

In the forward path, individual layers of the encoder are formalized as a linear transformation followed by Batch Normalization (Ioffe & Szegedy, 2015) and then application of a nonlinear activation function:

where $\odot$ is an element-wise multiplication operator and each per-element weight is initialized as:

In later sections, we will explore alternative initialization schemes on the vanilla combinator. Finally, the $\mbox{ReconsCost}(z^{(l)},\hat{z}^{(l)})$ in equation (4) is defined as the following:

where $\hat{z}^{(l)}$ is normalized using $\mu^{(l)}$ and $\sigma^{(l)}$ which are the encoder’s sample mean and standard deviation statistics of the current mini batch, respectively. The reason for this second normalization is to cancel the effect of unwanted noise introduced by the limited batch size of Batch Normalization.

Components of the Ladder Network

Now that the precise architecture of the Ladder Network has been described in details, we can identify a couple of important additions to the standard feed-forward neural network architecture that may have a pronounced impact on the performance. A distinction can also be made between those design choices that follow naturally from the motivation of the ladder network as a deep autoencoder and those that are more ad-hoc and task specific.

The most obvious addition is the extra reconstruction cost for every hidden layer and the input layer. While it is clear that the reconstruction cost provides an unsupervised objective to harness the unlabeled examples, it is not clear how important the penalization is for each layer and what role it plays for fully-supervised tasks.

A second important change is the addition of Gaussian noise to the input and the hidden representations. While adding noise to the first layer is a part of denoising autoencoder training, it is again not clear whether it is necessary to add this noise at every layer or not. We would also like to know if the noise helps by making the reconstruction task nontrivial and useful or just by regularizing the feed-forward network in a similar way that noise-based regularizers like dropout (Srivastava et al., 2014) and adaptive weight noise (Graves, 2011) do.

Experimental Setup

In this section, we introduce different variants of the Ladder Architecture and describe our experiment methodology. Some variants are derived by removal of one component of the model while other variants are derived by the replacement of that component with a new one. This enables us to isolate each component and observe its effects while other components remain unchanged. Table 1 depicts the hierarchy of the different variants and the baseline models.

Different configurations of noise injection, penalizing reconstruction errors, and the lateral connection removal suggest four different variants:

Add noise only to the first layer (FirstNoise).

Only penalize the reconstruction at the first layer (FirstRecons), i.e. $\lambda^{(l\geq 1)}$ are set to .

Apply both of the above changes: add noise and penalize the reconstruction only at the first layer (FirstN&R).

Remove all lateral connections from FirstN&R. Therefore, equivalent to to a denoising autoencoder with an additional supervised cost at the top, the encoder and the decoder are connected only through the topmost connection. We call this variant NoLateral.

1.2 Vanilla combinator variants

The simplest way of combining lateral connections with vertical connections is to simply add them in an element-wise fashion, similar to the nature of skip-connections in a recently published work on Residual Learning (He et al., 2015). We call this variant Linear combinator function. We also derive two more variants NoSig and NoMult in the way of stripping down the vanilla combinator function to the simple Linear one:

Remove sigmoid non-linearity (NoSig). The corresponding per-element weights are initialized in the same way as the vanilla combinator.

where the initialization scheme resembles the vanilla one in which $w_{z}$ is initialized to one while $w_{u}$ and $b$ are initialized to zero.

2 Variants derived by replacement of a component

We explore two variants of the initialization scheme:

Random initialization (RandInit): all per-element parameters are randomly initialized to $\mathcal{N}(0,0.2)$.

Reverse initialization (RevInit): all per-element parameters $w_{\{0,1\}z}$, $w_{\{0,1\}zu}$, and $b_{\{0,1\}}$ are initialized to zero while $w_{\{0,1\}u}$ and $w_{\sigma}$ are initialized to one.

2.2 Gaussian combinator variants

Strictly speaking, $\nu(u)$ is not a proper weight, because it is not guaranteed to be positive all the time. To make the Gaussian interpretation rigorous, we explore a variant that we call GatedGauss, where equations 20 and 21 stay the same but 22 is replaced by:

GatedGauss guarantees that $0<\nu(u)<1$. We expect that $\nu(u)_{i}$ will be close to 1 if the information from the lateral connection for unit $i$ is more helpful to reconstruction, and close to 0 if the vertical connection becomes more useful. The GatedGauss combinator is similar in nature to the gating mechanisms in other models such as Gated Recurrent Unit (Cho et al., 2014) and highway networks (Srivastava et al., 2015).

2.3 MLP (Multilayer Perceptron) combinator variants

We experiment with different numbers of layers and hidden units per layer in the MLP. We present results for three specific configurations: $$for a single hidden layer of 4 units,$$ for 2 hidden layers each with 2 units, and $$for 3 hidden layers. For example, in the$$ configuration, the MLP combinator function is defined as:

where $W_{1}$, $W_{2}$, and $W_{3}$ are $2\times 2$ weight matrices; $b_{1}$, $b_{2}$, and $b_{3}$ are $2\times 1$ bias vectors.

We do similar experiments as in the MLP case and include results for $$,$$ and $$ hidden layer configurations.

Both MLP and AMLP weight parameters are randomly initialized to $\mathcal{N}(0,\eta)$. $\eta$ is considered to be a hyperparameter and tuned on the validation set. Precise values for the best $\eta$ values are listed in Appendix.

3 Methodology

The experimental setup includes two semi-supervised classification tasks with 100 and 1000 labeled examples and a fully-supervised classification task with 60000 labeled examples for Permutation-Invariant MNIST handwritten digit classification. Labeled examples are chosen randomly but the number of examples for different classes is balanced. The test set is not used during all the hyperparameter search and tuning. Each experiment is repeated 10 times with 10 different but fixed random seeds to measure the standard error of the results for different parameter initializations and different selections of labeled examples.

All variants and the vanilla Ladder Network itself are trained using the ADAM optimization algorithm (Kingma & Ba, 2014) with a learning rate of 0.002 for 100 iterations followed by 50 iterations with a learning rate decaying linearly to 0. Hyperparameters including the standard deviation of the noise injection and the denoising weights at each layer are tuned separately for each variant and each experiment setting (100-, 1000-, and fully-labeled). Hyperparmeters are optimized by either a random search (Bergstra & Bengio, 2012), or a grid search, depending on the number of hyperparameters involved (see Appendix for precise configurations).

Results & Discussion

Table 2 collects all results for the variants and the baselines. The results are organized into five main categories for each of the three tasks. The Baseline model is a simple feed-forward neural network with no reconstruction penalty and Baseline+noise is the same network but with additive noise at each layer. The best results in terms of average error rate on the test set are achieved by the proposed AMLP combinator function: in the fully-supervised setting, the best average error rate is $0.569\pm 0.010$, while in the semi-supervised settings with 100 and 1000 labeled examples, the averages are $1.002\pm 0.037$ and $0.974\pm 0.021$ respectively. A visualization of the a learned combinator function is shown in Appendix.

The results in the table indicate that in the fully-supervised setting, adding noise either to the first layer only or to all layers leads to a lower error rate with respect to the baselines. Our intuition is that the effect of additive noise to layers is very similar to the weight noise regularization method (Graves, 2011) and dropout (Hinton et al., 2012).

In addition, it seems that removing the lateral connections hurts much more than the absence of noise injection or reconstruction penalty in the intermediate layers. It is also worth mentioning that hyperparameter tuning yields zero weights for penalizing the reconstruction errors in all layers except the input layer in the fully-supervised task for the vanilla model. Something similar happens for NoLateral as well, where hyperparameter tuning yields zero reconstruction weights for all layers including the input layer. In other words, NoLateral and Baseline+noise become the same models for the fully-supervised task. Moreover, the weights for the reconstruction penalty of the hidden layers are relatively small in the semi-supervised task. This is in line with similar observations (relatively small weights for the unsupervised part of the objective) for the hybrid discriminant RBM (Larochelle & Bengio, 2008).

The third part of Table 2 shows the relative performance of different combinator functions by removal. Unsurprisingly, the performance deteriorates considerably if we remove the sigmoid non-linearity (NoSig) or the multiplicative term (NoMult) or both (Linear) from the vanilla combinator. Judging from the size of the increase in average error rates, the multiplicative term is more important than the sigmoid unit.

2 Variants derived by replacements

3 Probabilistic Interpretations of the Ladder Network

Since many of the motivations behind regularized autoencoder architectures are based on observations about generative models, we briefly discuss how the Ladder Network can be related to some other models with varying degrees of probabilistic interpretability. Considering that the components that are most defining of the Ladder Network seem to be the most important ones for semi-supervised learning in particular, comparisons with generative models are at least intuitively appealing to get more insight about how the model learns about unlabeled examples.

By training the individual denoising autoencoders that make up the Ladder Network with a single objective function, this coupling goes as far as encouraging the lower levels to produce representations that are going to be easy to reconstruct by the upper levels. We find a similar term (-log of the top-level prior evaluated at the output of the encoder) in hierarchical extensions of the variational autoencoder (Rezende et al., 2014; Bengio, 2014). While the Ladder Network differs too much from an actual variational autoencoder to be treated as such, the similarities can still give one intuitions about the role of the noise and the interactions between the layers. Conversely, one also may wonder how a variational autoencoder might benefit from some of the components of Ladder Network like Batch Normalization and multiplicative connections.

When one simply views the Ladder Network as a peculiar type of denoising autoencoder, one could extend the recent work on the generative interpretation of denoising autoencoders (Alain & Bengio, 2013; Bengio et al., 2013) to interpret the Ladder Network as a generative model as well. It would be interesting to see if the Ladder Network architecture can be used to generate samples and if the architecture’s success at semi-supervised learning translates to this profoundly different use of the model.

Conclusion

The paper systematically compares different variants of the recent Ladder Network architecture (Rasmus et al., 2015; Valpola, 2014) with two feedforward neural networks as the baselines and the standard architecture (proposed in the original paper). Comparisons are done in a deconstructive way, starting from the standard architecture. Based on the comparisons of different variants we conclude that:

Unsurprisingly, the reconstruction cost is crucial to obtain the desired regularization from unlabeled data.

Applying additive noise to each layer and especially the first layer has a regularization effect which helps generalization. This seems to be one of the most important contributors to the performance on the fully supervised task.

The lateral connection is a vital component in the Ladder architecture to the extent that removing it considerably deteriorates the performance for all of the semi-supervised tasks.

The precise choice of the combinator function has a less dramatic impact, although the vanilla combinator can be replaced by the Augmented MLP to yield better performance, in fact allowing us to improve the record error rates on Permutation-Invariant MNIST for semi- and fully-supervised settings.

We hope that these comparisons between different architectural choices will help to improve understanding of semi-supervised learning’s success for the Ladder Network and like architectures, and perhaps even deep architectures in general.

References