L2-constrained Softmax Loss for Discriminative Face Verification

Introduction

Face verification in unconstrained settings is a challenging problem. Despite the excellent performance of recent face verification systems on curated datasets like Labeled Faces in the Wild (LFW) , it is still difficult to achieve similar accuracy on faces with extreme variations in viewpoints, resolution, occlusion and image quality. This is evident from the performance of the traditional algorithms on the publicly available IJB-A dataset. Data quality imbalance in the training set is one of the reason for this performance gap. Existing face recognition training datasets contain large amount of high quality and frontal faces, whereas the unconstrained and difficult faces occur rarely. Most of the DCNN-based methods trained with softmax loss for classification tend to over-fit to the high quality data and fail to correctly classify faces acquired in difficult conditions.

Using softmax loss function for training face verification system has its own pros and cons. On the one hand, it can be easily implemented using inbuilt functions from the publicly available deep leaning toolboxes such as Caffe , Torch and TensorFlow . Unlike triplet loss , it does not have any restrictions on the input batch size and converges quickly. The learned features are discriminative enough for efficient face verification without any metric learning.

On the other hand, the softmax loss is biased to the sample distribution. Unlike contrastive loss and triplet loss which specifically attend to hard samples, the softmax loss maximizes the conditional probability of all the samples in a given mini-batch. Hence, it fits well to the high quality faces, ignoring the rare difficult faces from a training mini-batch. We observe that the $L_{2}$-norm of features learned using softmax loss is informative of the quality of the face . Features for good quality frontal faces have a high $L_{2}$-norm while blurry faces with extreme pose have low $L_{2}$-norm (see Figure 1(b)). Moreover, the softmax loss does not optimize the verification requirement of keeping positive pairs closer and negative pairs far from each other. Due to this reason, many methods either apply metric learning on top of softmax features or train an auxiliary loss along with the softmax loss to achieve better verification performance.

In this paper, we provide a symptomatic treatment to issues associated with the softmax loss. We propose an $L_{2}$-softmax loss that adds a constraint on the features during training such that their $L_{2}$-norm remain constant. In other words, we restrict the features to lie on a hypersphere of a fixed radius. The proposed $L_{2}$-softmax loss has a dual advantage. Firstly, it provides similar attention to both good and bad quality faces since all the features have the same $L_{2}$-norm now, which is essential for better performance in unconstrained settings. Secondly, it strengthens the verification signal by forcing the same subject features to be closer and different subject features to be far from each other in the normalized space. Thus, it maximizes the margin for the normalized $L_{2}$ distance or cosine similarity score between negative and positive pairs. Thus, it overcomes the main disadvantages of the regular softmax loss.

The $L_{2}$-softmax loss also retains the advantages of the regular softmax loss. Similar to the softmax loss, it is a one network, one loss system. It doesn’t necessarily require any joint supervision as used by many recent methods . It can be easily implemented using inbuilt functions from Caffe , Torch and TensorFlow , and converges very fast. It introduces just a single scaling parameter to the network. Compared to the regular softmax loss, the $L_{2}$-softmax loss gains a significant boost in the performance. It achieves new state-of-the-art results on IJB-A dataset, and competing results on LFW and YouTube Face datasets. It surpasses the performance of several state-of-the-art systems, which use multiple networks or multiple loss functions or both. In summary, this paper contributes to the following aspects:

We propose a simple, novel and effective $L_{2}$-softmax loss for face verification that restricts the $L_{2}$-norm of the feature descriptor to a constant value $\alpha$.

We study the variations in the performance with respect to the scaling parameter $\alpha$ and provide suitable bounds on its value for achieving consistently high performance.

The proposed method yields a consistent and significant boost on all the three challenging face verification datasets namely LFW , YouTube Face and IJB-A

Moreover, the gains from $L_{2}$-softmax loss are complementary to metric learning (eg: TPE , joint-Bayes ) or auxiliary loss functions (eg: center loss , contrastive loss ). We show that applying these techniques on top of the $L_{2}$-softmax loss can further improve the verification performance. Combining with TPE , $L_{2}$-softmax loss achieves a record True Accept Rate (TAR) of 0.909 at False Accept Rate (FAR) of 0.0001 on the challenging IJB-A dataset.

Related Work

In recent years, there has been a significant improvement in the accuracy of face verification using deep learning methods . Most of these methods have even surpassed human performance on the LFW dataset. Although these methods use DCNNs, they differ by the type of loss function they use for training. For face verification, its essential for the positive subjects pair features to be closer and negative subjects pair features far apart. To solve this problem, researchers have adopted two major approaches.

In the first approach, pairs of face images are input to the training algorithm to learn a feature embedding where positive pairs are closer and negative pairs are far apart. In this direction, Chopra et al. proposed siamese networks with contrastive loss for training. Hu et al. designed a discriminative deep metric with a margin between positive and negative face pairs. FaceNet introduces triplet loss to learn the metric using hard triplet face samples.

In the second approach, the face images along with their subject labels are used to learn discriminative identification features in a classification framework. Most of the recent methods train a DCNN with softmax loss to learn these features which are later used either to directly compute the similarity score for a pair of faces or to train a discriminative metric embedding . Another strategy is to train the network for joint identification-verification task . Xiong et al. proposed transferred deep feature fusion (TDFF) which infolves two-stage fusion of features trained with different networks and datasets. Template adaptation is applied to further boost the performance.

A recent approach introduced center loss to learn better discriminative face features. Our proposed method is different from the center loss in the following aspects. First, we use one loss function (i.e., $L_{2}$-softmax loss) whereas uses center loss jointly with the softmax loss during training. Second, center loss introduces $C\times D$ additional parameters during training where $C$ is the number of classes and $D$ is the feature dimension. On the other hand, the $L_{2}$-softmax loss introduces just a single parameter that defines the fixed $L_{2}$-norm of the features. Moreover, the center loss can also be used in conjunction with $L_{2}$-softmax loss, which performs better than center loss trained with regular softmax loss (see Section 5.1.4).

Recently, a few algorithms have used feature normalization during training to improve performance. SphereFace proposes angular softmax (A-softmax) loss that enables DCNNs to learn angularly discriminative features. Another method called DeepVisage uses a special case of batch normalization technique to normalize the feature descriptor before applying the softmax loss. Our proposed method is different as it applies an $L_{2}$-constraint on the feature descriptors enforcing them to lie on a hypersphere of a given radius.

Motivation

We first summarize the general pipeline for training a face verification system using DCNN as shown in Figure 2. Given a training dataset with face images and corresponding identity labels, a DCNN is trained as a classification task where the network learns to classify a given face image to its correct identity label. A softmax loss function is used for training the network, given by Equation 1

where $M$ is the training batch size, $\mathbf{x}_{i}$ is the $i^{th}$ input face image in the batch, $f(\mathbf{x}_{i})$ is the corresponding output of the penultimate layer of the DCNN, $y_{i}$ is the corresponding class label, and $W$ and $b$ are the weights and bias for the last layer of the network which acts as a classifier.

At test time, feature descriptors $f(\mathbf{x}_{g})$ and $f(\mathbf{x}_{p})$ are extracted for the pair of test face images $\mathbf{x}_{g}$ and $\mathbf{x}_{p}$ respectively using the trained DCNN, and normalized to unit length. Then, a similarity score is computed on the feature vectors which provides a measure of distance or how close the features lie in the embedded space. If the similarity score is greater than a set threshold, the face pairs are decided to be of the same person. Usually, the similarity score is computed as the $L_{2}$-distance between the normalized features or by using cosine similarity $s$, as given by Equation 2. Both these similarity measures are equivalent and produce same results.

There are two major issues with this pipeline. First, the training and testing steps for face verification task are decoupled. Training with softmax loss doesn’t necessarily ensure the positive pairs to be closer and the negative pairs to be far separated in the normalized or angular space.

Secondly, the softmax classifier is weak in modeling difficult or extreme samples. In a typical training batch with data quality imbalance, the softmax loss gets minimized by increasing the $L_{2}$-norm of the features for easy samples, and ignoring the hard samples. The network thus learns to respond to the quality of the face by the $L_{2}$-norm of its feature descriptor. To validate this theory, we perform a simple experiment on the IJB-A dataset where we divide the templates (groups of images/frames of same subject) into three different sets based on the $L_{2}$-norm of their feature descriptors. The features were computed using Face-Resnet trained with regular softmax loss. Templates with descriptors’ $L_{2}$-norm <$90$ are assigned to set$1$. The templates with $L_{2}$-norm >$90$ but <$150$ are assigned to set$2$, while templates with $L_{2}$-norm >$150$ are assigned to set$3$. In total they form six sets of evaluation pairs. Figure 1(a) shows the performance of the these six different sets for the IJB-A face verification protocol. It can be clearly seen that pairs having low $L_{2}$-norm for both the templates perform very poor, while the pairs with high $L_{2}$-norm perform the best. The difference in performance between each set is quite significant. Figure 1(b) shows some sample templates from set$1$, set$2$ and set$3$ which confirms that the $L_{2}$-norm of the feature descriptor is informative of its quality.

To solve these issues, we enforce the $L_{2}$-norm of the features to be fixed for every face image. Specifically, we add an $L_{2}$-constraint to the feature descriptor such that it lies on a hypersphere of a fixed radius. This approach has two advantages. Firstly, on a hypersphere, minimizing the softmax loss is equivalent to maximizing the cosine similarity for the positive pairs and minimizing it for the negative pairs, which strengthens the verification signal of the features. Secondly, the softmax loss is able to model the extreme and difficult faces better, since all the face features have same $L_{2}$-norm.

Proposed Method

The proposed $L_{2}$-softmax loss is given by Equation 3

where $\mathbf{x}_{i}$ is the input image in a mini-batch of size $M$, $y_{i}$ is the corresponding class label, $f(\mathbf{x}_{i})$ is the feature descriptor obtained from the penultimate layer of DCNN, $C$ is the number of subject classes, and $W$ and $b$ are the weights and bias for the last layer of the network which acts as a classifier. This equation adds an additional $L_{2}$-constraint to the regular softmax loss defined in Equation 1. We show the effectiveness of this constraint using MNIST data.

We study the effect of $L_{2}$-softmax loss on the MNIST dataset . We use a deeper and wider version of LeNet mentioned in , where the last hidden layer output is restricted to $2$-dimensions for easy vizualization. For the first setup, we train the network end-to-end using the regular softmax loss for digits classifcation with number of classes = $10$. For the second setup, we add an $L_{2}$-normalize layer and scale layer to the $2$-dimensional features which enforces the $L_{2}$-constraint described in Equation 3 (seen Section 4.2 for details). Figure 3 depicts the $2$-D features for different classes for MNIST test set containing $10,000$ digit images. Each of the lobes shown in the figure represents $2$-D features of unique digits classes. The features for the second setup were obtained before the $L_{2}$-normalization layer.

We find two clear differences between the features learned using the two setups discussed above. First, the intra-class angular variance is large when using the regular softmax loss, which can be estimated by the average width of the lobes for each class. On the other hand, the features obtained with $L_{2}$-softmax loss have lower intra-class angular variability, and are represented by thinner lobes. Second, the magnitudes of the features are much higher with the softmax loss (ranging upto $150$), since larger feature norms result in a higher probability for a correctly classified class. In contrast, the feature norm has minimal effect on the $L_{2}$-softmax loss since every feature is normalized to a circle of fixed radius before computing the loss. Hence, the network focuses on bringing the features from the same class closer to each other and separating the features from different classes in the normalized or angular space. Table 1 lists the accuracy obtained with the two setups on MNIST test set. $L_{2}$-softmax loss achieves a higher performance, reducing the error by more than $15\%$. Note that these accuracy numbers are lower compared to a typical DCNN since we are using only $2$-dimensional features for classification.

2 Implementation Details

Here, we provide the details of implementing the $L_{2}$-constraint described in Equation 3 in the framework of DCNNs. The constraint is enforced by adding an $L_{2}$-normalize layer followed by a scale layer as shown in Figure 4.

This module is added just after the penultimate layer of DCNN which acts as a feature descriptor. The $L_{2}$-normalize layer normalizes the input feature $\mathbf{x}$ to a unit vector given by Equation 4. The scale layer scales the input unit vector to a fixed radius given by the parameter $\alpha$ (Equation 5). In total, we just introduce one scalar parameter ($\alpha$) which can be trained along with the other parameters of the network.

The module is fully differentiable and can be used in the end-to-end training of the network. At test time, the proposed module is redundant, since the features are eventually normalized to unit length while computing the cosine similarity. At training time, we backpropagate the gradients through the L2-normalize and the scale layer, as well as compute the gradients with respect to the scaling parameter $\alpha$ using the chain rule as given below.

3 Bounds on Parameter α𝛼\alpha

The scaling parameter $\alpha$ plays a crucial role in deciding the performance of $L_{2}$-softmax loss. There are two ways to enforce the $L_{2}$-constraint: 1) by keeping $\alpha$ fixed throughout the training, and 2) by letting the network to learn the parameter $\alpha$. The second way is elegant and always improves over the regular softmax loss. But, the $\alpha$ parameter learned by the network is high which results in a relaxed $L_{2}$-constraint. The softmax classifier aimed at increasing the feature norm for minimizing the overall loss, increases the $\alpha$ parameter instead, allowing it more freedom to fit to the easy samples. Hence, $\alpha$ learned by the network forms an upper bound for the parameter. A better performance is obtained by fixing $\alpha$ to a lower constant value.

On the other hand, with a very low value of $\alpha$, the training doesn’t converge. For instance, $\alpha=1$ performs very poorly on the LFW dataset, achieving an accuracy of $86.37\%$ (see Figure 7). The reason being that a hypersphere with small radius ($\alpha$) has limited surface area for embedding features from the same class together and those from different classes far from each other.

Here, we formulate a theoretical lower bound on $\alpha$. Assuming the number of classes $C$ to be lower than twice the feature dimension $D$, we can distribute the classes on a hypersphere of dimension $D$ such that any two class centers are at least $90^{\circ}$ apart. Figure 5(a) represents this case for $C=4$ class centers distributed on a circle of radius $\alpha$. We assume the classifier weights ($W_{i}$) to be a unit vector pointing in the direction of their respective class centers. We ignore the bias term. The average softmax probability $p$ for correctly classifying a feature is given by Equation 7

Ignoring the term $e^{-\alpha}$ and generalizing it for $C$ classes, the average probability becomes:

Figure 5(b) plots the probability score as a function of the parameter $\alpha$ for various number of classes $C$. We can infer that to achieve a given classification probability (say $p=0.9$), we need to have a higher $\alpha$ for larger $C$. Given the number of classes $C$ for a dataset, we can obtain the lower bound on $\alpha$ to achieve a probability score of $p$ by using Equation 9.

Results

We use the publicly available Face-Resnet DCNN for our experiments. Figure 6 shows the architecture of the network. It contains $27$ convolutional layers and $2$ fully-connected layers. The dimension of the feature descriptor is $512$. It utilizes the widely used residual skip-connections . We add an $L_{2}$-normalize layer and a scale layer after the fully-connected layer to enforce the $L_{2}$-constraint on the descriptor. All our experiments are carried out in Caffe .

In this subsection, we experimentally validate the usefulness of the $L_{2}$-softmax loss for face verification. We form two subsets of training dataset from the MS-Celeb-1M dataset: 1) MS-small containing $0.5$ million face images with the number of subjects being $13403$, and 2) MS-large containing $3.7$ million images of $58207$ subjects. The dataset was cleaned using the clustering algorithm mentioned in . We train the Face-Resnet network with softmax loss as well as $L_{2}$-softmax loss for various $\alpha$. While training with MS-small, we start with a base learning rate of $0.1$ and decrease it by ${1/10}^{th}$ after $16K$ and $24K$ iterations, upto a maximum of $28K$ iterations. For training on MS-large, we use the same learning rate but decrease it after $50K$ and $80K$ iterations upto a maximum of $100K$ iterations. A training batch size of $256$ was used. Both softmax and $L_{2}$-softmax loss functions consume the same amount of training time which is around $9$ hours for MS-small and $32$ hours for MS-large training set respectively, on two TITAN X GPUs. We set the learning multiplier and decay multiplier for the scale layer to $1$ for trainable $\alpha$, and for fixed $\alpha$ during the network training. We evaluate our baselines on the widely used LFW dataset for the unrestricted setting, and the challenging IJB-A dataset on the 1:1 face verification protocol. The faces were cropped and aligned to the size of $128\times 128$ in both training and testing phases by implementing the face detection and alignment algorithm mentioned in .

Here, we compare the network trained on MS-small dataset using our proposed $L_{2}$-softmax loss, against the one trained with regular softmax loss. Figure 7 shows that the regular softmax loss attains an accuracy of $98.1\%$ whereas the proposed $L_{2}$-softmax loss achieves the best accuracy of $99.28\%$, thereby reducing the error by more than $62\%$. It also shows the variations in performance with the scale parameter $\alpha$. The performance is poor when $\alpha$ is below a certain threshold and stable with $\alpha$ higher than the threshold. This behavior is consistent with the theoretical analysis presented in Section 4.3. From the figure, the performance of $L_{2}$-Softmax is better for $\alpha$ >$12$ which is close to its lower bound computed using equation 9 for $C=13403$ with a probability score of $0.9$.

A similar trend is observed for 1:1 verification protocol on IJB-A as shown in Table 2, where the numbers denote True Accept Rate (TAR) at False Accept Rates (FAR) of $0.0001$, $0.001$, $0.01$ and $0.1$. Our proposed approach improves the TAR@FAR=0.0001 by $19\%$ compared to the baseline softmax loss. The performance is consistent with $\alpha$ ranging between $16$ to $32$. Another point to note is that by allowing the network to learn the scale parameter $\alpha$ by itself results in a slight decrease in performance, which shows that having a tighter constraint is a better choice.

1.2 Experiment with large training set

We train the network on the MS-large dataset for this experiment. Figure 8 shows the performance on the LFW dataset. Similar to the small training set, the $L_{2}$-softmax loss significantly improves over the baseline, reducing the error by $60\%$ and achieving an accuracy of $99.6\%$. Similarly, it improves the TAR@FAR=0.0001 on IJB-A by more than $10\%$ (Table 3). The performance of $L_{2}$-softmax is consistent with $\alpha$ in the range $40$ and beyond. Unlike, the small set training, the self-trained $\alpha$ performs equally good compared to fixed $\alpha$ of $40$ and $50$. The theoretical lower bound on $\alpha$ is not of much use in this case since improved performance is achieved for $\alpha$ >$30$. We can deduce that as the number of subjects increases, the lower bound on $\alpha$ is less reliable, and the self-trained $\alpha$ is more reliable with performance. This experiment clearly suggests that the proposed $L_{2}$-softmax loss is consistent across the training and testing datasets.

1.3 Experiment with a different DCNN

To check the consistency of our proposed $L_{2}$-softmax loss, we apply it on the All-In-One Face instead of the Face-Resnet. We use the recognition branch of the All-In-One Face to fine-tune on the MS-small training set. The recognition branch of All-In-One Face consists of $7$ convolution layers followed by $3$ fully-connected layers and a softmax loss. We add an $L_{2}$-normalize and a scale layer after the $512$ dimension feature descriptor. Figure 9 shows the comparison of $L_{2}$-softmax loss and the base softmax loss on LFW dataset. Similar to the Face-Resnet, All-In-One Face with $L_{2}$-softmax loss improves over the base softmax performance, reducing the error by $40\%$, and achieving an accuracy of $98.82\%$. The improvement obtained by using All-In-One Face is smaller compared to the Face-Resnet. This shows that residual connections and depth of the network generate better feature embedding on a hypersphere. The performance variation with scaling parameter $\alpha$ is similar to that of Face-Resnet, indicating that the optimal scale parameter does not depend on the choice of the network.

1.4 Experiment with auxiliary loss

Similar to softmax loss, the $L_{2}$-softmax loss can be coupled with auxiliary losses such as center loss, contrastive loss, triplet loss, etc. to further improve the performance. Here we study the performance variation of $L_{2}$-softmax loss when coupled with the center loss. We use the MS-small dataset for training the networks. Table 4 lists the accuracy obtained on LFW dataset by different loss functions. The softmax loss performs the worst. The center loss improves the performance significantly when trained in conjunction with the softmax loss, and is comparable to the $L_{2}$-softmax loss. Training center loss with the $L_{2}$-softmax loss gives the best performance of $99.33\%$ accuracy. This shows that $L_{2}$-softmax loss is as versatile as the regular softmax loss and can be used efficiently with other auxiliary loss functions.

2 Comparison with recent methods

We compare our algorithm with recently reported face verification methods on LFW , YouTube Face and IJB-A datasets. We crop and align the images for all these datasets by implementing the algorithm mentioned in . We train the Face-Resnet (FR) with $L_{2}$-softmax as well as regular softmax loss using the MS-large training set. Additionally, we train ResNet-101(R101) and ResNeXt-101(RX101) deep networks for face recognition using MS-large training set with $L_{2}$-softmax loss. Both R101 and RX101 models were initialized with parameters pre-trained on ImageNet dataset. A fully-connected layer of dimension $512$ was added before the $L_{2}$-softmax classifier. The scaling parameter was kept fixed with a value of $\alpha=50$. Experimental results on different datasets show that $L_{2}$-softmax works efficiently with deeper models.

The LFW dataset contains $13,233$ web-collected images from $5749$ different identities. We evaluate our model following the standard protocol of unrestricted with labeled outside data. We test on 6,000 face pairs and report the experiment results in Table 5. Along with the accuracy values, we also compare with the number of images, networks and loss functions used by the methods for their overall training. The proposed method attains the state-of-the-art performance with the RX101 model, achieving an accuracy of $99.78\%$. Unlike other methods which use auxiliary loss functions such as center loss and contrastive loss along with the primary softmax loss, our method uses a single loss training paradigm which makes it easier and faster to train.

YouTube Face (YTF) dataset contains $3425$ videos of $1595$ different people, with an average length of $181.3$ frames per video. It contains $10$ folds of $500$ video pairs. We follow the standard verification protocol and report the average accuracy on splits with cross-validation in Table 5. We achieve the accuracy of $96.08\%$ using $L_{2}$-softmax loss with RX101 network. Our method outperforms many recent algorithms and is only behind DeepVisage which uses larger number of training samples, and VGG Face which further uses a discriminative metric learning on YTF.

The IJB-A dataset contains $500$ subjects with a total of $25,813$ images including $5,399$ still images and $20,414$ video frames. It contains faces with extreme viewpoints, resolution and illumination which makes it more challenging than the commonly used LFW dataset. Given a template containing multiple faces of the same individual, we generate a common vector representation by media pooling the individual face descriptors, as explained in . Table 6 lists the performance of recent DCNN-based methods on IJB-A dataset. We achieve state-of-the-art result for both verification and the identification protocols. Since the $L_{2}$-softmax loss can be coupled with any other auxiliary loss, we use the Triplet Probabilistic Embedding (TPE) to learn a $128$-dimensional embedding using the training splits of IJB-A. It further improves the performance and achieves a record TAR of 0.909 @ FAR = 0.0001. To the best of our knowledge, we are the first ones to surpass TAR of 0.9 @ FAR of 0.0001 on IJB-A. Our method performs significantly better than existing methods in most of the other metrics as well. The results on LFW , YTF and IJB-A datasets clearly suggests the effectiveness of the proposed $L_{2}$-softmax loss.

Conclusions

In this paper, we added a simple, yet effective, $L_{2}$-constraint to the regular softmax loss for training a face verification system. The constraint enforces the features to lie on a hypersphere of a fixed radius characterized by parameter $\alpha$. We also provided bounds on the value of $\alpha$ for achieving a consistent performance. Experiments on LFW, YTF and IJB-A datasets show that the proposed $L_{2}$-softmax loss provides a significant and consistent boost over the regular softmax loss and achieves the state-of-the-art result on IJB-A dataset. In conclusion, $L_{2}$-softmax loss is a valuable replacement for the existing softmax loss, for the task of face verification. In the future, we would further explore the possibility of exploiting the geometric structure of the feature encoding using manifold-based metric learning.

ACKNOWLEDGMENTS

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

References