Surpassing Human-Level Face Verification Performance on LFW with GaussianFace

Introduction

Face verification, which is the task of determining whether a pair of face images are from the same person, has been an active research topic in computer vision for decades . It has many important applications, including surveillance, access control, image retrieval, and automatic log-on for personal computer or mobile devices. However, various visual complications deteriorate the performance of face verification, as shown by numerous studies on real-world face images from the wild . The Labeled Faces in the Wild (LFW) dataset is well known as a challenging benchmark for face verification. The dataset provides a large set of relatively unconstrained face images with complex variations in pose, lighting, expression, race, ethnicity, age, gender, clothing, hairstyles, and other parameters. Not surprisingly, LFW has proven difficult for automatic face verification methods . Although there has been significant work on LFW and the accuracy rate has been improved from 60.02% to 97.35% since LFW is established in 2007, these studies have not closed the gap to human-level performance in face verification.

Why could not we surpass the human-level performance? Two possible reasons are found as follows:

1) Most existing face verification methods assume that the training data and the test data are drawn from the same feature space and follow the same distribution. When the distribution changes, these methods may suffer a large performance drop . However, many practical scenarios involve cross-domain data drawn from different facial appearance distributions. Learning a model solely on a single source data often leads to overfitting due to dataset bias . Moreover, it is difficult to collect sufficient and necessary training data to rebuild the model in new scenarios, for highly accurate face verification specific to the target domain. In such cases, it becomes critical to exploit more data from multiple source-domains to improve the generalization of face verification methods in the target-domain.

2) Modern face verification methods are mainly divided into two categories: extracting low-level features , and building classification models . Although these existing methods have made great progress in face verification, most of them are less flexible when dealing with complex data distributions. For the methods in the first category, for example, low-level features such as SIFT , LBP , and Gabor are handcrafted. Even for features learned from data , the algorithm parameters (such as the depth of random projection tree, or the number of centers in k-means) also need to be specified by users. Similarly, for the methods in the second category, the architectures of deep networks in (for example, the number of layers, the number of nodes in each layer, etc.), and the parameters of the models in (for example, the number of Gaussians, the number of classifiers, etc.) must also be determined in advance. Since most existing methods require some assumptions to be made about the structures of the data, they cannot work well when the assumptions are not valid. Moreover, due to the existence of the assumptions, it is hard to capture the intrinsic structures of data using these methods.

To this end, we propose the Multi-Task Learning approach based on Discriminative Gaussian Process Latent Variable Model (DGPLVM) , named GaussianFace, for face verification. Unlike most existing studies that rely on a single training data source, in order to take advantage of more data from multiple source-domains to improve the performance in the target-domain, we introduce the multi-task learning constraint to DGPLVM. Here, we investigate the asymmetric multi-task learning because we only focus on the performance improvement of the target task. From the perspective of information theory, this constraint aims to maximize the mutual information between the distributions of target-domain data and multiple source-domains data. Moreover, the GaussianFace model is a reformulation based on the Gaussian Processes (GPs) , which is a non-parametric Bayesian kernel method. Therefore, our model also can adapt its complexity flexibly to the complex data distributions in the real-world, without any heuristics or manual tuning of parameters.

Reformulating GPs for large-scale multi-task learning is non-trivial. To simplify calculations, we introduce a more efficient equivalent form of Kernel Fisher Discriminant Analysis (KFDA) to DGPLVM. Despite that the GaussianFace model can be optimized effectively using the Scaled Conjugate Gradient (SCG) technique, the inference is slow for large-scale data. We make use of GP approximations and anchor graphs to speed up the process of inference and prediction, so as to scale our model to large-scale data. Our model can be applied to face verification in two different ways: as a binary classifier and as a feature extractor. In the former mode, given a pair of face images, we can directly compute the posterior likelihood for each class to make a prediction. In the latter mode, our model can automatically extract high-dimensional features for each pair of face images, and then feed them to a classifier to make the final decision.

The main contributions of this paper are as follows:

We propose a novel GaussianFace model for face verification by virtue of the multi-task learning constraint to DGPLVM. Our model can adapt to complex distributions, avoid over-fitting, exploit discriminative information, and take advantage of multiple source-domains data.

We introduce a computationally more efficient equivalent form of KFDA to DGPLVM. This equivalent form reformulates KFDA to the kernel version consistent with the covariance function in GPs, which greatly simplifies calculations.

We introduce approximation in GPs and anchor graphs to speed up the process of inference and prediction.

We achieve superior performance on the challenging LFW benchmark , with an accuracy rate of 98.52%, beyond human-level performance reported in .

Related Work

Human and computer performance on face recognition has been compared extensively . These studies have shown that computer-based algorithms were more accurate than humans in well-controlled environments (e.g., frontal view, natural expression, and controlled illumination), whilst still comparable to humans in the poor condition (e.g., frontal view, natural expression, and uncontrolled illumination). However, the above conclusion is only verified on face datasets with controlled variations, where only one factor changes at a time . To date, there has been virtually no work showing that computer-based algorithms could surpass human performance on unconstrained face datasets, such as LFW, which exhibits natural (multifactor) variations in pose, lighting, expression, race, ethnicity, age, gender, clothing, hairstyles, and other parameters.

There has been much work dealing with multifactor variations in face verification. For example, Simonyan et al. applied the Fisher vector to face verification and achieved a good performance . However, the Fisher vector is derived from the Gaussian mixture model (GMM), where the number of Gaussians need to be specified by users, which means it cannot cover complex data automatically. Li et al. proposed a non-parametric subspace analysis , but it is only a linear transformation and cannot cover the complex distributions. Besides, there also exist some approaches for utilizing plentiful source-domain data. Based on the Joint Bayesian algorithm , Cao et al. proposed a transfer learning approach by merging source-domain data with limited target-domain data. Since this transfer learning approach is based on the joint Bayesian model of original visual features, it is not suitable for handling the complex nonlinear data and the data with complex manifold structures. Moreover, the transfer learning approach in only considered two different domains, restricting its wider applications in large-scale data from multiple domains. More recently, Zhu et al. learned the transformation from face images under various poses and lighting conditions to a canonical view with a deep convolutional network. Sun et al. learned face representation with a deep model through face identification, which is a challenging multi-class prediction task. Taigman et al. first utilized explicit 3D face modeling to apply a piecewise affine transformation, and then derived a face representation from a nine-layer deep neural network. Although these methods have achieved high performances on LFW, many parameters of them must be determined in advance so that they are less flexible when dealing with complex data distributions.

The core of our algorithm is GPs. To the best of our knowledge, GPs methods and Multi-task learning with related GPs methods (MTGP) have not been applied for face verification. Actually, MTGP/GPs have been extensively studied in machine learning and computer vision in recent years . However, most of them have only considered the symmetric multi-task learning, which means that all tasks have been assumed to be of equal importance, whereas our purpose is to enhance performance on a target task given all other source tasks. Leen et al. proposed a MTGP model in the asymmetric setting to focus on improving performance on the target task, and Kim et al. developed a GP model for clustering , but their methods do not take the discriminative information of the covariance function into special account like DGPLVM. Although the discriminative information is considered in , it does not apply multi-task learning to improve its performance. Salakhutdinov et al. used a deep belief net to learn a good covariance kernel for GPs . The limitation of such deep methods is that it is hard to determine which architecture for this network is optimal. Also, multi-task learning constraint was not considered in .

Preliminary

In this section, we briefly review Gaussian Processes (GPs) for classification and clustering , and Gaussian Process Latent Variable Model (GPLVM) . We use GPs method mainly due to the following three notable advantages. Firstly, as mentioned previously, it is a non-parametric method, which means it adapts its complexity flexibly to the complex data distributions in the real-world, without any heuristics or manual tuning of parameters. Secondly, GPs method can be computed effectively because of its closed-form marginal probability computation. Furthermore, its hyper-parameters can be learned from data automatically without using model selection methods such as cross validation, thereby avoiding the high computational cost. Thirdly, the inference of GPs is based on Bayesian rules, resulting in robustness to overfitting. We recommend Rasmussen and Williams’s excellent monograph for further reading .

Moreover, the posterior distribution over latent functions is

Since neither $p(\mathbf{f}|\mathbf{X},\mathbf{y},\boldsymbol{\theta})$ nor $p(\mathbf{y}|\mathbf{f})$ can be computed analytically, the Laplace method is utilized to approximate the posterior

where $\hat{\mathbf{f}}=\arg\max_{\mathbf{f}}p(\mathbf{f}|\mathbf{X},\mathbf{y},\boldsymbol{\theta})$ and $\mathbf{W}=-\triangledown\triangledown\log p(\mathbf{f}|\mathbf{X},\mathbf{y},\boldsymbol{\theta})|_{{\mathbf{f}}=\hat{\mathbf{f}}}$. Then, we can obtain

where $|\mathbf{B}|=|\mathbf{K}|\cdot|\mathbf{K}^{-1}+\mathbf{W}|=|\mathbf{I}_{n}+\mathbf{W}^{\frac{1}{2}}\mathbf{K}\mathbf{W}^{\frac{1}{2}}|$. The optimal value of $\boldsymbol{\theta}$ can be acquired by using the gradient method to maximize Equation (4). Given any unseen test point $x_{\ast}$, the probability of its latent function $f_{\ast}$ is

2 Gaussian Processes for Clustering

The principle of GP clustering is based on the key observation that the variances of predictive values are smaller in dense areas and larger in sparse areas. The variances can be employed as a good estimate of the support of a probability density function, where each separate support domain can be considered as a cluster. This observation can be explained from the variance function of any predictive data point $x_{\ast}$

To perform clustering, the following dynamic system associated with Equation (7) can be written as

The theorem in guarantees that almost all the trajectories approach one of the stable equilibrium points detected from Equation (8). After each data point finds its corresponding stable equilibrium point, we can employ a complete graph to assign cluster labels to data points with the stable equilibrium points. Obviously, the variance function in Equation (7) completely determines the performance of clustering.

3 Gaussian Process Latent Variable Model

where $\textbf{K}_{i,j}=k(\textbf{z}_{i},\textbf{z}_{j})$. Therefore, the posterior can be written as

where $\mathcal{Z}_{a}$ is a normalization constant, the uninformative priors over $\boldsymbol{\theta}$, and the simple spherical Gaussian priors over Z are introduced . To obtain the optimal $\boldsymbol{\theta}$ and Z, we need to optimize the above likelihood (10) with respect to $\boldsymbol{\theta}$ and Z, respectively.

GaussianFace

In order to automatically learn discriminative features or covariance function, and to take advantage of source-domain data to improve the performance in face verification, we develop a principled GaussianFace model by including the multi-task learning constraint into Discriminative Gaussian Process Latent Variable Model (DGPLVM) .

The DGPLVM is an extension of GPLVM, where the discriminative prior is placed over the latent positions, rather than a simple spherical Gaussian prior. The DGPLVM uses the discriminative prior to encourage latent positions of the same class to be close and those of different classes to be far. Since face verification is a binary classification problem and the GPs mainly depend on the kernel function, it is natural to use Kernel Fisher Discriminant Analysis (KFDA) to model class structures in kernel spaces. For simplicity of inference in the followings, we introduce another equivalent formulation of KFDA to replace the one in .

KFDA is a kernelized version of linear discriminant analysis method. It finds the direction defined by a kernel in a feature space, onto which the projections of positive and negative classes are well separated by maximizing the ratio of the between-class variance to the within-class variance. Formally, let $\{\textbf{z}_{1},\ldots,\textbf{z}_{N_{+}}\}$ denote the positive class and $\{\textbf{z}_{N_{+}+1},\ldots,\textbf{z}_{N}\}$ the negative class, where the numbers of positive and negative classes are $N_{+}$ and $N_{-}=N-N_{+}$, respectively. Let $\mathbf{K}$ be the kernel matrix. Therefore, in the feature space, the two sets $\{\phi_{\mathbf{K}}(\textbf{z}_{1}),\ldots,\phi_{\mathbf{K}}(\textbf{z}_{N_{+}})\}$ and $\{\phi_{\mathbf{K}}(\textbf{z}_{N_{+}+1}),\ldots,\phi_{\mathbf{K}}(\textbf{z}_{N})\}$ represent the positive class and the negative class, respectively. The optimization criterion of KFDA is to maximize the ratio of the between-class variance to the within-class variance

where $\lambda$ is a positive regularization parameter, $\mathbf{\mu}_{\mathbf{K}}^{+}=\frac{1}{N_{+}}\sum_{i=1}^{N_{+}}\phi_{\mathbf{K}}(\textbf{z}_{i})$, $\mathbf{\mu}_{\mathbf{K}}^{-}=\frac{1}{N_{-}}\sum_{i=N_{+}+1}^{N}\phi_{\mathbf{K}}(\textbf{z}_{i})$, $\mathbf{\Sigma}_{\mathbf{K}}^{+}=\frac{1}{N_{+}}\sum_{i=1}^{N_{+}}(\phi_{\mathbf{K}}(\textbf{z}_{i})-\mathbf{\mu}_{\mathbf{K}}^{+})(\phi_{\mathbf{K}}(\textbf{z}_{i})-\mathbf{\mu}_{\mathbf{K}}^{+})^{\top}$, and $\mathbf{\Sigma}_{\mathbf{K}}^{-}=\frac{1}{N_{-}}\sum_{i=N_{+}+1}^{N}(\phi_{\mathbf{K}}(\textbf{z}_{i})-\mathbf{\mu}_{\mathbf{K}}^{-})(\phi_{\mathbf{K}}(\textbf{z}_{i})-\mathbf{\mu}_{\mathbf{K}}^{-})^{\top}$.

In this paper, however, we focus on the covariance function rather than the latent positions. To simplify calculations, we represent Equation (11) with the kernel function, and let the kernel function have the same form as the covariance function. Therefore, it is natural to introduce a more efficient equivalent form of KFDA with certain assumptions as Kim et al. points out , i.e., maximizing Equation (11) is equivalent to maximizing the following equation

Therefore, the discriminative prior over the latent positions in DGPLVM can be written as

where $\mathcal{Z}_{b}$ is a normalization constant, and $\sigma^{2}$ represents a global scaling of the prior.

The covariance matrix obtained by DGPLVM is discriminative and more flexible than the one used in conventional GPs for classification (GPC), since they are learned based on a discriminative criterion, and more degrees of freedom are estimated than conventional kernel hyper-parameters.

2 Multi-task Learning Constraint

From an asymmetric multi-task learning perspective, the tasks should be allowed to share common hyper-parameters of the covariance function. Moreover, from an information theory perspective, the information cost between target task and multiple source tasks should be minimized. A natural way to quantify the information cost is to use the mutual entropy, because it is the measure of the mutual dependence of two distributions. For multi-task learning, we extend the mutual entropy to multiple distributions as follows

where $H(\cdot)$ is the marginal entropy, $H(\cdot|\cdot)$ is the conditional entropy, $S$ is the number of source tasks, $\{p_{i}\}_{i=1}^{S}$, and $p_{t}$ are the probability distributions of source tasks and target task, respectively.

3 GaussianFace Model

In this section, we describe our GaussianFace model in detail. Suppose we have $S$ source-domain datasets $\{\mathbf{X}_{1},\ldots,\mathbf{X}_{S}\}$ and a target-domain data $\mathbf{X}_{T}$. For each source-domain data or target-domain data $\mathbf{X}_{i}$, according to Equation (9), we write its marginal likelihood

where $\textbf{Z}_{i}$ represents the domain-relevant latent space. For each source-domain data and target-domain data, their covariance functions $\mathbf{K}$ have the same form because they share the same hyper-parameters $\boldsymbol{\theta}$. In this paper, we use a widely used kernel

where $\boldsymbol{\theta}=\{\theta_{i}\}_{i=0}^{d+2}$ and $d$ is the dimension of the data point. Then, from Equations (10), learning the DGPLVM is equivalent to optimizing

where $p(\textbf{X}_{i}|\textbf{Z}_{i},\boldsymbol{\theta})$ and $p(\textbf{Z}_{i})$ are respectively represented in (15) and (13). According to the multi-task learning constraint in Equation (14), we can attain

From Equations (15), (17), and (4.3), we know that learning the GaussianFace model amounts to minimizing the following marginal likelihood

where the parameter $\beta$ balances the relative importance between the target-domain data and the multi-task learning constraint.

4 Optimization

For the model optimization, we first expand Equation (19) to obtain the following equation (ignoring the constant items)

where $P_{i}=p(\textbf{Z}_{i},\boldsymbol{\theta}|\textbf{X}_{i})$ and $P_{i,j}$ means that its corresponding covariance function is computed on both $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$. We can now optimize Equation (4.4) with respect to the hyper-parameters $\boldsymbol{\theta}$ and the latent positions $\textbf{Z}_{i}$ by the Scaled Conjugate Gradient (SCG) technique. Since we focus on the covariance matrix in this paper, here we only present the derivations of hyper-parameters. It is easy to get

The above equation depends on the form $\frac{\partial P_{i}}{\partial\theta_{j}}$ as follows (ignoring the constant items)

The above three terms can be easily obtained (ignoring the constant items) by

5 Speedup

In the GaussianFace model, we need to invert the large matrix when doing inference and prediction. For large problems, both storing the matrix and solving the associated linear systems are computationally prohibitive. In this paper, we use the anchor graphs method to speed up this process. To put it simply, we first select $q$ ($q\ll n$) anchors to cover a cloud of $n$ data points, and form an $n\times q$ matrix $\mathbf{Q}$, where $\mathbf{Q}_{i,j}=k_{\boldsymbol{\theta}}(\mathbf{z}_{i},\mathbf{z}_{j})$. $\mathbf{z}_{i}$ and $\mathbf{z}_{j}$ are from $n$ latent data points and $q$ anchors, respectively. Then the original kernel matrix $\mathbf{K}$ can be approximated as $\mathbf{K}\approx\mathbf{Q}\mathbf{Q}^{\top}$. Using the Woodbury identity , computing the $n\times n$ matrix $\mathbf{Q}\mathbf{Q}^{\top}$ can be transformed into computing the $q\times q$ matrix $\mathbf{Q}^{\top}\mathbf{Q}$, which is more efficient.

Speedup on Inference When optimizing Equation (19), we need to invert the matrix $(\lambda\mathbf{I}_{n}+\mathbf{A}\mathbf{K}\mathbf{A})$. During inference, we take $q$ k-means clustering centers as anchors to form Q. Substituting $\mathbf{K}\approx\mathbf{Q}\mathbf{Q}^{\top}$ into $(\lambda\mathbf{I}_{n}+\mathbf{A}\mathbf{K}\mathbf{A})$, and then using the Woodbury identity, we get

Similarly, let $\mathbf{K}^{-1}\approx(\mathbf{K}+\tau\mathbf{I})^{-1}$ where $\tau$ a constant term, then we can get

Speedup on Prediction When we compute the predictive variance $\sigma(\mathbf{z}_{\ast})$, we need to invert the matrix $(\mathbf{K}+\mathbf{W}^{-1})$. At this time, we can use the method in Section 3.2 to calculate the accurate clustering centers that can be regarded as the anchors. Using the Woodbury identity again, we obtain

where $(\mathbf{I}_{q}+\mathbf{Q}^{\top}\mathbf{W}\mathbf{Q})$ is only a $q\times q$ matrix, and its inverse matrix can be computed more efficiently.

GaussianFace Model for Face Verification

In this section, we describe two applications of the GaussianFace model to face verification: as a binary classifier and as a feature extractor.

Each face image is first normalized to $150\times 120$ size by an affine transformation based on five landmarks (two eyes, nose, and two mouth corners). The image is then divided into overlapped patches of $25\times 25$ pixels with a stride of $2$ pixels. Each patch within the image is mapped to a vector by a certain descriptor, and the vector is regarded as the feature of the patch, denoted by $\{\mathbf{x}_{p}^{A}\}_{p=1}^{P}$ where $P$ is the number of patches within the face image $A$. In this paper, the multi-scale LBP feature of each patch is extracted . The difference is that the multi-scale LBP descriptors are extracted at the center of each patch instead of accurate landmarks.

2 GaussianFace Model as a Feature Extractor

As a feature extractor, our model can be regarded as an approach to automatically extract facial features, shown in Figure 1 (b). Here, for a pair of face images $A$ and $B$ from the same (or different) person, we regard the joint feature vector $\mathbf{x}_{i}=[(\mathbf{x}_{i}^{A})^{\top},(\mathbf{x}_{i}^{B})^{\top}]^{\top}$ as the input data point of the GaussianFace model, and its corresponding output is $y_{i}=1$ (or $-1$). To enhance the robustness of our approach, the flipped form of $\mathbf{x}_{i}$ is also included; for example, $\mathbf{x}_{i}=[(\mathbf{x}_{i}^{B})^{\top},(\mathbf{x}_{i}^{A})^{\top}]^{\top}$. After the hyper-parameters of covariance function are learnt from the training data, we first estimate the latent representations of the training data using the same method in , then can use the method in Section 3.2 to group the latent data points into different clusters automatically. Suppose that we finally obtain $C$ clusters. The centers of these clusters are denoted by $\{\mathbf{c}_{i}\}_{i=1}^{C}$, the variances of these clusters by $\{{\Sigma}_{i}^{2}\}_{i=1}^{C}$, and their weights by $\{w_{i}\}_{i=1}^{C}$ where $w_{i}$ is the ratio of the number of latent data points from the $i$-th cluster to the number of all latent data points. Then we refer to each $\mathbf{c}_{i}$ as the input of Equation (5), and we can obtain its corresponding probability $p_{i}$ and variance ${\sigma}_{i}^{2}$. In fact, $\{\mathbf{c}_{i}\}_{i=1}^{C}$ can be regarded as a codebook generated by our model.

For any un-seen pair of face images, we also first compute its joint feature vector $\mathbf{x}_{\ast}$ for each pair of patches, and estimate its latent representation $\mathbf{z}_{\ast}$. Then we compute its first-order and second-order statistics to the centers. Similarly, we regard $\mathbf{z}_{\ast}$ as the input of Equation (5), and can also obtain its corresponding probability $p_{\ast}$ and variance $\sigma_{\ast}^{2}$. The statistics and variance of $\mathbf{z}_{\ast}$ are represented as its high-dimensional facial features, denoted by $\hat{\mathbf{z}}_{\ast}=[{\Delta}_{1}^{1},{\Delta}_{1}^{2},{\Delta}_{1}^{3},{\Delta}_{1}^{4},\ldots,{\Delta}_{C}^{1},{\Delta}_{C}^{2},{\Delta}_{C}^{3},{\Delta}_{C}^{4}]^{\top}$, where {\Delta}_{i}^{1}=w_{i}\Big{(}\frac{\mathbf{z}_{\ast}-\mathbf{c}_{i}}{\Sigma_{i}}\Big{)}, {\Delta}_{i}^{2}=w_{i}\Big{(}\frac{\mathbf{z}_{\ast}-\mathbf{c}_{i}}{\Sigma_{i}}\Big{)}^{2}, ${\Delta}_{i}^{3}=\log\frac{p_{\ast}(1-p_{i})}{p_{i}(1-p_{\ast})}$, and ${\Delta}_{i}^{4}=\frac{\sigma_{\ast}^{2}}{{\sigma}_{i}^{2}}$. We then concatenate all of the new high-dimensional features from each pair of patches to form the final new high-dimensional feature for the pair of face images. The new high-dimensional facial features not only describe how the distribution of features of an un-seen face image differs from the distribution fitted to the features of all training images, but also encode the predictive information including the probabilities of label and uncertainty. We call this approach GaussianFace-FE.

Experimental Settings

In this section, we conduct experiments on face verification. We start by introducing the source-domain datasets and the target-domain dataset in all of our experiments (see Figure 2 for examples). The source-domain datasets include four different types of datasets as follows: Multi-PIE . This dataset contains face images from 337 subjects under 15 view points and 19 illumination conditions in four recording sessions. These images are collected under controlled conditions. MORPH . The MORPH database contains 55,000 images of more than 13,000 people within the age ranges of 16 to 77. There are an average of 4 images per individual. Web ImagesThese two datasets are collected by our own from the Web. It is guaranteed that these two datasets are mutually exclusive with the LFW dataset.. This dataset contains around 40,000 facial images from 3261 subjects; that is, approximately 10 images for each person. The images were collected from the Web with significant variations in pose, expression, and illumination conditions. \textbf{Life Photos}^{{\color[rgb]{0,0,0}\text{2}}}. This dataset contains approximately 5000 images of 400 subjects collected online. Each subject has roughly 10 images.

If not otherwise specified, the target-domain dataset is the benchmark of face verification as follows: LFW . This dataset contains 13,233 uncontrolled face images of 5749 public figures with variety of pose, lighting, expression, race, ethnicity, age, gender, clothing, hairstyles, and other parameters. All of these images are collected from the Web.

We use the LFW dataset as the target-domain dataset because it is well known as a challenging benchmark. Using it also allows us to compare directly with other existing face verification methods . Besides, this dataset provides a large set of relatively unconstrained face images with complex variations as described above, and has proven difficult for automatic face verification methods . In all the experiments conducted on LFW, we strictly follow the standard unrestricted protocol of LFW . More precisely, during the training procedure, the four source-domain datasets are: Web Images, Multi-PIE, MORPH, and Life Photos, the target-domain dataset is the training set in View 1 of LFW, and the validation set is the test set in View 1 of LFW. At the test time, we follow the standard 10-fold cross-validation protocol to test our model in View 2 of LFW.

For each one of the four source-domain datasets, we randomly sample 20,000 pairs of matched images and 20,000 pairs of mismatched images. The training partition and the testing partition in all of our experiments are mutually exclusive. In other words, there is no identity overlap among the two partitions.

For the experiments below, “The Number of SD” means “the Number of Source-Domain datasets that are fed into the GaussianFace model for training”. By parity of reasoning, if “The Number of SD” is $i$, that means the first $i$ source-domain datasets are used for model training. Therefore, if “The Number of SD” is 0, models are trained with the training data from target-domain data only.

Implementation details. Our model involves four important parameters: $\lambda$ in (12), $\sigma$ in (13), $\beta$ in (19), and the number of anchors $q$ in Speedup on Inference The other parameters, such as the hyper-parameters in the kernel function and the number of anchors in Speedup on Prediction, can be automatically learned from the data.. Following the same setting in , the regularization parameter $\lambda$ in (12) is fixed to $10^{-8}$. $\sigma$ reflects the tradeoff between our method’s ability to discriminate (small $\sigma$) and its ability to generalize (large $\sigma$), and $\beta$ balances the relative importance between the target-domain data and the multi-task learning constraint. Therefore, the validation set (the test set in View 1 of LFW) is used for selecting $\sigma$ and $\beta$. Each time we use different number of source-domain datasets for training, the corresponding optimal $\sigma$ and $\beta$ should be selected on the validation set.

Since we collected a large number of image pairs for training (20,000 matched pairs and 20,000 mismatched pairs from each source-domain dataset), and our model is based on the kernel method, thus an important consideration is how to efficiently approximate the kernel matrix using a low-rank method in the limited space and time. We adopt the anchor graphs method (see Section 4.5) for kernel approximation. In our experiments, we take two steps to determine the number of anchor points. In the first step, the optimal $\sigma$ and $\beta$ are selected on the validation set in each experiment. In the second step, we fix $\sigma$ and $\beta$, and then tune the number of anchor points. We vary the number of anchor points to train our model on the training set, and test it on the validation set. We report the average accuracy for our model over 10 trials. After we consider the trade-off between memory and running time in practice, the number of anchor points with the best average accuracy is determined in each experiments.

Experimental Results

In this section, we conduct five experiments to demonstrate the validity of the GaussianFace model.

Since our model is based on GPs, it is natural to compare our model with four popular GP models: GPC , MTGP prediction , GPLVM , and DGPLVM . For fair comparisons, all these models are trained on multiple source-domain datasets using the same two methods as our GaussianFace model described in Section 5. After the hyper-parameters of covariance function are learnt for each model, we can regard each model as a binary classifier and a feature extractor like ours, respectively. Figure 3 shows that our model significantly outperforms the other four GPs models, and the superiority of our model becomes more obvious as the number of source-domain datasets increases.

2 Comparisons with Other Binary Classifiers

Since our model can be regarded as a binary classifier, we have also compared our method with other classical binary classifiers. For this paper, we chose three popular representatives: SVM , logistic regression (LR) , and Adaboost . Table 1 demonstrates that the performance of our method GaussianFace-BC is much better than those of the other classifiers. Furthermore, these experimental results demonstrates the effectiveness of the multi-task learning constraint. For example, our GaussianFace-BC has about 7.5% improvement when all four source-domain datasets are used for training, while the best one of the other three binary classifiers has only around 4% improvement.

3 Comparisons with Other Feature Extractors

Our model can also be regarded as a feature extractor, which is implemented by clustering to generate a codebook. Therefore, we evaluate our method by comparing it with three popular clustering methods: K-means , Random Projection (RP) tree , and Gaussian Mixture Model (GMM) . Since our method can determine the number of clusters automatically, for fair comparison, all the other methods generate the same number of clusters as ours. As shown in Table 2, our method GaussianFace-FE significantly outperforms all of the compared approaches, which verifies the effectiveness of our method as a feature extractor. The results have also proved that the multi-task learning constraint is effective. Each time one different type of source-domain dataset is added for training, the performance can be improved significantly. Our GaussianFace-FE model achieves over 8% improvement when the number of SD varies from 0 to 4, which is much higher than the $\sim$3% improvement of the other methods.

4 Comparison with the state-of-art Methods

Motivated by the appealing performance of both GaussianFace-BC and GaussianFace-FE, we further combine them for face verification. Specifically, after facial features are extracted using GaussianFace-FE, GaussianFace-BC Here, the GaussianFace BC is trained with the extracted high-dimensional features using GaussianFace-FE. is used to make the final decision. Figure 4 shows the results of this combination compared with state-of-the-art methods . The best published result on the LFW benchmark is 97.35%, which is achieved by . Our GaussianFace model can improve the accuracy to 98.52%, which for the first time beats the human-level performance (97.53%, cropped) . Figure 5 presents some example pairs that were always incorrectly classified by our model. Obviously, even for humans, it is also difficult to verify some of them. Here, we emphasize that the centers of patches, instead of the accurate and dense facial landmarks like , are utilized to extract multi-scale features in our method. This makes our method simpler and easier to use.

5 Further Validations: Shuffling the Source-Target

To further prove the validity of our model, we also consider to treat Multi-PIE and MORPH respectively as the target-domain dataset and the others as the source-domain datasets. The target-domain dataset is split into two mutually exclusive parts: one consisting of 20,000 matched pairs and 20,000 mismatched pairs is used for training, the other is used for test. In the test set, similar to the protocol of LFW, we select 10 mutually exclusive subsets, where each subset consists of 300 matched pairs and 300 mismatched pairs. The experimental results are presented in Figure 6. Each time one dataset is added to the training set, the performance can be improved, even though the types of data are very different in the training set.

General Discussion

There is an implicit belief among many psychologists and computer scientists that human face verification abilities are currently beyond existing computer-based face verification algorithms . This belief, however, is supported more by anecdotal impression than by scientific evidence. By contrast, there have already been a number of papers comparing human and computer-based face verification performance . It has been shown that the best current face verification algorithms perform better than humans in the good and moderate conditions. So, it is really not that difficult to beat human performance in some specific scenarios.

As pointed out by , humans and computer-based algorithms have different strategies in face verification. Indeed, by contrast to performance with unfamiliar faces, human face verification abilities for familiar faces are relatively robust to changes in viewing parameters such as illumination and pose. For example, Bruce found human recognition memory for unfamiliar faces dropped substantially when there were changes in viewing parameters. Besides, humans can take advantages of non-face configurable information from the combination of the face and body (e.g., neck, shoulders). It has also been examined in , where the human performance drops from 99.20% (tested using the original LFW images) to 97.53% (tested using the cropped LFW images). Hence, the experiments comparing human and computer performance may not show human face verification skill at their best, because humans were asked to match the cropped faces of people previously unfamiliar to them. To the contrary, those experiments can fully show the performance of computer-based face verification algorithms. First, the algorithms can exploit information from enough training images with variations in all viewing parameters to improve face verification performance, which is similar to information humans acquire in developing face verification skills and in becoming familiar with individuals. Second, the algorithms might exploit useful, but subtle, image-based detailed information that give them a slight, but consistent, advantage over humans.

Therefore, surpassing the human-level performance may only be symbolically significant. In reality, a lot of challenges still lay ahead. To compete successfully with humans, more factors such as the robustness to familiar faces and the usage of non-face information, need to be considered in developing future face verification algorithms.

Conclusion and Future Work

This paper presents a principled Multi-Task Learning approach based on Discriminative Gaussian Process Latent Variable Model, named GaussianFace, for face verification by including a computationally more efficient equivalent form of KFDA and the multi-task learning constraint to the DGPLVM model. We use Gaussian Processes approximation and anchor graphs to speed up the inference and prediction of our model. Based on the GaussianFace model, we propose two different approaches for face verification. Extensive experiments on challenging datasets validate the efficacy of our model. The GaussianFace model finally surpassed human-level face verification accuracy, thanks to exploiting additional data from multiple source-domains to improve the generalization performance of face verification in the target-domain and adapting automatically to complex face variations.

Although several techniques such as the Laplace approximation and anchor graph are introduced to speed up the process of inference and prediction in our GaussianFace model, it still takes a long time to train our model for the high performance. In addition, large memory is also necessary. Therefore, for specific application, one needs to balance the three dimensions: memory, running time, and performance. Generally speaking, higher performance requires more memory and more running time. In the future, the issue of running time can be further addressed by the distributed parallel algorithm or the GPU implementation of large matrix inversion. To address the issue of memory, some online algorithms for training need to be developed. Another more intuitive method is to seek a more efficient sparse representation for the large covariance matrix.

Acknowledgements

We would like to thank Deli Zhao and Chen Change Loy for their insightful discussions. This work is partially supported by ”CUHK Computer Vision Cooperation” grant from Huawei, and by the General Research Fund sponsored by the Research Grants Council of Hong Kong (Project No.CUHK 416510 and 416312) and Guangdong Innovative Research Team Program (No.201001D0104648280).

References