DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

Introduction

Human body pose estimation methods have become increasingly reliable. Powerful body part detectors in combination with tree-structured body models show impressive results on diverse datasets . These benchmarks promote pose estimation of single pre-localized persons but exclude scenes with multiple people. This problem definition has been a driver for progress, but also falls short on representing a realistic sample of real-world images. Many photographs contain multiple people of interest (see Fig 1) and it is unclear whether single pose approaches generalize directly. We argue that the multi person case deserves more attention since it is an important real-world task.

Key challenges inherent to multi person pose estimation are the partial visibility of some people, significant overlap of bounding box regions of people, and the a-priori unknown number of people in an image. The problem thus is to infer the number of persons, assign part detections to person instances while respecting geometric and appearance constraints. Most strategies use a two-stage inference process to first detect and then independently estimate poses. This is unsuited for cases when people are in close proximity since they permit simultaneous assignment of the same body-part candidates to multiple people hypotheses.

As a principled solution for multi person pose estimation a model is proposed that jointly estimates poses of all people present in an image by minimizing a joint objective. The formulation is based on partitioning and labeling an initial pool of body part candidates into subsets that correspond to sets of mutually consistent body-part candidates and abide to mutual consistency and exclusion constraints. The proposed method has a number of appealing properties. (1) The formulation is able to deal with an unknown number of people, and also infers this number by linking part hypotheses. (2) The formulation allows to either deactivate or merge part hypotheses in the initial set of part candidates hence effectively performing non-maximum suppression (NMS). In contrast to NMS performed on individual part candidates, the model incorporates evidence from all other parts making the process more reliable. (3) The problem is cast in the form of an Integer Linear Program (ILP). Although the problem is NP-hard, the ILP formulation facilitates the computation of bounds and feasible solutions with a certified optimality gap.

This paper makes the following contributions. The main contribution is the derivation of a joint detection and pose estimation formulation cast as an integer linear program. Further, two CNN variants are proposed to generate representative sets of body part candidates. These, combined with the model, obtain state-of-the-art results for both single-person and multi-person pose estimation on different datasets.

Related work. Most work on pose estimation targets the single person case. Methods progressed from simple part detectors and elaborate body models to tree-structured pictorial structures (PS) models with strong part detectors . Impressive results are obtained predicting locations of parts with convolutional neural networks (CNN) . While body models are not a necessary component for effective part localization, constraints among parts allow to assemble independent detections into body configurations as demonstrated in by combining CNN-based body part detectors with a body model .

A popular approach to multi-person pose estimation is to detect people first and then estimate body pose independently . proposes a flexible mixture-of-parts model for detection and pose estimation. obtains multiple pose hypotheses corresponding to different root part positions and then performing non-maximum suppression. detects people using a flexible configuration of poselets and the body pose is predicted as a weighted average of activated poselets. detects people and then predicts poses of each person using a PS model. estimates poses of multiple people in 3D by constructing a shared space of 3D body part hypotheses, but uses 2D person detections to establish the number of people in the scene. These approaches are limited to cases with people sufficiently far from each other that do not have overlapping body parts.

Our work is closely related to who also propose a joint objective to estimate poses of multiple people. proposes a multi-person PS model that explicitly models depth ordering and person-person occlusions. Our formulation is not limited by a number of occlusion states among people. proposes a joint model for pose estimation and body segmentation coupling pose estimates of individuals by image segmentation. uses a person detector to generate initial hypotheses for the joint model. resorts to a greedy approach of adding one person hypothesis at a time until the joint objective can be reduced, whereas our formulation can be solved with a certified optimality gap. In addition relies on expensive labeling of body part segmentation, which the proposed approach does not require.

Similarly to we aim to distinguish between visible and occluded body parts. primarily focuse on the single-person case and handles multi-person scenes akin to . We consider the more difficult problem of full-body pose estimation, whereas focus on upper-body poses and consider a simplified case of people seen from the front.

Our work is related to early work on pose estimation that also relies on integer linear programming to assemble candidate body part hypotheses into valid configurations . Their single person method employs a tree graph augmented with weaker non-tree repulsive edges and expects the same number of parts. In contrast, our novel formulation relies on fully connected model to deal with unknown number of people per image and body parts per person.

The Minimum Cost Multicut Problem , known in machine learning as correlation clustering , has been used in computer vision for image segmentation but has not been used before in the context of pose estimation. It is known to be NP-hard .

Problem Formulation

In this section, the problem of estimating articulated poses of an unknown number of people in an image is cast as an optimization problem. The goal of this formulation is to state three problems jointly: 1. The selection of a subset of body parts from a set $D$ of body part candidates, estimated from an image as described in Section 4 and depicted as nodes of a graph in Fig. 1(a). 2. The labeling of each selected body part with one of $C$ body part classes, e.g., “arm”, “leg”, “torso”, as depicted in Fig. 1(c). 3. The partitioning of body parts that belong to the same person, as depicted in Fig. 1(b).

We encode labelings of the three problems jointly through triples $(x,y,z)$ of binary random variables with domains $x\in\{0,1\}^{D\times C},y\in\{0,1\}^{\tbinom{D}{2}}$ and $z\in\{0,1\}^{\tbinom{D}{2}\times C^{2}}$. Here, $x_{dc}=1$ indicates that body part candidate $d$ is of class $c$, $y_{dd^{\prime}}=1$ indicates that the body part candidates $d$ and $d^{\prime}$ belong to the same person, and $z_{dd^{\prime}cc^{\prime}}$ are auxiliary variables to relate $x$ and $y$ through $z_{dd^{\prime}cc^{\prime}}=x_{dc}x_{d^{\prime}c^{\prime}}y_{dd^{\prime}}$. Thus, $z_{dd^{\prime}cc^{\prime}}=1$ indicates that body part candidate $d$ is of class $c$ ($x_{dc}=1$), body part candidate $d^{\prime}$ is of class $c^{\prime}$ ($x_{d^{\prime}c^{\prime}}=1$), and body part candidates $d$ and $d^{\prime}$ belong to the same person ($y_{dd^{\prime}}=1$).

In order to constrain the 01-labelings $(x,y,z)$ to well-defined articulated poses of one or more people, we impose the linear inequalities (1)–(3) stated below. Here, the inequalities (1) guarantee that every body part is labeled with at most one body part class. (If it is labeled with no body part class, it is suppressed). The inequalities (2) guarantee that distinct body parts $d$ and $d^{\prime}$ belong to the same person only if neither $d$ nor $d^{\prime}$ is suppressed. The inequalities (3) guarantee, for any three pairwise distinct body parts, $d$, $d^{\prime}$ and $d^{\prime\prime}$, if $d$ and $d^{\prime}$ are the same person (as indicated by $y_{dd^{\prime}}=1$) and $d^{\prime}$ and $d^{\prime\prime}$ are the same person (as indicated by $y_{d^{\prime}d^{\prime\prime}}=1$), then also $d$ and $d^{\prime\prime}$ are the same person ($y_{dd^{\prime\prime}}=1$), that is, transitivity, cf. . Finally, the inequalities (4) guarantee, for any $dd^{\prime}\in\tbinom{D}{2}$ and any $cc^{\prime}\in C^{2}$ that $z_{dd^{\prime}cc^{\prime}}=x_{dc}x_{d^{\prime}c^{\prime}}y_{dd^{\prime}}$. These constraints allow us to write an objective function as a linear form in $z$ that would otherwise be written as a cubic form in $x$ and $y$. We denote by $X_{DC}$ the set of all $(x,y,z)$ that satisfy all inequalities, i.e., the set of feasible solutions.

When at most one person is in an image, we further constrain the feasible solutions to a well-defined pose of a single person. This is achieved by an additional class of inequalities which guarantee, for any two distinct body parts that are not suppressed, that they must be clustered together:

2 Objective Function

For every pair $(d,c)\in D\times C$, we will estimate a probability $p_{dc}\in$ of the body part $d$ being of class $c$. In the context of CRFs, these probabilities are called part unaries and we will detail their estimation in Section 4.

For every $dd^{\prime}\in\tbinom{D}{2}$ and every $cc^{\prime}\in C^{2}$, we consider a probability $p_{dd^{\prime}cc^{\prime}}\in(0,1)$ of the conditional probability of $d$ and $d^{\prime}$ belonging to the same person, given that $d$ and $d^{\prime}$ are body parts of classes $c$ and $c^{\prime}$, respectively. For $c\neq c^{\prime}$, these probabilities $p_{dd^{\prime}cc^{\prime}}$ are the pairwise terms in a graphical model of the human body. In contrast to the classic pictorial structures model, our model allows for a fully connected graph where each body part is connected to all other parts in the entire set $D$ by a pairwise term. For $c=c^{\prime}$, $p_{dd^{\prime}cc^{\prime}}$ is the probability of the part candidates $d$ and $d^{\prime}$ representing the same part of the same person. This facilitates clustering of multiple part candidates of the same part of the same person and a repulsive property that prevents nearby part candidates of the same type to be associated to different people.

The optimization problem that we call the subset partition and labeling problem is the ILP that minimizes over the set of feasible solutions $X_{DC}$:

The objective (6)–(10) is the MAP estimate of a probability measure of joint detections $x$ and clusterings $y,z$ of body parts, where prior probabilities $p_{dc}$ and $p_{dd^{\prime}cc^{\prime}}$ are estimated independently from data, and the likelihood is a positive constant if $(x,y,z)$ satisfies (1)–(4), and is 0, otherwise. The exact form (6)–(10) is obtained when minimizing the negative logarithm of this probability measure.

3 Optimization

In order to obtain feasible solutions of the ILP (6) with guaranteed bounds, we separate the inequalities (1)–(5) in the branch-and-cut loop of the state-of-the-art ILP solver Gurobi. More precisely, we solve a sequence of relaxations of the problem (6), starting with the (trivial) unconstrained problem. Each problem is solved using the cuts proposed by Gurobi. Once an integer feasible solution is found, we identify violated inequalities (1)–(5), if any, by breadth-first-search, add these to the constraint pool and re-solve the tightened relaxation. Once an integer solution satisfying all inequalities is found, together with a lower bound that certifies an optimality gap below 1%, we terminate.

Pairwise Probabilities

Here we describe the estimation of the pairwise terms. We define pairwise features $f_{dd^{\prime}}$ for the variable $z_{dd^{\prime}cc^{\prime}}$ (Sec. 2). Each part detection $d$ includes the probabilities $f_{p_{dc}}$ (Sec. 4.4), its location $(x_{d},y_{d})$, scale $h_{d}$ and bounding box $B_{d}$ coordinates. Given two detections $d$ and $d^{\prime}$, and the corresponding features $(f_{p_{dc}},x_{d},y_{d},h_{d},B_{d})$ and $(f_{p_{d^{\prime}c}},x_{d^{\prime}},y_{d^{\prime}},h_{d^{\prime}},B_{d^{\prime}})$, we define two sets of auxiliary variables for $z_{dd^{\prime}cc^{\prime}}$, one set for $c=c^{\prime}$ (same body part class clustering) and one for $c\neq c^{\prime}$ (across two body part classes labeling). These features capture the proximity, kinematic relation and appearance similarity between body parts.

The same body part class ($c=c^{\prime}$). Two detections denoting the same body part of the same person should be in close proximity to each other. We introduce the following auxiliary variables that capture the spatial relations: $\Delta x=|x_{d}-x_{d^{\prime}}|/\bar{h}$, $\Delta y=|y_{d}-y_{d^{\prime}}|/\bar{h}$, $\Delta h=|h_{d}-h_{d^{\prime}}|/\bar{h}$, $IOUnion$, $IOMin$, $IOMax$. The latter three are intersections over union/minimum/maximum of the two detection boxes, respectively, and $\bar{h}=(h_{d}+h_{d^{\prime}})/2$.

Non-linear Mapping. We augment the feature representation by appending quadratic and exponential terms. The final pairwise feature $f_{dd^{\prime}}$ for the variable $z_{dd^{\prime}cc}$ is $(\Delta x,\Delta y,\Delta h,IOUnion,IOMin,IOMax,{(\Delta x)}^{2},\\ \ldots,{(IOMax)}^{2},\exp{(-{\Delta x})},\ldots,\exp{(-{IOMax})})$.

Two different body part classes ($c\neq c^{\prime}$). We encode the kinematic body constraints into the pairwise feature by introducing auxiliary variables $S_{dd^{\prime}}$ and $R_{dd^{\prime}}$, where $S_{dd^{\prime}}$ and $R_{dd^{\prime}}$ are the Euclidean distance and the angle between two detections, respectively. To capture the joint distribution of $S_{dd^{\prime}}$ and $R_{dd^{\prime}}$, instead of using $S_{dd^{\prime}}$ and $R_{dd^{\prime}}$ directly, we employ the posterior probability $p(z_{dd^{\prime}cc^{\prime}}=1|S_{dd^{\prime}},R_{dd^{\prime}})$ as pairwise feature for $z_{dd^{\prime}cc^{\prime}}$ to encode the geometric relations between the body part class $c$ and $c^{\prime}$. More specifically, assuming the prior probability $p(z_{dd^{\prime}cc^{\prime}}=1)=p(z_{dd^{\prime}cc^{\prime}}=0)=0.5$, the posterior probability of detection $d$ and $d^{\prime}$ have the body part label $c$ and $c^{\prime}$, namely $z_{dd^{\prime}cc^{\prime}}=1$, is

where $p(S_{dd^{\prime}},R_{dd^{\prime}}|z_{dd^{\prime}cc^{\prime}}=1)$ is obtained by conducting a normalized 2D histogram of $S_{dd^{\prime}}$ and $R_{dd^{\prime}}$ from positive training examples, analogous to the negative likelihood $p(S_{dd^{\prime}},R_{dd^{\prime}}|z_{dd^{\prime}cc^{\prime}}=0)$. In Sec. 5.1 we also experiment with encoding the appearance into the pairwise feature by concatenating the feature $f_{p_{dc}}$ from $d$ and $f_{p_{d^{\prime}c}}$ from $d^{\prime}$, as $f_{p_{dc}}$ is the output of the CNN-based part detectors. The final pairwise feature is $(p(z_{dd^{\prime}cc^{\prime}}=1|S_{dd^{\prime}},R_{dd^{\prime}}),f_{p_{dc}},f_{p_{d^{\prime}c}})$.

The coefficients $\alpha$ and $\beta$ of the objective function (Eq. 6) are defined by the probability ratio in the log space (Eq. 7 and Eq. 8). Here we describe the estimation of the corresponding probability density: (1) For every pair of detection and part classes, namely for any $(d,c)\in D\times C$, we estimate a probability $p_{dc}\in(0,1)$ of the detection $d$ being a body part of class $c$. (2) For every combination of two distinct detections and two body part classes, namely for any $dd^{\prime}\in\tbinom{D}{2}$ and any $cc^{\prime}\in C^{2}$, we estimate a probability $p_{dd^{\prime}cc^{\prime}}\in(0,1)$ of $d$ and $d^{\prime}$ belonging to the same person, meanwhile $d$ and $d^{\prime}$ are body parts of classes $c$ and $c^{\prime}$, respectively.

Learning. Given the features $f_{dd^{\prime}}$ and a Gaussian prior $p(\theta_{cc^{\prime}})=\mathcal{N}(0,\sigma^{2})$ on the parameters, logistic model is

$(|C|\times(|C|+1))/2$ parameters are estimated using ML.

Inference Given two detections $d$ and $d^{\prime}$, the coefficients $\alpha_{dc}$ for $x_{dc}$ and $\alpha_{d^{\prime}c}$ for $x_{d^{\prime}c}$ are obtained by Eq. 7, the coefficient $\beta_{dd^{\prime}cc^{\prime}}$ for $z_{dd^{\prime}cc^{\prime}}$ has the form

Model parameters $\theta_{cc^{\prime}}$ are learned using logistic regression.

Body Part Detectors

We first introduce our deep learning-based part detection models and then evaluate them on two prominent benchmarks thereby significantly outperforming state of the art.

To obtain strong part detectors we adapt Fast R-CNN . FR-CNN takes as input an image and set of class-independent region proposals and outputs the softmax probabilities over all classes and refined bounding boxes. To adapt FR-CNN for part detection we alter it in two ways: 1) proposal generation and 2) detection region size. The adapted version is called AFR-CNN throughout the paper.

Detection proposals. Generating object proposals is essential for FR-CNN, meanwhile detecting body parts is challenging due to their small size and high intra-class variability. We use DPM-based part detectors for proposal generation. We collect $K$ top-scoring detections by each part detector in a common pool of $N$ part-independent proposals and use these proposals as input to AFR-CNN. $N$ is $2,000$ in case of single and $20,000$ in case of multiple people.

Larger context. Increasing the size of DPM detections by upscaling every bounding box by a fixed factor allows to capture more context around each part. In Sec. 4.3 we evaluate the influence of upscaling and show that using larger context around parts is crucial for best performance.

Details. Following standard FR-CNN training procedure ImageNet models are finetuned on pose estimation task. Center of a predicted bounding box is used for body part location prediction. See Appendix A for detailed parameter analysis.

2 Dense Architecture (Dense-CNN)

Using proposals for body part detection may be sub-optimal. We thus develop a fully convolutional architecture for computing part probability scoremaps.

Stride. We build on VGG . Fully convolutional VGG has stride of 32 px – too coarse for precise part localization. We thus use hole algorithm to reduce the stride to 8 px.

Scale. Selecting image scale is crucial. We found that scaling to a standing height of $340$ px performs best: VGG receptive field sees entire body to disambiguate body parts.

Loss function. We start with a softmax loss that outputs probabilities for each body part and background. The downside is inability to assign probabilities above $0.5$ to several close-by body parts. We thus re-formulate the detection as multi-label classification, where at each location a separate set of probability distributions is estimated for each part. We use sigmoid activation function on the output neurons and cross entropy loss. We found this loss to perform better than softmax and converge much faster compared to MSE . Target training scoremap for each joint is constructed by assigning a positive label 1 at each location within $15$ px to the ground truth, and negative label 0 otherwise.

Location refinement. In order to improve location precision we follow : we add a location refinement FC layer after the FC7 and use the relative offsets $(\Delta x,\Delta y)$ from a scoremap location to the ground truth as targets.

Regression to other parts. Similar to location refinement we add an extra term to the objective function where for each part we regress onto all other part locations. We found this auxiliary task to improve the performance (c.f. Sec. 4.3).

Training. We follow best practices and use SGD for CNN training. In each iteration we forward-pass a single image. After FC6 we select all positive and random negative samples to keep the pos/neg ratio as 25%/75%. We finetune VGG from Imagenet model to pose estimation task and use training data augmentation. We train for 430k iterations with the following learning rates (lr): 10k at lr=0.001, 180k at lr=0.002, 120k at lr=0.0002 and 120k at lr=0.0001. Pre-training at smaller lr prevents the gradients from diverging.

& 94.6 86.8 79.9 75.4 83.5 82.8 77.9 83.0 54.7

+ sigmoid 93.5 87.2 81.0 77.0 85.5 83.3 79.3 83.8 55.6

+ location refinement 95.0 88.4 81.5 76.4 88.0 83.3 80.8 84.8 61.5

+ auxiliary task 95.1 89.6 82.8 78.9 89.0 85.9 81.2 86.1 61.6

+ finetune LSP 97.2 90.8 83.0 79.3 90.6 85.6 83.1 87.1 63.6

4 Using Detections in DeepCut Models

The SPLP problem is NP-hard, to solve instances of it efficiently we select a subset of representative detections from the entire set produced by a model. In our experiments we use $|D|=100$ as default detection set size. In case of the AFR-CNN we directly use the softmax output as unary probabilities: $f_{p_{dc}}=(p_{d1},\ldots,p_{dc})$, where $p_{dc}$ is the probability of the detection $d$ being the part class $c$. For Dense-CNN detection model we use the sigmoid detection unary scores.

The aim of this paper is to tackle the multi person case. To that end, we evaluate the proposed DeepCut models on four diverse benchmarks. We confirm that both single person (SP) and multi person (MP) variants (Sec. 2) are effective on standard SP pose estimation datasets . Then, we demonstrate superior performance of DeepCut MP on the multi person pose estimation task.

We now evaluate single person (SP) and more general multi person (MP) DeepCut models on LSP and MPII SP benchmarks described in Sec. 4. Since this evaluation setting implicitly relies on the knowledge that all parts are present in the image we always output the full number of parts.

& 95.4 86.7 78.3 74.0 84.3 82.9 79.2 83.0 58.4

+ appearance pairwise 95.4 87.2 78.6 73.7 84.7 82.8 78.8 83.0 58.5 + DeepCut MP 95.2 86.7 78.2 73.5 84.6 82.8 79.0 82.9 58.0

Dense-CNN (unary) 97.2 90.8 83.0 79.3 90.6 85.6 83.1 87.1 63.6 + DeepCut SP 97.0 91.0 83.8 78.1 91.0 86.7 82.0 87.1 63.5 + DeepCut MP 96.2 91.2 83.3 77.6 91.3 87.0 80.4 86.7 62.6

Tompson et al. 90.6 79.2 67.9 63.4 69.5 71.0 64.2 72.3 47.3 Chen&Yuille 91.8 78.2 71.8 65.5 73.3 70.2 63.4 73.4 40.1 Fan et al. ∗ 92.4 75.2 65.3 64.0 75.7 68.3 70.4 73.0 43.2

∗ re-evaluated using the standard protocol, for details see project page of

Results on LSP. We report per-part PCK results (Tab. 5.1) and results for a variable distance threshold (Fig. 2 (a)). DeepCut SP AFR-CNN model using $100$ detections improves over unary only ($83.0$ vs. $82.8$% PCK, $58.4$ vs. $57$% AUC), as pairwise connections filter out some of the high-scoring detections on the background. The improvement is clear in Fig. 2 (a) for smaller thresholds. Using part appearance scores in addition to geometrical features in $c\neq c^{\prime}$ pairwise terms only slightly improves AUC, as the appearance of neighboring parts is mostly captured by a relatively large region centered at each part. The performance of DeepCut MP AFR-CNN matches the SP and improves over AFR-CNN alone: DeepCut MP correctly handles the SP case. Performance of DeepCut SP Dense-CNN is almost identical to unary only, unlike the results for AFR-CNN. Dense-CNN performance is noticeably higher compared to AFR-CNN, and “easy” cases that could have been corrected by a spatial model are resolved by stronger part detectors alone.

Comparison to the state of the art (LSP). Tab. 5.1 compares results of DeepCut models to other deep learning methods specifically designed for single person pose estimation. All DeepCuts significantly outperform the state of the art, with DeepCut SP Dense-CNN model improving by $13.7$% PCK over the best known result . The improvement is even more dramatic for lower thresholds (Fig. 2 (a)): for PCK @ $0.1$ the best model improves by $19.9$% over Tompson et al. , by $26.7$% over Fan et al. , and by $32.4$% PCK over Chen&Yuille . The latter is interesting, as use a stronger spatial model that predicts the pairwise conditioned on the CNN features, whereas DeepCuts use geometric-only pairwise connectivity. Including body part orientation information into DeepCuts should further improve the results.

Results on MPII Single Person. Results are shown in Tab. 5.1 and Fig. 2 (b). DeepCut SP AFR-CNN noticeably improves over AFR-CNN alone ($79.8$ vs. $78.8$% PCK, $51.1$ vs. $49.0$% AUC). The improvement is stronger for smaller thresholds (c.f. Fig. 2), as spatial model improves part localization. Dense-CNN alone trained on MPII outperforms AFR-CNN ($81.6$ vs. $78.8$% PCK), which shows the advantages of dense training and evaluation. As expected, Dense-CNN performs slightly better when trained on the larger MPII+LSPET. Finally, DeepCut Dense-CNN SP is slightly better than Dense-CNN alone leading to the best result on MPII dataset ($82.4$% PCK).

Comparison to the state of the art (MPII). We compare the performance of DeepCut models to the best deep learning approaches from the literature was re-trained and evaluated on MPII dataset by the authors.. DeepCut SP Dense-CNN outperforms both ($82.4$ vs $79.6$ and $82.0$% PCK, respectively). Similar to them DeepCuts rely on dense training and evaluation of part detectors, but unlike them use single size receptive field and do not include multi-resolution context information. Also, appearance and spatial components of DeepCuts are trained piece-wise, unlike . We observe that performance differences are higher for smaller thresholds (c.f. Fig. 2 (b)). This is remarkable, as a much simpler strategy for location refinement is used compared to . Using multi-resolution filters and joint training should improve the performance.

& 92.3 90.6 81.7 74.9 79.2 70.4 63.0 79.8 51.1

Dense-CNN (unary) 93.5 88.6 82.2 77.1 81.7 74.4 68.9 81.6 56.0 +LSPET 94.0 89.4 82.3 77.5 82.0 74.4 68.7 81.9 56.5 +DeepCut SP 94.1 90.2 83.4 77.3 82.6 75.7 68.6 82.4 56.5 Tompson et al. 95.8 90.3 80.5 74.3 77.6 69.7 62.8 79.6 51.8 Tompson et al. 96.1 91.9 83.9 77.8 80.9 72.3 64.8 82.0 54.9

2 Multi Person Pose Estimation

We now evaluate DeepCut MP models on the challenging task of MP pose estimation with an unknown number of people per image and visible body parts per person.

Datasets. For evaluation we use two public MP benchmarks: “We Are Family” (WAF) with $350$ training and $175$ testing group shots of people; “MPII Human Pose” (“Multi-Person”) consisting of $3844$ training and $1758$ testing groups of multiple interacting individuals in highly articulated poses with variable number of parts. On MPII, we use a subset of $288$ testing images for evaluation. We first pre-finetune both AFR-CNN and Dense-CNN from ImageNet to MPII and MPII+LSPET, respectively, and further finetune each model to WAF and MPII Multi-Person. For WAF, we re-train the spatial model on WAF training set.

WAF evaluation measure. Approaches are evaluated using the official toolkit , thus results are directly comparable to prior work. The toolkit implements occlusion-aware “Percentage of Correct Parts ($m$PCP)” metric. In addition, we report “Accuracy of Occlusion Prediction (AOP)” .

MPII Multi-Person evaluation measure. PCK metric is suitable for SP pose estimation with known number of parts and does not penalize for false positives that are not a part of the ground truth. Thus, for MP pose estimation we use “Mean Average Precision (mAP)” measure, similar to . In contrast to evaluating the detection of any part instance in the image disrespecting inconsistent pose predictions, we evaluate consistent part configurations. First, multiple body pose predictions are generated and then assigned to the ground truth (GT) based on the highest PCKh . Only single pose can be assigned to GT. Unassigned predictions are counted as false positives. Finally, AP for each body part is computed and mAP is reported.

Baselines. To assess the performance of AFR-CNN and Dense-CNN we follow a traditional route from the literature based on two stage approach: first a set of regions of interest (ROI) is generated and then the SP pose estimation is performed in the ROIs. This corresponds to unary only performance. ROI are either based on a ground truth (GT ROI) or on the people detector output (det ROI).

& 99.0 79.5 74.3 87.1 82.2 85.6 Dense-CNN det ROI 76.0 46.0 40.2 83.7 55.3 73.8 DeepCut MP Dense-CNN 99.3 81.5 79.5 87.1 84.7 86.5 Ghiasi et. al. - - - - 63.6 74.0 Eichner&Ferrari 97.6 68.2 48.1 86.1 69.4 80.0 Chen&Yuille 98.5 77.2 71.3 88.5 80.7 84.9

Results on WAF. Results are shown in Tab. 5.2. det ROI is obtained by extending provided upper body detection boxes. AFR-CNN det ROI achieves $57.6$% $m$PCP and $73.9$% AOP. DeepCut MP AFR-CNN significantly improves over AFR-CNN det ROI achieving $82.2$% $m$PCP. This improvement is stronger compared to LSP and MPII due to several reasons. First, $m$PCP requires consistent prediction of body sticks as opposite to body joints, and including spatial model enforces consistency. Second, $m$PCP metric is occlusion-aware. DeepCuts can deactivate detections for the occluded parts thus effectively reasoning about occlusion. This is supported by strong increase in AOP ($85.6$ vs. $73.9$%). Results by DeepCut MP Dense-CNN follow the same tendency achieving the best performance of $84.7$% $m$PCP and $86.5$% AOP. Both increase in $m$PCP and AOP show the advantages of DeepCuts over traditional det ROI approaches.

Tab. 5.2 shows that DeepCuts outperform all prior methods. Deep learning method is outperformed both for $m$PCP (84.7 vs. 80.7%) and AOP (86.5 vs. 84.9%) measures. This is remarkable, as DeepCuts reason about part interactions across several people, whereas primarily focuses on the single-person case and handles multi-person scenes akin to . In contrast to , DeepCuts are not limited by the number of possible occlusion patterns and cover person-person occlusions and other types as truncation and occlusion by objects in one formulation. DeepCuts significantly outperform while being more general: unlike DeepCuts do not require person detector and not limited by a number of occlusion states among people.

Qualitative comparison to is provided in Fig. 3.

Results on MPII Multi-Person. Obtaining a strong detector of highly articulated people having strong occlusions and truncations is difficult. We employ a neck detector as a person detector as it turned out to be the most reliable part. Full body bounding box is created around a neck detection and used as det ROI. GT ROIs were provided by the authors . As the MP approach is not public, we compare to SP state-of-the-art method applied to GT ROI image crops.

Results are shown in Tab. 4.2. DeepCut MP AFR-CNN improves over AFR-CNN det ROI by 4.3% achieving 51.4% AP. The largest differences are observed for the ankle, knee, elbow and wrist, as those parts benefit more from the connections to other parts. DeepCut MP UB AFR-CNN using upper body parts only slightly improves over the full body model when compared on common parts (60.5 vs 58.2% AP). Similar tendencies are observed for Dense-CNNs, though improvements of MP UB over MP are more significant.

All DeepCuts outperform Chen&Yuille SP GT ROI, partially due to stronger part detectors compared to (c.f. Tab. 5.1). Another reason is that Chen&Yuille SP GT ROI does not model body part occlusion and truncation always predicting the full set of parts, which is penalized by the AP measure. In contrast, our formulation allows to deactivate the part hypothesis in the initial set of part candidates thus effectively performing non-maximum suppression. In DeepCuts part hypotheses are suppressed based on the evidence from all other body parts making this process more reliable.

& 71.8 67.8 54.9 38.1 52.0 41.2 30.4 58.2 51.4

AFR-CNN MP UB 75.2 71.0 56.4 39.6 - - - 60.5 -

Dense-CNN det ROI 77.2 71.8 55.9 42.1 53.8 39.9 27.4 61.8 53.2

Dense-CNN MP 73.4 71.8 57.9 39.9 56.7 44.0 32.0 60.7 54.1

Dense-CNN MP UB 81.5 77.3 65.8 50.0 - - - 68.7 -

AFR-CNN GT ROI 73.2 66.5 54.6 42.3 50.1 44.3 37.8 59.1 53.1

Dense-CNN GT ROI 78.1 74.1 62.2 52.0 56.9 48.7 46.1 66.6 60.2

Chen&Yuille SP GT ROI 65.0 34.2 22.0 15.7 19.2 15.8 14.2 34.2 27.1

Conclusion

Articulated pose estimation of multiple people in uncontrolled real world images is challenging but of real world interest. In this work, we proposed a new formulation as a joint subset partitioning and labeling problem (SPLP). Different to previous two-stage strategies that separate the detection and pose estimation steps, the SPLP model jointly infers the number of people, their poses, spatial proximity, and part level occlusions. Empirical results on four diverse and challenging datasets show significant improvements over all previous methods not only for the multi person, but also for the single person pose estimation problem. On multi person WAF dataset we improve by $30$% PCP over the traditional two-stage approach. This shows that a joint formulation is crucial to disambiguate multiple and potentially overlapping persons. Models and code available at http://pose.mpi-inf.mpg.de.

We provide additional quantitative results on LSP dataset using person-centric (PC) and observer-centric (OC) evaluation settings.

First, detailed performance analysis is performed when evaluating various parameters of AFR-CNN and results are reported using PCK evaluation measure. Then, performance of the proposed AFR-CNN and Dense-CNN part detection models is evaluated using strict PCP measure.

Detailed AFR-CNN performance analysis (PCK). Detailed parameter analysis of AFR-CNN is provided in Tab. 4.2 and results are reported using PCK evaluation measure. Respecting parameters for each experiment are shown in the first column and parameter differences between the neighboring rows in the table are highlighted in bold. Re-scoring the $2000$ DPM proposals using AFR-CNN with AlexNet leads to $56.9$% PCK. This is achieved using basis scale $1$ ($\approx$ head size) of proposals and training with initial learning rate (lr) of $0.001$ for $80$k iterations, after which lr is reduced by $0.1$, for a total number of $140$k SGD iterations. In addition, bounding box regression and default IoU threshold of $0.5$ for positive/negative label assignment have been used. Extending the regions by $4$x increases the performance to 65.1% PCK, as it incorporates more context including the information about symmetric body parts and allows to implicitly encode higher-order body part relations into the part detector. No improvements observed for larger scales. Increasing lr to $0.003$, lr reduction step to $160$k and training for a larger number of iterations ($240$k) improves the results to $67.4$, as higher lr allows for for more significant updates of model parameters when finetuned on the task of human body part detection. Increasing the number of training examples by reducing the training IoU threshold to $0.4$ results into slight performance improvement ($68.8$ vs. $67.4$% PCK). Further increasing the number of training samples by horizontally flipping each image and performing translation and scale jittering of the ground truth training samples improves the performance to $69.6$% PCK and $42.3$% AUC. The improvement is more pronounced for smaller distance thresholds ($42.3$ vs. $40.9$% AUC): localization of body parts is improved due to the increased number of jittered samples that significantly overlap with the ground truth. Further increasing the lr, lr reduction step and total number of iterations altogether improves the performance to $72.4$% PCK, and very minor improvements are observed when training longer. All results above are achieved by finetuning the AlexNet architecture from the ImageNet model on the MPII training set. Further finetuning the MPII-finetuned model on the LSP training set increases the performance to $77.9$% PCK, as the network learns LSP-specific image representations. Using the deeper VGG architecture improves over more shallow AlexNet (77.9 vs. 72.4% PCK, 50.0 vs. 44.6% AUC). Funetuning VGG on LSP achieves remarkable 82.8% PCK and 57.0% AUC. Strong increase in AUC (57.0 vs. 50%) characterizes the improvement for smaller PCK evaluation thresholds. Switching off bounding box regression results into performance drop (81.3% PCK, 53.2% AUC) thus showing the importance of the bounding box regression for better part localization. Overall, we demonstrate that proper adaptation and tweaking of the state-of-the-art generic object detector FR-CNN leads to a strong body part detection model that dramatically improves over the vanilla FR-CNN ($82.8$ vs. $56.9$% PCK, $57.8$ vs. $35.9$% AUC) and significantly outperforms the state of the art ($+9.4$% PCK over the best known PCK result and $+9.7$% AUC over the best known AUC result .

& 85.7 74.4 61.3 53.2 64.1 63.1 53.8 65.1 39.0

AlexNet scale 4, lr 0.003, lr step 160k, # iter 240k, IoU pos/neg 0.5 87.0 75.1 63.0 56.3 67.0 65.7 58.0 67.4 40.8

AlexNet scale 4, lr 0.003, lr step 160k, # iter 240k, IoU pos/neg 0.4 87.5 76.7 64.8 56.0 68.2 68.7 59.6 68.8 40.9

AlexNet scale 4, lr 0.003, lr step 160k, # iter 240k, IoU pos/neg 0.4, data augment 87.8 77.8 66.0 58.1 70.9 66.9 59.8 69.6 42.3

AlexNet scale 4, lr 0.004, lr step 320k, # iter 1M, IoU pos/neg 0.4, data augment 88.1 79.3 68.9 62.6 73.5 69.3 64.7 72.4 44.6

+ finetune LSP, lr 0.0005, lr step 10k, # iter 40k 92.9 81.0 72.1 66.4 80.6 77.6 75.0 77.9 51.6

VGG scale 4, lr 0.003, lr step 160k, # iter 320k, IoU pos/neg 0.4, data augment 91.0 84.2 74.6 67.7 77.4 77.3 72.8 77.9 50.0

+ finetune LSP lr 0.0005, lr step 10k, # iter 40k 95.4 86.5 77.8 74.0 84.5 78.8 82.6 82.8 57.0

A.2 LSP Observer-Centric (OC)

We now evaluate the performance of the proposed part detection models on LSP dataset using the observer-centric (OC) annotations . In contrast to the person-centric (PC) annotations used in all previous experiments, OC annotations do not penalize for the right/left body part prediction flips and count a body part to be the right body part, if it is on the right side of the line connecting pelvis and neck, and a body part to be the left body part otherwise.

Evaluation is performed using the official OC annotations provided by . Prior to evaluation, we first finetune the AFR-CNN and Dense-CNN part detection models from ImageNet on MPII and MPII+LSPET training sets, respectively, (same as for PC evaluation), and then further finetuned the models on LSP OC training set.

PCK evaluation measure. Results using OC annotations and PCK evaluation measure are shown in Tab. 9 and in Fig. 4. AFR-CNN achieves $84.2$% PCK and $58.1$% AUC. This result is only slightly better compared to AFR-CNN evaluated using PC annotations (84.2 vs 82.8% PCK, $58.1$ vs. $57.0$% AUC). Although PC annotations correspond to a harder task, only small drop in performance when using PC annotations shows that the network can learn to accurately predict person’s viewpoint and correctly label left/right limbs in most cases. This is contrast to earlier approaches based on hand-crafted features whose performance drops much stronger when evaluated in PC evaluation setting (e.g. drops from 71.0% PCK when using OC annotations to 58.0% PCK when using PC annotations). Similar to PC case, Dense-CNN detection model outperforms AFR-CNN (88.2 vs. 84.2% PCK and 65.0 vs. 58.1% AUC). The differences are more pronounced when examining the entire PCK curve for smaller distance thresholds (c.f. Fig. 4).

Comparing the performance by AFR-CNN and Dense-CNN to the state of the art, we observe that both proposed approaches significantly outperform other methods. Both deep learning based approaches of Chen&Yuille and Ouyang et al. are outperformed by $+10.7$ and $+18.2$% PCK when compared to the best performing Dense-CNN. Analysis of PCK curve for the entire range of PCK distance thresholds reveals even larger performance differences (c.f. Fig. 4). The results using OC annotations confirm our findings from PC evaluation and clearly show the advantages of the proposed part detection models over the state-of-the-art deep learning methods , as well as over earlier pose estimation methods based on hand-crafted image features .

PCP evaluation measure. Results using OC annotations and PCP evaluation measure are shown in Tab. 10. Overall, the trend is similar to PC evaluation: both proposed approaches significantly outperform the state-of-the-art methods with Dense-CNN achieving the best result of 85.0% PCP thereby improving by $+10$% PCP over the best published result .

Qualitative comparison of our joint formulation DeepCut MP Dense-CNN to the traditional two-stage approach Dense-CNN det ROI relying on person detector, and to the approach of Chen&Yuille on WAF dataset is shown in Fig. 5. See figure caption for visual performance analysis.

Appendix C Additional Results on MPII Multi-Person

Qualitative comparison of our joint formulation DeepCut MP Dense-CNN to the traditional two-stage approach Dense-CNN det ROI on MPII Multi-Person dataset is shown in Fig. 6 and 7. Dense-CNN det ROI works well when multiple fully visible individuals are sufficiently separated and thus their body parts can be partitioned based on the person detection bounding box. In this case the strong Dense-CNN body part detection model can correctly estimate most of the visible body parts (image 16, 17, 19). However, Dense-CNN det ROI cannot tell apart the body parts of multiple individuals located next to each other and possibly occluding each other, and often links the body parts across the individuals (images 1-16, 19-20). In addition, Dense-CNN det ROI cannot reason about occlusions and truncations always providing a prediction for each body part (image 4, 6, 10). In contrast, DeepCut MP Dense-CNN is able to correctly partition and label an initial pool of body part candidates (each image, top row) into subsets that correspond to sets of mutually consistent body part candidates and abide to mutual consistency and exclusion constraints (each image, row 2), thereby outputting consistent body pose predictions (each image, row 3). $c\neq c^{\prime}$ pairwise terms allow to partition the initial set of part detection candidates into valid pose configurations (each image, row 2: person-clusters highlighted by dense colored connections). $c=c^{\prime}$ pairwise terms facilitate clustering of multiple body part candidates of the same body part of the same person (each image, row 2: markers of the same type and color). In addition, $c=c^{\prime}$ pairwise terms facilitate a repulsive property that prevents nearby part candidates of the same type to be associated to different people (image 1: detections of the left shoulder are assigned to the front person only). Furthermore, DeepCut MP Dense-CNN allows to either merge or deactivate part hypotheses thus effectively performing non-maximum suppression and reasoning about body part occlusions and truncations (image 3, row 2: body part hypotheses on the background are deactivated (black crosses); image 6, row 2: body part hypotheses for the truncated body parts are deactivated (black crosses); image 1-6, 8-9, 13-14, row 3: only visible body parts of the partially occluded people are estimated, while non-visible body parts are correctly predicted to be occluded). These qualitative examples show that DeepCuts MP can successfully deal with the unknown number of people per image and the unknown number of visible body parts per person.

References