Channel Pruning for Accelerating Very Deep Neural Networks

Introduction

Recent CNN acceleration works fall into three categories: optimized implementation (e.g., FFT ), quantization (e.g., BinaryNet ), and structured simplification that convert a CNN into compact one . This work focuses on the last one.

Structured simplification mainly involves: tensor factorization , sparse connection , and channel pruning . Tensor factorization factorizes a convolutional layer into several efficient ones (Fig. 1(c)). However, feature map width (number of channels) could not be reduced, which makes it difficult to decompose $1\times 1$ convolutional layer favored by modern networks (e.g., GoogleNet , ResNet , Xception ). This type of method also introduces extra computation overhead. Sparse connection deactivates connections between neurons or channels (Fig. 1(b)). Though it is able to achieves high theoretical ,/the sparse convolutional layers have an ”irregular” shape which is not implementation friendly. In contrast, channel pruning directly reduces feature map width, which shrinks a network into thinner one, as shown in Fig. 1(d). It is efficient on both CPU and GPU because no special implementation is required.

Pruning channels is simple but challenging because removing channels in one layer might dramatically change the input of the following layer. Recently, training-based channel pruning works have focused on imposing sparse constrain on weights during training, which could adaptively determine hyper-parameters. However, training from scratch is very costly and results for very deep CNNs on ImageNet have been rarely reported. Inference-time attempts have focused on analysis of the importance of individual weight. The reported /is very limited.

In this paper, we propose a new inference-time approach for channel pruning, utilizing redundancy inter channels. Inspired by tensor factorization improvement by feature maps reconstruction , instead of analyzing filter weights , we fully exploits redundancy inter feature maps. Specifically, given a trained CNN model, pruning each layer is achieved by minimizing reconstruction error on its output feature maps, as showned in Fig. 2. We solve this minimization problem by two alternative steps: channels selection and feature map reconstruction. In one step, we figure out the most representative channels, and prune redundant ones, based on LASSO regression. In the other step, we reconstruct the outputs with remaining channels with linear least squares. We alternatively take two steps. Further, we approximate the network layer-by-layer, with accumulated error accounted. We also discuss methodologies to prune multi-branch networks (e.g., ResNet , Xception ).

For VGG-16, we achieve $4\times$ acceleration, with only 1.0% increase of top-5 error. Combined with tensor factorization, we reach $5\times$ acceleration but merely suffer 0.3% increase of error, which outperforms previous state-of-the-arts. We further speed up ResNet-50 and Xception-50 by $2\times$ with only 1.4%, 1.0% accuracy loss respectively.

Related Work

There has been a significant amount of work on accelerating CNNs. Many of them fall into three categories: optimized implementation , quantization , and structured simplification .

Optimized implementation based methods accelerate convolution, with special convolution algorithms like FFT . Quantization reduces floating point computational complexity.

Sparse connection eliminates connections between neurons . prunes connections based on weights magnitude. could accelerate fully connected layers up to $50\times$. However, in practice, the actual speed-up maybe very related to implementation.

Tensor factorization decompose weights into several pieces. accelerate fully connected layers with truncated SVD. factorize a layer into $3\times 3$ and $1\times 1$ combination, driven by feature map redundancy.

Channel pruning removes redundant channels on feature maps. There are several training-based approaches. regularize networks to improve accuracy. Channel-wise SSL reaches high compression ratio for first few conv layers of LeNet and AlexNet . could work well for fully connected layers. However, training-based approaches are more costly, and the effectiveness for very deep networks on large datasets is rarely exploited.

Inference-time channel pruning is challenging, as reported by previous works . Some works focus on model size compression, which mainly operate the fully connected layers. Data-free approaches results for /(e.g., $5\times$) have not been reported, and requires long retraining procedure. select channels via over 100 random trials, however it need long time to evaluate each trial on a deep network, which makes it infeasible to work on very deep models and large datasets. is even worse than naive solution from our observation sometimes (Sec. 4.1.1).

Approach

In this section, we first propose a channel pruning algorithm for a single layer, then generalize this approach to multiple layers or the whole model. Furthermore, we discuss variants of our approach for multi-branch networks.

Fig. 2 illustrates our channel pruning algorithm for a single convolutional layer. We aim to reduce the number of channels of feature map B, while maintaining outputs in feature map C. Once channels are pruned, we can remove corresponding channels of the filters that take these channels as input. Also, filters that produce these channels can also be removed. It is clear that channel pruning involves two key points. The first is channel selection, since we need to select proper channel combination to maintain as much information. The second is reconstruction. We need to reconstruct the following feature maps using the selected channels.

Motivated by this, we propose an iterative two-step algorithm. In one step, we aim to select most representative channels. Since an exhaustive search is infeasible even for tiny networks, we come up with a LASSO regression based method to figure out representative channels and prune redundant ones. In the other step, we reconstruct the outputs with remaining channels with linear least squares. We alternatively take two steps.

2 Whole Model Pruning

Inspired by , we apply our approach layer by layer sequentially. For each layer, we obtain input volumes from the current input feature map, and output volumes from the output feature map of the un-pruned model. This could be formalized as:

3 Pruning Multi-Branch Networks

The whole model pruning discussed above is enough for single-branch networks like LeNet , AlexNet and VGG Nets . However, it is insufficient for multi-branch networks like GoogLeNet and ResNet . We mainly focus on pruning the widely used residual structure (e.g., ResNet , Xception ). Given a residual block shown in Fig. 3 (left), the input bifurcates into shortcut and residual branch. On the residual branch, there are several convolutional layers (e.g., 3 convolutional layers which have spatial size of $1\times 1,3\times 3,1\times 1$, Fig. 3, left). Other layers except the first and last layer can be pruned as is described previously. For the first layer, the challenge is that the large input feature map width (for ResNet, 4 times of its output) can’t be easily pruned, since it’s shared with shortcut. For the last layer, accumulated error from the shortcut is hard to be recovered, since there’s no parameter on the shortcut. To address these challenges, we propose several variants of our approach as follows.

First layer of residual branch: Illustrated in Fig. 3(left), the input feature map of the residual block could not be pruned, since it is also shared with the shortcut branch. In this condition, we could perform feature map sampling before the first convolution to save computation. We still apply our algorithm as Eqn. 1. Differently, we sample the selected channels on the shared feature maps to construct a new input for the later convolution, shown in Fig. 3(right). Computational cost for this operation could be ignored. More importantly, after introducing feature map sampling, the convolution is still ”regular”.

Filter-wise pruning is another option for the first convolution on the residual branch. Since the input channels of parameter-free shortcut branch could not be pruned, we apply our Eqn. 1 to each filter independently (each filter chooses its own representative input channels). Under single layer acceleration, filter-wise pruning is more accurate than our original one. From our experiments, it improves 0.5% top-5 accuracy for $2\times$ ResNet-50 (applied on the first layer of each residual branch) without fine-tuning. However, after fine-tuning, there’s no noticeable improvement. In addition, it outputs ”irregular” convolutional layers, which need special library implementation support. We do not adopt it in the following experiments.

Experiment

We evaluation our approach for the popular VGG Nets , ResNet , Xception on ImageNet , CIFAR-10 and PASCAL VOC 2007 .

For Batch Normalization , we first merge it into convolutional weights, which do not affect the outputs of the networks. So that each convolutional layer is followed by ReLU . We use Caffe for deep network evaluation, and scikit-learn for solvers implementation. For channel pruning, we found that it is enough to extract 5000 images, and 10 samples per image, which is also efficient (i.e. several minutes for VGG-16 On Intel Xeon E5-2670 CPU). On ImageNet, we evaluate the top-5 accuracy with single view. Images are resized such that the shorter side is 256. The testing is on center crop of $224\times 224$ pixels. We could gain more performance with fine-tuning. We use a batch size of 128 and learning rate $1e^{-5}$. We fine-tune our pruned models for 10 epoches (less than 1/10 iterations of training from scratch). The augmentation for fine-tuning is random crop of $224\times 224$ and mirror.

VGG-16 is a 16 layers single-branch convolutional neural network, with 13 convolutional layers. It is widely used in recognition, detection and segmentation, etc. Single view top-5 accuracy for VGG-16 is 89.9%http://www.vlfeat.org/matconvnet/pretrained/.

In this subsection, we evaluate single layer acceleration performance using our algorithm in Sec. 3.1. For better understanding, we compare our algorithm with two naive channel selection strategies. first k selects the first k channels. max response selects channels based on corresponding filters that have high absolute weights sum . For fair comparison, we obtain the feature map indexes selected by each of them, then perform reconstruction (Sec. 3.1 (ii)). We hope that this could demonstrate the importance of channel selection. Performance is measured by increase of error after a certain layer is pruned without fine-tuning, shown in Fig. 4.

As expected, error increases as /increases. Our approach is consistently better than other approaches in different convolutional layers under different ./Unexpectedly, sometimes max response is even worse than first k. We argue that max response ignores correlations between different filters. Filters with large absolute weight may have strong correlation. Thus selection based on filter weights is less meaningful. Correlation on feature maps is worth exploiting. We can find that channel selection affects reconstruction error a lot. Therefore, it is important for channel pruning.

Also notice that channel pruning gradually becomes hard, from shallower to deeper layers. It indicates that shallower layers have much more redundancy, which is consistent with . We could prune more aggressively on shallower layers in whole model acceleration.

1.2 Whole Model Pruning

Shown in Table 1, whole model acceleration results under $2\times$, $4\times$, $5\times$ are demonstrated. We adopt whole model pruning proposed in Sec. 3.2. Guided by single layer experiments above, we prune more aggressive for shallower layers. Remaining channels ratios for shallow layers (conv1_x to conv3_x) and deep layers (conv4_x) is $1:1.5$. conv5_x are not pruned, since they only contribute 9% computation in total and are not redundant.

After fine-tuning, we could reach $2\times$ speed-up without losing accuracy. Under $4\times$, we only suffers 1.0% drops. Consistent with single layer analysis, our approach outperforms previous channel pruning approach (Li et al. ) by large margin. This is because we fully exploits channel redundancy within feature maps. Compared with tensor factorization algorithms, our approach is better than Jaderberg et al. , without fine-tuning. Though worse than Asym. , our combined model outperforms its combined Asym. 3D (Table 2). This may indicate that channel pruning is more challenging than tensor factorization, since removing channels in one layer might dramatically change the input of the following layer. However, channel pruning keeps the original model architecture, do not introduce additional layers, and the absolute /on GPU is much higher (Table 3).

Since our approach exploits a new cardinality, we further combine our channel pruning with spatial factorization and channel factorization . Demonstrated in Table 2, our 3 cardinalities acceleration (spatial, channel factorization, and channel pruning, denoted by 3C) outperforms previous state-of-the-arts. Asym. 3D (spatial and channel factorization), factorizes a convolutional layer to three parts: $1\times 3,\ 3\times 1,\ 1\times 1$.

We apply spatial factorization, channel factorization, and our channel pruning together sequentially layer-by-layer. We fine-tune the accelerated models for 20 epoches, since they are 3 times deeper than the original ones. After fine-tuning, our $4\times$ model suffers no degradation. Clearly, a combination of different acceleration techniques is better than any single one. This indicates that a model is redundant in each cardinality.

1.3 Comparisons of Absolute Performance

We further evaluate absolute performance of acceleration on GPU. Results in Table 3 are obtained under Caffe , CUDA8 and cuDNN5 , with a mini-batch of 32 on a GPU (GeForce GTX TITAN X). Results are averaged from 50 runs. Tensor factorization approaches decompose weights into too many pieces, which heavily increase overhead. They could not gain much absolute speed-up. Though our approach also encountered performance decadence, it generalizes better on GPU than other approaches. Our results for tensor factorization differ from previous research , maybe because current library and hardware prefer single large convolution instead of several small ones.

1.4 Comparisons with Training from Scratch

Though training a compact model from scratch is time-consuming (usually 120 epoches), it worths comparing our approach and from scratch counterparts. To be fair, we evaluated both from scratch counterpart, and normal setting network that has the same computational complexity and same architecture.

Shown in Table 4, we observed that it’s difficult for from scratch counterparts to reach competitive accuracy. our model outperforms from scratch one. Our approach successfully picks out informative channels and constructs highly compact models. We can safely draw the conclusion that the same model is difficult to be obtained from scratch. This coincides with architecture design researches that the model could be easier to train if there are more channels in shallower layers. However, channel pruning favors shallower layers.

For from scratch (uniformed), the filters in each layers is reduced by half (eg. reduce conv1_1 from 64 to 32). We can observe that normal setting networks of the same complexity couldn’t reach same accuracy either. This consolidates our idea that there’s much redundancy in networks while training. However, redundancy can be opt out at inference-time. This maybe an advantage of inference-time acceleration approaches over training-based approaches.

Notice that there’s a 0.6% gap between the from scratch model and uniformed one, which indicates that there’s room for model exploration. Adopting our approach is much faster than training a model from scratch, even for a thinner one. Further researches could alleviate our approach to do thin model exploring.

1.5 Acceleration for Detection

VGG-16 is popular among object detection tasks . We evaluate transfer learning ability of our $2\times$/$4\times$ pruned VGG-16, for Faster R-CNN object detections. PASCAL VOC 2007 object detection benchmark contains 5k trainval images and 5k test images. The performance is evaluated by mean Average Precision (mAP) and mmAP (primary challenge metric of COCO ). In our experiments, we first perform channel pruning for VGG-16 on the ImageNet. Then we use the pruned model as the pre-trained model for Faster R-CNN.

The actual running time of Faster R-CNN is 220ms / image. The convolutional layers contributes about 64%. We got actual time of 94ms for $4\times$ acceleration. From Table 5, we observe 0.4% mAP drops of our $2\times$ model, which is not harmful for practice consideration. Observed from mmAP, For higher localization requirements our speedup model does not suffer from large degradation.

2 Experiments with Residual Architecture Nets

For Multi-path networks , we further explore the popular ResNet and latest Xception , on ImageNet and CIFAR-10. Pruning residual architecture nets is more challenging. These networks are designed for both efficiency and high accuracy. Tensor factorization algorithms have difficult to accelerate these model. Spatially, $1\times 1$ convolution is favored, which could hardly be factorized.

ResNet complexity uniformly drops on each residual block. Guided by single layer experiments (Sec. 4.1.1), we still prefer reducing shallower layers heavier than deeper ones.

Following similar setting as Filter pruning , we keep 70% channels for sensitive residual blocks (res5 and blocks close to the position where spatial size change, e.g. res3a,res3d). As for other blocks, we keep 30% channels. With multi-branch enhancement, we prune branch2a more aggressively within each residual block. The preserving channels ratios for branch2a,branch2b,branch2c is $2:4:3$ (e.g., Given 30%, we keep 40%, 80%, 60% respectively).

We evaluate performance of multi-branch variants of our approach (Sec. 3). From Table 6, we improve 4.0% with our multi-branch enhancement. This is because we accounted the accumulated error from shortcut connection which could broadcast to every layer after it. And the large input feature map width at the entry of each residual block is well reduced by our feature map sampling.

2.2 Xception Pruning

Since computational complexity becomes important in model design, separable convolution has been payed much attention . Xception is already spatially optimized and tensor factorization on $1\times 1$ convolutional layer is destructive. Thanks to our approach, it could still be accelerated with graceful degradation. For the ease of comparison, we adopt Xception convolution on ResNet-50, denoted by Xception-50. Based on ResNet-50, we swap all convolutional layers with spatial conv blocks. To keep the same computational complexity, we increase the input channels of all branch2b layers by $2\times$. The baseline Xception-50 has a top-5 accuracy of 92.8% and complexity of 4450 MFLOPs.

We apply multi-branch variants of our approach as described in Sec. 3, and adopt the same pruning ratio setting as ResNet in previous section. Maybe because of Xception block is unstable, Batch Normalization layers must be maintained during pruning. Otherwise it becomes nontrivial to fine-tune the pruned model.

Shown in Table 7, after fine-tuning, we only suffer 1.0% increase of error under $2\times$. Filter pruning could also apply on Xception, though it is designed for small ./Without fine-tuning, top-5 error is 100%. After training 20 epochs which is like training from scratch, increased error reach 4.3%. Our results for Xception-50 are not as graceful as results for VGG-16, since modern networks tend to have less redundancy by design.

2.3 Experiments on CIFAR-10

Even though our approach is designed for large datasets, it could generalize well on small datasets. We perform experiments on CIFAR-10 dataset , which is favored by many acceleration researches. It consists of 50k images for training and 10k for testing in 10 classes.

We reproduce ResNet-56, which has accuracy of 92.8% (Serve as a reference, the official ResNet-56 has accuracy of 93.0%). For $2\times$ acceleration, we follow similar setting as Sec. 4.2.1 (keep the final stage unchanged, where the spatial size is $8\times 8$). Shown in Table 8, our approach is competitive with scratch trained one, without fine-tuning, under $2\times$ speed-up. After fine-tuning, our result is significantly better than Filter pruning and scratch trained one.

Conclusion

To conclude, current deep CNNs are accurate with high inference costs. In this paper, we have presented an inference-time channel pruning method for very deep networks. The reduced CNNs are inference efficient networks while maintaining accuracy, and only require off-the-shelf libraries. Compelling speed-ups and accuracy are demonstrated for both VGG Net and ResNet-like networks on ImageNet, CIFAR-10 and PASCAL VOC.

In the future, we plan to involve our approaches into training time, instead of inference time only, which may also accelerate training procedure.

References