Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang

Introduction

Single-image super-resolution (SR) aims to reconstruct a high-resolution (HR) image from a single low-resolution (LR) input image. In recent years, example-based SR methods have demonstrated the state-of-the-art performance by learning a mapping from LR to HR image patches using large image databases. Numerous learning algorithms have been applied to learn such a mapping, including dictionary learning , local linear regression , and random forest .

To address these drawbacks, we propose the Laplacian Pyramid Super-Resolution Network (LapSRN) based on a cascade of convolutional neural networks (CNNs). Our network takes an LR image as input and progressively predicts the sub-band residuals in a coarse-to-fine fashion. At each level, we first apply a cascade of convolutional layers to extract feature maps. We then use a transposed convolutional layer for upsampling the feature maps to a finer level. Finally, we use a convolutional layer to predict the sub-band residuals (the differences between the upsampled image and the ground truth HR image at the respective level). The predicted residuals at each level are used to efficiently reconstruct the HR image through upsampling and addition operations. While the proposed LapSRN consists of a set of cascaded sub-networks, we train the network with a robust Charbonnier loss function in an end-to-end fashion (i.e., without stage-wise optimization). As depicted in Figure 1(e), our network architecture naturally accommodates deep supervision (i.e., supervisory signals can be applied simultaneously at each level of the pyramid).

Our algorithm differs from existing CNN-based methods in the following three aspects:

(1) Accuracy. The proposed LapSRN extracts feature maps directly from LR images and jointly optimizes the upsampling filters with deep convolutional layers to predict sub-band residuals. The deep supervision with the Charbonnier loss improves the performance thanks to the ability to better handle outliers. As a result, our model has a large capacity to learn complicated mappings and effectively reduces the undesired visual artifacts.

(2) Speed. Our LapSRN embraces both fast processing speed and high capacity of deep networks. Experimental results demonstrate that our method is faster than several CNN based super-resolution models, e.g., SRCNN , SCN , VDSR , and DRCN . Similar to FSRCNN , our LapSRN achieves real-time speed on most of the evaluated datasets. In addition, our method provides significantly better reconstruction accuracy.

(3) Progressive reconstruction. Our model generates multiple intermediate SR predictions in one feed-forward pass through progressive reconstruction using the Laplacian pyramid. This characteristic renders our technique applicable to a wide range of applications that require resource-aware adaptability. For example, the same network can be used to enhance the spatial resolution of videos depending on the available computational resources. For scenarios with limited computing resources, our $8\times$ model can still perform 2 $\times$ or $4\times$ SR by simply bypassing the computation of residuals at finer levels. Existing CNN-based methods, however, do not offer such flexibility.

SR based on external databases.

Numerous SR methods learn the LR-HR mapping with image pairs collected from external databases using supervised learning algorithms, such as nearest neighbor , manifold embedding , kernel ridge regression , and sparse representation . Instead of directly modeling the complex patch space over the entire database, several methods partition the image database by K-means , sparse dictionary or random forest , and learn locally linear regressors for each cluster.

Convolutional neural networks based SR.

In contrast to modeling the LR-HR mapping in the patch space, SRCNN jointly optimize all the steps and learn the non-linear mapping in the image space. The VDSR network demonstrates significant improvement over SRCNN by increasing the network depth from 3 to 20 convolutional layers. To facilitate training a deeper model with a fast convergence speed, VDSR trains the network to predict the residuals rather the actual pixel values. Wang et al. combine the domain knowledge of sparse coding with a deep CNN and train a cascade network (SCN) to upsample images to the desired scale factor progressively. Kim et al. propose a shallow network with deeply recursive layers (DRCN) to reduce the number of parameters.

Laplacian pyramid.

The Laplacian pyramid has been used in a wide range of applications, such as image blending , texture synthesis , edge-aware filtering and semantic segmentation . Denton et al. propose a generative model based on a Laplacian pyramid framework (LAPGAN) to generate realistic images in , which is the most related to our work. However, the proposed LapSRN differs from LAPGAN in three aspects.

First, LAPGAN is a generative model which is designed to synthesize diverse natural images from random noise and sample inputs. On the contrary, our LapSRN is a super-resolution model that predicts a particular HR image based on the given LR image. LAPGAN uses a cross-entropy loss function to encourage the output images to respect the data distribution of training datasets. In contrast, we use the Charbonnier penalty function to penalize the deviation of the prediction from the ground truth sub-band residuals.

Second, the sub-networks of LAPGAN are independent (i.e., no weight sharing). As a result, the network capacity is limited by the depth of each sub-network. Unlike LAPGAN, the convolutional layers at each level in LapSRN are connected through multi-channel transposed convolutional layers. The residual images at a higher level are therefore predicted by a deeper network with shared feature representations at lower levels. The feature sharing at lower levels increases the non-linearity at finer convolutional layers to learn complex mappings. Also, the sub-networks in LAPGAN are independently trained. On the other hand, all the convolutional filters for feature extraction, upsampling, and residual prediction layers in the LapSRN are jointly trained in an end-to-end, deeply supervised fashion.

Third, LAPGAN applies convolutions on the upsampled images, so the speed depends on the size of HR images. On the contrary, our design of LapSRN effectively increases the size of the receptive field and accelerates the speed by extracting features from the LR space. We provide comparisons with LAPGAN in the supplementary material.

Adversarial training.

The SRGAN method optimizes the network using the perceptual loss and the adversarial loss for photo-realistic SR. We note that our LapSRN can be easily extended to the adversarial training framework. As it is not our contribution, we provide experiments on the adversarial loss in the supplementary material.

Deep Laplacian Pyramid Network for SR

In this section, we describe the design methodology of the proposed Laplacian pyramid network, the optimization using robust loss functions with deep supervision, and the details for network training.

We propose to construct our network based on the Laplacian pyramid framework, as shown in Figure 1(e). Our model takes an LR image as input (rather than an upscaled version of the LR image) and progressively predicts residual images at $\log_{2}S$ levels where $S$ is the scale factor. For example, the network consists of $3$ sub-networks for super-resolving an LR image at a scale factor of $8$ . Our model has two branches: (1) feature extraction and (2) image reconstruction.

Feature extraction. At level $s$ , the feature extraction branch consists of $d$ convolutional layers and one transposed convolutional layer to upsample the extracted features by a scale of 2. The output of each transposed convolutional layer is connected to two different layers: (1) a convolutional layer for reconstructing a residual image at level $s$ , and (2) a convolutional layer for extracting features at the finer level $s+1$ . Note that we perform the feature extraction at the coarse resolution and generate feature maps at the finer resolution with only one transposed convolutional layer. In contrast to existing networks that perform all feature extraction and reconstruction at the fine resolution, our network design significantly reduces the computational complexity. Note that the feature representations at lower levels are shared with higher levels, and thus can increase the non-linearity of the network to learn complex mappings at the finer levels.

Image reconstruction. At level $s$ , the input image is upsampled by a scale of 2 with a transposed convolutional (upsampling) layer. We initialize this layer with the bilinear kernel and allow it to be jointly optimized with all the other layers. The upsampled image is then combined (using element-wise summation) with the predicted residual image from the feature extraction branch to produce a high-resolution output image. The output HR image at level $s$ is then fed into the image reconstruction branch of level $s+1$ . The entire network is a cascade of CNNs with a similar structure at each level.

2 Loss function

Let $x$ be the input LR image and $\theta$ be the set of network parameters to be optimized. Our goal is to learn a mapping function $f$ for generating a high-resolution image $\hat{y}=f(x;\theta)$ that is close to the ground truth HR image $y$ . We denote the residual image at level $s$ by $r_{s}$ , the upscaled LR image by $x_{s}$ and the corresponding HR images by $y_{s}$ . The desired output HR images at level $s$ is modeled by $y_{s}=x_{s}+r_{s}$ . We use the bicubic downsampling to resize the ground truth HR image $y$ to $y_{s}$ at each level. Instead of minimizing the mean square errors between $y_{s}$ and $\hat{y}_{s}$ , we propose to use a robust loss function to handle outliers. The overall loss function is defined as:

In the proposed LapSRN, each level $s$ has its loss function and the corresponding ground truth HR image $y_{s}$ . This multi-loss structure resembles the deeply-supervised nets for classification and edge detection . However, the labels used to supervise intermediate layers in are the same across the networks. In our model, we use different scales of HR images at the corresponding level as supervision. The deep supervision guides the network training to predict sub-band residual images at different levels and produce multi-scale output images. For example, our $8\times$ model can produce $2\times$ , $4\times$ and $8\times$ super-resolution results in one feed-forward pass. This property is particularly useful for resource-aware applications, e.g., mobile devices or network applications.

3 Implementation and training details

In the proposed LapSRN, each convolutional layer consists of 64 filters with the size of $3\times 3$ . We initialize the convolutional filters using the method of He et al. . The size of the transposed convolutional filters is $4\times 4$ and the weights are initialized from a bilinear filter. All the convolutional and transposed convolutional layers (except the reconstruction layers) are followed by leaky rectified linear units (LReLUs) with a negative slope of 0.2. We pad zeros around the boundaries before applying convolution to keep the size of all feature maps the same as the input of each level. The convolutional filters have small spatial supports ( $3\times 3$ ). However, we can achieve high non-linearity and increase the size of receptive fields with a deep structure.

We use 91 images from Yang et al. and 200 images from the training set of Berkeley Segmentation Dataset as our training data. The same training dataset is used in as well. In each training batch, we randomly sample $64$ patches with the size of $128\times 128$ . An epoch has $1,000$ iterations of back-propagation. We augment the training data in three ways: (1) Scaling: randomly downscale between $[0.5,1.0]$ . (2) Rotation: randomly rotate image by $90^{\circ}$ , $180^{\circ}$ , or $270^{\circ}$ . (3) Flipping: flip images horizontally or vertically with a probability of $0.5$ . Following the protocol of existing methods , we generate the LR training patches using the bicubic downsampling. We train our model with the MatConvNet toolbox . We set momentum parameter to $0.9$ and the weight decay to $1e-4$ . The learning rate is initialized to $1e-5$ for all layers and decreased by a factor of 2 for every 50 epochs.

Experiment Results

We first analyze the contributions of different components of the proposed network. We then compare our LapSRN with state-of-the-art algorithms on five benchmark datasets and demonstrate the applications of our method on super-resolving real-world photos and videos.

Residual learning. To demonstrate the effect of residual learning, we remove the image reconstruction branch and directly predict the HR images at each level. Figure 2 shows the convergence curves in terms of PSNR on the Set14 for $4\times$ SR. The performance of the “non-residual” network (blue curve) converges slowly and fluctuates significantly. The proposed LapSRN (red curve), on the other hand, outperforms SRCNN within 10 epochs.

Pyramid structure. By removing the pyramid structure, our model falls back to a network similar to FSRCNN but with the residual learning. To use the same number of convolutional layers as LapSRN, we train a network with 10 convolutional layers and one transposed convolutional layer. The quantitative results in Table 2 shows that the pyramid structure leads to moderate performance improvement (e.g. 0.7 dB on Set5 and 0.4 dB on Set14).

Network depth. We train the proposed model with different depth, $d=3,5,10,15$ , at each level and show the trade-offs between performance and speed in Table 3. In general, deep networks perform better shallow ones at the expense of increased computational cost. We choose $d=10$ for our $2\times$ and $4\times$ SR models to strike a balance between performance and speed. We show that the speed of our LapSRN with $d=10$ is faster than most of the existing CNN-based SR algorithms (see Figure 6). For $8\times$ model, we choose $d=5$ because we do not observe significant performance gain by using more convolutional layers.

2 Comparisons with the state-of-the-arts

We compare the proposed LapSRN with 8 state-of-the-art SR algorithms: A+ , SRCNN , FSRCNN , SelfExSR , RFL , SCN , VDSR and DRCN . We carry out extensive experiments using 5 datasets: Set5 , Set14 , BSDS100 , Urban100 and manga109 . Among these datasets, Set5, Set14 and BSDS100 consist of natural scenes; Urban100 contains challenging urban scenes images with details in different frequency bands; and manga109 is a dataset of Japanese manga. We train the LapSRN until the learning rate decreases to $1e-6$ and the training time is around three days on a Titan X GPU.

We evaluate the SR images with three commonly used image quality metrics: PSNR, SSIM , and IFC . Table 4 shows quantitative comparisons for $2\times$ , $4\times$ and $8\times$ SR. Our LapSRN performs favorably against existing methods on most datasets. In particular, our algorithm achieves higher IFC values, which has been shown to be correlated well with human perception of image super-resolution . We note that the best results can be achieved by training with specific scale factors (Ours $2\times$ and Ours $4\times$ ). As the intermediate convolutional layers are trained to minimize the prediction errors for both the corresponding level and higher levels, the intermediate predictions of our $8\times$ model are slightly inferior to our $2\times$ and $4\times$ models. Nevertheless, our $8\times$ model provides a competitive performance to the state-of-the-art methods in $2\times$ and $4\times$ SR.

In Figure 4, we show visual comparisons on Urban100, BSDS100 and Manga109 with the a scale factor of $4\times$ . Our method accurately reconstructs parallel straight lines and grid patterns such as windows and the stripes on tigers. We observe that methods using the bicubic upsampling for pre-processing generate results with noticeable artifacts . In contrast, our approach effectively suppresses such artifacts through progressive reconstruction and the robust loss function.

For $8\times$ SR, we re-train the model of A+, SRCNN, FSRCNN, RFL and VDSR using the publicly available codeWe do not re-train DRCN because the training code is not available.. Both SelfExSR and SCN methods can handle different scale factors using progressive reconstruction. We show $8\times$ SR results on BSDS100 and Urban100 in Figure 5. For $8\times$ SR, it is challenging to predict HR images from bicubic-upsampled images or using one-step upsampling . The state-of-the-art methods do not super-resolve the fine structures well. In contrast, the LapSRN reconstructs high-quality HR images at a relatively fast speed. We present SR images generated by all the evaluated methods in the supplementary material.

3 Execution time

We use the original codes of state-of-the-art methods to evaluate the runtime on the same machine with 3.4 GHz Intel i7 CPU (64G RAM) and NVIDIA Titan X GPU (12G Memory). Since the codes of SRCNN and FSRCNN for testing are based on CPU implementations, we reconstruct these models in MatConvNet with the same network weights to measure the run time on GPU. Figure 6 shows the trade-offs between the run time and performance (in terms of PSNR) on Set14 for $4\times$ SR. The speed of the proposed LapSRN is faster than all the existing methods except FSRCNN. We present detailed evaluations on run time of all evaluated datasets in the supplementary material.

4 Super-resolving real-world photos

We demonstrate an application of super-resolving historical photographs with JPEG compression artifacts. In these cases, neither the ground-truth images nor the downsampling kernels are available. As shown in Figure 7, our method can reconstruct sharper and more accurate images than the state-of-the-art approaches.

5 Super-resolving video sequences

We conduct frame-based SR experiments on two video sequences from with a spatial resolution of $1200\times 800$ pixels.Our method is not a video super-resolution algorithm as temporal coherence or motion blur are not considered. We downsample each frame by $8\times$ , and then apply super-resolution frame by frame for $2\times$ , $4\times$ and $8\times$ , respectively. The computational cost depends on the size of input images since we extract features from the LR space. On the contrary, the speed of SRCNN and VDSR is limited by the size of output images. Both FSRCNN and our approach achieve real-time performance (i.e., over 30 frames per second) on all upsampling scales. In contrast, the FPS is 8.43 for SRCNN and 1.98 for VDSR on $8\times$ SR. Figure 8 visualizes results of $8\times$ SR on one representative frame.

6 Limitations

While our model is capable of generating clean and sharp HR images on a large scale factor, e.g., $8\times$ , it does not “hallucinate” fine details. As shown in Figure 9, the top of the building is significantly blurred in the $8\times$ downscaled LR image. All SR algorithms fail to recover the fine structure except SelfExSR , which explicitly detects the 3D scene geometry and uses self-similarity to hallucinate the regular structure. This is a common limitation shared by parametric SR methods . Another limitation of the proposed network is the relative large model size. To reduce the number of parameters, one can replace the deep convolutional layers at each level with recursive layers.

Conclusions

In this work, we propose a deep convolutional network within a Laplacian pyramid framework for fast and accurate single-image super-resolution. Our model progressively predicts high-frequency residuals in a coarse-to-fine manner. By replacing the pre-defined bicubic interpolation with the learned transposed convolutional layers and optimizing the network with a robust loss function, the proposed LapSRN alleviates issues with undesired artifacts and reduces the computational complexity. Extensive evaluations on benchmark datasets demonstrate that the proposed model performs favorably against the state-of-the-art SR algorithms in terms of visual quality and run time.

Acknowledgments

This work is supported in part by the NSF CAREER Grant $\#1149783$ , gifts from Adobe and Nvidia. J.-B. Huang and N. Ahuja are supported in part by Office of Naval Research under Grant N00014-16-1-2314.