Universal Adversarial Perturbations Against Semantic Image Segmentation

Introduction

While deep learning has led to significant performance increases for numerous visual perceptual tasks and is relatively robust to random noise , several studies have found it to be vulnerable to adversarial perturbations . Adversarial attacks involve generating slightly perturbed versions of the input data that fool the classifier (i.e., change its output) but stay almost imperceptible to the human eye. Adversarial perturbations transfer between different network architectures, and networks trained on disjoint subsets of data . Furthermore, Papernot et al. showed that adversarial examples for a network of unknown architecture can be constructed by training an auxiliary network on similar data and exploiting the transferability of adversarial examples.

Prior work on adversarial examples focuses on the task of image classification. In this paper, we investigate the effect of adversarial attacks on tasks involving a localization component, more specifically: semantic image segmentation. Semantic image segmentation is an important methodology for scene understanding that can be used for example for automated driving, video surveillance, or robotics. With the wide-spread applicability in those domains comes the risk of being confronted with an adversary trying to fool the system. Thus, studying adversarial attacks on semantic segmentation systems deployed in the physical world becomes an important problem.

Adversarial attacks that aim at systems grounded in the physical world should be physically realizable and inconspicuous . One prerequisite for physical realizability is that perturbations do not depend on the specific input since this input is not known in advance when the perturbations (which need to be placed in the physical world) are determined. This work proposes a method for generating image-agnostic universal perturbations. Universal perturbations have been proposed by Moosavi-Dezfooli et al. ; however, we extend the idea to the task of semantic image segmentation. We leave further prerequisites for physical realizability as detailed by Sharif et al. to future work.

An attack is inconspicuous if it does not raise the suspicion of humans monitoring the system (at least not under cursory investigation). This requires that the system inputs are modified only subtly, and, for a semantic image segmentation task, also requires that system output (the scene segmentation) looks mostly as a human would expect for the given scene. If an adversary’s objective is to remove all occurrences of a specific class (e.g., an adversary trying to hide all pedestrians to deceive an emergency braking system) then the attack is maximally inconspicuous if it leaves the prediction for all other classes unchanged and only hides the target class. We present one adversarial attack which explicitly targets this dynamic target segmentation scenario.

While inconspicuous attacks require that target scenes mostly match what a human expects, we also present an attack yielding an static target segmentations. This attack generates universal perturbations that let the system output always essentially the same segmentation regardless of the input, even when the input is from a completely different scene (see Figure 1). The main motivation for this experiment is to show how fragile current approaches for semantic segmentation are when confronted with an adversary. In practice, such attacks could be used in scenarios in which a static camera monitors a scene (for instance in surveillance scenarios) as it would allow an attacker to always output the segmentation of the background scene and blend out all activity like, e.g., burglars robbing a jewelry shop.

We summarize our main contributions as follows:

We show the existence of (targeted) universal perturbations for semantic image segmentation models. Their existence was not clear a priori because the receptive fields of different output elements largely overlap. Thus perturbations cannot be chosen independently for each output target. This makes the space of adversarial perturbations for semantic image segmentation presumably smaller than for recognition tasks like image classification and the existence of universal perturbations even more surprising.

We propose two efficient methods for generating these universal perturbations. These methods optimize the perturbations on a training set. The objective of the first methods is to let the network yield a fixed target segmentation as output. The second method’s objective is to leave the segmentation unchanged except for removing a designated target class.

We show empirically that the generated perturbations are generalizable: they fool unseen validation images with high probability. Controlling the capacity of universal perturbations is important for achieving this generalization from small training sets.

We show that universal perturbations generated for a fixed target segmentation have a local structure that resembles the target scene (see Figure 4).

Background

Let $f_{\theta}$ be a function with parameters $\theta$. Moreover, let $\mathbf{x}$ be an input of $f_{\theta}$, $f_{\theta}(\mathbf{x})$ be the output of $f_{\theta}$, and $\mathbf{y}^{\text{true}}$ be the corresponding ground-truth target. More specifically for the scenario studied in this work, $f_{\theta}$ denotes a deep neural network, $\mathbf{x}$ an image, $f_{\theta}(\mathbf{x})$ the conditional probability $p(\mathbf{y}|\mathbf{x};\theta)$ encoded as a class probability vector, and $\mathbf{y}^{\text{true}}$ a one-hot encoding of the class. Furthermore, let $J_{\text{cls}}(f_{\theta}(\mathbf{x}),\mathbf{y}^{\text{true}})$ be the basic classification loss such as cross-entropy. We assume that $J_{\text{cls}}$ is differentiable with respect to $\theta$ and with respect to $\mathbf{x}$.

Semantic image segmentation denotes a dense prediction task that addresses the “what is where in an image?” question by assigning a class label to each pixel of the image. Recently, deep learning based approaches (oftentimes combined with conditional random fields) have become the dominant and best performing class of methods for this task . In this work, we focus on one of the first and most prominent architectures, the fully convolutional network architecture FCN-8s introduced by Long et al. for the VGG16 model .

The FCN-8s architecture can roughly be divided into two parts: an encoder part which transforms a given image into a low resolution semantic representation and a decoder part which increases the localization accuracy and yields the final semantic segmentation at the resolution of the input image. The encoder part is based on a VGG16 pretrained on ImageNet where the fully connected layers are reinterpreted as convolutions making the network “fully convolutional”. The output of the last encoder layer can be interpreted as a low-resolution semantic representation of the image and is the input to five upsampling layers which recover the high spatial resolution of the image via successive bilinear-interpolation (FCN-32s). For FCN-8s, additionally two parallel paths merge higher-resolution, less abstract layers of the VGG16 into the upsampling path via convolutions and element-wise summation. This enables the network to utilize features with a higher spatial resolution.

2 Adversarial Examples

The first method for generating adversarial examples was proposed by Szegedy et al. . While this method was able to generate adversarial examples successfully for many inputs and networks, it was also relatively slow computationally since it involved an L-BFGS-based optimization. Since then, several methods for generating adversarial examples have been proposed. These methods either maximize the predicted probability for all but the true class or minimize the probability of the true class.

Here, $\varepsilon$ is a hyper-parameter governing the distance between original image and adversarial image. FGSM is a targeted method. This means that the adversary is solely trying to make the predicted probability of the true class smaller. However, it does not control which of the other classes becomes more probable.

Kurakin et al. proposed an extension of FGSM which is iterative and targeted. The proposed least-likely method (LLM) makes the least likely class $\mathbf{y}^{\text{LL}}=\arg\min_{\mathbf{y}}p(\mathbf{y}|\mathbf{x})$ under the prediction of the model more probable. LLM is in principle not specific for the least-likely class $\mathbf{y}^{LL}$; it can rather be used with an arbitrary target class $\mathbf{y}^{\text{target}}$. The method tries to find $\mathbf{x}^{\text{adv}}$ which maximizes the predictive probability of class $y^{\text{target}}$ under $f_{\theta}$. This can be achieved by the following iterative procedure:

Here $\alpha$ denotes a step size and all entries of $\mathbf{\xi}$ are clipped after each iteration such that their absolute value remains smaller than $\varepsilon$. We use $\alpha=1$ throughout all experiments. Concurrent with this work, adversarial examples have been extended to semantic image segmentation and object detection . Moreover, training with adversarial examples has been applied to mammographic mass segmentation to reduce overfitting .

For the methods outlined above, the adversarial perturbation $\mathbf{\xi}$ depends on the input $\mathbf{x}$. Recently, Moosavi-Dezfooli et al. proposed a method for generating universal, image-agnostic perturbations $\mathbf{\Xi}$ that, when added to arbitrary data points, fool deep nets on a large fraction of images. The method for generating these adversarial perturbations is based on the adversarial attack method DeepFool . DeepFool is applied to a set of $m$ images (the train set). These images are presented sequentially in a round-robin manner to DeepFool. For the first image, DeepFool identifies a standard image-dependent perturbation. For subsequent images, it is checked whether adding the previous adversarial perturbation already fools the classifier; if yes the algorithm continues with the next image, otherwise it updates the perturbation using DeepFool such that also the current image becomes adversarial. The algorithm stops once the perturbation is adversarial on a large fraction of the train set.

The authors show impressive results on ImageNet , where they show that the perturbations are adversarial for a large fraction of test images, which the method did not see while generating the perturbation. One potential short-coming of the approach is that the attack is not targeted, i.e., the adversary cannot control which class the classifier shall assign to an adversarial example. Moreover, for high-resolution images and a small train set, the perturbation might overfit the train set and not generalize to unseen test data since the number of “tunable parameters” is proportional to the number of pixels. Thus, high-resolution images will need a large train set and a large computational budget. In this paper, we propose a method which overcomes these shortcomings.

Adversarial Perturbations Against Semantic Image Segmentation

For semantic image segmentation, the loss is a sum over the spatial dimensions $(i,j)\in\mathcal{I}$ of the target such as:

In this section, we describe how to find an input $\mathbf{x}^{\text{adv}}$ for $f_{\theta}$ such that $J_{\text{ss}}(f_{\theta}(\mathbf{x}^{\text{adv}}),\mathbf{y}^{\text{target}})$ becomes minimal, i.e., how an adversary can do quasi-imperceptible changes to the input such that it achieves a desired target segmentation $\mathbf{y}^{\text{target}}$. We start by describing how an adversary can choose $\mathbf{y}^{\text{target}}$.

In principle, an adversary may choose $\mathbf{y}^{\text{target}}$ arbitrarily. Crucially, however, an adversary may not choose $\mathbf{y}^{\text{target}}$ based on $\mathbf{y}^{\text{true}}$ since the ground-truth is also unknown to the adversary. Instead, the adversary may use $\mathbf{y}^{\text{pred}}=f_{\theta}(\mathbf{x})$ as basis as we assume that the adversary has access to $f_{\theta}$.

As motivated in Section 1, typical scenarios involve an adversary whose primary objective is to hide certain kinds of objects such as, e.g., pedestrians. As a secondary objective, an adversary may try to perform attacks that are inconspicuous, i.e., do not call the attention of humans monitoring the system (at least not under cursory investigation) . Thus the input must be modified only subtly. For a semantic image segmentation task, however, it is also required that the output of the system looks mostly as a human would expect for the given scene. This can be achieved, for instance, by keeping $\mathbf{y}^{\text{target}}$ as similar as possible to $\mathbf{y}^{\text{pred}}$ where the primary objective does not apply. We define two different ways of generating the target segmentation:

In this scenario, the adversary defines a fixed segmentation, such as the system’s prediction at a time step $t_{0}$, as target for all subsequent time steps: $\mathbf{y}^{\text{target}}_{t}=\mathbf{y}^{\text{pred}}_{t_{0}}\,\forall t>t_{0}$. This target segmentation is suited for instance in situations where an adversary wants to attack a system based on a static camera and wants to hide suspicious activity in a certain time span $t>t_{0}$ that had not yet started at time $t_{0}$.

Dynamic target segmentation:

In situations involving ego-motion, a static target segmentation is not suited as it would not account for changes in the scene caused by the movement of the camera. In contrast, dynamic target segmentation aims at keeping the network’s segmentation unchanged with the exception of removing certain target classes. Let $o$ be the class of objects the adversary wants to hide, and let $\mathcal{I}_{o}=\left\{(i,j)\mid f_{\theta}(x_{ij})=o\right\}$ and $\mathcal{I}_{bg}=\mathcal{I}\setminus\mathcal{I}_{o}$. We assign $\mathbf{y}^{\text{target}}_{ij}=\mathbf{y}^{\text{pred}}_{ij}$ for all $(i,j)\in\mathcal{I}_{bg}$, and $\mathbf{y}^{\text{target}}_{ij}=\mathbf{y}^{\text{pred}}_{i^{\prime}j^{\prime}}$ for all $(i,j)\in\mathcal{I}_{o}$ with $i^{\prime},j^{\prime}=\operatorname*{arg\,min}\limits_{i^{\prime},j^{\prime}\in\mathcal{I}_{bg}}(i^{\prime}-i)^{2}+(j^{\prime}-j)^{2}$. The latter corresponds to filling the gaps left in the target segmentation by removing elements predicted to be $o$ using a nearest-neighbor heuristic. An illustration of the adversarial target generation is shown in Figure 2.

2 Image-Dependent Perturbations

Before turning to image-agnostic universal perturbations, we first define how an adversary might choose an image-dependent perturbation. Given $\mathbf{y}^{\text{target}}$, we formulate the objective of the adversary as follows:

An alternative formulation which takes into consideration that the primary objective (hiding objects) and the secondary objective (being inconspicuous) are not necessarily equally important can be achieved by a modified version of the loss including a weighting parameter $\omega$:

Here, $\omega=1$ lets the adversary solely focus on removing target-class predictions, $\omega=0$ forces the adversary only to keep the background constant, and $J_{\text{ss}}^{\omega}=0.5J_{\text{ss}}$ for $\omega=0.5$.

An additional issue for $J_{\text{ss}}$ (and $J_{\text{ss}}^{\omega}$) is that there is potentially competition between different target pixels, i.e., the gradient of the loss for $(i_{1},j_{1})$ might point in the opposite direction as the loss gradient for $(i_{2},j_{2})$. Standard classification losses such as the cross entropy in general encourage target predictions which are already correct to become more confident as this reduces the loss. This is not necessarily desirable in face of competition between different targets. The reason for this is that loss gradients for making correct predictions more confident might counteract loss gradients which would make wrong predictions correct. Note that this issue does not exist for adversaries targeted at image classification as there is essentially only a single target output. To address this issue, we set the loss of target pixels which are predicted as the desired target with a confidence above $\tau$ to . Throughout this paper, we use $\tau=0.75$.

3 Universal Perturbations

In this section, we propose a method for generating universal adversarial perturbations $\mathbf{\Xi}$ in the context of semantic segmentation. The general setting is that we generate $\mathbf{\Xi}$ on a set of $m$ training inputs $\mathcal{D}^{\text{train}}=\{(\mathbf{x}^{(k)},\mathbf{y}^{\text{target},k})\}_{k=1}^{m}$, where $\mathbf{y}^{\text{target},k}$ was generated with either of the two methods presented in Section 3.1. We are interested in the generalization of $\mathbf{\Xi}$ to test inputs $\mathbf{x}$ for which it was not optimized and for which no target $\mathbf{y}^{\text{target}}$ exists. This generalization to inputs for which no target exists is required because generating $\mathbf{y}^{\text{target}}$ would require evaluating $f_{\theta}$ which might not be possible at test time or under real-time constraints. We propose the following extension of the attack presented in Section 3.2:

with $\nabla^{\mathcal{D}}(\mathbf{\Xi})=\frac{1}{m}\sum\limits_{k=1}^{m}\nabla_{x}J_{\text{ss}}^{\omega}(f_{\theta}(\mathbf{x}^{(k)}+\mathbf{\Xi}),\mathbf{y}^{\text{target},k})$ being the loss gradient averaged over the entire training data. A potential issue of this approach is overfitting to the training data which would reduce generalization of $\mathbf{\Xi}$ to unseen inputs. Overfitting is actually likely given that $\mathbf{\Xi}$ has the same dimensionality as the input image and is thus high-dimensional. We adopt a relatively simple regularization approach by enforcing $\mathbf{\Xi}$ to be periodic in both spatial dimensions. More specifically, we enforce for all $i,j\in\mathcal{I}$ the constraints $\mathbf{\Xi}_{i,j}=\mathbf{\Xi}_{i+h,j}$ and $\mathbf{\Xi}_{i,j}=\mathbf{\Xi}_{i,j+w}$ for a predefined spatial periodicity $h,w$. This can be achieved by optimizing a proto-perturbation $\mathbf{\hat{\Xi}}$ of size $h\times w$ and tile it to the full $\mathbf{\Xi}$. This results in a gradient averaged over the training data and all tiles:

with $R$, $S$ denoting the number of tiles per dimension and $[r,s]=\{i,j\mid[rh\leq i<(r+1)h]\wedge[sw\leq j<(s+1)w]\}$.

As we will show in Section 4, the quality of the generated universal perturbation depends crucially on the size $m$ of the train set. As our method for generating universal perturbations does not require ground-truth labels, we may in principle use arbitrary large unlabeled data sets. Nevertheless, we also investigate how well universal perturbations can be generated for small $m$ since large $m$ requires considerable computational resources and also more queries to $f_{\theta}$, which might increase monetary costs or the risk of being identified.

Experimental Results

We evaluated the proposed adversarial attacks against semantic image segmentation on the Cityscapes dataset , which consists of $3475$ publicly available labeled RGB images ($2975$ for training and $500$ for validation) with a resolution of $2048\times 1024$ pixels from $44$ different cities. We used the pixel-wise fine annotations covering $19$ frequent classes. For computational reasons, all images and labels were downsampled to a resolution of $1024\times 512$ pixels, where for images a bilinear interpolation and for labels a nearest-neighbor approach was used for down-sampling. We trained the FCN-8s network architecture (see Section 2.1) for semantic image segmentation on the whole training data and achieved a class-wise intersection-over-union (IoU) on the validation data of 64.8%.

We generated the universal perturbations on (subsets of) the training data and evaluated them on unseen validation data. When not noted otherwise, we used $\varepsilon=10$ in the experiments. This value of $\varepsilon$ was also used by Moosavi-Dezfooli et al. and corresponds to a level of noise which is only perceptible for humans at closer inspection. Moreover, we set the number of iterations to $n=60$.

As Cityscapes does not involve static scenes, we evaluated an even more challenging scenario: namely to output a static target scene segmentation which has nothing in common with the actual input scene present in the image. For this, we selected an arbitrary ground-truth segmentation (monchengladbach_000000_026602_gtFine) from Cityscapes as target. We set the number of training images to $m=2975$, which corresponds to the number of images in the Cityscapes train set. Moreover, we used the unweighted loss $J_{ss}$, and did not use periodic tiles, i.e., $h=512,w=1024$. An illustration for this setting on unseen validation images is shown in Figure 3. The adversary achieved the desired target segmentation nearly perfectly when adding the universal perturbation that was generated on the training images. This is even more striking as for a human, the original scene, which has nothing in common with the target scene, remains clearly dominant.

Figure 4 shows an illustration of the generated universal perturbation for $\varepsilon=20$. This perturbation is highly structured and the local structure depends strongly on the target class. When comparing the perturbation with the static target segmentation, it is fairly easy to recognize the structure of the target in the perturbation. For instance, man-made structures such as buildings and fences correspond to mostly horizontal and vertical edges. This property indicates that the adversarial attack might exploit the (generally desirable) robustness of deep networks to contrast changes. This allows low contrast noise structures to have stronger impact than the high-contrast structures in the actual image.

Table 1 shows a quantitative analysis of the success rate for different values of $\varepsilon$. Here, we define the success rate as the categorical accuracy between static target segmentation and predicted segmentation of the network on the adversarial example. The success rate on training and validation data is nearly on par, which shows that overfitting is not an issue even for high-dimensional perturbations. This is probably due to the large number of training images and the consistent target. Unsurprisingly, larger $\varepsilon$ leads to higher success rates. The value $\varepsilon=10$ strikes a good balance between high success rate and being quasi-imperceptible.

Dynamic Target Segmentation

In this experiment, we focused on an adversary which tries to hide all pedestrians (Cityscapes class “person”) in an image while leaving the segmentation unchanged otherwise. When not noted otherwise, we set the number of training images to $m=1700$ (this value corresponds to the number of images containing pedestrians in the Cityscapes train set), the periodic tile size to $h=w=512$ and use $J^{\omega}_{ss}$ with $\omega=0.9999$ as motivated empirically (see Figure 6 and Table 2 and 3). An illustration for this setting on unseen validation images is shown in Figure 5. We note that qualitatively, the adversary succeeds in removing nearly all pedestrian pixels while leaving the background mostly unchanged. However, closer inspection by a human would probably raise suspicion as the predicted segmentation looks relatively inhomogeneous.

For quantifying how well an adversary achieves its primary objective of hiding a target class, we measure which percentage of the pixels that were predicted as pedestrians on the original input are assigned to any of the other classes for the adversarial example (“Pedestrian pixels hidden”). We measure the categorical accuracy on background pixels (i.e., pixels that were not predicted as pedestrians on the original input) between dynamic adversarial target segmentation and the segmentation predicted by the network on the adversarial example (“Background pixels preserved”). This quantifies the secondary objective of being inconspicuous by preserving the background. Note that this comparison does not involve the ground-truth segmentation; we solely measure if the network’s original background segmentation is preserved.

Figure 6 shows how the periodic tile-size and $m$, the number of training images, affects the results of the adversary. In general, more training images and smaller tile-sizes increase the number of hidden pedestrian pixels. This indicates that failures in hiding pedestrian pixels on validation data are mostly due to overfitting to the training data; in fact the adversary succeeds in hiding nearly $100\%$ of all pedestrian pixels on the train set for any combination of number of training images and tile-size (not shown). The number of background pixels preserved typically decreases with increased score on hiding pedestrians. As this is also the case on training images, it is likely an underfitting or optimization issue which could be improved in the future by alternative regularization methods (other than periodic noise) or more sophisticated adversarial attacks. For the presented method and $m=1700$ , a tile-size of $512\times 512$ achieves a good trade-off and is used in the remaining experiments.

Table 2 illustrates the influence of the maximum noise level $\varepsilon$. Values of $\varepsilon$ below $10$ clearly correspond to an underfitting regime as the adversary is not capable of hiding all pedestrian pixels on the train data. For $\varepsilon=10$, failures of the adversary in hiding pedestrian pixels on validation data are mostly due to overfitting (see above). Additional capacity in the perturbation ($\varepsilon=20$) is then used by the adversary to preserve the background even better but does not help in reducing overfitting. The influence of parameter $\omega$, which allows controlling the trade-off between the primary and secondary objective, is investigated in Table 3: the larger $\omega$, the more pedestrian pixels are hidden (but the background is preserved less well). Since the number of pedestrian pixels is considerably smaller than the number of background pixels, setting $\omega$ close to $1$, e.g., $\omega=0.9999$ presents a reasonable trade-off. In contrast, the unweighted loss $J_{ss}$ with no $\omega$ ($\omega=$ no) fails since it focuses too much on preserving the background.

Generalizability

We have tested the effect of the universal perturbation generated for Cityscapes on CamVid (without any fine-tuning on CamVid). An average of 78% of the pixels are transformed to the adversarial target for the static target segmentation. For dynamic target segmentation, an average of 84.5% pedestrian pixels are hidden and 79.6% of the background pixels are preserved. Thus, the perturbations generalize to a similar dataset with only a small decrease in performance. Moreover, we have evaluated the FCN’s static target universal perturbation on a PSPNet . Adding the universal perturbation reduced the IoU between PSPNet’s predictions and the ground truth on Cityscapes from 75.8% to 8.8%. However, the IoU between the prediction and the adversarial target was also only 9.5%. In summary, the universal perturbation generalizes over networks as an untargeted attack but not as a targeted attack.

Conclusion and Outlook

We have proposed a method for generating universal adversarial perturbations that change the semantic segmentation of images in close to arbitrary ways: an adversary can achieve (approximately) the same desired static target segmentation for arbitrary input images that have nothing in common. Moreover, an adversary can blend out certain classes (like pedestrians) almost completely while leaving the rest of the class map nearly unchanged. These results emphasize the necessity of future work to address how machine learning can become more robust against (adversarial) perturbations and how adversarial attacks can be detected . This is especially important in safety- or security-critical applications. On the other hand, the presented method does not directly allow an adversarial attack in the physical world since it requires that the adversary is able to precisely control the digital representation of the scene. While first works have shown that adversarial attacks might be extended to the physical world and deceive face recognition systems , a practical attack against, e.g., an automated driving or surveillance system has not been presented yet. Investigating whether such practical attacks are feasible presents an important direction for future work. Furthermore, investigating whether other architectures for semantic image segmentation are less vulnerable to adversarial perturbations is equally important.

References

Appendix A Supplementary material

In this supplementary material, we show additional higher-resolution illustrations of Figure 3 and Figure 6. We show three examples for a static target segmentation which correspond to the images where the approach worked best (Section B) and worst (Section C), and a randomly selected image (Section D). Similarly, we show three examples for a dynamic target segmentation which correspond to the images where the approach worked best (Section E) and worst (Section F), and a randomly selected image (Section G). Moreover, we also show the predictions of the network on the noise itself in Section H.

Appendix B Static Target Segmentation - Best Example

Appendix C Static Target Segmentation - Worst Example

Appendix D Static Target Segmentation - Random Example

Appendix E Dynamic Target Segmentation - Best Example

Appendix F Dynamic Target Segmentation - Worst Example

Appendix G Dynamic Target Segmentation - Random Example

Appendix H Network Predictions on Universal Perturbations