Adversarial Manipulation of Deep Representations

Introduction

Recent papers have shown that deep neural networks (DNNs) for image classification can be fooled, often using relatively simple methods to generate so-called adversarial images (Fawzi et al., 2015; Goodfellow et al., 2014; Gu & Rigazio, 2014; Nguyen et al., 2015; Szegedy et al., 2014; Tabacof & Valle, 2015). The existence of adversarial images is important, not just because they reveal weaknesses in learned representations and classifiers, but because 1) they provide opportunities to explore fundamental questions about the nature of DNNs, e.g., whether they are inherent in the network structure per se or in the learned models, and 2) such adversarial images might be harnessed to improve learning algorithms that yield better generalization and robustness (Goodfellow et al., 2014; Gu & Rigazio, 2014).

Research on adversarial images to date has focused mainly on disrupting classification, i.e., on algorithms that produce images classified with labels that are patently inconsistent with human perception. Given the large, potentially unbounded regions of feature space associated with a given class label, it may not be surprising that it is easy to disrupt classification. In this paper, in constrast to such label adversaries, we consider a new, somewhat more incidious class of adversarial images, called feature adversaries, which are confused with other images not just in the class label, but in their internal representations as well.

Given a source image, a target (guide) image, and a trained DNN, we find small perturbations to the source image that produce an internal representation that is remarkably similar to that of the guide image, and hence far from that of the source. With this new class of adversarial phenomena we demonstrate that it is possible to fool a DNN to confound almost any image with any other chosen image. We further show that the deep representations of such adversarial images are not outliers per se. Rather, they appear generic, indistinguishable from representations of natural images at multiple layers of a DNN. This phenomena raises questions about DNN representations, as well as the properties of natural images themselves.

Related Work

Several methods for generating adversarial images have appeared in recent years. Nguyen et al. (2015) describe an evolutionary algorithm to generate images comprising 2D patterns that are classified by DNNs as common objects with high confidence (often $99\%$). While interesting, such adversarial images are quite different from the natural images used as training data. Because natural images only occupy a small volume of the space of all possible images, it is not surprising that discriminative DNNs trained on natural images have trouble coping with such out-of-sample data.

In Sec. 4, we show that our new category of adversarial images exhibits qualitatively different properties from those above. In particular, the DNN representations of our adversarial images are very similar to those of natural images. They do not appear unnatural in any obvious way, except for the fact that they remain inconsistent with human perception.

Adversarial Image Generation

Let $I_{s}$ and $I_{g}$ denote the source and guide images. Let $\phi_{k}$ be the mapping from an image to its internal DNN representation at layer $k$. Our goal is to find a new image, $I_{\alpha}$, such that the Euclidian distance between $\phi_{k}(I_{\alpha})$ and $\phi_{k}(I_{g})$ is as small as possible, while $I_{\alpha}$ remains close to the source $I_{s}$. More precisely, $I_{\alpha}$ is defined to be the solution to a constrained optimization problem:

The constraint on the distance between $I_{\alpha}$ and $I_{s}$ is formulated in terms of the $L_{\infty}$ norm to limit the maximum deviation of any single pixel color to $\delta$. The goal is to constrain the degree to which the perturbation is perceptible. While the $L_{\infty}$ norm is not the best available measure of human visual discriminability (e.g., compared to SSIM (Wang et al., 2004)), it is superior to the $L_{2}$ norm often used by others.

Rather than optimizing $\delta$ for each image, we find that a fixed value of $\delta=10$ (out of 255) produces compelling adversarial images with negligible perceptual distortion. Further, it works well with different intermediate layers, different networks and most images. We only set $\delta$ larger when optimizing lower layers, close to the input (e.g., see Fig. 5). As $\delta$ increases distortion becomes perceptible, but there is little or no perceptible trace of the guide image in the distortion. For numerical optimization, we use l-BFGS-b, with the inequality (2) expressed as a box constraint around $I_{s}$.

Figure 1 shows nine adversarial images generated in this way, all using the well-known BVLC Caffe Reference model (Caffenet) (Jia et al., 2014). Each row in Fig. 1 shows a source, a guide, and three adversarial images along with their differences from the corresponding source. The adversarial examples were optimized with different perturbation bounds ($\delta$), and using different layers, namely FC$7$ (fully connected level 7), P$5$ (pooling layer 5), and C3 (convolution layer 3). Inspecting the adversarial images, one can see that larger values of $\delta$ allow more noticeable perturbations. That said, we have found no natural images in which the guide image is perceptible in the adversarial image. Nor is there a significant amount of salient structure readily visible in the difference images.

While the class label was not an explicit factor in the optimization, we find that class labels assigned to adversarial images by the DNN are almost always that of the guide. For example, we took 100 random source-guide pairs of images from Imagenet ILSVRC data (Deng et al., 2009), and applied optimization using layer FC7 of Caffenet, with $\delta=10$. We found that class labels assigned to adversarial images were never equal to those of source images. Instead, in 95% of cases they matched the guide class. This remains true for source images from training, validation, and test ILSVRC data.

We found a similar pattern of behavior with other networks and datasets, including AlexNet (Krizhevsky et al., 2012), GoogleNet (Szegedy et al., 2015), and VGG CNN-S (Chatfield et al., 2014), all trained on the Imagenet ILSVRC dataset. We also used AlexNet trained on the Places205 dataset, and on a hybrid dataset comprising 205 scene classes and 977 classes from ImageNet (Zhou et al., 2014). In all cases, using 100 random source-guide pairs the class labels assigned to the adversarial images do not match the source. Rather, in 97% to 100% of all cases the predicted class label is that of the guide.

Like other approaches to generating adversarial images (e.g., Szegedy et al. (2014)), we find that those generated on one network are usually misclassified by other networks Using the same 100 source-guide pairs with each of the models above, we find that, on average, 54% of adversarial images obtained from one network are misclassified by other networks. That said, they are usually not consistently classified with the same label as the guide on different netowrks.

We next turn to consider internal representations – do they resemble those of the source, the guide, or some combination of the two? One way to probe the internal representations, following Mahendran & Vedaldi (2014), is to invert the mapping, thereby reconstructing images from internal representations at specific layers. The top panel in Fig. 2 shows reconstructed images for a source-guide pair. The Input row displays a source (left), a guide (right) and adervarisal images optimized to match representations at layers FC7, P5 and C3 of Caffenet (middle). Subsequent rows show reconstructions from the internal representations of these five images, again from layers C3, P5 and FC7. Note how lower layers bear more similarity to the source, while higher layers resemble the guide. When optimized using C3, the reconstructions from C3 shows a mixture of source and guide. In almost all cases we find that internal representations begin to mimic the guide at the layer targeted by the optimization. These reconstructions suggest that human perception and the DNN representations of these adversarial images are clearly at odds with one another.

The bottom panel of Fig. 2 depicts FC7 and P5 activation patterns for the source and guide images in Fig. 2, along with those for their corresponding adversarial images. We note that the adversarial activations are sparse and much more closely resemble the guide encoding than the source encoding. The supplementary material includes several more examples of adversarial images, their activation patterns, and reconstructions from intermediate layers.

Experimental Evaluation

We investigate further properties of adversarial images by asking two questions. To what extent do internal representations of adversarial images resemble those of the respective guides, and are the representations unnatural in any obvious way? To answer these questions we focus mainly on Caffenet, with random pairs of source-guide images drawn from the ImageNet ILSVRC datasets.

We first report quantitative measures of proximity between the source, guide, and adversarial image encodings at intermediate layers. Surprisingly, despite the constraint that forces adversarial and source images to remain perceptually indistinguishable, the intermediate representations of the adversarial images are much closer to guides than source images. More interestingly, the adversarial representations are often nearest neighbors of their respective guides. We find this is true for a remarkably wide range of natural images.

For optimizations at layer FC7, we test on a dataset comprising over 20,000 source-guide pairs, sampled from training, test and validation sets of ILSVRC, plus some images from Wikipedia to increase diversity. For layers with higher dimensionality (e.g., P5), for computational expedience, we use a smaller set of 2,000 pairs. Additional details about how images are sampled can be found in the supplementary material. To simplify the exposition in what follows, we use ${\boldsymbol{\mathbf{s}}}$, $\boldsymbol{\mathbf{g}}$ and $\boldsymbol{\mathbf{\alpha}}$ to denote DNN representations of source, guide and adversarial images, whenever there is no confusion about the layer of the representations.

As a means of quantifying the qualitative results in Fig. 2, for a large ensemble of source-guide pairs, all optimized at layer FC7, Fig. 33(a) shows a histogram of the ratio of Euclidean distance between adversarial $\boldsymbol{\mathbf{\alpha}}$ and guide $\boldsymbol{\mathbf{g}}$ in FC7, to the distance between source ${\boldsymbol{\mathbf{s}}}$ and guide $\boldsymbol{\mathbf{g}}$ in FC7. Ratios less than 0.5 indicate that the adversarial FC7 encoding is closer to $\boldsymbol{\mathbf{g}}$ than ${\boldsymbol{\mathbf{s}}}$. While one might think that the $L_{\infty}$ norm constraint on the perturbation will limit the extent to which adversarial encodings can deviate from the source, we find that the optimization fails to reduce the FC7 distance ratio to less than $0.8$ in only $0.1\%$ of pairs when $\delta=5$. Figure 5 below shows that if we relax the $L_{\infty}$ bound on the deviation from the source image, then $\alpha$ is even closer to $\boldsymbol{\mathbf{g}}$, and that adversarial encodings become closer to $\boldsymbol{\mathbf{g}}$ as one goes from low to higher layers of a DNN.

Figure 33(b) compares the FC$7$ distances between $\boldsymbol{\mathbf{\alpha}}$ and $\boldsymbol{\mathbf{g}}$ to the average FC7 distance between representations of all ILSVRC training images from the same class as the guide and their FC7 nearest neighbors (NN). Not only is $\boldsymbol{\mathbf{\alpha}}$ often the 1-NN of $\boldsymbol{\mathbf{g}}$, but the distance between $\boldsymbol{\mathbf{\alpha}}$ and $\boldsymbol{\mathbf{g}}$ is much smaller than the distance between other points and their NN in the same class. Fig. 33(c) shows that the FC7 distance between $\boldsymbol{\mathbf{\alpha}}$ and ${\boldsymbol{\mathbf{s}}}$ is relatively large compared to typical pairwise distances between FC7 encodings of images of the source class. Only $8\%$ of adversarial images (at $\delta=10$) are closer to their source than the average pairwise FC7 distance within the source class.

Intersection and Average Distance to Nearest Neighbors:

Looking at one’s nearest neighbors provides another measure of similarity. It is useful when densities of points changes significantly through feature space, in which case Euclidean distance may be less meaningful. To this end we quantify similarity through rank statistics on near neighbors. We take the average distance to a point’s $K$ NNs as a scalar score for the point. We then rank that point along with all other points of the same label class within the training set. As such, the rank is a non-parametric transformation of average distance, but independant of the unit of distance. We denote the rank of a point $x$ as $r_{K}({x})$; we use $K=3$ below. Since $\boldsymbol{\mathbf{\alpha}}$ is close to $\boldsymbol{\mathbf{g}}$ by construction, we exclude $\boldsymbol{\mathbf{g}}$ when finding NNs for adversarial points $\boldsymbol{\mathbf{\alpha}}$.

Table 1 shows 3NN intersection as well as the difference in rank between adversarial and guide encodings, $\Delta{r_{3}}(\boldsymbol{\mathbf{\alpha}},\boldsymbol{\mathbf{g}})=r_{3}({\boldsymbol{\mathbf{\alpha}}})-r_{3}({\boldsymbol{\mathbf{g}}})$. When $\boldsymbol{\mathbf{\alpha}}$ is close enough to $\boldsymbol{\mathbf{g}}$, we expect the intersection to be high, and rank differences to be small in magnitude. As shown in Table 1, in most cases they share exactly the same 3NN; and in at least $50\%$ of cases their rank is more similar than $90\%$ of data points in that class. These results are for sources and guides taken from the ILSVRC training set. The same statistics are observed for data from test or validation sets.

2 Similarity to Natural representations

Having established that internal representations of adversarial images ($\boldsymbol{\mathbf{\alpha}}$) are close to those of guides ($\boldsymbol{\mathbf{g}}$), we then ask, to what extent are they typical of natural images? That is, in the vicinity of $\boldsymbol{\mathbf{g}}$, is $\boldsymbol{\mathbf{\alpha}}$ an inlier, with the same characteristics as other points in the neighborhood? We answer this question by examining two neighborhood properties: 1) a probabilistic parametric measure giving the log likelihood of a point relative to the local manifold at $\boldsymbol{\mathbf{g}}$; 2) a geometric non-parametric measure inspired by high dimensional outlier detection methods.

For the analysis that follows, let $\mathcal{N}_{{K}}({x})$ denote the set of $K$ NNs of point $x$. Also, let $N_{ref}$ be a set of reference points comprising $15$ random points from $\mathcal{N}_{{20}}({\boldsymbol{\mathbf{g}}})$, and let $N_{c}$ be the remaining “close” NNs of the guide, $N_{c}=\mathcal{N}_{{20}}({\boldsymbol{\mathbf{g}}})\setminus N_{ref}$. Finally, let $N_{f}=\mathcal{N}_{{50}}({\boldsymbol{\mathbf{g}}})\setminus\mathcal{N}_{{40}}({\boldsymbol{\mathbf{g}}})$ be the set of “far” NNs of the guide. The reference set $N_{ref}$ is used for measurement construction, while $\boldsymbol{\mathbf{\alpha}}$, $N_{c}$ and $N_{f}$ are scored relative to $\boldsymbol{\mathbf{g}}$ by the two measures mentioned above. Because we use up to $50$ NNs, for which Euclidean distance might not be meaningful similarity measure for points in a high-dimensional space like P5, we use cosine distance for defining NNs. (The source images used below are the same $20$ used in Sec. 4.1. For expedience, the guide set is a smaller version of that used in Sec. 4.1, comprising three images from each of only $30$ random classes.)

We build a probabilistic subspace model with probabilistic PCA (PPCA) around $\boldsymbol{\mathbf{g}}$ and compare the likelihood of $\boldsymbol{\mathbf{\alpha}}$ to other points. More precisely, PPCA is applied to $N_{ref}$, whose principal space is a secant plane that has approximately the same normal direction as the tangent plane, but generally does not pass through $\boldsymbol{\mathbf{g}}$ because of the curvature of the manifold. We correct this small offset by shifting the plane to pass through $\boldsymbol{\mathbf{g}}$; with PPCA this is achieved by moving the mean of the high-dimensional Gaussian to $\boldsymbol{\mathbf{g}}$. We then evaluate the log likelihood of points under the model, relative to the log likelihood of $\boldsymbol{\mathbf{g}}$, denoted $\Delta L({\cdot},{\boldsymbol{\mathbf{g}}})=L({\cdot})-L({\boldsymbol{\mathbf{g}}})$. We repeat this measurement for a large number of guide and source pairs, and compare the distribution of $\Delta L$ for $\boldsymbol{\mathbf{\alpha}}$ with points in $N_{c}$ and $N_{f}$.

For guide images sampled from ILSVRC training and validation sets, results for FC7 and P5 are shown in the first two columns of Fig. 4. Since the Gaussian is centred at $\boldsymbol{\mathbf{g}}$, $\Delta L$ is bounded above by zero. The plots show that $\boldsymbol{\mathbf{\alpha}}$ is well explained locally by the manifold tangent plane. Comparing $\boldsymbol{\mathbf{\alpha}}$ obtained when $\boldsymbol{\mathbf{g}}$ is sampled from training or validation sets (Fig. 4(a) vs 4(b), 4(d) vs 4(e)), we observe patterns very similar to those in plots of the log likelihood under the local subspace models. This suggests that the phenomenon of adversarial perturbation in Eqn. (1) is an intrinsic property of the representation itself, rather than the generalization of the model.

Angular Consistency Measure:

If the NNs of $\boldsymbol{\mathbf{g}}$ are sparse in the high-dimensional feature space, or the manifold has high curvature, a linear Gaussian model will be a poor fit. So we consider a way to test whether $\boldsymbol{\mathbf{\alpha}}$ is an inlier in the vicinity of $\boldsymbol{\mathbf{g}}$ that does not rely on a manifold assumption. We take a set of reference points near a $\boldsymbol{\mathbf{g}}$, $N_{ref}$, and measure directions from $\boldsymbol{\mathbf{g}}$ to each point. We then compare the directions from $\boldsymbol{\mathbf{g}}$ with those from $\boldsymbol{\mathbf{\alpha}}$ and other nearby points, e.g., in $N_{c}$ or $N_{f}$, to see whether $\boldsymbol{\mathbf{\alpha}}$ is similar to other points around $\boldsymbol{\mathbf{g}}$ in terms of angular consistency. Compared to points within the local manifold, a point far from the manifold will tend to exhibit a narrower range of directions to others points in the manifold. Specifically, given reference set $N_{ref}$, with cardinality $k$, and with $z$ being $\boldsymbol{\mathbf{\alpha}}$ or a point from $N_{c}$ or $N_{f}$, our angular consistency measure is defined as

Fig. 4(c) and 4(f) show histograms of $\Omega(\boldsymbol{\mathbf{\alpha}},\boldsymbol{\mathbf{g}})$ compared to $\Omega(n_{c},\boldsymbol{\mathbf{g}})$ where $n_{c}\in N_{c}$ and $\Omega(n_{f},\boldsymbol{\mathbf{g}})$ where $n_{f}\in N_{f}$. Note that maximum angular consistency is $1$, in which case the point behaves like $\boldsymbol{\mathbf{g}}$. Other than differences in scaling and upper bound, the angular consistency plots 4(c) and 4(f) are strikingly similar to those for the likelihood comparisons in the first two columns of Fig. 4, supporting the conclusion that $\boldsymbol{\mathbf{\alpha}}$ is an inlier with respect to representations of natural images.

3 Comparisons and analysis

We now compare our feature adversaries to images created to optimize mis-classification (Szegedy et al., 2014), in part to illustrate qualitative differences. We also investigate if the linearity hypothesis for mis-classification adversaries of Goodfellow et al. (2014) is consistent with and explains with our class of adversarial examples. We hereby refer to our results as feature adversaries via optimization (feature-opt). The adversarial images designed to trigger mis-classification via optimization (Szegedy et al., 2014), described briefly in Sec. 2, are referred to as label adversaries via optimization (label-opt).

To demonstrate that label-opt differs qualitatively from feature-opt, we report three empirical results. First, we rank $\boldsymbol{\mathbf{\alpha}}$, $\boldsymbol{\mathbf{g}}$, and other points assigned the same class label as $\boldsymbol{\mathbf{g}}$, according to their average distance to three nearest neighbours, as in Sec. 4.1. Fig. 5 shows rank of $\boldsymbol{\mathbf{\alpha}}$ versus rank of its nearest neighbor-$n_{1}(\boldsymbol{\mathbf{\alpha}})$ for the two types of adversaries. Unlike feature-opt, for label-opt, the rank of $\boldsymbol{\mathbf{\alpha}}$ does not correlate well with the rank of $n_{1}(\boldsymbol{\mathbf{\alpha}})$. In other words, for feature-opt $\boldsymbol{\mathbf{\alpha}}$ is close to $n_{1}(\boldsymbol{\mathbf{\alpha}})$, while for label-opt it is not.

Second, we use the manifold PPCA approach in Sec. 4.2. Comparing to peaked histogram of standardized likelihood of feature-opt shown in Fig. 4, Fig. 5 shows that label-opt examples are not represented well by the Gaussian around the first NN of $\boldsymbol{\mathbf{\alpha}}$.

Third, we analyze the sparsity patterns on different DNN layers for different adversarial construction methods. It is well known that DNNs with ReLU activation units produce sparse activations (Glorot et al. (2011)). Therefore, if the degree of sparsity increases after the adversarial perturbation, the adversarial example is using additional paths to manipulate the resulting represenation. We also investigate how many activated units are shared between the source and the adversary, by computing the intersection over union $I/U$ of active units. If the $I/U$ is high on all layers, then two represenations share most active paths. On the other hand, if $I/U$ is low, while the degree of sparsity remains the same, then the adversary must have closed some activation paths and opened new ones. In Table 2, $\Delta S$ is the difference between the proportion of non-zero activations on selected layers between the source image represenation for the two types of adversaries. One can see that for all except FC$7$ of label-opt, the difference is significant. The column “$I/U$ with ${\boldsymbol{\mathbf{s}}}$” also shows that feature-opt uses very different activation paths from ${\boldsymbol{\mathbf{s}}}$ when compared to label-opt.

Testing The Linearity Hypothesis for feature-opt:

Goodfellow et al. (2014) suggests that the existence of label adversaries is a consequence of networks being too linear. If this linearity hypothesis applies to our class of adversaries, it should be possible to linearize the DNN around the source image, and then obtain similar adversaries via optimization. Formally, let $J_{s}=J(\phi(I_{s}))$ be the Jacobian matrix of the internal layer encoding with respect to source image input. Then, the linearity hypothesis implies $\phi(I)\approx\phi(I_{s})+J_{s}^{\top}(I-I_{s})$. Hence, we optimize $\|\phi(I_{s})+J_{s}^{\top}(I-I_{s})-\phi(I_{g})\|^{2}_{2}$ subject to the same infinity norm constraint in Eqn. 2. We refer to these adversaries as feature-linear.

As shown in Fig. 5, such adversaries do not get particularly close to the guide. They get no closer than 80%, while for feature-opt the distance is reduced to $50\%$ or less for layers down to C2. Note that unlike feature-opt, the objective of feature-linear does not guarantee a reduction in distance when the constraint on $\delta$ is relaxed. These results suggest that the linearity hypothesis may not explain the existence of feature-opt adversaries.

Networks with Random Weights:

We further explored whether the existence of feature-opt adversaries is due to the learning algorithm and the training set, or to the structure of deep networks per se. For this purpose, we randomly initialized layers of Caffenet with orthonormal weights. We then optimized for adversarial images as above, and looked at distance ratios (as in Fig. 3). Interestingly, the distance ratios for FC$7$ and Norm$2$ are similar to Fig. 5 with at most $2\%$ deviation. On C$2$, the results are at most $10\%$ greater than those on C$2$ for the trained Caffenet. We note that both Norm$2$ and C$2$ are overcomplete representations of the input. The table of distance ratios can be found in the Supplementary Material. These results with random networks suggest that the existence of feature-opt adversaries may be a property of the network architecture.

Discussion

We introduce a new method for generating adversarial images that appear perceptually similar to a given source image, but whose deep representations mimic the characteristics of natural guide images. Indeed, the adversarial images have representations at intermediate layers appear quite natural and very much like the guide images used in their construction. We demonstrate empirically that these imposters capture the generic nature of their guides at different levels of deep representations. This includes their proximity to the guide, and their locations in high density regions of the feature space. We show further that such properties are not shared by other categories of adversarial images.

We also find that the linearity hypothesis (Goodfellow et al., 2014) does not provide an obvious explanation for these new adversarial phenomena. It appears that the existence of these adversarial images is not predicated on a network trained with natural images per se. For example, results on random networks indicate that the structure of the network itself may be one significant factor. Nevertheless, further experiments and analysis are required to determine the true underlying reasons for this discrepancy between human and DNN representations of images.

Another future direction concerns the exploration of failure cases we observed in optimizing feature adversaries. As mentioned in supplementary material, such cases involve images of hand-written digits, and networks that are fine-tuned with images from a narrow domain (e.g., the Flicker Style dataset). Such failures suggest that our adversarial phenomena may be due to factors such as network depth, receptive field size, or the class of natural images used. Since our aim here was to analyze the representation of well-known networks, we leave the exploration of these factors to future work. Another interesting question concerns whether existing discriminative models might be trained to detect feature adversaries. Since training such models requires a diverse and relatively large dataset of adversarial images we also leave this to future work.

Financial support for this research was provided, in part, by MITACS, NSERC Canada, and the Canadian Institute for Advanced Research (CIFAR). We would like to thank Foteini Agrafioti for her support. We would also like to thank Ian Goodfellow, Xavier Boix, as well as the anoynomous reviewers for helpful feedback.

References

Supplementary Material

Fig. S1 illustrates the achieved goal in this paper. The image of the fancy car on the left is a training example from the ILSVRC dataset. On the right of it, there is an adversarial image that was generated by guiding the source image by an image of Max (the dog). While the two fancy car images are very close in image space, the activation pattern of the adversarial car is almost identical to that of Max. This shows that the mapping from the image space to the representation space is such that for each natural image, there exists a point in a small neighborhood in the image space that is mapped by the network to a point in the representation space that is in a small neighborhood of the representation of a very different natural image.

S2 Datasets for Empirical Analysis

Unless stated otherwise, we have used the following two sets of source and guide images. The first set is used for experiments on layer FC$7$ and the second set is used for computational expedience on other layers (e.g. P$5$). The source images are guided by all guide images to show that the convergence does not depend on the class of images. To simplify the reporting of classification behavior, we only used guides from training set whose labels are correctly predicted by Caffenet.

In both sets we used $20$ source images, with five drawn at random from each of the ILSVRC train, test and validation sets, and five more selected manually from Wikipedia and the ILSVRC validation set to provide greater diversity. The guide set for the first set consisted of three images from each of $1000$ classes, drawn at random from ILSVRC training images, and another $30$ images from each of the validation and test sets. For the second set, we drew guide images from just $100$ classes.

S3 Examples of Adversaries

Fig. S2 shows a random sample of source and guide pairs along with their FC$7$ or Pool$5$ adversarial images. In none of the images the guide is perceptable in the adversary, regardless of the choice of source, guide or layer. The only parameter that affects the visibility of the noise is $\delta$.

S4 Dimensionality of Representations

The main focus of this study is on the well-known Caffenet model. The layer names of this model and their representation dimensionalities are provided in Tab. S1.

S5 Results for Networks with Random Weights

As described in Sec. 4.3, we attempt at analyzing the architecture of Caffenet independent of the training by initializing the model with random weights and generating feature adversaries. Results in Tab. S2 show that we can generate feature adversaries on random networks as well. We use the ratio of distances of the adversary to the guide over the source to the guide for this analysis. In each cell, the mean and standard deviation of this ratio is shown for each of the three random, orthonormal random and trained Caffenet networks. The weights of the random network are drawn from the same distribution that Caffenet is initialized with. Orthorgonal random weights are obtained using singular value decomposition of the regular random weights.

Results in Tab. S2 indicate that convergence on Norm$2$ and Conv$2$ is almost similar while the dimensionality of Norm$2$ is quite smaller than Conv$2$. On the other hand, Fig. 5 shows that although Norm$2$ has smaller dimensionality than Conv$3$, the optimization converges to a closer point on Conv$3$ rather than Conv$2$ and hence Norm$2$. This means that the relation between dimensionality and the achieved distance of the adversary is not straightforward.

S6 Adversaries by Fast Gradient

As we discussed in Sec. 4.3, Goodfellow et al. (2014) also proposed a method to construct label adversaries efficiently by taking a small step consistent with the gradient. While this fast gradient method shines light on the label adversary misclassifications, and is useful for adversarial training, it is not relevant to whether the linearity hypothesis explains the feature adversaries. Therefore we omitted the comparison in Sec. 4.3 to fast gradient method, and continue the discussion here.

The same experimental setup as in Sec. 4.3 is used here. In Fig. S3, we show the nearest neighbor rank analysis and manifold analysis as done in Sec. 4.2 and Sec. 4.3. Moreover, Figs. 3(a)-3(b) in compare to Figs. 4(a)-4(b) from feature-opt results and Fig. 5 from label-opt results indicates that this adversaries are not represented as well as feature-opt by a Gaussian around the NN of the adversary too. Also, Figs. 3(c)-3(d) in compare to Fig. 5 show the obvious difference in adversarial distribution for the same set of source and guide.

S7 Failure Cases

There are cases in which our optimization was not successful in generating good adversaries. We observed that for low resolution images or hand-drawn characters, the method does not always work well. It was successful on LeNet with some images from MNIST or CIFAR10, but for other cases we found it necessary to relax the magnitude bound on the perturbations to the point that traces of guide images were perceptible. With Caffenet, pre-trained on ImageNet and then fine-tuned on the Flickr Style dataset, we could readily generate adversarial images using FC$8$ in the optimization (i.e., the unnormalized class scores), however, with FC$7$ the optimization often terminated without producing adversaries close to guide images. One possible cause may be that the fine-tuning distorts the original natural image representation to benefit style classification. As a consequence, the FC$7$ layer no longer gives a good generic image represenation, and Euclidean distance on FC$7$ is no longer useful for the loss function.

S8 More Examples with Activation Patterns

Finally, we dedicate the remaining pages to several pairs of source and guide along with their adversaries, activation patterns and inverted images as a complementary to Fig. 2. Figs. S4, S5, S6, S7 and S8 all have similar setup as it is discussed in Sec. 3.