Radioactive data: tracing through training

Introduction

The availability of large-scale public datasets has accelerated the development of machine learning. The Imagenet collection (Deng et al., 2009) and challenge (Russakovsky et al., 2015) contributed to the success of the deep learning architectures (Krizhevsky et al., 2012). The annotation of precise instance segmentation on the large-scale COCO dataset (Lin et al., 2014) enabled large improvements of object detectors and instance segmentation models (He et al., 2017). Even in weakly-supervised (Joulin et al., 2016; Mahajan et al., 2018) and unsupervised learning (Caron et al., 2019) where annotations are scarcer, state-of-the-art results are obtained on large-scale datasets collected from the Web (Thomee et al., 2015).

Machine learning and deep learning models are trained to solve specific tasks (e.g. classification, segmentation), but as a side-effect reproduce the bias in the datasets (Torralba et al., 2011). Such a bias is a weak signal that a particular dataset has been used to solve a task. Our objective in this paper is to enable the traceability for datasets. By introducing a specific mark in a dataset, we want to provide a strong signal that a dataset has been used to train a model.

We thus slightly change the dataset, effectively substituting the data for similar-looking marked data (isotopes).

Let us assume that this data, as well as other collected data, is used to train a convolutional neural network (convnet). After training, the model is inspected to assess the use of radioactive data. The convnet is accessed either (1) explicitly when the model and corresponding weights are available (white-box setting), or (2) implicitly if only the decision scores are accessible (black-box setting). From that information, we answer the question of whether any radioactive data has been used to train the model, or if only vanilla data was used. We want to provide a statistical guarantee with the answer, in the form of a $p$-value.

Passive techniques such as those employed to measure dataset bias (Torralba et al., 2011) or to do membership inference (Sablayrolles et al., 2019; Shokri et al., 2017) cannot provide sufficient empirical or statistical guarantees. More importantly, their measurement is relatively weak and therefore cannot be considered as an evidence: they are likely to confuse datasets having the same underlying statistics. In contrast, we target a $p$-value much below $0.1\%$, meaning there is a very low probability that the results we observe are obtained by chance.

Therefore, we focus on active techniques, where we apply visually imperceptible changes to the images. We consider the following three criteria: (1) The change should be tiny, as measured by an image quality metric like PSNR (Peak Signal to Noise Ratio); (2) The technique should be reasonably neutral with respect to the end-task, i.e., the accuracy of the model trained with the marked dataset should not be significantly modified; (3) The method should not be detectable by a visual analysis of failure cases and should be immune to a re-annotation of the dataset. This disqualifies techniques that employ incorrect labels as a mark, which are easy to detect by a simple analysis of the failure cases. Similarly the “backdoor” techniques are easy to identify and circumvent with outlier detection (Tran et al., 2018).

At this point, one may draw the analogy between this problem and watermarking (Cox et al., 2002), whose goal is to imprint a mark into an image such that it can be re-identified with high probability. We point out that traditional image-based watermarking is ineffective in our context: the learning procedure ignores the watermarks if they are not useful to guide the classification decision (Tishby et al., 2000). Therefore regular watermarking leaves no exploitable trace after training. We need to force the network to keep the mark through the learning process, whatever the learning procedure or architecture.

To that goal, we propose radioactive data. As illustrated in Figure 1 and similarly to radioactive markers in medical applications, we introduce marks (data isotopes) that remain through the learning process and that are detectable with high confidence in a neural network. Our idea is to craft a class-specific additive mark in the latent space before the classification layer. This mark is propagated back to the pixels with a marking (pretrained) network.

This behaviour is confirmed by an analysis of the latent space before classification. It shows that the network devotes a small part of its capacity to keep track of our “radioactive tracers”.

Our experiments on Imagenet confirm that our radioactive marking technique is effective: with almost invisible changes to the images ($\text{PSNR}=42~{}\text{dB}$), and when marking only a fraction of the images ($q=1\%$), we are able to detect the use of our radioactive images with very strong confidence. Note that our radioactive marks, while visually imperceptible, might be detected by a statistical analysis of the latent space of the network. Our aim in this paper is to provide a proof of concept that marking data is possible with statistical guarantees, and the analysis of defense mechanisms lies outside the scope of this paper. The deep learning community has developed a variety of defense mechanisms against “adversarial attacks”: these techniques prevent test-time tampering, but are not designed to prevent training-time attacks on neural networks.

Our conclusions are supported in various settings: we consider both the black-box and white-box settings; we change the tested architecture such that it differs from the one employed to insert the mark. We also depart from the common restrictions of many data-poisoning works (Shafahi et al., 2018; Biggio et al., 2012), where only the logistic layer is retrained, and which consider small datasets (CIFAR) and/or limited data augmentation. We verify that the radioactive mark holds when the network is trained from scratch on a radioactive Imagenet dataset with standard random data augmentations. As an example, for a ResNet-18 trained from scratch, we achieve a $p$-value of $10^{-4}$ when only $1\%$ of the training data is radioactive. The accuracy of the network is not noticeably changed ($\pm 0.1\%$).

The paper is organized as follows. Section 2 reviews the related literature. We discuss related works in watermarking, and explain how the problem that we tackle is related to and differs from data poisoning. In Section 3, after introducing a few mathematical notions, we describe how we add markers, and discuss the detection methods in both the white-box and black-box settings. Section 4 provides an analysis of the latent space learned with our procedure and compares it to the original one. We present qualitative and quantitative results in different settings in the experimental section 5. We conclude the paper in Section 6.

Related work

is a way of tracking media content by adding a mark to it. In its simplest form, a watermark is an addition in the pixel space of an image, that is not visually perceptible. Zero-bit watermarking techniques (Cayre et al., 2005) modify the pixels of an image so that its Fourier transform lies in the cone generated by an arbitrary random direction, the “carrier”. When the same image or a slightly perturbed version of it are encountered, the presence of the watermark is assessed by verifying whether the Fourier representation lies in the cone generated by the carrier. Zero-bit watermarking detects whether an image is marked or not, but in general watermarking also considers the case where the marks carry a number of bits of information (Cox et al., 2002).

Traditional watermarking is notoriously not robust to geometrical attacks (Vukotić et al., 2018). In contrast, the latent space associated with deep networks is almost invariant to such transformations, due to the train-time data augmentations. This observation has motivated several authors to employ convnets to watermark images (Vukotić et al., 2018; Zhu et al., 2018) by inserting marks in this latent space. HiDDeN (Zhu et al., 2018) is an example of these approaches, applied either for steganographic or watermarking purposes.

Adversarial examples.

Privacy and membership inference.

Differential privacy (Dwork et al., 2006) protects the privacy of training data by bounding the impact that an element of the training set has on a trained model. The privacy budget $\epsilon>0$ limits the impact that the substitution of one training example can have on the log-likelihood of the estimated parameter vector. It has become the standard for privacy in the industry and the privacy budget $\epsilon$ trades off between learning statistical facts and hiding the presence of individual records in the training set. Recent work (Abadi et al., 2016; Papernot et al., 2018) has shown that it is possible to learn deep models with differential privacy on small datasets (MNIST, SVHN) with a budget as small as $\epsilon=1$. Individual privacy degrades gracefully to group privacy: when testing for the joint presence of a group of $k$ samples in the training set of a model, an $\epsilon$-private algorithm provides guarantees of $k\epsilon$.

Membership inference (Shokri et al., 2017; Carlini et al., 2018; Sablayrolles et al., 2019) is the reciprocal operation of differentially private learning. It predicts from a trained model and a sample, whether the sample was part of the model’s training set. These classification approaches do not provide any guarantee: if a membership inference model predicts that an image belongs to the training set, it does not give a level of statistical significance. Furthermore, these techniques require training multiple models to simulate datasets with and without an image, which is computationally intensive.

Data poisoning

(Biggio et al., 2012; Steinhardt et al., 2017; Shafahi et al., 2018) studies how modifying training data points affects a model’s behavior at inference time. Backdoor attacks (Chen et al., 2017; Gu et al., 2017) are a recent trend in machine learning attacks. They choose a class $c$, and add unrelated samples from other classes to this class $c$, along with an overlayed “trigger” pattern; at test time, any sample having the same trigger will be classified in this class $c$. Backdoor techniques bear similarity with our radioactive tracers, in particular their trigger is close to our carrier. However, our method differs in two main aspects. First we do “clean-label” attacks, i.e., we perturb training points without changing their labels. Second, we provide statistical guarantees in the form of a $p$-value.

Watermarking deep learning models.

A few works (Adi et al., 2018; Yeom et al., 2018) focus on watermarking deep learning models: these works modify the parameters of a neural network so that any downstream use of the network can be verified. Our assumption is different: in our case, we control the training data, but the training process is not controlled.

Our method

In this section, we describe our method for marking data. It consists of three stages: the marking stage where the radioactive mark is added to the vanilla training images, without changing their labels. The training stage uses vanilla and/or marked images to train a multi-class classifier using regular learning algorithms. Finally, in the detection stage, we examine the model to determine whether marked data was used or not.

Given a fixed vector $v$ and a random vector $u$ distributed uniformly over the unit sphere in dimension $d$ ($\|u\|_{2}=1$), we are interested in the distribution of their cosine similarity $c(u,v)=u^{T}v/(\|u\|_{2}\|v\|_{2})$. A classic result from statistics (Iscen et al., 2017) shows that this cosine similarity follows an incomplete beta distribution with parameters $a=\frac{1}{2}$ and $b=\frac{d-1}{2}$:

In particular, it has expectation and variance $1/d$.

Combination of p𝑝p-values.

Fisher’s method (Fisher, 1925) enables to combine $p$-values of multiple tests. We consider statistical tests $T_{1},\dots,T_{k}$, independent under the null hypothesis $\mathcal{H}_{0}$. Under $\mathcal{H}_{0}$, the corresponding $p$-values $p_{1},\dots,p_{k}$ are distributed uniformly in $$. Hence$-\log(p_{i})$follows an exponential distribution, which corresponds to a$\chi^{2}$distribution with two degrees of freedom. The quantity$Z=-2\sum_{i=1}^{k}\log(p_{i})$thus follows a$\chi^{2}$distribution with$2k$degrees of freedom. The combined$p$-value of tests$T_{1},\dots,T_{k}$is thus the probability that the random variable$Z$ has a value higher than the threshold we observe.

2 Additive marks in feature space

If radioactive data is used at training time, the linear classifier of the corresponding class $w$ is updated with weighted sums of $\phi(x)+\alpha u$, where $\alpha$ is the strength of the mark. The linear classifier $w$ is thus likely to have a positive dot product with the direction $u$, as shown in Figure 2.

At detection time, we examine the linear classifier $w$ to determine if $w$ was trained on radioactive or vanilla data. We test the statistical hypothesis $\mathcal{H}_{1}$: “$w$ was trained using radioactive data” against the null hypothesis $\mathcal{H}_{0}$: “$w$ was trained using vanilla data”. Under the null hypothesis $\mathcal{H}_{0}$, $u$ is a random vector independent of $w$. Their cosine similarity $c(u,w)$ follows the beta-incomplete distribution with parameters $a=\frac{1}{2}$ and $b=\frac{d-1}{2}$. Under hypothesis $\mathcal{H}_{1}$, the classifier vector $w$ is more aligned with the direction $u$ so and $c(u,w)$ is likely to be higher.

Thus if we observe a high value of $c(u,w)$, its corresponding $p$-value (the probability of it happening under the null hypothesis $\mathcal{H}_{0}$) is low, and we can conclude with high significance that radioactive data has been used.

The extension to $C$ classes follows. In the marking stage we sample i.i.d. random directions $(u_{i})_{i=1..C}$ and add them to the features of images of class $i$. At detection time, under the null hypothesis, the cosine similarities $c(u_{i},w_{i})$ are independent (since $u_{i}$ are independent) and we can thus combine the $p$ values for each class using Fisher’s combined probability test (Section 3.1) to obtain the $p$ value for the whole dataset.

3 Image-space perturbations

We now assume that we have a fixed known feature extractor $\phi$. At marking time, we wish to modify pixels of image $x$ such that the features $\phi(x)$ move in the direction $u$. We can achieve this by backpropagating gradients in the image space. This setup is very similar to adversarial examples (Goodfellow et al., 2015; Szegedy et al., 2014). More precisely, we optimize over the pixel space by running the following optimization program:

where the radius $R$ is a hard upper bound on the change of color levels of the image that we can accept. The loss is a combination of three terms:

The first term encourages the features to align with $u$, the two other terms penalize the $L_{2}$ distance in both pixel and feature space. In practice, we optimize this objective by running SGD with a constant learning rate in the pixel space, projecting back into the $L_{\infty}$ ball at each step and rounding to integral pixel values every $T=10$ iterations.

This procedure is a generalization of classical watermarking in the Fourier space. In that case the “feature extractor” is invertible via the inverse Fourier transform, so the marking does not need to be iterative.

Figure 3 shows examples of radioactive images and their vanilla version. We can see that the radioactive mark is not visible to the naked eye, except when we amplify it for visualization purposes (last column).

4 White-box test with subspace alignment

In practice, we use vanilla images of a held-out set (the validation set) to do the estimation.

The classifier we manipulate at detection time is thus $W\phi_{t}(x)\approx WM\phi_{0}(x)$. The lines of $WM$ form classification vectors aligned with the output space of $\phi_{0}$, and we can compare these vectors to $u_{i}$ in cosine similarity. Under the null hypothesis, $u_{i}$ are random vectors independent of $\phi_{0}$, $\phi_{t}$, $W$ and $M$ and thus the cosine similarity is still given by the beta incomplete function, and we can apply the techniques of subsection 3.2.

5 Black-box test

Analysis of the latent feature space

In this section, we analyze how the classifier learned on a radioactive dataset is related to (1) a classifier learned on unmarked images ; and (2) the direction of the carrier. For the sake of analysis, we take the simplest case where the mark is added in the latent feature space just before the classification layer, and we assume that only the logistic regression has been re-trained.

For a given class, we analyze how the classifier learned with a mark is explained by

the “semantic” space, that is the classifier learned by a vanilla classifier. This is a 1-dimensional subspace identified by a vector $w^{*}$;

the direction of the carrier, favored by the insertion of our class-specific mark. We denote it by $u$.

the noise space $\mathcal{F}$, which is in direct sum with the span of vectors $w^{*}$ and $u$ of the previous space. This noise space is due to the randomness of the initialization and the optimization procedure (SGD and random data augmentations).

The rationale of performing this decomposition is to quantify, with respect to the norm of the vector, what is the dominant subspace depending on the fraction of marked data.

This decomposition is analyzed in Figure 4, where we make two important observations. First, the 2-dimensional subspace contains most of the projection of the new vector, which can be seen by the fact that the norm of the vector projected onto that subspace is close to 1 (which translates visually as to be close to the unit circle). Second and unsurprisingly, the contribution of the semantic vector is significant and still dominant compared to the mark, even when most of the dataset is marked. This property explains why our procedure has only a little impact on the accuracy.

Figure 5 shows the histograms of cosine similarities between the classifiers and random directions, the mark direction and the semantic direction. We can see that the classifiers are well aligned with the mark when $q=20\%$ or $2\%$ of the data is marked.

Experiments

In order to provide a comparison on the widely-used vision benchmarks, we use Imagenet (Deng et al., 2009), a dataset of natural images with 1,281,167 images belonging to $1,000$ classes. We first consider the Resnet-18 and Resnet-50 models (He et al., 2016). We perform training using the standard set of data augmentations from Pytorch (Paszke et al., 2017). We train with SGD with a momentum of $0.9$ and a weight decay of $10^{-4}$ for $90$ epochs, using a batch size of $2048$ across $8$ GPUs. We use Pytorch (Paszke et al., 2017) and adopt its standard data augmentation settings (random crop resized to $224\times 224$). We use the waterfall learning rate schedule: the learning starts at $0.8$, (as recommended in (Goyal et al., 2017)) and is divided by $10$ every $30$ epochs. On a vanilla Imagenet, we obtain a top 1 accuracy of $69.6\%$ and a top-5 accuracy of $89.1\%$ with our Resnet18. We ran experiments by varying the random initialization and the order of elements seen during SGD, and found that the top 1 accuracy varies by $0.1\%$ from one experiment to the other.

2 Experimental setup and metrics

We modify Imagenet images by inserting our radioactive mark, and retrain models on this radioactive data using the learning algorithm described above. We then analyze these “contaminated” models for the presence of our mark. We report several measures of performance. On the images, we report the PSNR, i.e. the magnitude of the perturbation necessary to add the radioactive mark. On the model, we report the $p$-value that measures how confident we are that radioactive data was used to train the model, as well as the accuracy of this model on vanilla (held-out) data. We conduct experiments where we only mark a fraction $q$ of the data, with $q\in\{0.01,0.02,0.05,0.1,0.2\}$.

As a sanity check, we ran our radioactive detector on pretrained models of the Pytorch zoo and found $p$ values of $15\%$ for Resnet-18 and $51\%$ for Resnet-50, which is reasonable: in the absence of radioactive data, these values should be uniformly distributed between and $1$.

3 Preliminary experiment: comparison to the backdoor technique

We experimented with the backdoor technique of Chen et al. (Chen et al., 2017) in the context of our marking problem. In general, the backdoor technique adds unrelated images to a class, plus a “trigger” that is consistent across these added images. In their work, Chen et al. need to poison approximately $10\%$ of the data in a class to activate their trigger. We adapted their technique to the “clean-label” setup on Imagenet: we blend a trigger (a Gaussian pattern) to images of a class. We observed that it is possible to detect this trigger at train time, albeit with a low image quality (PSNR $<30$dB) that is visually perceptible. In this case, the model is more confident on images that have the trigger than on vanilla images in about $90\%$ of the cases. However, we also observed that any Gaussian noise activates the trigger: hence we have no guarantee that images with our particular mark were used.

4 Results

We first analyze the results in Table 1 of a ResNet-18 model with fixed features trained on Imagenet. We can see that we are able to detect that our model was trained on radioactive data with a very high confidence for both center crop and random crop. The model overfits more on the center crop, hence it learns more the radioactive mark, which is why the $p$-value is lower on center crop images. Conversely on random crops, marking data has less impact on the accuracy of the model ($-0.24$ as opposed to $-0.48$ for $q=1\%$ marked data).

Table 2 shows the results of retraining a Resnet-18 from scratch on radioactive data. The results confirm that our watermark can be detected when only $q=1\%$ of the data is used at train time. This setup is more complicated for our marks because since the network is retrained from scratch, the directions that will be learned in the new feature space have no a priori reason to be aligned with the directions of the network we used. Table 2 shows two interesting results: first, the gap in accuracy is less important than when retraining only the logistic regression layer, in particular using $1\%$ of radioactive data does not impact accuracy; second, data augmentation is actually helping the radioactive process. We hypothesize that the multiple crops make the network believe it sees more variety, but in reality all the feature representations of these crops are aligned with our carrier which makes the network learn the carrier direction.

Black-box results.

We report in Figure 6 the results of our black-box detection test. We measure the difference between the loss on vanilla samples and the loss on radioactive samples: when this gap is positive, it means that the model fares better on radioactive images, and thus that it has been trained on the radioactive data. We can see that the use of radioactive data can be detected when a fraction of $q=20\%$ or more of the training set is radioactive. When a smaller portion of the data is radioactive, the model fares better on vanilla data than on radioactive data and thus it is difficult to tell.

Distillation.

Given only black-box access to a model (assuming access to the full softmax), we experiment distillation of this model, and test the distilled model for radioactivity. In this setup, it is possible to detect the use of radioactive data on the distilled model, with a slightly lower performance compared to white-box access to the model. We give detailed results in Appendix A.

5 Ablation analysis

We ran experiments on different architectures with the same training procedure: Resnet-50, VGG-16 and Densenet121. The results are shown in Table 3: the values and trend are similar to what we obtain with Resnet-18 (Table 2). This is non-trivial, as there is no reason that the feature space of a VGG-16 would behave in the same way as that of a Resnet-18: yet, after alignment, we are able to detect the presence of our radioactive mark with high statistical significance. Specifically, when $q=2\%$ of the data is radioactive, we are able to detect it with a $p$-value of $10^{-15}$. This $p$-value is even stronger than the one we obtain when retraining the same architecture as our marking architecture (Resnet-18). We hypothesize that larger model overfit more in general, and thus in this case will learn the mark more acutely.

Transfer to other datasets.

We conducted experiments on a slightly different setup: we mark images from the dataset Places205, but use a network pretrained on Imagenet for the marking phase. These experiments show that even if the marking network is fit for a different distribution, the marking still works and we are able to detect it. Results are shown in Table 4. We can see that when a fraction q higher than $10\%$ of the training data is marked, we can detect radioactivity with a strong statistical significance ($p<10^{-3})$.

Correlation with class difficulty.

Given that radioactive data adds a marker in the features that is correlated with the class label, we expect this mark to be learned by the network more when the class accuracy is low. To validate this hypothesis, we compute the Spearman correlation between the class accuracy for each class and the cosine between the classifier and the carrier: this correlation is negative, with a $p$-value of $4\times 10^{-5}$. This confirms that the network relies more on the mark when learning with difficult classes.

6 Discussion

The experiments validate that our radioactive marks do indeed imprint on the trained models. We also observe two beneficial effects: data augmentation improves the strength of the mark, and transferring the mask to a larger and more realistic architectures makes its detection more reliable. These two observations suggest that our radioactive method is appropriate for real use cases.

We assume that at training time, there is no special procedure to take into account the radioactive data, but rather training is conducted as if it was vanilla data. In particular, a subspace analysis would likely reveal the marking direction. This adversarial scenario becomes akin to that considered in the watermarking literature, where strategies have been developed to reduce the detectability of the carrier. Our current proposal is therefore restricted to the proof of concept that we can mark a model through training that is only resilient to blind attacks such as architectural or training changes. We hope that follow-up works will address a more challenging scenario under Kerckhoffs assumptions (Kerckhoffs, 1883).

Conclusion

The method proposed in this paper, radioactive data, is a way to verify if some data was used to train a model, with statistical guarantees.

We have shown in this paper that such radioactive contamination is effective on large-scale computer vision tasks such as classification on Imagenet with modern architecture (Resnet-18 and Resnet-50), even when only a very small fraction ($1\%$) of the training data is radioactive. Although it is not the core topic of our paper, our method incidentally offers a way to watermark images in the classical sense (Cayre et al., 2005).

References

Appendix A Distillation

Given a marked resnet-18 on which we only have black-box access, we use distillation (Hinton et al., 2015) to train a second network. On this distilled network, we perform the radioactivity test. We show in Table 5 the results of this radioactivity test on distilled networks. We can see that when $2\%$ or more of the original training data is radioactive, the radioactivity propagates through distillation with statistical significance ($p<10^{-3}$).