UnQovering Stereotyping Biases via Underspecified Questions

Introduction

Training vector representations (contextual or non-contextual) from large textual corpora has been the dominant technical paradigm for building NLP models in recent years (Pennington et al., 2014; Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019, inter alia). Unfortunately, these representations learn stereotypes often enmeshed in the massive body of text used to train them Sun et al. (2019). These biases are subsequently passed on to downstream tasks such as co-reference resolution Rudinger et al. (2018); Zhao et al. (2018), textual entailment Dev et al. (2020a), and translation Stanovsky et al. (2019).

Inspired by such prior works, we propose using underspecified questions to uncover stereotyping biases in downstream QA models. We find, however, that there are confounding factors that often overwhelm the effect of bias in such questions, making it difficult to reveal the true stereotype. To address this challenge, we develop UnQover, a general approach to probe biases by building minimal contexts and peeling off confounding factors, such that any choice made by a model would indicate its stereotyping bias. For instance, if the model favors either subject We refer to the two mentions of the the protected groups in our examples as subjects, not to be confused with their grammatical roles. ( Asian or Caucasian for the second question in Fig 1) it would suggest a stereotyping association of the preferred subject towards the attribute bad driver embedded in the model’s parameters. We call such queries underspecified since there is no factual support for either of the choices, based on the context laid out in the paragraph.

We observe that one cannot directly use a QA model’s predicted probabilities to quantify its stereotyping bias, because model predictions are often influenced by factors completely unrelated to the bias being probed. Specifically, we show that QA models have two strong confounding factors: (1) predictions depend on the position of the subject in the question, and (2) predictions are often unchanged even when the attribute (such as being a bad driver) in the question is negated. Such factors, which are reflections of reasoning errors, can lead to incorrect bias estimation. To circumvent this, we design a metric that factors them out, to more accurately uncover underlying stereotyping biases.

Note that prior approaches have often focused on discovering biases by recognizing when a model is categorically incorrect Stanovsky et al. (2019); Dev et al. (2020a); Nadeem et al. (2020). Such approaches, by design, are unable to identify biases not strong enough to change the predicted category. Instead, by using underspecified questions to compare two potential candidates, we make it easier to surface underlying stereotypes in the model.

We introduce a general framework, UnQover, to measure stereotyping biases in QA models via underspecified questions.https://github.com/allenai/unqover

We present two forms of reasoning errors that can affect the study of biases in QA models.

We design a metric that removes these factors to reveal stereotyping biases.

Our broad study spanning five models, two QA datasets and four bias classes shows that (1) larger models (RoBERTa${}_{\textrm{L}}$, BERT${}_{\textrm{L}}$) tend to have more bias than their smaller counterparts (RoBERTa${}_{\textrm{B}}$ and BERT${}_{\textrm{B}}$); (2) fine-tuning on QA datasets affects the degree of bias in a model (increases with SQuAD and decreases with NewsQA); and (3) fine-tuning a distilled model reduces its bias while fine-tuning larger ones can amplify their bias.

We hypothesize that QA models make unfair predictions. We construct a framework to verify this hypothesis and consider it an effort to facilitate future bias evaluation and mitigation in QA models.

The decisions made by models trained on large human-generated data are typically a mixture of some forms of reasoning and stereotyping associations, among other forms of biases. In particular, we focus on studying a model’s underlying associations between protected groups (defined by gender, race, etc.) and certain activities/attributes. Even though we study these associations in underspecified contexts, these stereotypes are part of the QA systems. Such QA systems, if blindly deployed in real life settings (e.g., seeking information in the context of job applications or cybercrimes), could run the risk of conflating their decisions with stereotyped associations. Hence, if unchecked, such representational harms in model predictions would percolate into allocational harms (cf. Crawford, 2017; Abbasi et al., 2019; Blodgett et al., 2020).

For our analysis of gender stereotypes (Sec 5.3), we assume a binary view of gender and acknowledge that this is a simplification of the more complex concept of gender, as noted, e.g., by Larson (2017). We aim to use this assumption to answer the following question: Does our metric, after ruling out confounding factors, actually reveal stereotyping biases? We answer this by confirming that our metric reveals, among other things, harmful gender biases that have been identified in prior literature that also took a binary view of gender. We note that the proposed framework for analysis (Sec 4) is more general, and can be adapted to more nuanced perspectives of gender.

While our methodology is general, the models and datasets we use are built on English resources that, we believe, are only representative of Western societies. We acknowledge that there could thus be a WEIRD skew Henrich et al. (2010) in the presented analysis, focusing on a Western, Educated, Industrialized, Rich, and Democratic subset of the human population. Moreover, our choices of members in the protected groups as well as the attributes might also carry a Western view. Hence we emphasize here (and in Sec 5) that the negative sentiment carried in biased associations are dependent on these choices. However, as noted above, our methodology is general and can be adapted to other cultural contexts.

Related Work

The study of biases in NLP systems is an active subfield. The majority of the work in the area is dedicated to pre-trained models, often via similarity-based analysis of the biases in input representations Bolukbasi et al. (2016a); Garg et al. (2018); Chaloner and Maldonado (2019); Bordia and Bowman (2019); Tan and Celis (2019); Zhao et al. (2019, 2020), or an intermediate classification task Recasens et al. (2013).

Some recent works have focused on biases in downstream tasks, in the form of prediction-based analysis where changes in the predicted labels can be used to discover biases. Arguably this setting is more natural, as it better aligns with how systems are used in real life. Several notable examples are coreference resolution Rudinger et al. (2018); Zhao et al. (2018); Kurita et al. (2019), machine translation Stanovsky et al. (2019); Cho et al. (2019), textual entailment Dev et al. (2020a), language generation Sheng et al. (2019), or clinical classification Zhang et al. (2020).

Our work (UnQover) is similar in spirit where we also rely on model predictions. But we use underspecified inputs to probe comparative biases in QA as well as the underlying LMs. By using the model scores (instead of just changes in labels) in this underspecified setting, we can reveal hard to observe stereotypes inherent in model parameters.

Such studies on model bias have led to many bias mitigation techniques (e.g., Bolukbasi et al., 2016b; Dev et al., 2020a; Ravfogel et al., 2020; Dev et al., 2020b). In this work, we focus on exploring biases across QA models and expect that our framework could also help future efforts on bias mitigation.

Constructing Underspecified Inputs

Let us first examine the question of what it means for a model to be biased. We consider model predictions are represented as conditional probabilities given input texts and model parameters. Imagine that inputs do not have any bearing on what are the outputs, and yet the model is highly confident in its predictions. In this case, what the model predicts exposes an unwarranted preference embedded in its parameters. This idea is the recipe for our construction of underspecified inputs. We apply this notion in the form of question answering.

Consider the task of uncovering gender stereotypes related to occupations in QA models. We have two classes of subjects: $\{$male, female$\}$ and we want to probe the model’s bias towards certain attributes, in this case, occupations.

With that in mind, we define a template $\tau$ with three slots to fill: two subjects $x_{1},x_{2}$ and an attribute $a$. The template is then instantiated by iterating over lists of subjects (i.e., gendered names) and attributes (i.e., occupations). For example, consider the template:

Paragraph: [$x_{1}$] got off the flight to visit [$x_{2}$]. Question ($a$): Who [$a$]?

which can be instantiated given the filler values:

[$x_{1}$]= John, [$x_{2}$]= Mary, [$a$]= was a senator Paragraph: John got off the flight to visit Mary. Question: Who was a senator?

To ensure that stereotype information is not inadvertently introduced into our templates, we design them with the following guidelines:

Questions are designed such that each subject is equally likely (e.g., there are no gender hints in the question)

Attributes are selected such that favoring any subject over another would be unfair, and not considered common knowledge.

We describe the specific details of our templates and instantiations for each bias in Sec 5.

While ideally a QA model should select either subject with equal probability, it is likely for it to have minor deviations from the ideal distribution. Hence, we aggregate the model scores across examples to identify and measure a true bias despite such minor perturbations (described in Sec 4.3).

2 Underspecified Questions for Masked Language Models

We can generalize the above design for masked language models (LMs), allowing us to study their comparative biases as well as potential bias shift brought by downstream training. Using the same slots, we could instantiate the following example:

Template: [$x_{1}$] got off the flight to visit [$x_{2}$]. [MASK] [$a$]. Example: John got off the flight to visit Mary. [MASK] was a senator.

Unlike QA, a masked LM is free to make predictions other than the provided choices in the context ( John and Mary). Here, our underspecified examples differ from prior works in that we present both candidates in the context to elicit model predictions. As a result, we will only use the score assigned to these specific fillers.

Uncovering Stereotypes

Ideally, a perfect model would score each subject purely based on the semantics of the input. We can then quantify stereotyping by directly comparing predicted probabilities on the two subjects (e.g., De-Arteaga et al., 2019). However, in reality, model predictions are influenced by reasoning errors. We discover two such errors and address them next.

Similarly, for masked LM, we use the unnormalized scores and only single-token subjects.

When evaluating our probe, we discovered that the predictions of QA models can heavily depend on the order of the subjects, even if the information content is unchanged! Let $\tau_{1,2}(a)$ denote the (paragraph, question) pair generated by grounding a template $\tau$ with subjects $x_{1},x_{2}$ and attribute $a$. Similarly $\tau_{2,1}(a)$ refers to a filling of the template with flipped ordering of the subjects. Consider the examples $\tau_{1,2}(a)$ and $\tau_{2,1}(a)$ in Fig 2 (left column) which are evaluated with a RoBERTa model Liu et al. (2019) fine-tuned on SQuAD v1.1 Rajpurkar et al. (2016).

where $\operatorname*{avg}$ denotes arithmetic mean over $X_{1},X_{2}$, the sets of subjects, $A$, the set of attributes, and $T$, the set of templates.

1.2 Attribute Independence

A more subtle issue is the model’s indifference to the attribute in the question. This is easy to miss until we ask a negated version of the original question. For instance, consider $\tau_{1,2}(\bar{a})$ and similarly $\tau_{2,1}(\bar{a})$, in Fig 2.

To state it more formally, model prediction should flip when questions are negated:

In practice, models can be oblivious to simple question negations (is versus isn’t) making it hard to probe the underlying bias. For example, if the model scores do not change with negation, it is impossible to know if it even understood the question. We explored few options and found that models are much better at recognizing antonyms and “never” as a negation marker (as shown in our example).

2 Uncovering Stereotyping Biases

Given these confounding factors arising from reasoning errors, how can we reveal a more accurate estimate of stereotyping biases of QA models? What we want to know is the stereotyping bias associated with $x_{1}$, in a template $\tau$ that has another subject $x_{2}$ and an attribute $a$. To isolate both positional dependence and attribute indifference, we define the bias measurement on $x_{1}$ as:

We compute the biases towards $x_{1}$ and $x_{2}$ to compute a comparative measure of bias score:

Our metrics can indeed help isolate other confounding factors. For instance, if there are potential association between subjects and lexical items that affects model predictions, it would play the same role in the negated questions, and hence our metric defined in Eq 6 will cancel out their first-order components.

3 Aggregated Metrics

Let $X_{1},X_{2}$ denote two sets of subjects, $A$ a set of attributes, and $T$ a set of templates. The bias between $x_{1}$ and $a$ is measured by averaging our scores across over $X_{2}$ and $T$:

We can further aggregate over attributes to get a bias score $\gamma(x_{1})$ to capture how subject $x_{1}$ is preferred across all activities. Such a metric can be used to gauge the sentiment associated with $x_{1}$ across many negative sentiment attributes.

Given a dataset, we can compare different models using the intensity of their biases. In practice, model could yield lots of predictions that have low $\gamma$ scores and relatively fewer predictions that have high $\gamma$. In this case, taking median or average of $\gamma$ scores over the dataset would wash away biased predictions. To this end, we first compute the extremeness of the bias for/against each subject as $\max_{a\in A}|\gamma(x_{1},a)|$. To compute the overall bias intensity, we then average this subject bias across all subjects:

where $\mu\in$. Higher score indicates more intensive bias.

A few high scoring outliers can skew our bias estimates when aggregating $\gamma$ values. To address this, we also consider a count-based aggregation that quantifies, for each attribute $a$, which indicates how often is a subject $x_{1}$ preferred (or not) over other subjects, irrespective of the model’s scores:

For a model, if the $\eta\sim 0$, the bias could be explained by a few outliers. However, we found all our datasets and models have $\eta\sim$ 0.5, i.e., the bias is systematic (Appendix A.3).

Experiments

The biased associations presented in the following sections are mined based on the introduced framework and existing models. The examples are meant to highlight issues with current NLP models and should not be taken out of the context of this paper.

In this section, we will show how different transformer-based QA models differ in the degree of their biases, and how biases shift after fine-tuning the underlying language model. We focus on reporting bias intensities, i.e., how much bias percolates to model decisions. We explore biases in four subject classes: (1) gender, (2) nationality, (3) ethnicity, and (4) religion. With gender, we explore the bias associated with occupations, while for the latter three, we focus on negative-activity bias.

We use five models: DistilBERT Sanh et al. (2019), BERT base/large, and RoBERTa base/large. These are evaluated under three settings: (1) pre-trained LM, (2) fine-tuned on SQuAD, and (3) fine-tuned on NewsQA Trischler et al. (2017). To the best of our knowledge, this is the broadest study of model biases across bias classes and models.

We define templates ($T$) for all four bias classes, and select common names, nationalities, ethnicities, and religions for our subject list ($X$). We use the occupations from Dev et al. (2020a) and statements that capture prejudices from StereoSet Nadeem et al. (2020) to create our attribute list ($A$). Table 1 shows the sizes of slot-fillers in our templates and the resulted data sizes.

Each subject and activity appear the same number of times relative to others. Further, the number of examples in Table 1 is not necessarily the product of $|T|$, $|X|$, and $|A|$, since, e.g., some templates only accept country demonyms while some only take country names. Finally, we should note that these datasets are meant for evaluation only. More details are in Appendix A.4.

2 Biases in Models: General Trends

We use the bias intensity $\mu$ introduced in Sec 4.3 to rank models. With five masked LMs and their fine-tuned versions on SQuAD and NewsQA datasets, we compare $15$ models for each type of bias, and summarize them in Fig 3. We start with broad findings that are shared across models and biases.

For QA models, we see that BERT${}_{\textrm{Dist}}$ is among the least biased models across different biases. The large models (RoBERTa${}_{\textrm{L}}$ and BERT${}_{\textrm{L}}$) show more intensive biases than their base versions with few exceptions (RoBERTa models fine-tuned on NewsQA on the gender and religion class).

We also observe that fine-tuning on QA dataset results in a bias shift. The BERT${}_{\textrm{Dist}}$ model, after fine-tuning on SQuAD or NewsQA, shows much less biases across different bias classes. For the larger and stronger models, downstream training can amplify biases, e.g. RoBERTa${}_{\textrm{B/L}}$ become more biased on gender-occupation and nationality.

As seen in Fig 3, NewsQA models show substantially lower biases than SQuAD models, consistently across all four bias classes. Moreover, for ethnicity and religions, NewsQA models have an even lower bias intensity then their masked LM peers. This suggests less biases are picked up from this datasets, and biases that already exist in masked LMs can be mitigated during fine-tuning.

We next explore specific biases in details.

3 Gender-Occupation Bias

Prior works (e.g., Sheng et al., 2019; Rudinger et al., 2018) have shown that gender-occupation bias is predominant in textual corpora, and consequently in learned representations. We will use this bias as a proof of concept for our metrics. We use the names most commonly associated with the genders in the binary viewhttps://www.ssa.gov/oact/babynames/decades/century.html being male or female to show the associated occupation stereotypes.

In Table 2, we aggregate over gendered names and show the top-3 gender-biased occupations. As seen in recent work, these models generally associate jobs that are considered stereotypically feminine with female names and masculine ones with male names. Furthermore, comparing the biased occupations shared across different models in Table 3, we see that these models consistently associate “nurse”, “model”, and “dancer” with female names. In contrast, the occupations associated with male names vary between BERT and RoBERTa. We also present the top biased occupations for NewsQA models and masked LM in Appendix A.5.

Interestingly, we see that even the highest female bias score of BERT${}_{\textrm{Dist}}$ is negative (Table 2). This suggests that the model has a general preference for male names for all occupations. Despite this, the highest ranked occupations for females identified by $\gamma$ are consistent with those for other models.

4 Nationality Bias

For nationalities, we focus on the associations between nations and negative attributes such as crime, violence, poverty, etc. In an effort to anonymize the prejudiced associations, here, we show abstract categories of attributes rather than their raw form (e.g., full of savages). Table 4 summarizes the most biased nationality-attribute pairs for SQuAD models. It is clear that the most biased pairs reflect a non-Western stereotype. Comparing the subject bias metrics $\gamma$ and $\eta$, RoBERTa models are more intensively biased than BERT (as also seen in Fig 3). Among SQuAD models, BERT${}_{\textrm{Dist}}$ is the least biased one where scores are fairly low. Note that, in Table 4, the count-based metric $\eta$’s are all close to $1$, meaning that the listed countries are almost always preferred over other candidates. In Appendix A.6, we also show bias samples from NewsQA model.

To further examine how model bias varies across models, we use the aggregated subject score $\gamma(x)$ introduced in Sec 4.3 which reflects the sentiment associated with each country: the higher the bias, the more negative the sentiment (as the attributes are all negative). Fig 4 shows ranked nationalities according to $\gamma(x)$ scores. We see that, across different models, there is a clear boundary separating Western and non-Western geoschemes.

5 Ethnicity/Religion BiasWe group these due to smaller data and similar findings.

We adopt the same strategy used in Sec 5.4 and show the shared sentiment of ethnicity and religion groups across different models in Figure 5. For ethnicity, we see that there is a clear polarity between the two extremes. Those being ranked high (smaller avg. rank), e.g., Arab and African-American, are far from those being ranked low, e.g., European. However, the variance is large, e.g. Arab appears among the top-4 in both BERT and RoBERTa models, but is ranked neutral, i.e.,$\gamma(x){\sim}0$ in BERT${}_{\textrm{Dist}}$. For religion, Muslim is ranked the most negative but with low variance. While Jewish ethnicity ranks higher among other religions, it is one of the lowest ranked ethnicities. In both cases, the intensity has fairly small scales ($|\gamma(x)|{\leq}0.03$).

Quite similar to the nationality bias, all of the top-biased subject-attribute pairs have $\eta(x,a){\sim}1$, meaning those subjects are almost always chosen over others. In Appendix A.7, we demonstrate with model scores in more details.

6 Quantifying Reasoning Errors

As we show in Sec 4.1, there are reasoning errors in the scores elicited from QA models. In Table 5, we show these two reasoning errors are substantial across different models on our gender-occupation dataset. Comparing QA models, we see that RoBERTa models suffer more from positional errors compared to similar sized BERT models (higher $\delta$). Smaller models do not necessarily fare better where BERT${}_{\textrm{Dist}}$ NewsQA model has strong positional error, even higher than RoBERTa${}_{\textrm{L}}$.

For attribute errors ($\epsilon$), both QA models and masked LMs perform poorly due to the generally observed inconsistency in models (e.g., Ribeiro et al., 2019). Surprisingly the more robustly trained RoBERTa is no better at recognizing the change in question attributes than BERT (similar $\epsilon$ scores) and gets even worse with fine-tuning.

Conclusions & Future Work

We presented UnQover, a general framework for measuring stereotyping biases in QA models and their masked LM peers. Our framework consists of underspecified input construction (Sec 3) and evaluation metrics that factor out effects of reasoning errors (Sec 4). Our broad experiments span over $15$ transformer models on four stereotype classes, and result in interesting findings about how different models behave and how fine-tuning shifts bias (Sec 5). The proposed framework is an effort to facilitate bias evaluation and mitigation.

Our analysis (Sec 5) is based on a binary view of gender and common choices of nationality, ethnicity, and religion groups. Further, the prejudiced statements (Sec 3.1) we extracted from the StereoSet data might carry a Western-specific view of bias, just like the training data for QA models. Future work should address these limitations by providing more inclusive studies.

Acknowledgements

We thank Noah Smith, Suresh Venkatasubramanian and Maarten Sap for their valuable insights and suggestions, and also the reviewers and the ethics committee of EMNLP for constructive comments and pointers.

References

Appendix A Appendix

In this appendix, we present details of our experiments, proofs to our propositions, and model prediction samples. Given the number of models we evaluated in our paper, it is impractical to show all model predictions here. Thus, we present broader experiment results and when presenting predictions from a specific model, we use RoBERTa${}_{\textrm{B}}$ fine-tuned on SQuAD.

We use the pre-trained transformer LMs released by Wolf et al. (2019). For SQuAD models, we either use the their released versions or fine-tune on our end with standard hyperparameter settings.

For BERT${}_{\textrm{Dist}}$ models, we directly fine-tune the distilled language model without extra distillation on the downstream corpus. This allows us to better study the effect of fine-tuning.

In Table 6, we show the F1 scores of QA models on the corresponding official development sets (which are the test sets in our practice). Our training and evaluation use a window size $384$ of tokens that contains the ground truth answer.

A.2 Proof of Propositions in Sec 4.2

A.3 Count-based Bias Metric

In Fig 6, we show the model-wise $\eta$ metric. We see that when counting the win/lose ratio, models are mostly very biased on the same level. With $\eta$ values close to $0.5$, it means most of the biases showing Fig 3 are aggregated by small margins.

A.4 Dataset Generation

For gender-occupation dataset, we list the gendered names in Table 7, occupations in Table 10, and templates in Table 16. For nationality dataset, Table 8 contains the list of country names while Table 17 has the set of templates. Ethnicity and religion subjects are in Table 9, and templates in Table 18. Across all templates, we automate grammar correction for each time of instantiation.

A.5 Gender Bias

A.6 Nationality Bias

In Table 11, we show the top-3 biased nationality-attribute pairs using RoBERTa${}_{\textrm{B}}$ fine-tuned on NewsQA.

A.7 Ethnicity/Religion Biases

In Table 12 and Table 13, we present the sentiments associated with the list of ethnic and religion groups.