EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

Introduction

Numbers play a vital role in our lives. We reason with numbers in day-to-day tasks ranging from handling currency to reading news articles to understanding sports results, elections and stock markets. As numbers are used to communicate information accurately, reasoning with them is an essential core competence in understanding natural language Levinson (2001); Frank et al. (2008); Dehaene (2011). A benchmark task in natural language understanding is natural language inference (NLI)(or recognizing textual entailment (RTE)) Cooper et al. (1996); Condoravdi et al. (2003); Bos and Markert (2005); Dagan et al. (2006), wherein a model determines if a natural language hypothesis can be justifiably inferred from a given premiseOften, this is posed as a three-way decision where the hypothesis can be inferred to be true (entailment), false (contradiction) or cannot be determined (neutral).. Making such inferences often necessitates reasoning about quantities.

Consider the following example from Table 1,

P: With 99.6% of precincts counted , Dewhurst held 48% of the vote to 30% for Cruz . H: Lt. Gov. David Dewhurst fails to get 50% of primary vote.

To conclude the hypothesis is inferable, a model must reason that since 99.6% precincts are counted, even if all remaining precincts vote for Dewhurst, he would fail to get 50% of the primary vote. Scant attention has been paid to building datasets to evaluate this reasoning ability. To address this gap, we present EQUATE (Evaluating Quantity Understanding Aptitude in Textual Entailment) (§3), which consists of five evaluation sets, each featuring different facets of quantitative reasoning in textual entailment (Table 1) (including verbal reasoning with quantities, basic arithmetic computation, dealing with approximations and range comparisons.).

We evaluate the ability of existing state-of-the-art NLI models to perform quantitative reasoning (§4.1), by benchmarking 9 published models on EQUATE. Our results show that most models are incapable of quantitative reasoning, instead relying on lexical cues for prediction. Additionally, we build Q-REAS , a shallow semantic reasoning baseline for quantitative reasoning in NLI (§4.2). Q-REAS is effective on synthetic test sets which contain more quantity-based inference, but shows limited success on natural test sets which require deeper linguistic reasoning. However, the hardest cases require a complex interplay between linguistic and numerical reasoning. The EQUATE evaluation framework makes it clear where this new challenge area for textual entailment stands.

Related Work

NLI has attracted community-wide interest as a stringent test for natural language understanding Cooper et al. (1996); Fyodorov ; Glickman et al. (2005); Haghighi et al. (2005); Harabagiu and Hickl (2006); Romano et al. (2006); Dagan et al. (2006); Giampiccolo et al. (2007); Zanzotto et al. (2006); Malakasiotis and Androutsopoulos (2007); MacCartney (2009); DeMarneffe et al. (2009); Dagan et al. (2010); Angeli and Manning (2014); Marelli et al. (2014). Recently, the creation of large-scale datasets Bowman et al. (2015); Williams et al. (2017); Khot et al. (2018) spurred the development of many neural models Parikh et al. (2016); Nie and Bansal (2017); Conneau et al. (2017); Balazs et al. (2017); Chen et al. (2017a); Radford et al. (2018); Devlin et al. (2018).

However, state-of-the-art models for NLI treat the task like a matching problem, which appears to work in many cases, but breaks down in others. As the field moves past current models of the matching variety to ones that embody more of the reasoning we know is part of the task, we need benchmarks that will enable us to mark progress in the field. Prior work on challenge tasks has already made headway in defining tasks for subproblems such as lexical inference with hypernymy, co-hyponymy, antonymy Glockner et al. (2018); Naik et al. (2018). In this work, we specifically probe into quantitative reasoning.

De Marneffe et al. (2008) find that in a corpus of real-life contradiction pairs collected from Wikipedia and Google News, 29% contradictions arise from numeric discrepancies, and in the RTE-3 (Recognizing Textual Entailment) development set, numeric contradictions make up 8.8% of contradictory pairs. Naik et al. (2018) find that model inability to do numerical reasoning causes 4% of errors made by state-of-the-art models. Sammons et al. (2010); Clark (2018) argue for a systematic knowledge-oriented approach in NLI by evaluating specific semantic analysis tasks, identifying quantitative reasoning in particular as a focus area. Bentivogli et al. (2010) propose creating specialized datasets, but feature only 6 examples with quantitative reasoning. Our work bridges this gap by providing a more comprehensive examination of quantitative reasoning in NLI.

While to the best of our knowledge, prior work has not studied quantitative reasoning in NLI, Roy (2017) propose a model for a related subtask called quantity entailment, which aims to determine if a given quantity can be inferred from a sentence. In contrast, our work is concerned with general-purpose textual entailment which considers if a given sentence can be inferred from another. Our work also relates to solving arithmetic word problems Hosseini et al. (2014); Mitra and Baral (2016); Zhou et al. (2015); Upadhyay et al. (2016); Huang et al. (2017); Kushman et al. (2014a); Koncel-Kedziorski et al. (2015); roy2016solving; Roy (2017); Ling et al. (2017a). A key difference is that word problems focus on arithmetic reasoning, while the requirement for linguistic reasoning and world knowledge is limited as the text is concise, straightforward, and self-contained Hosseini et al. (2014); Kushman et al. (2014b). Our work provides a testbed that evaluates basic arithmetic reasoning while incorporating the complexity of natural language.

Recently, Dua2019DROPAR also recognize the importance of quantitative reasoning for text understanding. They propose DROP, a reading comprehension dataset focused on a limited set of discrete operations such as counting, comparison, sorting and arithmetic. In contrast, EQUATE features diverse phenomena that occur naturally in text, including reasoning with approximation, ordinals, implicit quantities and quantifiers, requiring NLI models to reason comprehensively about the interplay between quantities and language. Additionally, through EQUATE we suggest the inclusion of controlled synthetic tests in evaluation benchmarks. Controlled tests act as basic validation of model behaviour, isolating model ability to reason about a property of interest.

Quantitative Reasoning in NLI

Our interpretation of “quantitative reasoning” draws from cognitive testing and education Stafford (1972); Ekstrom et al. (1976), which considers it “verbal problem-solving ability”. While inextricably linked to mathematics, it is an inclusive skill involving everyday language rather than a specialized lexicon. To excel at quantitative reasoning, one must interpret quantities expressed in language, perform basic calculations, judge their accuracy, and justify quantitative claims using verbal and numeric reasoning. These requirements show a reciprocity: NLI lends itself as a test bed for quantitative reasoning, which conversely, is important for NLI Sammons et al. (2010); Clark (2018). Motivated by this, we present the EQUATE (Evaluating Quantity Understanding Aptitude in Textual Entailment) framework.

EQUATE consists of five NLI test sets featuring quantities. Three of these tests for quantitative reasoning feature language from real-world sources such as news articles and social media (§3.2; §3.3; §3.4). We focus on sentences containing quantities with numerical values, and consider an entailment pair to feature quantitative reasoning if it is at least one component of the reasoning required to determine the entailment label (but not necessarily the only reasoning component). Quantitative reasoning features quantity matching, quantity comparison, quantity conversion, arithmetic, qualitative processes, ordinality and quantifiers, quantity noun and adverb resolutionSuch as the quantities represented in dozen, twice, teenagers. as well as verbal reasoning with the quantity’s textual contextFor example, $\langle$Obama cuts tax rate to 28%, Obama wants to cut tax rate to 28% as part of overhaul$\rangle$.. Appendix B gives some examples for these quantitative phenomena. We further filter sentence pairs which require only temporal reasoning, since specialized knowledge is needed to reason about time. These three test sets contain pairs which conflate multiple lexical and quantitative reasoning phenomena. In order to study aspects of quantitative reasoning in isolation, EQUATE further features two controlled synthetic tests (§3.5; §3.6), evaluating model ability to reason with quantifiers and perform simple arithmetic. We intend that models reporting performance on any NLI dataset additionally evaluate on the EQUATE benchmark, to demonstrate competence at quantitative reasoning.

2 RTE-Quant

This test set is constructed from the RTE sub-corpus for quantity entailment Roy (2017), originally drawn from the RTE2-RTE4 datasets Dagan et al. (2006). The original sub-corpus conflates temporal and quantitative reasoning. We discarded pairs requiring temporal reasoning, obtaining a set of 166 entailment pairs.

3 NewsNLI

This test set is created from the CNN corpus Hermann et al. (2015) of news articles with abstractive summaries. We identify summary points with quantities, filtering out temporal expressions. For a summary point, the two most similar sentencesAccording to Jaccard similarity. from the article are chosen, flipping pairs where the premise begins with a first-person pronoun (eg:$\langle$“He had nine pears”, “Bob had nine pears”$\rangle$ becomes $\langle$“Bob had nine pears”, “He had nine pears”$\rangle$). The top 50% of similar pairs are retained to avoid lexical overlap bias. We crowdsource annotations for a subset of this data from Amazon Mechanical Turk. CrowdworkersWe require crowdworkers to have an approval rate of 95% on at least 100 tasks and pass a qualification test. are shown two sentences and asked to determine whether the second sentence is definitely true, definitely false, or not inferable given the first. We collect 5 annotations per pair, and consider pairs with lowest token overlap between premise and hypothesis and least difference in premise-hypothesis lengths when stratified by entailment label. Top 1000 samples meeting these criteria form our final set. To validate crowdsourced labels, experts are asked to annotate 100 pairs. Crowdsourced gold labels match expert gold labels in 85% cases, while individual crowdworker labels match expert gold labels in 75.8%. Disagreements are manually resolved by experts and examples not featuring quantitiative reasoning are filtered, leaving a set of 968 samples.

4 RedditNLI

This test set is sourced from the popular social forum \redditAccording to the Reddit User Agreement, users grant Reddit the right to make their content available to other organizations or individuals.. Since reasoning about quantities is important in domains like finance or economics, we scrape all headlines from the posts on \r\economics, considering titles that contain quantities and do not have meta-forum information. Titles appearing within three days of each other are clustered by Jaccard similarity, and the top 300 pairs are extracted. After filtering out nonsensical titles, such as concatenated stock prices, we are left with 250 sentence pairs. Similar to RTE, two expert annotators label these pairs, achieving a Cohen’s kappa of 0.82. Disagreements are discussed to resolve final labels.

5 Stress Test

We include the numerical reasoning stress test from Naik et al. (2018) as a synthetic sanity check. The stress test consists of 7500 entailment pairs constructed from sentences in algebra word problems Ling et al. (2017b). Focusing on quantifiers, it requires models to compare entities from hypothesis to the premise while incorporating quantifiers, but does not require them to perform the computation from the original algebra word problem (eg: $\langle$“NHAI employs 100 men to build a highway of 2 km in 50 days working 8 hours a day”,“NHAI employs less than 700 men to build a highway of 2 km in 50 days working 8 hours a day”$\rangle$).

6 AwpNLI

Models

We describe the 9 NLI modelsAccuracy of all models on MultiNLI closely matches original publications (numbers in appendix A). used in this study, as well as our new baseline. The interested reader is invited to refer to the corresponding publications for further details.

1) Majority Class (MAJ): Simple baseline that always predicts the majority class in test set. 2) Hypothesis-Only (HYP): FastText classifier Mikolov et al. (2018) trained on only hypotheses Gururangan et al. (2018). 3) ALIGN: A bag-of-words alignment model inspired by MacCartney (2009).Model accuracy on RTE-3 test is 61.12%, comparable to the reported average model performance in the RTE competition of 62.4% . 4) CBOW: A simple bag-of-embeddings sentence representation model Williams et al. (2017). 5) BiLSTM: The simple BiLSTM model described by Williams et al. (2017). 6) Chen (CH): Stacked BiLSTM-RNNs with shortcut connections and character word embeddings Chen et al. (2017b). 7) InferSent: A single-layer BiLSTM-RNN model with max-pooling Conneau et al. (2017). 8) SSEN: Stacked BiLSTM-RNNs with shortcut connections Nie and Bansal (2017). 9) ESIM: Sequential inference model proposed by Chen et al. (2017a) which uses BiLSTMs with an attention mechanism. 10) OpenAI GPT: Transformer-based language model Vaswani et al. (2017), with finetuning on NLI Radford et al. (2018). 11) BERT: Transformer-based language model Vaswani et al. (2017), with a cloze-style and next-sentence prediction objective, and finetuning on NLI Devlin et al. (2018).

2 Q-REAS Baseline System

Figure 1 describes the Q-REAS baseline for quantitative reasoning in NLI. The model manipulates quantity representations symbolically to make entailment decisions, and is intended to serve as a strong heuristic baseline for numerical reasoning on the equate benchmark. This model has four stages: Quantity mentions are extracted and parsed into semantic representations called Numsets (§4.2.1, §4.2.2); compatible Numsets are extracted (§4.2.3) and composed (§4.2.4) to form justifications; Justifications are analyzed to determine entailment labels (§4.2.5).

We follow Barwise and Cooper (1981) in defining quantities as having a number, unit, and an optional approximator. Quantity mentions are identified as least ancestor noun phrases from the constituency parse of the sentence containing cardinal numbers.

2.2 Quantity Parser

The quantity parser constructs a grounded representation for each quantity mention in the premise or hypothesis, henceforth known as a Numset A Numset may be a composition of other Numsets .. A Numset is a tuple (val, unit, ent, adj, loc, verb, freq, flux)As in Koncel-Kedziorski et al. (2015) S denotes all possible spans in the sentence, ${\phi}$ represents the empty span, and $S^{\phi}$=$S\cup\phi$ with:

To extract values for a quantity, we extract cardinal numbers, recording contiguity. We normalize the number(remove “,”s, convert written numbers to float, decide the numerical value, for example hundred fifty eight thousand is 158000, two fifty eight is 258, 374m is 3740000 etc.). If cardinal numbers are non-adjacent, we look for an explicitly mentioned range such as ‘to’ and ‘between’.. We also handle simple ratios such as quarter, half etc, and extract bounds (eg: fewer than 10 apples is parsed to $[-\infty,10]$ apples.)

To extract units, we examine tokens adjacent to cardinal numbers in the quantity mention and identify known units. If no known units are found, we assign the token in a numerical modifier relationship with the cardinal number, else we assign the nearest noun to the cardinal number as the unit. A quantity is determined to be approximate if the word in an adverbial modifier relation with the cardinal number appears in a gazetteerroughly, approximately, about, nearly, roundabout, around, circa, almost, approaching, pushing, more or less, in the neighborhood of, in the region of, on the order of,something like, give or take (a few), near to, close to, in the ballpark of. If approximate, range is extended to (+/-)2% of the current value.

2.3 Quantity Pruner

The pruner constructs “compatible” premise-hypothesis Numset pairs. Consider the pair “Insurgents killed 7 U.S. soldiers, set off a car bomb that killed four Iraqi policemen” and “7 US soldiers were killed, and at least 10 Iraqis died”. Our parser extracts Numsets corresponding to “four Iraqi policemen” and “7 US soldiers” from premise and hypothesis respectively. But these Numsets should not be compared as they involve different units. The pruner discards such incompatible pairs. Heuristics to identify unit-compatible Numset pairs include three cases- 1) direct string match, 2) synonymy/hypernymy relations from WordNet, 3) one unit is a nationality/ jobLists of jobs, nationalities scraped from Wikipedia. and the other unit is synonymous with person Roy (2017).

2.4 Quantity Composition

The composition module detects whether a hypothesis Numset is justified by composing “compatible” premise Numsets . For example, consider the pair “I had 3 apples but gave one to my brother” and “I have two apples”. Here, the premise Numsets $P_{1}$ (“3 apples”) and $P_{2}$ (“one apple”) must be composed to deduce that the hypothesis Numset $H_{1}$ (“2 apples”) is justified. Our framework accomplishes this by generating postfix arithmetic equationsNote that arithmetic equations differ from algebraic equations in that they do not contain unknown variables from premise Numsets , that justify the hypothesis Numset Direct comparisons are incorporated by adding “=” as an operator.. In this example, the expression $<P_{1},P_{2},-,H_{1},=>$ will be generated.

The set of possible equations is exponential in number of Numsets , making exhaustive generation intractable. But a large number of equations are invalid as they violate constraints such as unit consistency. Thus, our framework uses integer linear programming (ILP) to constrain the equation space. It is inspired by prior work on algebra word problems Koncel-Kedziorski et al. (2015), with some key differences:

1. Arithmetic equations: We focus on arithmetic equations instead of algebraic ones. 2. Range arithmetic: Quantitative reasoning involves ranges, which are handled by representing then as endpoint-inclusive intervals and adding the four operators ($\cup,\cap,\setminus,\subseteq$) 3. Hypothesis quantity-driven: We optimize an ILP model for each hypothesis Numset because a sentence pair is marked “entailment” iff every hypothesis quantity is justified.

Table 3 describes ILP variables. We impose the following types of constraints: 1. Definitional Constraints: Ensure that ILP variables take on valid values by constraining initialization, range, and update. 2. Syntactic Constraints: Assure syntactic validity of generated postfix expressions by limiting operator-operand ordering. 3. Operand Access: Simulate stack-based evaluation correctly by choosing correct operator-operand assignments. 4. Type Consistency: Ensure that all operations are type-compatible. 5. Operator Consistency: Force range operators to have range operands and mathematical operators to have single-valued operands.

Definitional, syntactic, and operand access constraints ensure mathematical validity while type and operator consistency constraints add linguistic consistency. Constraint formulations are provided in Tables 4 and 5. We limit tree depth to 3 and retrieve a maximum of 50 solutions per hypothesis Numset , then solve to determine whether the equation is mathematically correct. We discard equations that use invalid operations (division by 0) or add unnecessary complexity (multiplication/ division by 1). The remaining equations are considered plausible justifications .

2.5 Global Reasoner

The global reasoner predicts the final entailment label as shown in Algorithm 1MaxSimilarityClass() takes two quantities and returns a probability distribution over entailment labels based on unit match. Similarly, ValueMatch() detects whether two quantities match in value (this function can also handle ranges)., on the assumption that every Numset in the hypothesis has to be justified This is a necessary but not sufficient condition for entailment. Consider the example, $\langle$‘Sam believed Joan had 5 apples’, ‘Joan had 5 apples’$\rangle$. The hypothesis quantities of 5 apples is justified but is not a sufficient condition for entailment. for entailment.

Results and Discussion

Table 6 presents results on EQUATE. All models, except Q-REAS are trained on MultiNLI. Q-REAS utilizes WordNet and lists from Wikipedia. We observe that neural models, particularly OpenAI GPT excel at verbal aspects of quantitative reasoning (RTE-Quant, NewsNLI), whereas Q-REAS excels at numerical aspects (Stress Test, AwpNLI).

To tease apart contributory effects of numerical and verbal reasoning in natural data, we experiment with NewsNLI. We extract all entailed pairs where a quantity appears in both premise and hypothesis, and perturb the quantity in the hypothesis generating contradictory pairs. For example, the pair $\langle$‘In addition to 79 fatalities , some 170 passengers were injured.’$\rangle$ ‘The crash took the lives of 79 people and injured some 170’, ‘entailment’ is changed to $\langle$‘In addition to 79 fatalities , some 170 passengers were injured.’, ‘The crash took the lives of 80 people and injured some 170’, ‘contradiction’$\rangle$, assuming scalar implicature and event coreference. Our perturbed test set contains 218 pairs. On this set, GPTthe best-performing neural model on EQUATE. achieves an accuracy of 51.18%, as compared to 72.04% on the unperturbed set, suggesting the model relies on verbal cues rather than numerical reasoning. In comparison, Q-REAS achieves an accuracy of 98.1% on the perturbed set, compared to 75.36% on the unperturbed set, highlighting reliance on quantities rather than verbal information. Closer examination reveals that OpenAI switches to predicting the ‘neutral’ category for perturbed samples instead of entailment, accounting for 42.7% of its errors, possibly symptomatic of lexical bias issues Naik et al. (2018).

2 What Quantitative Phenomena Are Hard?

We sample 100 errors made by Q-REAS on each test in EQUATE, to identify phenomena not addressed by simple quantity comparison. Our analysis of causes for error suggest avenues for future research:

1. Multi-step numerical-verbal reasoning: Models do not perform well on examples requiring interleaved verbal and quantitative reasoning, especially multi-step deduction. Consider the pair $\langle$“Two people were injured in the attack”, “Two people perpetrated the attack”$\rangle$. Quantities “two people” and “two people” are unit-compatible, but must not be compared. Another example is the NewsNLI entailment pair in Table 1. This pair requires us to identify that 16 and 17 refer to Emmanuel and Zachary’s ages (quantitative), deduce that this implies they are teenagers (verbal) and finally count them (quantitative) to get the hypothesis quantity “two teens”. Numbers and language are intricately interleaved and developing a reasoner capable of handling such complex interplay is challenging.

2. Lexical inference: Lack of real world knowledge causes errors in identifying quantities and valid comparisons. Errors include mapping abbreviations to correct units (“m” to “meters”), detecting part-whole coreference (“seats” can be used to refer to “buses”), and resolving hypernymy/hyponymy (“young men” to “boys”).

3. Inferring underspecified quantities: Quantity attributes can be implicitly specified, requiring inference to generate a complete representation. Consider “A mortar attack killed four people and injured 80”. A system must infer that the quantity “80” refers to people. On RTE-Quant, 20% of such cases stem from zero anaphora, a hard problem in coreference resolution.

4. Arithmetic comparison limitations: These examples require composition between incompatible quantities. For example, consider $\langle$“There were 3 birds and 6 nests”, “There were 3 more nests than birds”$\rangle$. To correctly label this pair “3 birds” and “6 nests” must be composed.

Conclusion

In this work, we present EQUATE, an evaluation framework to estimate the ability of models to reason quantitatively in textual entailment. We observe that existing neural approaches rely heavily on the lexical matching aspect of the task to succeed rather than reasoning about quantities. We implement a strong symbolic baseline Q-REAS that achieves success at numerical reasoning, but lacks sophisticated verbal reasoning capabilities. The EQUATE resource presents an opportunity for the community to develop powerful hybrid neuro-symbolic architectures, combining the strengths of neural models with specialized reasoners such as Q-REAS . We hope our insights lead to the development of models that can more precisely reason about the important, frequent, but understudied, phenomena of quantities in natural language.

Acknowledgments

This research was supported in part by grants from the National Science Foundation Secure and Trustworthy Computing program (CNS-1330596, CNS-15-13957, CNS-1801316, CNS-1914486) and a DARPA Brandeis grant (FA8750-15-2-0277). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, DARPA, or the US Government. The author Naik was supported by a fellowship from the Center of Machine Learning and Health at Carnegie Mellon University. The authors would like to thank Graham Neubig, Mohit Bansal and Dongyeop Kang for helpful discussion regarding this work, and Shruti Rijhwani and Siddharth Dalmia for reviews while drafting this paper. The authors are also grateful to Lisa Carey Lohmueller and Xinru Yan for volunteering their time for pilot studies.

Literatur

Appendix

Anhang A Baseline performance on MultiNLI-Dev Matched

Table 7 presents classification accuracies of all baseline models used in this work on the matched subset of MultiNLI-Dev. These scores are very close to the numbers reported by the original publications, affirming the correctness of our baseline setup.

Anhang B Examples of quantitative phenomena present in EQUATE

Table 8 presents some examples from EQUATE which demonstrate interesting quantitative phenomena that must be understood to label the pair correctly.