A Simple Method for Commonsense Reasoning

Introduction

Although deep neural networks have achieved remarkable successes (e.g., ), their dependence on supervised learning has been challenged as a significant weakness. This dependence prevents deep neural networks from being applied to problems where labeled data is scarce. An example of such problems is common sense reasoning, such as the Winograd Schema Challenge , where the labeled set is typically very small, on the order of hundreds of examples. Below is an example question from this dataset:

The trophy doesn’t fit in the suitcase because it is too big. What is too big? Answer 0: the trophy. Answer 1: the suitcase

Although it is straightforward for us to choose the answer to be "the trophy" according to our common sense, answering this type of question is a great challenge for machines because there is no training data, or very little of it.

In this paper, we present a surprisingly simple method for common sense reasoning with Winograd schema multiple choice questions. Key to our method is th e use of language models (LMs), trained on a large amount of unlabeled data, to score multiple choice questions posed by the challenge and similar datasets. More concretely, in the above example, we will first substitute the pronoun ("it") with the candidates ("the trophy" and "the suitcase"), and then use LMs to compute the probability of the two resulting sentences ("The trophy doesn’t fit in the suitcase because the trophy is too big." and "The trophy doesn’t fit in the suitcase because the suitcase is too big."). The substitution that results in a more probable sentence will be the correct answer.

On both Pronoun Disambiguation and Winograd Schema challenges, our method outperforms previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. On a Pronoun Disambiguation dataset, PDP-60, our method achieves 70.0% accuracy, which is better than the state-of-art accuracy of 66.7%. On a Winograd Schema dataset, WSC-273, our method achieves 63.7% accuracy, 11% above that of the current state-of-art result (52.8%)Code to reproduce these results are available at https://github.com/tensorflow/models/tree/master/research/lm_commonsense.

A unique feature of Winograd Schema questions is the presence of a special word that decides the correct reference choice. In the above example, "big" is this special word. When "big" is replaced by "small", the correct answer switches to "the suitcase". Although detecting this feature is not part of the challenge, further analysis shows that our system successfully discovers this special word to make its decisions in many cases, indicating a good grasp of commonsense knowledge.

Related Work

Unsupervised learning has been used to discover simple commonsense relationships. For example, Mikolov et al. show that by learning to predict adjacent words in a sentence, word vectors can be used to answer analogy questions such as: Man:King::Woman:?. Our work uses a similar intuition that language modeling can naturally capture common sense knowledge. The difference is that Winograd Schema questions require more contextual information, hence our use of LMs instead of just word vectors.

Neural LMs have also been applied successfully to improve downstream applications . In , researchers have shown that pre-trained LMs can be used as feature representations for a sentence, or a paragraph to improve NLP applications such as document classification, machine translation, question answering, etc. The combined evidence suggests that LMs trained on a massive amount of unlabeled data can capture many aspects of natural language and the world’s knowledge, especially commonsense information.

Previous attempts on solving the Winograd Schema Challenge usually involve heavy utilization of annotated knowledge bases, rule-based reasoning, or hand-crafted features . In particular, Rahman and Ng employ human annotators to build more supervised training data. Their model utilizes nearly 70K hand-crafted features, including querying data from Google Search API. Sharma et al. rely on a semantic parser to understand the question, query texts through Google Search, and reason on the graph produced by the parser. Similarly, Schüller formalizes the knowledge-graph data structure and a reasoning process based on cognitive linguistics theories. Bailey et al. introduces a framework for reasoning, using expensive annotated knowledge bases as axioms.

The current best approach makes use of the skip-gram model to learn word representations . The model incorporates several knowledge bases to regularize its training process, resulting in Knowledge Enhanced Embeddings (KEE). A semantic similarity scorer and a deep neural network classifier are then combined on top of KEE to predict the answers. The final system, therefore, includes both supervised and unsupervised models, besides three different knowledge bases. In contrast, our unsupervised method is simpler while having significantly higher accuracy. Unsupervised training is done on text corpora which can be cheaply curated.

Using language models in reading comprehension tests also produced many great successes. Namely Chu et al. used bi-directional RNNs to predict the last word of a passage in the LAMBADA challenge. Similarly, LMs are also used to produce features for a classifier in the Store Close Test 2017, giving best accuracy against other methods . In a broader context, LMs are used to produce good word embeddings, significantly improved a wide variety of downstream tasks, including the general problem of question answering .

Methods

We first substitute the pronoun in the original sentence with each of the candidate choices. The problem of coreference resolution then reduces to identifying which substitution results in a more probable sentence. By reframing the problem this way, language modeling becomes a natural solution by its definition. Namely, LMs are trained on text corpora, which encodes human knowledge in the form of natural language. During inference, LMs are able to assign probability to any given text based on what they have learned from training data. An overview of our method is shown in Figure 1.

Suppose the sentence $S$ of $n$ consecutive words has its pronoun to be resolved specified at the $k^{th}$ position: $S=\{w_{1},..,w_{k-1},w_{k}\equiv p,w_{k+1},..,w_{n}\}$. We make use of a trained language model $P_{\theta}(w_{t}|w_{1},w_{2},..,w_{t-1})$, which defines the probability of word $w_{t}$ conditioned on the previous words $w_{1},...,w_{t-1}$. The substitution of a candidate reference $c$ in to the pronoun position $k$ results in a new sentence $S_{w_{k}\leftarrow c}$ (we use notation $w_{k}\leftarrow c$ to mean that word $w_{k}$ is substituted by candidate $c$). We consider two different ways of scoring the substitution:

$Score_{full}(w_{k}\leftarrow c)=P_{\theta}(w_{1},w_{2},...,w_{k-1},c,w_{k+1},...,w_{n})$

which scores how probable the resulting full sentence is, and

$Score_{partial}(w_{k}\leftarrow c)=P_{\theta}(w_{k+1},...,w_{n}|w_{1},...,w_{k-1},c)$

which scores how probable the part of the resulting sentence following $c$ is, given its antecedent. In other words, it only scores a part $S_{w_{k}\leftarrow c}$ conditioned on the rest of the substituted sentence. An example of these two scores is shown in Table 1. In our experiments, we find that partial scoring strategy is generally better than the naive full scoring strategy.

Experimental settings

In this section we describe tests for commonsense reasoning and the LMs used to solve these tasks. We also detail training text corpora used in our experiments.

We conduct experiments to evaluate our methods on two tasks: Pronoun Disambiguation Problems and Winograd Schema Challenge. These two tasks have been proposed as potential alternatives to the Turing Test, specifically targeting its potential weaknesses and inadequacy .

On the former task, we use the original set of 60 questions (PDP-60) as the main benchmarkhttps://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml. Later analysis augments this test with 62 questions from the development set to avoid bias presented in the original smaller set.http://commonsensereasoning.org/disambiguation.html The second task (WSC-273) is qualitatively much more difficulthttps://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.xml. Its recent best reported result is only 3% of accuracy above random guess . This task consists of 273 questions and is designed to work against techniques such as traditional linguistic restrictions, common heuristics or simple statistical test over text corpora ("Google-proof") .

Recurrent language models.

We consider two types of recurrent LMs, one processes word inputs and the other processes character inputs. Their output layer, however, is constructed to only produce word outputs, allowing both types of input processing to join in ensembles. Namely, our LMs predict a distribution over a large vocabulary (800K words) at each time step, using a softmax layer. Following , we employ importance sampling at the softmax layer with 8,192 negative samples for each mini-batch to significantly speed up training. We use two layers of LSTM with 8,192 hidden units and a projection layer to a smaller dimensionality at output gates for faster processing.

For models that process words, we use a big embedding look up matrix with vocabulary size 800K and embedding size 1,024. For character-level input, we use a vocabulary size of 256 characters and embedding size 16. Characters in the same word are concatenated and used as input at a single time step. The resulting character embedding is processed using eight convolutions before going into the LSTM layers. More details about our LMs can be found in Appendix A.

Training text corpora.

We perform experiments on several different text copora to examine the effect of training data type on test accuracy. Namely, we consider LM-1-Billion, CommonCrawlWe evaluate all models trained on CommonCrawl after approximately 10-billion words are consumed., SQuAD and Gutenberg Books. For SQuAD, we collect context passages from the Stanford Question-Answering Dataset to form its training and validation sets accordingly.

Main results

Our experiments start with testing LMs trained on all text corpora with PDP-60 and WSC-273. Next, we show that it is possible to customize training data to obtain even better results.

We first examine unsupervised single-model resolvers on PDP-60 by training one character-level and one word-level LM on the Gutenberg corpus. In Table 2, these two resolvers outperform previous results by a large margin. For this task, we found full scoring gives better results than partial scoring. In Section 6.2, we provide evidences that this is an atypical case due to the very small size of PDP-60.

Next, we allow systems to take in necessary components to maximize their test performance. This includes making use of supervised training data that maps commonsense reasoning questions to their correct answer. Here we simply train another three variants of LMs on LM-1-Billion, CommonCrawl, and SQuAD and ensemble all of them. As reported in Table 3, this ensemble of five unsupervised models outperform the best system in the 2016 competition (58.3%) by a large margin. Specifically, we achieve 70.0% accuracy, better than the more recent reported results from Quan Liu et al (66.7%) , who makes use of three knowledge bases and a supervised deep neural network.

2 Winograd Schema Challenge

On the harder task WSC-273, our single-model resolvers also outperform the current state-of-the-art by a large margin, as shown in Table 4. Namely, our word-level resolver achieves an accuracy of 56.4%. By training another 4 LMs, each on one of the 4 text corpora LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and add to the previous ensemble, we are able to reach 61.5%, nearly 10% of accuracy above the previous best result. This is a drastic improvement considering this previous best system outperforms random guess by only 3% in accuracy.

This task is more difficult than PDP-60. First, the overall performance of all competing systems are much lower than that of PDP-60. Second, incorporating supervised learning and expensive annotated knowledge bases to USSM provides insignificant gain this time (+3%), comparing to the large gain on PDP-60 (+19%).Our results so far have been with recurrent language models. As a comparison, we also trained a subword-level Transformer LM on Wikipedia texts and obtain competitive performance (58.3% on PDP-60 and 54.1% on WSC-273).

3 Customized training data for Winograd Schema Challenge

As previous systems collect relevant data from knowledge bases after observing questions during evaluation , we also explore using this option. Namely, we build a customized text corpus based on questions in commonsense reasoning tasks. It is important to note that this does not include the answers and therefore does not provide supervision to our resolvers. In particular, we aggregate documents from the CommonCrawl dataset that has the most overlapping n-grams with the questions. The score for each document is a weighted sum of $F_{1}(n)$ scores when counting overlapping n-grams:

The top 0.1% of highest ranked documents is chosen as our new training corpus. Details of the ranking is shown in Figure 2. This procedure resulted in nearly 1,000,000 documents, with the highest ranking document having a score of $8\times 10^{-2}$, still relatively small to a perfect score of $1.0$. We name this dataset STORIES since most of the constituent documents take the form of a story with long chain of coherent events.

We train four different LMs on STORIES and add them to the previous ensemble of 10 LMs, resulting in a gain of 2% accuracy in the final system as shown in Table 5. Remarkably, single models trained on this corpus are already extremely strong, with a word-level LM achieving 62.6% accuracy, even better than the ensemble of 10 models previously trained on 4 other text corpora (61.5%).

Analysis

We introduce a method to potentially detect keywords at which our proposed resolvers make decision between the two candidates $c_{correct}$ and $c_{incorrect}$. Namely, we look at the following ratio:

Where $1\leq t\leq n$ for full scoring, and $k+1\leq t\leq n$ for partial scoring. It follows that the choice between $c_{correct}$ or $c_{incorrect}$ is made by the value of $Q=\prod_{t}q_{t}$ being bigger than $1.0$ or not. By looking at the value of each individual $q_{t}$, it is possible to retrieve words with the largest values of $q_{t}$ and hence most responsible for the final value of $Q$.

We visualize the probability ratios $q_{t}$ to have more insights into the decisions of our resolvers. Figure 3 displays a sample of incorrect decisions made by full scoring and is corrected by partial scoring. Interestingly, we found $q_{t}$ with large values coincides with the special keyword of each Winograd Schema in several cases. Intuitively, this means the LMs assigned very low probability for the keyword after observing the wrong substitution. It follows that we can predict the keyword in each the Winograd Schema question by selecting top word positions with the highest value of $q_{t}$.

For questions with keyword appearing before the reference, we detect them by backward-scoring models. Namely, we ensemble 6 LMs, each trained on one text corpora with word order reversed. This ensemble also outperforms the previous best system on WSC-273 with a remarkable accuracy of 58.2%. Overall, we are able to discover a significant amount of special keywords (115 out of 178 correctly answered questions) as shown in Table 6. This strongly indicates a correct understanding of the context and a good grasp of commonsense knowledge in the resolver’s decision process.

2 Partial scoring is better than full scoring.

In this set of experiments, we look at wrong predictions from a word-level LM. With full scoring strategy, we observe that $q_{t}$ at the pronoun position is most responsible for a very large percentage of incorrect decisions as shown in Figfure 3 and Table 7. For example, with the test "The trophy cannot fit in the suitcase because it is too big.", the system might return $c_{incorrect}=$"suitcase" simply because $c_{correct}=$ "trophy" is a very rare word in its training corpus and therefore, is assigned a very low probability, overpowering subsequent $q_{t}$ values.

Following this reasoning, we apply a simple fix to full scoring by normalizing its score with the unigram count of $c$: $Score_{full~{}normalized}=Score_{full}/Count(c)$. Partial scoring, on the other hand, disregards $c$ altogether. As shown in Figure 4, this normalization fixes full scoring in 9 out of 10 tested LMs on PDP-122. On WSC-273, the result is very decisive as partial scoring strongly outperforms the other two scoring in all cases. Since PDP-122 is a larger superset of PDP-60, we attribute the different behaviour observed on PDP-60 as an atypical case due to its very small size.

3 Importance of training corpus

In this set of experiments, we examine the effect of training data on commonsense reasoning test performance. Namely, we train both word-level and character-level LMs on each of the five corpora: LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and STORIES. A held-out dataset from each text corpus is used for early stopping on the corresponding training data.

To speed up training on these large corpora, we first train the models on the LM-1-Billion text corpus. Each trained model is then divided into three groups of parameters: Embedding, Recurrent Cell, and Softmax. Each of the three is optionally transferred to train the same architectures on CommonCrawl, SQuAD and Gutenberg Books. The best transferring combination is chosen by cross-validation.

Figure 5-left and middle show that STORIES always yield the highest accuracy for both types of input processing. We next rank the text corpora based on ensemble performance for more reliable results. Namely, we compare the previous ensemble of 10 models against the same set of models trained on each single text corpus. This time, the original ensemble trained on a diverse set of text corpora outperforms all other single-corpus ensembles including STORIES. This highlights the important role of diversity in training data for commonsense reasoning accuracy of the final system.

Conclusion

We introduce a simple unsupervised method for Commonsense Reasoning tasks. Key to our proposal are large language models, trained on a number of massive and diverse text corpora. The resulting systems outperform previous best systems on both Pronoun Disambiguation Problems and Winograd Schema Challenge. Remarkably on the later benchmark, we are able to achieve 63.7% accuracy, comparing to 52.8% accuracy of the previous state-of-the-art, who utilizes supervised learning and expensively annotated knowledge bases. We analyze our system’s answers and observe that it discovers key features of the question that decides the correct answer, indicating good understanding of the context and commonsense knowledge. We also demonstrated that ensembles of models benefit the most when trained on a diverse set of text corpora.

We anticipate that this simple technique will be a strong building block for future systems that utilize reasoning ability on commonsense knowledge.

References

Appendix A Recurrent language models

The base model consists of two layers of Long-Short Term Memory (LSTM) with 8192 hidden units. The output gate of each LSTM uses peepholes and a projection layer to reduce its output dimensionality to 1024. We perform drop-out on LSTM’s outputs with probability 0.25.

For word inputs, we use an embedding lookup of 800000 words, each with dimension 1024. For character inputs, we use an embedding lookup of 256 characters, each with dimension 16. We concatenate all characters in each word into a tensor of shape (word length, 16) and add to its two ends the and tokens. The resulting concatenation is zero-padded to produce a fixed size tensor of shape (50, 16). This tensor is then processed by eight different 1-D convolution (Conv) kernels of different sizes and number of output channels, listed in Table 8, each followed by a ReLU acitvation. The output of all CNNs are then concatenated and processed by two other fully-connected layers with highway connection that persist the input dimensionality. The resulting tensor is projected down to a 1024-feature vector. For both word input and character input, we perform dropout on the tensors that go into LSTM layers with probability 0.25.

We use a single fully-connected layer followed by a $Softmax$ operator to process the LSTM’s output and produce a distribution over word vocabulary of size 800K. During training, LM loss is evaluated using importance sampling with negative sample size of 8192. This loss is minimized using the AdaGrad algorithm with a learning rate of 0.2. All gradients on LSTM parameters and Character Embedding parameters are clipped by their global norm at 1.0. To avoid storing large matrices in memory, we shard them into 32 equal-sized smaller pieces. In our experiments, we used 8 different variants of this base model as listed in Table 9.

In Table 10, we listed all LMs and their training text corpora used in each of the experiments in Section 5.

Appendix B Data contamination in CommonCrawl

Using the similarity scoring technique in section 5.3, we observe a large amount of low quality training text on the lower end of the ranking. Namely, these are documents whose content are mostly unintelligible or unrecognized by our vocabulary. Training LMs for commonsense reasoning tasks on full CommonCrawl, therefore, might not be ideal. On the other hand, we detected and removed a portion of PDP-122 questions presented as an extremely high ranked document.