Comparative Study of CNN and RNN for Natural Language Processing

Introduction

Natural language processing (NLP) has benefited greatly from the resurgence of deep neural networks (DNNs), due to their high performance with less need of engineered features. There are two main DNN architectures: convolutional neural network (CNN) LeCun et al. (1998) and recurrent neural network (RNN) Elman (1990). Gating mechanisms have been developed to alleviate some limitations of the basic RNN, resulting in two prevailing RNN types: long short-term memory (LSTM) Hochreiter and Schmidhuber (1997) and gated recurrent unit (GRU) Cho et al. (2014).

Generally speaking, CNNs are hierarchical and RNNs sequential architectures. How should we choose between them for processing language? Based on the characterization “hierarchical (CNN) vs. sequential (RNN)”, it is tempting to choose a CNN for classification tasks like sentiment classification since sentiment is usually determined by some key phrases; and to choose RNNs for a sequence modeling task like language modeling as it requires flexible modeling of context dependencies. But current NLP literature does not support such a clear conclusion. For example, RNNs perform well on document-level sentiment classification Tang et al. (2015); and Dauphin et al. (2016) recently showed that gated CNNs outperform LSTMs on language modeling tasks, even though LSTMs had long been seen as better suited. In summary, there is no consensus on DNN selection for any particular NLP problem.

This work compares CNNs, GRUs and LSTMs systematically on a broad array of NLP tasks: sentiment/relation classification, textual entailment, answer selection, question-relation matching in Freebase, Freebase path query answering and part-of-speech tagging.

Our experiments support two key findings. (i) CNNs and RNNs provide complementary information for text classification tasks. Which architecture performs better depends on how important it is to semantically understand the whole sequence. (ii) Learning rate changes performance relatively smoothly, while changes to hidden size and batch size result in large fluctuations.

Related Work

To our knowledge, there has been no systematic comparison of CNN and RNN on a large array of NLP tasks.

Vu et al. (2016) investigate CNN and basic RNN (i.e., no gating mechanisms) for relation classification. They report higher performance of CNN than RNN and give evidence that CNN and RNN provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative ngrams for the relation and only considers their resulting activations. Both Wen et al. (2016) and Adel and Schütze (2017) support CNN over GRU/LSTM for classification of long sentences. In addition, Yin et al. (2016) achieve better performance of attention-based CNN than attention-based LSTM for answer selection. Dauphin et al. (2016) further argue that a fine-tuned gated CNN can also model long-context dependency, getting new state-of-the-art in language modeling above all RNN competitors

In contrast, Arkhipenko et al. (2016) compare word2vec Mikolov et al. (2013), CNN, GRU and LSTM in sentiment analysis of Russian tweets, and find GRU outperforms LSTM and CNN.

In empirical evaluations, Chung et al. (2014) and Jozefowicz et al. (2015) found there is no clear winner between GRU and LSTM. In many tasks, they yield comparable performance and tuning hyperparameters like layer size is often more important than picking the ideal architecture.

Models

This section gives a brief introduction of CNN, GRU and LSTM.

Sequence $x$ contains $n$ entries. Each entry is represented by a $d$-dimensional dense vector; thus the input $x$ is represented as a feature map of dimensionality $d\times n$. Figure 1(a) shows the input layer as the lower rectangle with multiple columns.

Convolution Layer

Maxpooling

2 Gated Recurrent Unit (GRU)

GRU, as shown in Figure 1(b), models text $x$ as follows:

3 Long Short-Time Memory (LSTM)

LSTM is denoted in Figure 1(c). It models the word sequence $x$ as follows:

LSTM has three gates: input gate $i_{t}$, forget gate $f_{t}$ and output gate $o_{t}$. All gates are generated by a sigmoid function over the ensemble of input $x_{t}$ and the preceding hidden state $h_{t-1}$. In order to generate the hidden state at current step $t$, it first generates a temporary result $q_{t}$ by a tanh non-linearity over the ensemble of input $x_{t}$ and the preceding hidden state $h_{t-1}$, then combines this temporary result $q_{t}$ with history $p_{t-1}$ by input gate $i_{t}$ and forget gate $f_{t}$ respectively to get an updated history $p_{t}$, finally uses output gate $o_{t}$ over this updated history $p_{t}$ to get the final hidden state $h_{t}$.

Experiments

Sentiment Classification (SentiC) on Stanford Sentiment Treebank (SST) Socher et al. (2013). This dataset predicts the sentiment (positive or negative) of movie reviews. We use the given split of 6920 train, 872 dev and 1821 test sentences. As in Kalchbrenner et al. (2014); Le and Mikolov (2014), we treat labeled phrases that occur as subparts of training sentences as independent training instances. Measure: accuracy.

Relation Classification (RC) on SemEval 2010 task 8 Hendrickx et al. (2009). It consists of sentences which have been manually labeled with 19 relations (18 directed relations and Other), 8000 sentences in train and 2717 in test. As there is no dev set, we use 1500 training examples as dev, similar to Vu et al. (2016). Measure: F1.

Textual Entailment (TE) on Stanford Natural Language Inference (SNLI) Bowman et al. (2015). SNLI contains premise-hypothesis pairs, labeled with a relation (entailment, contradiction, neutral). After removing unlabeled pairs, we end up having 549,367 pairs for train, 9,842 for dev and 9,824 for test. Measure: accuracy.

Answer Selection (AS) on WikiQA Yang et al. (2015), an open domain question-answer dataset. We use the subtask that assumes that there is at least one correct answer for a question. The corresponding dataset consists of 20,360 question-candidate pairs in train, 1,130 in dev and 2,352 in test where we adopt the standard setup of only considering questions with correct answers in test. The task is to choose the correct answer(s) from some candidates for a question. Measures: MAP and MRR.

Question Relation Match (QRM). We utilize WebQSP Yih et al. (2016) dataset to create a large-scale relation detection task, benefitting from the availability of labeled semantic parses of questions. For each question, we (i) select the topic entity from the parse; (ii) select all the relations/relation chains (length $\leq$ 2) connecting to the topic entity; and (iii) set the relations/relation-chains in the labeled parse as positive and all the others as negative. Following Yih et al. (2016) and Xu et al. (2016), we formulate this task as a sequence matching problem. Ranking-loss is used for training. Measure: accuracy.

Path Query Answering (PQA) on the path query dataset released by Guu et al. (2015). It contains KB paths like $e_{h},r_{0},r_{1},\cdots,r_{t},e_{t}$, where head entity $e_{h}$ and relation sequence $r_{0},r_{1},\cdots,r_{t}$ are encoded to predict the tail entity $e_{t}$. There are 6,266,058/27,163/109,557 paths in train/dev/test, respectively. Measure: hit@10.

Part-of-Speech Tagging on WSJ. We use the setup of Blitzer et al. (2006); Petrov and McDonald (2012): sections 2-21 are train, section 22 is dev and section 23 is test. Measure: accuracy.

We organize above tasks in four categories. (i) TextC. Text classification, including SentiC and RC. (ii) SemMatch including TE, AS and QRM. (iii) SeqOrder. Sequence order, i.e., PQA. (iv) ContextDep. Context dependency, i.e., POS tagging. By investigating these four categories, we aim to discover some basic principles involved in utilizing CNNs / RNNs.

2 Experimental Setup

To fairly study the encoding capability of different basic DNNs, our experiments have the following design. (i) Always train from scratch, no extra knowledge, e.g., no pretrained word embeddings. (ii) Always train using a basic setup without complex tricks such as batch normalization. (iii) Search for optimal hyperparameters for each task and each model separately, so that all results are based on optimal hyperparameters. (iv) Investigate the basic architecture and utilization of each model: CNN consists of a convolution layer and a max-pooling layer; GRU and LSTM model the input from left to right and always use the last hidden state as the final representation of the input. An exception is for POS tagging, we also report bi-directional RNNs as this can make sure each word’s representation can encode the word’s context of both sides, like the CNN does.

Hyperparameters are tuned on dev: hidden size, minibatch size, learning rate, maximal sentence length, filter size (for CNN only) and margin in ranking loss in AS, QRM and PQA tasks.

3 Results & Analysis

Table 1 shows experimental results for all tasks and models and corresponding hyperparameters. For TextC, GRU performs best on SentiC and comparably with CNN in RC. For SemMatch, CNN performs best on AS and QRM while GRU (and also LSTM) outperforms CNN on TE. For SeqOrder (PQA), both GRU and LSTM outperform CNN. For ContextDep (POS tagging), CNN outperforms one-directional RNNs, but lags behind bi-directional RNNs.

The results for SeqOrder and ContextDep are as expected: RNNs are well suited to encode order information (for PQA) and long-range context dependency (for POS tagging). But for the other two categories, TextC and SemMatch, some unexpected observations appear. CNNs are considered good at extracting local and position-invariant features and therefore should perform well on TextC; but in our experiments they are outperformed by RNNs, especially in SentiC. How can this be explained? RNNs can encode the structure-dependent semantics of the whole input, but how likely is this helpful for TextC tasks that mostly depend on a few local regions? To investigate the unexpected observations, we do some error analysis on SentiC.

Table 2 shows examples (1) – (4) in which CNN predicts correctly while GRU predicts falsely or vice versa. We find that GRU is better when sentiment is determined by the entire sentence or a long-range semantic dependency – rather than some local key-phrases – is involved. Example (1) contains the phrases “won’t” and “miss” that usually appear with negative sentiment, but the whole sentence describes a positive sentiment; thus, an architecture like GRU is needed that handles long sequences correctly. On the other hand, modeling the whole sentence sometimes is a burden – neglecting the key parts. The GRU encodes the entire word sequence of the long example (3), making it hard for the negative keyphrase “loosens” to play a main role in the final representation. The first part of example (4) seems positive while the second part seems negative. As GRU chooses the last hidden state to represent the sentence, this might result in the wrong prediction.

Studying acc vs sentence length can also support this. Figure 2 (left) shows sentence lengths in SST are mostly short in train while close to normal distribution around 20 in dev and test. Figure 2 (right) depicts the accuracies w.r.t length ranges. We found that GRU and CNN are comparable when lengths are small, e.g., $<$10, then GRU gets increasing advantage over CNN when meet longer sentences. Error analysis shows that long sentences in SST mostly consist of clauses of inverse semantic such as “this version is not classic like its predecessor, but its pleasures are still plentiful”. This kind of clause often include a local strong indicator for one sentiment polarity, like “is not” above, but the successful classification relies on the comprehension of the whole clause.

Hence, which DNN type performs better in text classification task depends on how often the comprehension of global/long-range semantics is required.

This can also explain the phenomenon in SemMatch – GRU/LSTM surpass CNN in TE while CNN dominates in AS, as textual entailment relies on the comprehension of the whole sentence Bowman et al. (2015), question-answer in AS instead can be effectively identified by key-phrase matching Yin et al. (2016).

Sensitivity to hyperparameters

We next check how stable the performance of CNN and GRU are when hyperparameter values are varied. Figure 3 shows the performance of CNN, GRU and LSTM for different learning rates, hidden sizes and batch sizes. All models are relativly smooth with respect to learning rate changes. In contrast, variation in hidden size and batch size cause large oscillations. Nevertheless, we still can observe that CNN curve is mostly below the curves of GRU and LSTM in SentiC task, contrarily located at the higher place in AS task.

Conclusions

This work compared the three most widely used DNNs – CNN, GRU and LSTM – in representative sample of NLP tasks. We found that RNNs perform well and robust in a broad range of tasks except when the task is essentially a keyphrase recognition task as in some sentiment detection and question-answer matching settings. In addition, hidden size and batch size can make DNN performance vary dramatically. This suggests that optimization of these two parameters is crucial to good performance of both CNNs and RNNs.

References