Data-to-Text Generation with Content Selection and Planning

Introduction

Data-to-text generation broadly refers to the task of automatically producing text from non-linguistic input (?; ?). The input may be in various forms including databases of records, spreadsheets, expert system knowledge bases, simulations of physical systems, and so on. Table 1 shows an example in the form of a database containing statistics on NBA basketball games, and a corresponding game summary.

Traditional methods for data-to-text generation (?; ?) implement a pipeline of modules including content planning (selecting specific content from some input and determining the structure of the output text), sentence planning (determining the structure and lexical content of each sentence) and surface realization (converting the sentence plan to a surface string). Recent neural generation systems (Lebret et al. ?; Mei et al. ?; Wiseman et al. ?) do not explicitly model any of these stages, rather they are trained in an end-to-end fashion using the very successful encoder-decoder architecture (Bahdanau et al. ?) as their backbone.

Despite producing overall fluent text, neural systems have difficulty capturing long-term structure and generating documents more than a few sentences long. Wiseman et al. (?) show that neural text generation techniques perform poorly at content selection, they struggle to maintain inter-sentential coherence, and more generally a reasonable ordering of the selected facts in the output text. Additional challenges include avoiding redundancy and being faithful to the input. Interestingly, comparisons against template-based methods show that neural techniques do not fare well on metrics of content selection recall and factual output generation (i.e., they often hallucinate statements which are not supported by the facts in the database).

In this paper, we address these shortcomings by explicitly modeling content selection and planning within a neural data-to-text architecture. Our model learns a content plan from the input and conditions on the content plan in order to generate the output document (see Figure 1 for an illustration). An explicit content planning mechanism has at least three advantages for multi-sentence document generation: it represents a high-level organization of the document structure allowing the decoder to concentrate on the easier tasks of sentence planning and surface realization; it makes the process of data-to-document generation more interpretable by generating an intermediate representation; and reduces redundancy in the output, since it is less likely for the content plan to contain the same information in multiple places.

We train our model end-to-end using neural networks and evaluate its performance on RotoWire (Wiseman et al. ?), a recently released dataset which contains statistics of NBA basketball games paired with human-written summaries (see Table 1). Automatic and human evaluation shows that modeling content selection and planning improves generation considerably over competitive baselines.

Related Work

The generation literature provides multiple examples of content selection components developed for various domains which are either hand-built (?; ?; ?; ?) or learned from data (Barzilay and Lapata ?; Duboue and McKeown ?; ?; Liang et al. ?; Angeli et al. ?; Kim and Mooney ?; Konstas and Lapata ?). Likewise, creating summaries of sports games has been a topic of interest since the early beginnings of generation systems (Robin ?; Tanaka-Ishii et al. ?).

Earlier work on content planning has relied on generic planners (?), based on Rhetorical Structure Theory (?) and schemas (?). They defined content planners by analysing the target texts and devising hand-crafted rules. Duboue and McKeown (?) studied ordering constraints for content plans and Duboue and McKeown (?) learn a content planner from an aligned corpus of inputs and human outputs. A few researchers (?; ?) rank content plans according to a ranking function.

More recent work focuses on end-to-end systems instead of individual components. However, most models make simplistic assumptions such as generation without any content selection or planning (?; ?) or content selection without planning (Konstas and Lapata ?; Angeli et al. ?; Kim and Mooney ?). An exception are Konstas and Lapata (?) who incorporate content plans represented as grammar rules operating on the document level. Their approach works reasonably well with weather forecasts, but does not scale easily to larger databases, with richer vocabularies, and longer text descriptions. The model relies on the EM algorithm (?) to learn the weights of the grammar rules which can be very many even when tokens are aligned to database records as a preprocessing step.

Our work is closest to recent neural network models which learn generators from data and accompanying text resources. Most previous approaches generate from Wikipedia infoboxes focusing either on single sentences (Lebret et al. ?; ?; Sha et al. ?; Liu et al. ?) or short texts (?). Mei et al. (?) use a neural encoder-decoder model to generate weather forecasts and soccer commentaries, while Wiseman et al. (?) generate NBA game summaries (see Table 1). They introduce a new dataset for data-to-document generation which is sufficiently large for neural network training and adequately challenging for testing the capabilities of document-scale text generation (e.g., the average summary length is 330 words and the average number of input records is 628). Moreover, they propose various automatic evaluation measures for assessing the quality of system output. Our model follows on from Wiseman et al. (?) addressing the challenges for data-to-text generation identified in their work. We are not aware of any previous neural network-based approaches which incorporate content selection and planning mechanisms and generate multi-sentence documents. ? (?) introduce a content selection component (based on multi-instance learning) without content planning, while ? (?) propose a sentence planning mechanism which orders the contents of a Wikipedia infobox in order to generate a single sentence.

Problem Formulation

The input to our model is a table of records (see Table 1 left hand-side). Each record $r_{j}$ has four features including its type ($r_{j,1}$; e.g., LOSS, CITY), entity ($r_{j,2}$; e.g., Pacers, Miles Turner), value ($r_{j,3}$; e.g., 11, Indiana), and whether a player is on the home- or away-team ($r_{j,4}$; see column H/V in Table 1), represented as {$r_{j,k}\}_{k=1}^{4}$. The output $y$ is a document containing words $y=y_{1}\cdots y_{|y|}$ where $|y|$ is the document length. Figure 2 shows the overall architecture of our model which consists of two stages: (a) content selection and planning operates on the input records of a database and produces a content plan specifying which records are to be verbalized in the document and in which order (see Table 2) and (b) text generation produces the output text given the content plan as input; at each decoding step, the generation model attends over vector representations of the records in the content plan.

Let $r=\{r_{j}\}_{j=1}^{|r|}$ denote a table of input records and $y$ the output text. We model $p(y|r)$ as the joint probability of text $y$ and content plan $z$, given input $r$. We further decompose $p(y,z|r)$ into $p(z|r)$, a content selection and planning phase, and $p(y|r,z)$, a text generation phase:

In the following we explain how the components $p(z|r)$ and $p(y|r,z)$ are estimated.

The input to our model is a table of unordered records, each represented as features {$r_{j,k}\}_{k=1}^{4}$. Following previous work (Yang et al. ?; Wiseman et al. ?), we embed features into vectors, and then use a multilayer perceptron to obtain a vector representation $\mathbf{r}_{j}$ for each record:

Content Selection Gate

The context of a record can be useful in determining its importance vis-a-vis other records in the table. For example, if a player scores many points, it is likely that other meaningfully related records such as field goals, three-pointers, or rebounds will be mentioned in the output summary. To better capture such dependencies among records, we make use of the content selection gate mechanism as shown in Figure 3.

We first compute the attention scores $\alpha_{j,k}$ over the input table and use them to obtain an attentional vector $\mathbf{r}_{j}^{att}$ for each record $r_{j}$:

We next apply the content selection gating mechanism to $\mathbf{r}_{j}$, and obtain the new record representation $\mathbf{r}_{j}^{cs}$ via:

where $\odot$ denotes element-wise multiplication, and gate $\mathbf{g}_{j}\in{}^{n}$ controls the amount of information flowing from $\mathbf{r}_{j}$. In other words, each element in $\mathbf{r_{j}}$ is weighed by the corresponding element of the content selection gate $\mathbf{g}_{j}$.

Content Planning

In our generation task, the output text is long but follows a canonical structure. Game summaries typically begin by discussing which team won/lost, following with various statistics involving individual players and their teams (e.g., who performed exceptionally well or under-performed), and finishing with any upcoming games. We hypothesize that generation would benefit from an explicit plan specifying both what to say and in which order. Our model learns such content plans from training data. However, notice that RotoWire (see Table 1) and most similar data-to-text datasets do not naturally contain content plans. Fortunately, we can obtain these relatively straightforwardly following an information extraction approach (which we explain in Section 4).

Suffice it to say that plans are extracted by mapping the text in the summaries onto entities in the input table, their values, and types (i.e., relations). A plan is a sequence of pointers with each entry pointing to an input record $\{r_{j}\}_{j=1}^{|r|}$. An excerpt of a plan is shown in Table 2. The order in the plan corresponds to the sequence in which entities appear in the game summary. Let $z=z_{1}\dots z_{|z|}$ denote the content planning sequence. Each $z_{k}$ points to an input record, i.e., $z_{k}\in\{r_{j}\}_{j=1}^{|r|}$. Given the input records, the probability $p(z|r)$ is decomposed as:

Since the output tokens of the content planning stage correspond to positions in the input sequence, we make use of Pointer Networks (Vinyals et al. ?). The latter use attention to point to the tokens of the input sequence rather than creating a weighted representation of source encodings. As shown in Figure 2, given $\{r_{j}\}_{j=1}^{|r|}$, we use an LSTM decoder to generate tokens corresponding to positions in the input. The first hidden state of the decoder is initialized by $\operatorname{\operatorname{avg}}(\{\mathbf{r}_{j}^{cs}\}_{j=1}^{|r|})$, i.e., the average of record vectors. At decoding step $k$, let $\mathbf{h}_{k}$ be the hidden state of the LSTM. We model $p(z_{k}=r_{j}|z_{<k},r)$ as the attention over input records:

where the probability is normalized to $1$, and $\mathbf{W}_{c}$ are parameters. Once $z_{k}$ points to record $r_{j}$, we use the corresponding vector $\mathbf{r}_{j}^{cs}$ as the input of the next LSTM unit in the decoder.

Text Generation

The probability of output text $y$ conditioned on content plan $z$ and input table $r$ is modeled as:

where $y_{<t}=y_{1}\dots y_{t-1}$. We use the encoder-decoder architecture with an attention mechanism to compute $p(y|r,z)$.

We first encode the content plan $z$ into $\{\mathbf{e}_{k}\}_{k=1}^{|z|}$ using a bidirectional LSTM. Because the content plan is a sequence of input records, we directly feed the corresponding record vectors $\{\mathbf{r}_{j}^{cs}\}_{j=1}^{|r|}$ as input to the LSTM units, which share the record encoder with the first stage.

The text decoder is also based on a recurrent neural network with LSTM units. The decoder is initialized with the hidden states of the final step in the encoder. At decoding step $t$, the input of the LSTM unit is the embedding of the previously predicted word $y_{t-1}$. Let $\mathbf{d}_{t}$ be the hidden state of the $t$-th LSTM unit. The probability of predicting $y_{t}$ from the output vocabulary is computed via:

We further augment the decoder with a copy mechanism, i.e., the ability to copy words directly from the value portions of records in the content plan (i.e., $\{z_{k}\}_{k=1}^{|z|}$). We experimented with joint (?) and conditional copy methods (?). Specifically, we introduce a variable $u_{t}\in\{0,1\}$ for each time step to indicate whether the predicted token $y_{t}$ is copied ($u_{t}=1$) or not ($u_{t}=0$). The probability of generating $y_{t}$ is computed by:

The probability of copying from record values and generating from the vocabulary is globally normalized:

where $y_{t}\leftarrow z_{k}$ indicates that $y_{t}$ can be copied from $z_{k}$, $\mathbf{W}_{b}$ is shared as in Equation (1), and $\mathbf{W}_{y},\mathbf{b}_{y}$ are shared as in Equation (2).

Conditional Copy

The variable $u_{t}$ is first computed as a switch gate, and then is used to obtain the output probability:

Training and Inference

Our model is trained to maximize the log-likelihood of the goldStrictly speaking, the content plan is not gold since it was not created by an expert but is the output of a fairly accurate IE system. content plan given table records $r$ and the gold output text given the content plan and table records:

where $\mathcal{D}$ represents training examples (input records, plans, and game summaries). During inference, the output for input $r$ is predicted by:

where $z^{\prime}$ and $y^{\prime}$ represent content plan and output text candidates, respectively. For each stage, we utilize beam search to approximately obtain the best results.

Experimental Setup

We trained and evaluated our model on RotoWire Wiseman et al. (?), a dataset of basketball game summaries, paired with corresponding box- and line-score tables. The summaries are professionally written, relatively well structured and long (337 words on average). The number of record types is 39, the average number of records is 628, the vocabulary size is 11.3K words and token count is 1.6M. The dataset is ideally suited for document-scale generation. We followed the data partitions introduced in Wiseman et al. (?): we trained on 3,398 summaries, tested on 728, and used 727 for validation.

Content Plan Extraction

We extracted content plans from the RotoWire game summaries following an information extraction (IE) approach. Specifically, we used the IE system introduced in Wiseman et al. (?) which identifies candidate entity (i.e., player, team, and city) and value (i.e., number or string) pairs that appear in the text, and then predicts the type (aka relation) of each candidate pair. For instance, in the document in Table 1, the IE system might identify the pair “Jeff Teague, 20” and then predict that that their relation is “PTS”, extracting the record (Jeff Teague, 20, PTS). Wiseman et al. (?) train an IE system on RotoWire by determining word spans which could represent entities (i.e., by matching them against players, teams or cities in the database) and numbers. They then consider each entity-number pair in the same sentence, and if there is a record in the database with matching entities and values, the pair is assigned the corresponding record type or otherwise given the label “none” to indicate unrelated pairs.

We adopted their IE system architecture which predicts relations by ensembling 3 convolutional models and 3 bidirectional LSTM models. We trained this system on the training portion of the RotoWire corpus.A bug in the code of Wiseman et al. (?) excluded number words from the output summary. We corrected the bug and this resulted in greater recall for the relations extracted from the summaries. See the supplementary material for more details. On held-out data it achieved 94% accuracy, and recalled approximately 80% of the relations licensed by the records. Given the output of the IE system, a content plan simply consists of (entity, value, record type, h/v) tuples in their order of appearance in a game summary (the content plan for the summary in Table 1 is shown in Table 2). Player names are pre-processed to indicate the individual’s first name and surname (see Isaiah and Thomas in Table 2); team records are also pre-processed to indicate the name of team’s city and the team itself (see Boston and Celtics in Table 2).

Training Configuration

We validated model hyperparameters on the development set. We did not tune the dimensions of word embeddings and LSTM hidden layers; we used the same value of $600$ reported in Wiseman et al. (?). We used one-layer pointer networks during content planning, and two-layer LSTMs during text generation. Input feeding (Luong et al. ?) was employed for the text decoder. We applied dropout (Zaremba et al. ?) at a rate of $0.3$. Models were trained for $25$ epochs with the Adagrad optimizer (Duchi et al. ?); the initial learning rate was $0.15$, learning rate decay was selected from $\{0.5,0.97\}$, and batch size was 5. For text decoding, we made use of BPTT (?) and set the truncation size to $100$. We set the beam size to $5$ during inference. All models are implemented in OpenNMT-py (?).

Results

We evaluated model output using the metrics defined in Wiseman et al. (?). The idea is to employ a fairly accurate IE system (see the description in Section 4) on the gold and automatic summaries and compare whether the identified relations align or diverge.

Let $\hat{y}$ be the gold output, and $y$ the system output. Content selection (CS) measures how well (in terms of precision and recall) the records extracted from $y$ match those found in $\hat{y}$. Relation generation (RG) measures the factuality of the generation system as the proportion of records extracted from $y$ which are also found in $r$ (in terms of precision and number of unique relations). Content ordering (CO) measures how well the system orders the records it has chosen and is computed as the normalized Damerau-Levenshtein Distance between the sequence of records extracted from $y$ and $\hat{y}$. In addition to these metrics, we report BLEU (?), with human-written game summaries as reference.

Our results on the development set are summarized in Table 3. We compare our Neural Content Planning model (NCP for short) against the two encoder-decoder (ED) models presented in Wiseman et al. (?) with joint copy (JC) and conditional copy (CC), respectively. In addition to our own re-implementation of these models, we include the best scores reported in Wiseman et al. (?) which were obtained with an encoder-decoder model enhanced with conditional copy. Table 3 also shows results when NCP uses oracle content plans (OR) as input. In addition, we report the performance of a template-based generator (Wiseman et al. ?) which creates a document consisting of eight template sentences: an introductory sentence (who won/lost), six player-specific sentences (based on the six highest-scoring players in the game), and a conclusion sentence.

As can be seen, NCP improves upon vanilla encoder-decoder models (ED+JC, ED+CC), irrespective of the copy mechanism being employed. In fact, NCP achieves comparable scores with either joint or conditional copy mechanism which indicates that it is the content planner which brings performance improvements. Overall, NCP+CC achieves best content selection and content ordering scores in terms of BLEU. Compared to the best reported system in Wiseman et al. (?), we achieve an absolute improvement of approximately 12% in terms of relation generation; content selection precision also improves by 5% and recall by 15%, content ordering increases by 3%, and BLEU by 1.5 points. The results of the oracle system (NCP+OR) show that content selection and ordering do indeed correlate with the quality of the content plan and that any improvements in our planning component would result in better output. As far as the template-based system is concerned, we observe that it obtains low BLEU and CS precision but scores high on CS recall and RG metrics. This is not surprising as the template system is provided with domain knowledge which our model does not have, and thus represents an upper-bound on content selection and relation generation. We also measured the degree to which the game summaries generated by our model contain redundant information as the proportion of non-duplicate records extracted from the summary by the IE system. 84.5% of the records in NCP+CC are non-duplicates compared to Wiseman et al. (?) who obtain 72.9% showing that our model is less repetitive.

We further conducted an ablation study with the conditional copy variant of our model (NCP+CC) to establish whether improvements are due to better content selection (CS) and/or content planning (CP). We see in Table 4 that content selection and planning individually contribute to performance improvements over the baseline (ED+CC), and accuracy further increases when both components are taken into account. In addition we evaluated these components on their own (independently of text generation) by comparing the output of the planner (see $p(z|r)$ block in Figure 2) against gold content plans obtained using the IE system (see row NCP in Table 4. Compared to the full system (NCP+CC), content selection precision and recall are higher (by 4.5% and 2%, respectively) as well as content ordering (by 1.8%). In another study, we used the CS and CO metrics to measure how well the generated text follows the content plan produced by the planner (instead of arbitrarily adding or removing information). We found out that NCP+CC generates game summaries which follow the content plan closely: CS precision is higher than 85%, CS recall is higher than 93%, and CO higher than 84%. This reinforces our claim that higher accuracies in the content selection and planning phases will result in further improvements in text generation.

The test set results in Table 5 follow a pattern similar to the development set. NCP achieves higher accuracy in all metrics including relation generation, content selection, content ordering, and BLEU compared to Wiseman et al. (?). We provide examples of system output in Table 6 and the supplementary material.

Human-Based Evaluation

We conducted two human evaluation experiments using the Amazon Mechanical Turk (AMT) crowdsourcing platform. The first study assessed relation generation by examining whether improvements in relation generation attested by automatic evaluation metrics are indeed corroborated by human judgments. We compared our best performing model (NCP+CC), with gold reference summaries, a template system and the best model of Wiseman et al. (?). AMT workers were presented with a specific NBA game’s box score and line score, and four (randomly selected) sentences from the summary. They were asked to identify supporting and contradicting facts mentioned in each sentence. We randomly selected 30 games from the test set. Each sentence was rated by three workers.

The left two columns in Table 7 contain the average number of supporting and contradicting facts per sentence as determined by the crowdworkers, for each model. The template-based system has the highest number of supporting facts, even compared to the human gold standard. TEMPL does not perform any content selection, it includes a large number of facts from the database and since it does not perform any generation either, it exhibits a few contradictions. Compared to WS-2017 and the Gold summaries, NCP+CC displays a larger number of supporting facts. All models are significantlyAll significance differences reported throughout this paper are with a level less than 0.05. different in the number of supporting facts (#Supp) from TEMPL (using a one-way ANOVA with post-hoc Tukey HSD tests). NCP+CC is significantly different from WS-2017 and Gold. With respect to contradicting facts (#Cont), Gold and TEMPL are not significantly different from each other but are significantly different from the neural systems (WS-2017, NCP+CC).

In the second experiment, we assessed the generation quality of our model. We elicited judgments for the same 30 games used in the first study. For each game, participants were asked to compare a human-written summary, NCP with conditional copy (NCP+CC), Wiseman et al.’s (?) best model, and the template system. Our study used Best-Worst Scaling (BWS; ? ?), a technique shown to be less labor-intensive and providing more reliable results as compared to rating scales (?). We arranged every 4-tuple of competing summaries into 6 pairs. Every pair was shown to three crowdworkers, who were asked to choose which summary was best and which was worst according to three criteria: Grammaticality (is the summary fluent and grammatical?), Coherence (is the summary easy to read? does it follow a natural ordering of facts?), and Conciseness (does the summary avoid redundant information and repetitions?). The score of a system for each criterion is computed as the difference between the percentage of times the system was selected as the best and the percentage of times it was selected as the worst (?). The scores range from $-100$ (absolutely worst) to $+100$ (absolutely best).

The results of the second study are summarized in Table 7. Gold summaries were perceived as significantly better compared to the automatic systems across all criteria (again using a one-way ANOVA with post-hoc Tukey HSD tests). NCP+CC was perceived as significantly more grammatical than WS-2017 but not compared to TEMPL which does not suffer from fluency errors since it does not perform any generation. NCP+CC was perceived as significantly more coherent than TEMPL and WS-2017. The template fairs poorly on coherence, its output is stilted and exhibits no variability (see top block in Table 6). With regard to conciseness, the neural systems are significantly worse than TEMPL, while NCP+CC is significantly better than WS-2017. By design the template cannot repeat information since there is no redundancy in the sentences chosen to verbalize the summary.

Taken together, our results show that content planning improves data-to-text generation across metrics and systems. We find that NCP+CC overall performs best, however there is a significant gap between automatically generated summaries and human-authored ones.

Conclusions

We presented a data-to-text generation model which is enhanced with content selection and planning modules. Experimental results (based on automatic metrics and judgment elicitation studies) demonstrate that generation quality improves both in terms of the number of relevant facts contained in the output text, and the order according to which these are presented. Positive side-effects of content planning are additional improvements in the grammaticality, and conciseness of the generated text. In the future, we would like to learn more detail-oriented plans involving inference over multiple facts and entities. We would also like to verify our approach across domains and languages.

References

Comparison with the Results in Wiseman et al.’s (?) Webpage

There was a bug in the dataset creation of Wiseman et al. (?) which they identified and corrected. They also posted updated scores on their webpage.https://github.com/harvardnlp/data2text We have used this corrected dataset in our experiments. We then discovered a bug in their code which computes the automatic metrics. The scores reported in this paper are using the corrected automatic metrics. To make the scores on our paper comparable to the numbers published on the webpage of Wiseman et al. (?), we recompute here our scores with older IE metrics (without the bug fix) in Table 8 (development set) and Table 9 (test set).

Qualitative Examples

Table 10 shows two sample documents generated using the template system, Wiseman et al. (?) (WS-2017) and our neural content planning model with conditional copy (NCP+CC). The text is highlighted in blue if it agrees with respective box/line scores and red if the text contradicts box/line scores. We also use the orange color to highlight repetitions.

The template documents are gold standard in relation generation accuracy and they appear all in blue. The documents of Wiseman et al. (?) show instances of contradictions and tend to be verbose containing duplicate text too. In contrast, our neural content planning model generates more factual text with fewer contradictions to box/line scores and less duplicate information.