Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

Introduction

With advances in neural machine learning Sutskever et al. (2014); Gehring et al. (2017); Vaswani et al. (2017) and availability of the huge amount of human conversations on social media Adiwardana et al. (2020), building an open domain dialogue system with data-driven approaches has attracted increasing attention from the community of artificial intelligence and natural language processing. In this work, we are interested in generative approaches. Generative models for open domain dialogues are notorious for replying with generic and bland responses, resulting in meaningless and boring conversations Li et al. (2015). Such deficiency is particularly severe when human participants attempt to dive into specific topics in conversation Dinan et al. (2019). As a result, there is still a big gap between conversation with existing systems and conversation with humans.

Very recently, there emerge two lines of research that seem promising to bridge the gap. One is to apply large-scale pre-trained language models, such as GPT-2 Radford et al. (2019), to the task of open domain dialogue generation. Prototypes such as DialoGPT Zhang et al. (2019c) have exhibited compelling performance on generating responses that make sense under conversation contexts and at the same time carry specific content for keeping the conversation going. While the giant language models can memorize enough patterns in language during pre-training, they only capture “average” semantics of the data Zhang et al. (2019c). As a result, responses could still be bland or inappropriate when specific knowledge is required, as illustrated by the example in Table 1. The other line is to ground dialogue generation by extra knowledge such as unstructured documents Zhao et al. (2020). By the means, the documents (e.g., wiki articles) serve as content sources, and make a dialogue system knowledgeable regarding to a variety of concepts in discussion. However, collecting enough dialogues that are naturally grounded on documents for model training is not trivial. Although some benchmarks built upon crowd-sourcing have been released by recent papers Zhou et al. (2018b); Dinan et al. (2019); Gopalakrishnan et al. (2019), the small training size makes the generation models generalize badly on unseen topics Dinan et al. (2019) and the cost of building such data also prevents from transferring the techniques proved on the benchmarks to new domains and new languages.

Encouraged by the results on pre-training for dialogue generation and knowledge-grounded dialogue generation, and motivated by the problems in both sides, we consider bringing the two together in this work. Specifically, we propose knowledge-grounded dialogue generation with pre-trained language models in order to endow a generative model with both rich knowledge and good generalization abilityIn this paper, we assume that knowledge is retrieved from documents.. The challenge is that pre-trained language models often set constraints on the maximum number of tokens they can handle (e.g., the maximum number for GPT-2 Radford et al. (2019) is $1024$), and thus hinders exploitation of the knowledge text which could be rather long and redundant (e.g., in Wizard of Wikipedia Dinan et al. (2019), on average each conversation context is associated with $61.2$ sentences retrieved from wiki articles, and the average number of tokens in the extra knowledge is $1625.6$). Indeed, the conflict between model capacity and the ability required for processing long knowledge input represents an essential obstacle for applying pre-trained language models to knowledge-grounded dialogue generation, since on the one hand we always have to set up an upper bound to the capacity of pre-trained models in order to handle massive text corpus, and on the other hand we need to keep sufficient candidates with rich enough content in the procedure of response generation in order to guarantee the recall of relevant knowledge.

To overcome the challenge, we consider equipping the pre-trained response generation model with a knowledge selection module whereby the redundant knowledge input is slimmed with relevant information (regarding to conversation contexts) kept to meet the capacity constraint. While some recent papers on knowledge-grounded dialogues have paid attention to the problem of knowledge selection Lian et al. (2019); Kim et al. (2020); Ren et al. (2019), the knowledge selection module is either deeply coupled with the specially configured models Lian et al. (2019); Ren et al. (2019) and thus is incompatible with the pre-trained language models, or it is learned with human annotations Dinan et al. (2019); Kim et al. (2018) which are difficult to obtain in practice (e.g., the dataset in Zhou et al. (2018b) does not contain annotations for knowledge selection). Therefore, we propose an unsupervised approach where learning of knowledge selection and fine-tuning of response generation are jointly conducted with unlabeled dialogues. Specifically, we build the knowledge selection module on the basis of BERT, and formalize knowledge selection as a sequence prediction process, by which the model can take advantage of the pre-training techniques and dynamically determine the relevant knowledge for a given context. The learning algorithm starts from training with pseudo ground-truth that is constructed by making full use of responses as an alternation of human annotations, and then alternatively updates the knowledge selection model and the response generation model through a reinforcement learning approach and a curriculum learning approach respectively. Thus, knowledge selection is further optimized with the feedback from response generation, and the knowledge used for fine-tuning the response generation model gradually moves from the pseudo ground-truth to the prediction of the knowledge selection module.

We test the proposed method on two benchmarks of knowledge-grounded dialogue generation: Wizard of Wikipedia Dinan et al. (2019) and CMU Document Grounded Conversations Zhou et al. (2018b). Evaluation results indicate that our model can significantly outperform state-of-the-art methods as well as a few pre-trained models used in heuristic ways, and thus achieves new state-of-the-art on the benchmarks. Moreover, as a byproduct, the knowledge selection module also outperforms the state-of-the-art model in terms of accuracy of knowledge selection on Wizard of Wikipedia, implying that other models could also benefit from the component.

Our contributions in this paper are three-fold: (1) proposal of a knowledge selection module for applying pre-trained language models to the task of knowledge-grounded dialogue generation; (2) proposal of an unsupervised approach in which learning of knowledge selection and fine-tuning of the pre-trained model are conducted in a joint manner; and (3) empirical verification of the effectiveness of the proposed method on benchmarks of knowledge-grounded dialogue generation.

Related Work

Early work on end-to-end open domain dialogue generation is inspired by the research of machine translation (Ritter et al., 2011; Shang et al., 2015; Vinyals and Le, 2015). Later, the vanilla encoder-decoder architecture is widely extended to improve diversity of responses Li et al. (2015); Xing et al. (2017a); Zhao et al. (2017); Tao et al. (2018); to model the structure of conversation contexts Serban et al. (2016, 2017); Xing et al. (2017b); Zhang et al. (2019a); to control attributes of responses Xu et al. (2019); Zhou et al. (2017); Zhang et al. (2018a); Wang et al. (2018); See et al. (2019); and to bias responses to some specific personas Li et al. (2016); Zhang et al. (2018b). Recently, grounding dialogue generation by extra knowledge is emerging as an important step towards human-like conversational AI, where the knowledge could be obtained from knowledge graphs Zhou et al. (2018a); Moon et al. (2019); Tuan et al. (2019), retrieved from unstructured documents Dinan et al. (2019); Lian et al. (2019); Zhao et al. (2020); Kim et al. (2020), or extracted from visual background Mostafazadeh et al. (2017); Shuster et al. (2018); Huber et al. (2018). In this work, we study document-grounded dialogue generation. Rather than learning from scratch like most existing work, we take advantage of the pre-trained language models and achieve new state-of-the-art on the benchmarks of the task.

Big, deep neural language models pre-trained on huge unlabeled text corpus have led to strong improvements on numerous natural language understanding and natural language generation benchmarks Devlin et al. (2018); Yang et al. (2019); Liu et al. (2019); Radford et al. (2019); Song et al. (2019); Dong et al. (2019); Lewis et al. (2019), and therefore are revolutionizing almost the full spectrum of NLP applications Raffel et al. (2019); Sun et al. (2019b); Qiao et al. (2019); Zhang et al. (2019b); Lample and Conneau (2019) and some interdisciplinary applications in NLP and computer vision Lu et al. (2019); Su et al. (2019); Sun et al. (2019a). In the context of dialogue generation, by fine-tuning GPT-2 Radford et al. (2019) in different sizes on social media data, recent work has Zhang et al. (2019c); Wolf et al. (2019) shown promising progress on conversation engagement and commonsense question-answering. In this work, we further explore the application of pre-training to the task of open domain dialogue generation by equipping the pre-trained language models with external knowledge. Different from a very recent paper on pre-training for low-resource knowledge-grounded dialogue generation Zhao et al. (2020), the work presents an in-depth investigation on how to release the power of the existing pre-trained language models on the task when input exceeds the capacity of the models.

Preliminary

Suppose that we have a dataset $\mathcal{D}=\{(U_{i},D_{i},r_{i})\}_{i=1}^{N}$, where $\forall i\in\{1,\ldots,N\}$, $U_{i}$ is a dialogue context, $D_{i}$ is a document that contains relevant knowledge regarding to $U_{i}$, and $r_{i}$ is a response to $U_{i}$ based on $D_{i}$. The goal is to learn a generation model $P(r|U,D;\theta)$ ($\theta$ denotes the parameters of the model) from $\mathcal{D}$, and thus given a new dialogue context $U$ associated with a document $D$, one can generate a response $r$ following $P(r|U,D;\theta)$.

2 Pre-trained Language Models

We define $P(r|U,D;\theta)$ on the basis of GPT-2 from OpenAI Radford et al. (2019). GPT-2 are transformer language models with a stack of masked multi-head self-attention layers, and are learned from large scale web text. To apply GPT-2 to the task of knowledge-grounded dialogue generation, we formulate the generation problem as

where $g(U,D)$ tailors $U\cup D$ to meet the length constraint of a GPT-2 model as the input of generation, and $r_{t}$ refers to the $t$-th token of $r$ whose length is supposed to be $l_{r}$. The problem then boils down to (1) how to define $g(U,D)$; and (2) how to fine-tune $\theta$ (and probably learn $g(U,D)$) with $\mathcal{D}$.

In this work, we assume that labels that indicate the ground-truth knowledge are not available, which is practical but makes the problem even more challenging. Since $D$ could be rather redundant with a lot of information irrelevant with the topic or the context of the conversation, simply truncating the concatenation of sentences of $U$ and $D$ as $g(U,D)$ may cut the relevant knowledge and introduce noise into response generation, which hurts the performance of the GPT-2 model, as will be demonstrated in the experiments. Therefore, we consider learning a $g(U,D)$ that can distill useful information from $D$ for the GPT-2 model, as will be elaborated in the next section.

Approach

Heading for learning a $g(U,D)$ for applying GPT-2 to the task of knowledge-grounded dialogue generation, we need to deal with several challenges: (1) how to model the correlation between a context and the external knowledge; (2) how to learn $g(U,D)$ when labels of ground-truth knowledge are absent; and (3) how to jointly optimize $g(U,D)$ and the GPT-2 model with $\mathcal{D}$, and thus the two can boost each other. Figure 1 illustrates the architecture of the model. On the basis of the transformer architecture, the knowledge selection module is made up of a context-aware knowledge encoder and a sequential knowledge selector. The former captures interaction patterns between a context $U$ and each sentence in $D$ through a stack of self-attention layers, and the patterns are then fed to the latter to decode useful knowledge one sentence per step. Since human annotations are not accessible, the learning method begins with pseudo ground-truth constructed by making full use of responses, and optimization of $g(U,D)$ and optimization of the GPT-2 generation model are alternatively conducted with a reinforcement learning approach and a curriculum learning approach respectively.

We choose BERT Devlin et al. (2018) as the backbone of the encoder. Thus, the encoder can take advantage of pre-training, and the multi-layer bi-directional attention mechanism in BERT allows a dialogue context and the associated knowledge to sufficiently interact with each other, resulting in context-aware knowledge representations. Specifically, let $U=(u_{1},\ldots,u_{n})$ and $D=(d_{1},\ldots,d_{m})$ be the context and the knowledge respectively, then we concatenate $\{u_{i}\}_{i=1}^{n}$ as $(w_{1}^{u},\cdots,w_{l_{u}}^{u})$ with $w_{i}^{u}$ the $i$-th word and $l_{u}$ the length of the sequence, and define the input of the encoder as $\mathcal{S}=(S_{1},\ldots,S_{m})$ with $S_{i}$ formulated as

2 Sequential Knowledge Selector

With $E$ as input, the sequential knowledge selector determines a subset of $D$ (denoted as $D^{\prime}$) as the relevant knowledge and exploits $D^{\prime}$ to construct $g(U,D)$. Since there may exist one-to-many relations between a context and the relevant knowledge Kim et al. (2020), the size of $D^{\prime}$ could vary from context to context. Therefore, we regard the construction of $D^{\prime}$ as a sequence prediction process in which $D^{\prime}$ starts from an empty set and gradually expands by adding one sentence from $D$ per step. By this means, the size of $D^{\prime}$ can also be viewed as a parameter and is dynamically determined according to the given context. Formally, we maintain a sequence of hidden states $\{s_{t}\}_{t=0}^{T_{U,D}}$ with the initial state $s_{0}$ a trainable parameter, and weight $\{d_{i}\}_{i=1}^{m}$ by an attention mechanism which can be formulated as

where $W_{e}$, $W_{s}$, $b$ and $v$ are trainable parameters. Then $d_{j_{t}}$ will be added to $D^{\prime}$ if $j_{t}=\operatorname{argmax}_{i\in\{1,\ldots,m\}}P(d_{i}|U,d_{j_{1:t-1}})$. After that, $s_{t+1}$ is calculated by

To determine $T_{U,D}$, we introduce a special embedding $e_{spe}$ into $E$, and terminate the prediction process if $e_{spe}$ is selected or an upper bound $T_{max}$ is reached. Finally, $g(U,D)$ is defined as the concatenation of the sentences in $U\cup D^{\prime}$.

3 Learning Method

Learning a $g(U,D)$ without human annotations is not trivial. For example, in a recent paper Kim et al. (2020), when human labels are removed, the accuracy of knowledge selection drops from $27$% to $0.3$%. Moreover, since knowledge selection and response generation are entangled, ideally we hope $g(U,D)$ and the GPT-2 model can enhance each other in learning. However, as the parameters of $g(U,D)$ are far from optimal at the early stage, it is very possible that noise from $g(U,D)$ will be fed to the GPT-2 model and then flows back to the learning procedure of $g(U,D)$, resulting in inferior models on both sides. To cope with the challenges, we propose a joint optimization strategy with weak supervision as follows. The learning algorithm is summarized in Algorithm 1.

To alleviate error accumulation in joint optimization, we consider constructing weak supervision and utilize the signals to warm up the learning of $g(U,D)$ and the fine-tuning of GPT-2 beforehand. The intuition is that responses from humans carry clues to relevance of the knowledge candidates, and thus can be used to construct pseudo ground-truth. To be specific, we first sort $D=\{d_{t}\}_{t=1}^{m}$ in a descending order as $\{d_{j_{t}}\}_{t=1}^{m}$ according to $\{\operatorname{Sim}(d_{t},r)\}_{t=1}^{m}$ where $\operatorname{Sim}(\cdot,\cdot)$ denotes a similarity function, and then build a subset of $D$ by

where $d_{j_{1:t}}$ refers to the concatenation of $\{d_{j_{i}}\}_{i=1}^{t}$. With $\bar{D}$, $g(U,D)$ and the GPT-2 model are optimized via maximum likelihood estimation (MLE) on $\mathcal{D}_{K}=\{(U_{i},D_{i},\bar{D}_{i})\}_{i=1}^{N}$ and $\mathcal{D}_{G}=\{(U_{i},\bar{D}_{i},r_{i})\}_{i=1}^{N}$ respectively.

Joint Optimization: the Reinforcement Step.

Joint Optimization: the Curriculum Step.

Though $g(U,D)$ has been pre-trained with the pseudo ground-truth $\bar{D}$, the relevant knowledge provided by the model (i.e., $D^{\prime}$) may still be worse than $\bar{D}$ at the beginning of fine-tuning. Therefore, we mix $D^{\prime}$ and $\bar{D}$ and exploit a curriculum learning strategy to fine-tune the GPT-2 model where $D^{\prime}$ and $\bar{D}$ are regarded as hard materials and easy materials respectively and fine-tuning gradually moves from $\bar{D}$ to $D^{\prime}$. Formally, the loss function for fine-tuning the GPT-2 model is defined by

where $\{z_{i}\}$ are sampled from a Bernoulli distribution parameterized by $p$. By gradually shrinking $p$, the generation model will be exposed to more hard materials with the learning procedure going on.

Experiments

We conduct experiments on Wizard of Wikipedia (Wizard) and CMU Document Grounded Conversations (CMU$\_$DoG) Zhou et al. (2018b).

Both datasets are built with crowd-sourcing on Amazon Mechanical Turk, employ Wikipedia as the knowledge base, and are split into training sets, validation sets, and test sets by the data owners. Topics in Wizard cover a wide range ($1,365$ in total), and each conversation happens between a wizard who has access to the knowledge about a specific topic and an apprentice who is just eager to learn from the wizard about the topic. The test set is split into two subsets: Test Seen and Test Unseen. Test Seen contains new dialogues with topics appearing in the training set, while topics in Test Unseen never appear in the training set and the validation set. We follow Dinan et al. (2019) and conduct the pre-processing with the code published on ParlAIhttps://github.com/facebookresearch/ParlAI/blob/master/projects/wizard_of_wikipedia. Different from Wizard, CMU$\_$DoG focuses on movie domain, and besides wizard-apprentice conversations, the data also contain conversations between two workers who know the document and try to discuss the content in depth. To better compare with the baselines, we adopt the version shared at https://github.com/lizekang/ITDD. In both data, only the turns where knowledge is accessible are considered in response generation. More details are described in supplementary material.

We choose perplexity (PPL) of the ground-truth responses, BOW Embedding (Liu et al., 2016), and unigram F1 Dinan et al. (2019) as metrics, where Embedding-based metrics are computed with an NLG evaluation open source available at https://github.com/Maluuba/nlg-eval, and F1 is calculated with the code published at https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/metrics.py.

Besides automatic evaluation, we randomly sample $300$ examples from Test Seen, Test Unseen, and the test set of CMU$\_$DoG respectively, and recruit $3$ well-educated native speakers as annotators for human evaluation. To each annotator, an example is presented with a context, the associated external knowledgeFor ease of labeling, only the ground-truth knowledge is shown to the annotators in Wizard., and model responses (top 1 in greedy search) that are randomly shuffled to hide their sources. The annotators then judge the quality of the responses from three aspects, including fluency, context coherence and knowledge relevance, and assign a score in $\{0,1,2\}$ (representing “bad”, “fair”, and “good”) to each response for each aspect. Each response receives $3$ scores per aspect, and the agreement among the annotators is measured via Fleiss’ kappa Fleiss (1971).

2 Baselines

The following models are selected as baselines:

Transformer Memory Network (TMN): the model proposed in Dinan et al. (2019) along with the release of the Wizard data. We implement it using the code shared at https://github.com/facebookresearch/ParlAI/blob/master/projects/wizard_of_wikipedia.

Incremental Transformer with Deliberation Decoder (ITDD): a transformer-based model Li et al. (2019) that incrementally encodes multi-turn dialogues and knowledge and decodes responses with a deliberation technique. We implement it using the code shared at https://github.com/lizekang/ITDD.

Sequential Knowledge Transformer (SKT): a sequential latent variable model with state-of-the-art performance on knowledge selection published in a very recent paper (Kim et al., 2020). Since human labels that indicate ground-truth knowledge are crucial to the performance of the model, we only involve it as a baseline on the Wizard data. The model is implemented with the code shared at https://github.com/bckim92/sequential-knowledge-transformer.

Disentangled Response Decoder (DRD): a model that tackles the low-resource challenge with pre-training techniques Zhao et al. (2020). We choose the one in which all parameters are fine-tuned with the full training data after pre-training as the baseline, since such a configuration results in state-of-the-art performance on Wizard, as reported in Zhao et al. (2020).

We name our model KnowledGPT. Besides the baselines described above, the following pre-trained models are also included in comparison in order to have a thorough understanding towards the proposed method: (1) GPT-2trunc. We concatenate a context and the associated knowledge as a long document, and then truncate the document to meet the length constraint of the GPT-2 model. This is to check if the simple heuristics work for the task. Note that in Wizard, we randomly mix the ground-truth knowledge with others and repeat the procedure $8$ times. The means with standard deviation (i.e., numbers in “( )”) are reported to remove randomness; and (2) SKT+GPT-2. We feed the candidate selected by SKT to GPT-2 for response generation. This is to examine if we can simply replace the proposed knowledge selection module as well as the learning approach with an off-the-shelf knowledge selection model. Similar to SKT, the comparison is only conducted on Wizard.

3 Implementation Details

In both Wizard and CMU_DoG, we set the hidden size and the number of layers of the sequential knowledge selector as $256$ and $1$ respectively. $T_{max}$ for $D^{\prime}$ is set as $1$ in Wizard, and $2$ in CMU_DoG. We choose BERT (110M) and GPT-2 (117M) as the pre-trained language models in KnowledGPT, and implement the models with the code in https://github.com/huggingface/transformers. We employ greedy search in response decoding. All models are learned with Adam Kingma and Ba (2015) optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. In warming up, we define $\operatorname{Sim}(\cdot,\cdot)$ as unigram F1, and optimize $g(U,D)$ and the GPT-2 model with the pseudo ground-truth for $1000$ steps with a batch size of $64$. In joint optimization, the batch size is set as $128$, and the learning rates for $g(U,D)$ and GPT-2 are set as $5e-6$ and $5e-5$ respectively. The learning rate will be halved if there is no improvement in terms of PPL on the validation sets. The parameter $p$ of the Bernoulli distribution in the curriculum step is initially set as $1.0$ and anneals with a rate of $1e-5$. Early stopping on validation is adopted as a regularization strategy.

4 Evaluation Results

Table 2 and Table 3 report evaluation results on Wizard and CMU$\_$DoG respectively. KnowledGPT achieves new state-of-the-art on most metrics in both datasets, which demonstrates the effectiveness of large-scale pre-trained language models on the task of knowledge-grounded dialogue generation. GPT-2trunc is worse than KnowledGPT, due to (1) knowledge loss: we find that in $53\%$ test examples (Test Seen+Test Unseen), the ground-truth knowledge is cut. In this case, GPT-2trunc only relies on the context, the related knowledge in other candidates (thanks to the one-to-many relations between a context and knowledge), and the knowledge packed in the parameters of GPT-2 for responding, which explains the comparable performance with SKT and DRD; and (2) noisy input: even though the ground-truth knowledge is kept, the redundant and irrelevant information in the knowledge candidates are still harmful. Evidence is that GPT-2trunc is worse than KnowledGPT on CMU_DoG even though we do not cut anything on the knowledge (the maximum length of the knowledge input is $502$, and thus is within the constraint of GPT-2). KnowledGPT also outperforms SKT+GPT-2 on Wizard, because (1) KnowledGPT is more accurate than SKT on knowledge selection, even though it does not leverage any human annotations in learning. In fact, the accuracy scores of knowledge selection for SKT are $26.8$ and $18.3$ on Test Seen and Test Unseen respectively, while the two numbers are $28.0$ and $25.4$ respectively for KnowledGPT; and (2) in KnowledGPT, knowledge selection and response generation are jointly optimized.

Table 4 shows human evaluation results. While the three models are comparable on fluency, KnowledGPT is superior to the others on both context coherence and knowledge relevance, which is consistent with the results on automatic metrics. All kappa values are no less than $0.6$, indicating substantial agreement among the annotators. We present a case study in supplementary material.

5 Discussions

Ablation study. To understand the impact of the learning strategies on model performance, we compare the full KnowledGPT with the following variants: (1) -pseudo: the warming up stage is removed; (2) -joint: the joint optimization stage is removed; (3) -reinforcement: $g(U,D)$ is fixed after it is optimized with MLE on $\mathcal{D}_{K}$; and (4) -curriculum: GPT-2 is fixed after it is optimized with MLE on $\mathcal{D}_{G}$. Table 5 reports the evaluation results. We can conclude that (1) the pseudo ground-truth plays a crucial role in Wizard, as removing the step causes dramatic performance drop. This is because in Wizard, there is a strong correlation between the knowledge and human responses. The results indicate that though the pseudo ground-truth is constructed with heuristics, it still contains valuable information and thus allows the following joint optimization to start from a good point. On the other hand, in CMU_DoG, the crowd-workers do not refer to the external knowledge as much as those workers do in Wizard when they form the responses; (2) the reinforcement step and curriculum step are useful because the reinforcement step allows the knowledge selection module to make better use of GPT-2’s feedback, and through the curriculum step GPT-2 can take advantage of the output of knowledge selection module progressively; (3) joint optimization is meaningful, as removing this stage results in performance drop.

Impact of $T_{max}$ (i.e., the upper bound in knowledge selection). Besides the learning strategies, we are also curious about how $T_{max}$, as part of the termination criterion in knowledge selection described at the end of Section 4.2, influences the performance of KnowledGPT. To this end, we vary the value of $T_{max}$ in $\{1,2,3\}$ and report the evaluation results in Table 6. The larger $T_{max}$ is, the more chances KnowledGPT has to involve the ground-truth candidate into generation, and the lower PPL is. This also explains why the PPL of GPT-2trunc is lower than that of KnowledGPT in Table 2 and Table 3. On the other hand, a larger $T_{max}$ also means more noise in generation. That is why when $T_{max}$ exceeds a value, F1 begins to drop.

Conclusions

We apply large-scaled pre-trained language models to the task of knowledge-grounded dialogue generation. To this end, we devise a knowledge selection module, and propose an unsupervised approach to jointly optimizing knowledge selection and response generation. Evaluation results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods.

We would like to thank the reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0105200), the National Science Foundation of China (NSFC No. 61876196 and NSFC No. 61672058). Rui Yan was sponsored as the young fellow of Beijing Academy of Artificial Intelligence (BAAI). Rui Yan is the corresponding author.

References

Appendix A Details of Datasets

Table 7 reports the statistics of the Wizard data and the CMU$\_$DoG data.

Appendix B Comparison with DialoGPT

We compare KnowledGPT and with DialoGPT in order to learn if a pre-trained generation model with state-of-the-art performance on open domain dialogues is already good enough when it is fine-tuned with knowledge-grounded dialogues. We discard the associated knowledge and fine-tune DialoGPT on the knowledge-grounded dialogues. We choose the model trained from OpenAI GPT-2 with $345$M parameters, as it shows the best performance in the evaluation in the original paper. The model is implemented based on the code shared at https://github.com/microsoft/DialoGPT.

Table 8 shows the results, indicating that external knowledge is necessary even though one has exploited a powerful pre-trained language model for dialogue generation. In CMU$\_$DoG the gap between DialoGPT and KnowledGPT is narrowed because about 35% of the conversation has a weak correlation with the document (e.g. BLEU $<$ 0.1).

Appendix C Impact of Maximum Tokens of GPT-2

To further justify our claims on why GPT-2trunc is worse than KnowledGPT, we keep the ground-truth knowledge in the input sequence of GPT-2 and gradually increase the constraint of the maximum number of tokens on Wizard. As the maximum token limit increases, more irrelevant knowledge is introduced. Note that in practice, one has no way to perfectly locate the ground-truth, and this experiment is only to provide more insights to GPT-2trunc. Table 9 shows the performance of GPT-2trunc with the increase of the maximum number of tokens where Ground-truth Percentage indicates the percentage of ground-truth in the input knowledge. First, when the ground-truth is forced to be kept, GPT-2trunc is always better than the one where the ground-truth is randomly mixed with other candidates and bears the risk to be cut. This echoes our claim that knowledge loss is one of the reasons for the poor performance of GPT-2trunc used with the practical setting. Second, even if ground-truth is retained, once more noise is introduced, the performance of GPT-2trunc will become worse. When the length is limited to $128$ tokens, the PPL of the model is not good, mainly because under this limitation, the input sequence of some cases only contains the dialogue context and response.

Appendix D Impact of the Size of GPT-2

We further check if the performance of KnowledGPT can be further improved when the GPT-2 model is replaced with a larger one. Table 10 shows the results. Though GPT-2 (345M) can further reduce PPL, it does not bring significant improvement to F1 over GPT-2 (117M), probably because the larger model can not provide more accurate feedback to the knowledge selection module in learning. Therefore, to balance efficacy and cost, GPT-2 (117M) is still favored in practice.

Appendix E Case Study

Table 11 and Table 12 show the examples from Test Seen and Test Unseen of Wizard, each example contains the dialogue context and the background knowledge which is retrieved from Wikipedia given the last two turns of dialogue and the original topic. We can see that KnowledGPT can locate the knowledge more accurately due to its knowledge selection module and reinforcement learning, and make better use of the associated knowledge with the help of curriculum learning.