CTRL: A Conditional Transformer Language Model for Controllable Generation

Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, Richard Socher

Introduction

With enough data, model capacity, and compute, generative models can learn distributions powerful enough to produce high-quality samples from complex domains. In computer vision, the advent of generative adversarial networks (Goodfellow et al., 2014) improved image generation. Much research then focused on methods for controlling the generation process and improving estimation of generative distributions (Arjovsky et al., 2017; Chen et al., 2016; Kingma & Welling, 2013).

In natural language processing, language models are often trained as conditional language models for specific tasks that require text generation (Brants et al., 2007; Sutskever et al., 2014; Rush et al., 2015). They are also used as a means of learning word vectors (Mikolov et al., 2013), document vectors (Kiros et al., 2015), or contextualized word vectors (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018) for transfer learning. The language models themselves have been transferred to new tasks through fine-tuning as well (Dai & Le, 2015; Radford et al., 2018; Howard & Ruder, 2018). Less is understood about generation that is not constrained to any specific task. Typically prompts generated by models (Fan et al., 2018) or written by humans can only be used to provide a rough guide or starting point for the generated text. This raises the question of how text generation can be controlled more explicitly.

Inspired by the degree of control available in image generation as well as the recent progress in text generation (Radford et al., 2019) and multitask learning McCann et al. (2018), we train a language model that is conditioned on a variety of control codes (Pfaff, 1979; Poplack, 1980) that make desired features of generated text more explicit. With 1.63 billion parameters, our Conditional Transformer Language (CTRL) model can generate text conditioned on control codes that specify domain, style, topics, dates, entities, relationships between entities, plot points, and task-related behavior. To preserve the generality of the language model trained in an unsupervised setting, we train CTRL on control codes derived from structure that naturally co-occurs with the raw text typically collected for training large language models. For example, large resources like Wikipedia, Project Gutenberg, and Amazon Reviews can each be assigned a domain-related control code. Smaller resources, like the content extracted from individual subreddits, often occur with both a broader domain name, reddit, as well as subdomain information, r/subdomain. In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents. Humans can use these codes to trigger generation of text from different linguistic communities without having to understand how to prompt with particular linguistic patterns. Text can be generated in more predictable ways by controlling for content or changing the domain even when the initial prompt remains fixed.

Because all control codes can be traced back to a particular subset of the training data, CTRL can be used to predict the subset of training data that is most likely given a sequence. This explicit relationship between CTRL and its training data can be exploited to analyze the correlations that the language model has learned from each domain, and it provides a means of studying large amounts of text through the language model.

These control codes also allow for the straightforward inclusion of task-specific data in a way that improves important skills without harming the generality of the model. Control codes for question answering and machine translation make these skills easily accessible with CTRL. These codes can be combined with codes during generation to create novel cross-over between control codes that are task-specific behavior and those that are related to domain and content.

In order to push towards more controllable, general models for natural language processing, we have released multiple full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl. We hope that the release leads to further research into how controllable generation can enhance natural language understanding.

Language Modeling

Given example sequences of the form $x=(x_{1},\ldots,x_{n})$ where each $x_{i}$ comes from a fixed set of symbols, the goal of language modeling is to learn $p(x)$ . Because $x$ is a sequence, it is natural to factorize this distribution using the chain rule of probability (Bengio et al., 2003):

This decomposes language modeling into next-word prediction. Current state-of-the-art methods (Dai et al., 2019; Radford et al., 2019) train a neural network with parameters $\theta$ to minimize the negative log-likelihood over a dataset $D=\{x^{1},\ldots,x^{|D|}\}$ :

Language Modeling with CTRL

CTRL is a conditional language model that is always conditioned on a control code $c$ and learns the distribution $p(x|c)$ . The distribution can still be decomposed using the chain rule of probability and trained with a loss that takes the control code into account.

The control code $c$ provides a point of control over the generation process. This is true even when sampling $x_{0}$ , in contrast to the traditional language modeling framework described in Sec. 2.

The core of the first block is multi-head attention with $k$ heads that uses a causal mask to preclude attending to future tokens:

Each block precedes core functionality with layer normalization (Ba et al., 2016; Child et al., 2019) and follows it with a residual connection (He et al., 2016). Together, they yield $X_{i+1}$ :

Scores for each token in the vocabulary are computed from the output of the last layer:

During training, these scores are the inputs of a cross-entropy loss function. During generation, the scores corresponding to the final token are normalized with a softmax, yielding a distribution for sampling a new token.

We train on $140$ GB of text drawing from a wide variety of domains: Wikipedia (En, De, Es, Fr), Project GutenbergWe use a modified version of https://github.com/chiphuyen/lazynlp, submissions from 45 subreddits, OpenWebTextWe use a modified version of https://github.com/jcpeterson/openwebtext.git, a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McAuley et al., 2015), Europarl and UN data from WMT (En-De, En-Es, En-Fr) (Barrault et al., 2019), question-answer pairs (no context documents) from ELI5 (Fan et al., 2019) and the MRQA shared taskhttps://github.com/mrqa/MRQA-Shared-Task-2019, which includes the Stanford Question Answering Dataset (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019). A full account of training data and associated control codes can be found in Table 7 in the Appendix.

2 Experimental Settings

We learn BPE (Sennrich et al., 2015) codes and tokenize the data using fastBPEhttps://github.com/glample/fastBPE, but we use a large vocabulary of roughly $250$ K tokens. This includes the sub-word tokens necessary to mitigate problems with rare words, but it also reduces the average number of tokens required to generate long text by including most common words. We use English Wikipedia and a 5% split of our collected OpenWebText data for learning BPE codes. We also introduce an unknown token so that during preprocessing we can filter out sequences that contain more than $2$ unknown tokens. This, along with the compressed storage for efficient training (TFRecords) (Abadi et al., 2016), reduces our training data to $140$ GB from the total $180$ GB collected. Data was treated as a single stream of tokens with non-domain control codes inserted where appropriate (often at document boundaries). The stream was chunked into contiguous sequences of tokens. Each sequence originated from a domain, and it has the corresponding domain control code prepended as the first token in the sequence. In this way, domain control codes receive special treatment (Kobus et al., 2016). They are propagated to all text in the domain as the first token. This is similar to how codes and natural language sequences have been used in multi-task settings (Wu et al., 2016; Johnson et al., 2017; McCann et al., 2018) to control conditional language models. All other control codes are injected into the data without such special treatment (Moryossef et al., 2019; Caswell et al., 2019). We experimented with sequence lengths of $256$ and $512$ due to memory and optimization constraints. Despite training on relatively short sequences compared to other approaches, we found that a sliding-window approach allows for generation beyond these windows, and we also found little difference in quality between the two models within the first $256$ tokens. Further, we note that our vocabulary is approximately 4 times larger than similar approaches, hence the effective sequence length in characters is comparable.

CTRL has model dimension $d=1280$ , inner dimension $f=8192$ , $48$ layers, and $16$ heads per layer. Dropout with probability $0.1$ follows the residual connections in each layer. Token embeddings were tied with the final output embedding layer (Inan et al., 2016; Press & Wolf, 2016).

CTRL was implemented in TensorFlow (Abadi et al., 2016) and trained with a global batch size of $1024$ distributed across $256$ cores of a Cloud TPU v $3$ Pod for $800$ k iterations. Training took approximately 2 weeks using Adagrad (Duchi et al., 2011) with a linear warmup from to $0.05$ over $25$ k steps. The norm of gradients were clipped to $0.25$ as in (Merity et al., 2017). Learning rate decay was not necessary due to the monotonic nature of the Adagrad accumulator. We compared to the Adam optimizer (Kingma & Ba, 2014) while training smaller models, but we noticed comparable convergence rates and significant memory savings with Adagrad. We also experimented with explicit memory-saving optimizers including SM3 (Anil et al., 2019), Adafactor (Shazeer & Stern, 2018), and NovoGrad (Ginsburg et al., 2019) with mixed results.

Controllable Generation

The next token is then chosen by sampling through a multinomial distribution with probabilities $p_{i}$ clipped at the top- $k$ tokens. In the equation above, $T\to 0$ approximates a greedy distribution which magnifies the peaks in the probability distribution while $T\to\infty$ flattens the distribution to make it more uniform. Rather than choosing a fixed value of $k$ , as is common practice, Holtzman et al. (2019) suggested adapting $k$ heuristically. The nucleus sampling approach chooses a probability threshold $p_{t}$ and sets $k$ to be the lowest value such that $\sum_{i}\text{sort}(p_{i})>p_{t}$ . If the model is confident in its next-word prediction, then $k$ will be lower and vice versa. Despite the improved generative capabilities of models with such heuristics, there still exists a trade-off between these parameters depending on the generation intended.

Given a prompt: Q: What is the capital of Australia?, a well-trained model assigns higher probability mass to the correct answer, Canberra, but a non-zero probability mass to other cities such as Melbourne, Sydney, Brisbane, Darwin, and Perth, see Figure 1.

By choosing to sample, we mistrust the model, despite it being correct. A natural solution to this is to choose the next token greedily. However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019). To reconcile the two, we propose a new sampling scheme that trusts the model distribution through near-greedy sampling but prevents repetitions through a penalty. This penalized sampling works by discounting the scores of previously generated tokens. The motivation is similar to coverage mechanisms (See et al., 2017) and other losses designed to discourage repetition (Welleck et al., 2019), but penalized sampling is not used during training. Given a list of generated tokens $g$ , using the notation from equation 1, the probability distribution $p_{i}$ for the next token is defined as:

We find that using a greedy sampling and $\theta\approx 1.2$ yields a good balance between truthful generation and lack of repetition. Note that $\theta=1$ is equivalent to equation 1. We note in passing that this approach succeeds only if the model has learned a sufficiently reliable distribution.

2 Control Codes

A small number of control codes are related to specific tasks like question answering and translation. These codes constrain the generation process the most, by triggering task-specific generation. In Table 4, we demonstrate relatively complex control codes for question answering and machine translation that act as a template mixed with a natural language prompt.

In the first example we mix a diet subreddit (r/keto) with machine translation control codes for English and German. In contrast to using Translation in 4.2, the generated text with mixed codes is coherent across multiple translated lines. This structure is an influence of Diet because it had multiline examples in the training data, whereas the translation data consisted of shuffled single lines. In the second example we mix the politics subreddit (r/politics) with a prompt that starts in French though no examples of this kind were found in the training data.

The domain control codes can be used to partition the training data into mutually exclusive sets. This supports a simple method for determining which subsets of the training data the language model considers most likely given a sequence. Recall that the language model has learned a distribution $p_{\theta}(x|c)$ . By specifying a prior over domain control codes for $p(c)$ , it is straightforward to compute a ranking of domains:

We found that the empirical prior of the training data weights domains with large amounts of data too heavily. Instead, we use a uniform prior over the domain control codes. Examples can be found in Table 6.

We note that the data used to train this model does not have universal coverage and contains the cultural associations present in the original sources. All applications of the model inherently depend on those original associations for prediction. In fact, this method of source attribution relies on exploiting the original associations to establish relationships between the language model and its training data.

The model does not have a notion of whether any particular cultural association is good or bad, right or wrong, true or false. It only learns correlations between cultural associations and domains. This is evidenced by the fact that contradictory statements are often attributed to the same sources: competing claims often appear in the same contexts. CTRL provides model-based evidence that certain domains are more likely to contain language similar to given statements, but it should not be used to make normative or prescriptive claims. It is a descriptive tool for analyzing correlations in large amounts of text.

Language models (Bengio et al., 2003) have played an important role in natural language processing through transferrable word vectors (Mikolov et al., 2013), contextualized word vectors (Peters et al., 2018; Devlin et al., 2018; Lample & Conneau, 2019), and models (Howard & Ruder, 2018; Radford et al., 2018). Recent work on memory mechanisms (Dai et al., 2019; Lample et al., 2019) has improved perplexities on the most common benchmarks, and even without these memories, large Transformer architectures (Vaswani et al., 2017) like GPT-2 (Radford et al., 2019), OpenGPT-2 https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc, and Megatron https://github.com/NVIDIA/Megatron-LM can achieve state-of-the-art results without directly training for any particular language modeling benchmark. Because these latter language models are trained on far more diverse data than is used in the supervised setting, they demonstrate impressive text generation capabilities (Radford et al., 2019; Zellers et al., 2019).

Multi-task learning.

These models demonstrate the potential to learn multiple tasks as well as quick adaptation to patterns in input prompts (Radford et al., 2019). This potential showed that language models can offer an alternative to supervised multi-task learning as framed by several recent benchmarks (Wang et al., 2018; McCann et al., 2018). Language models might also offer a foundation to extend proposals of unified, multi-task systems for all of NLP (Collobert & Weston, 2008; Collobert et al., 2011), parsing and tagging (Hashimoto et al., 2016), multiple languages (Wu et al., 2016; Johnson et al., 2017), and multiple modalities (Luong et al., 2015; Kaiser et al., 2017). Several works have pointed to natural language as a means for controlling these multi-task systems (McCann et al., 2018; Radford et al., 2019; Keskar et al., 2019), and several point to the benefits of a code book either specified explicitly (Wu et al., 2016) or learned in a latent space (Kaiser et al., 2018). This work attempts to balance these approaches.

Sampling methods and coverage mechanisms.

Recent work in sampling methods for text generation has focused on reducing repetition by replacing it with novel, coherent text (Fan et al., 2018; Holtzman et al., 2019). The problem of repetition can instead be approached by altering the training objectives, as with coverage mechanisms (See et al., 2017) and context-based losses (Welleck et al., 2019). When prioritizing control, the trade-off between novelty in the generated text and consistency with prompts and prior generated text remains a difficult challenge, but this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.

The particular choice of control codes in this work is intended to represent a reasonably large variety in control over domain, topic, entities, entity relations, and dates. A very flexible means of control is through the natural structure of the internet in the form of URLs. Many of the domains that were mapped in this work to a single control code (e.g. Wikipedia, Project Gutenberg), could be refined to provide more fine-grained control either through further exploitation of URL structure (en.wikipedia.org, de.wikipedia.org, en.wikipedia.org/wiki/Anarchism, en.wikipedia.org/wiki/Anarchism#History) or through the manual extraction of structure already present in the data (e.g. Books Author Title Chapter). We hope future work explores extensions of CTRL to new domains in ways that provide further insight into controllable text generation.

Extensions to other areas in NLP.

This work suggests that including data for specific tasks need not harm the general nature of an unsupervised learning process. For important skills, the inclusion of supervised data or task-specific data generated through unsupervised means (Artetxe et al., 2017; Lewis et al., 2019) can lead to obvious improvements. While this work experimented with trivia-style question answering (without context documents) and small amounts of machine translation data, it remains an open question whether these language models can learn to effectively perform tasks like extractive question answering or state-of-the-art multilingual machine translation while still preserving general pattern recognition and text generation functionality.

Many tasks present difficult challenges to the supervised setting. Commonsense reasoning (Levesque et al., 2012) and abstractive summarization (Rush et al., 2015) represent two areas where these challenges remain readily apparent (Kryściński et al., 2019). Yet language models show potential for mitigating these problems directly (Trinh & Le, 2018; Radford et al., 2019) or indirectly (Rajani et al., 2019; Xenouleas et al., 2019; Scialom et al., 2019). We hope that in future work CTRL can be extended to far more tasks through the use of both unsupervised and supervised techniques.

Analyzing the relationships between language models and training data.

CTRL is trained on a small subset of the possible data available. Therefore the model is biased towards the patterns of language used in the training data. The data is likely not representative of many linguistic communities, but CTRL offers an explicit method for analyzing the relationship between the model and its current training data. As methods improve, more data is collected, and training of these large models continues, we hope to use this tool to better understand the particular cultural associations the model learns from each data source.

Making the interface between humans and language models more explicit and intuitive.

CTRL is designed to make the interface between humans and language models more intuitive. Text generation can be a powerful tool for enhancing creativity and exploration. In future work, we hope to study how the beneficial applications of such models can be enhanced by providing more control to human users.

Openness and replicability are central aspects of the scientific ethos that, prima facie, suggest the release of complete scientific research results. We reify these principles by releasing all trained CTRL models.

Although much scientific research and innovation can benefit the public, it may also be diverted to harmful uses or have unintended negative impacts (without animus). Brundage et al. (2019), among others, have argued artificial intelligence has such an omni-use character and have suggested governance policies emerging from the responsible innovation literature (Brundage, 2016). Historical evidence has pointed to the inadequacy of self-moratoriums for governing omni-use technologies (Kaiser & Moreno, 2012); we take a course of action that differs from such self-regulation. Our actions reflect principles from a recent sociology-based AI governance framework that aims to expand responsible innovation to consider networks of users, dynamics, and feedback (Varshney et al., 2019).

Rather than self-governance, we sought to diversify inputs to governance through pre-release review from experts at the Partnership on AI (PAI). These experts, in turn, drew on emerging norms and governance processes that incorporate a broad set of values from across society.

Prior to release, the research team conducted a technology foresight exercise to anticipate possible malicious use cases. In particular, we used a scenario planning approach to technology foresight that systematically attempts to envision plausible longer-term future states of science, technology, and society. This anticipatory focus on possibilities rather than probabilities lessens several shortcomings of formal risk assessment in the face of contested assumptions, which has proven ineffective in identifying the most profound future impacts of innovation (Stilgoe et al., 2013).

As part of our model release, we include a code of conduct in the README at https://github.com/salesforce/ctrl. This code of conduct is modeled after emerging community norms ensconced in the Do No Harm and Just World Licenses. Simultaneously recognizing that it has no legal force and that users are agents of technological change embedded in social networks, the aim is to encourage reflection at the consumption junction (Cowan, 1987) through norm-setting and reduce unintended uses.

The README also includes a subset of the questions that the team discussed when deliberating release of the models, drawn from early drafts of community-driven PAI documents (to be released in the near future). This may further encourage users to reflect on norms and responsibilities associated with models that generate artificial content. In particular, users are asked to share answers to the included questions, to pose further questions, and suggest solutions by emailing ctrl-monitoring@salesforce.com.

Finally, the README asks users to develop appropriate documentation (PAI, 2019; Arnold et al., 2018; Mitchell et al., 2019) when building on CTRL and to tell the research team how they are using CTRL by emailing ctrl-monitoring@salesforce.com. This facilitates a post-release monitoring plan that observes how people are using CTRL in the wild (together with active observations). Such post-market plans recognize that most innovations are unexpected and hard to forecast. It is intended to enable a responsive approach to responsible innovation, not just with respect to harmful uses but also unintended negative impacts without animus.

With 1.63 billion parameters, CTRL is the largest publicly released language model to date. It is trained with control codes so that text generation can be more easily controlled by human users. These codes allow users to explicitly specify domain, subdomain, entities, relationships between entities, dates, and task-specific behavior. We hope that the release of this model at https://github.com/salesforce/ctrl pushes towards more controllable, general models for natural language processing, and we encourage future discussion about artificial generation with our team by emailing ctrl-monitoring@salesforce.com.

We would like to thank Kathy Baxter for her help in the ethical considerations of our work and facilitating the external review process; Srinath Meadusani, Lavanya Karanam, Ning Dong, and Navin Ramineni for their help with setting up and maintaining compute infrastructure; Zak Stone and his team at Google for assistance with TPU infrastructure and code; and Joseph Olsen, Roy Davis, Joshua Simmons, Denise Lo, and Sam Edwards for their help with open sourcing.