Skip-Thought Vectors

Introduction

Developing learning algorithms for distributed compositional semantics of words has been a long-standing open problem at the intersection of language understanding and machine learning. In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks socher2013recursive , recurrent networks hochreiter1997long , convolutional networks kalchbrenner2014convolutional ; kim2014convolutional and recursive-convolutional methods cho2014properties ; zhao2015self among others. All of these methods produce sentence representations that are passed to a supervised task and depend on a class label in order to backpropagate through the composition weights. Consequently, these methods learn high-quality sentence representations but are tuned only for their respective task. The paragraph vector of le2014distributed is an alternative to the above models in that it can learn unsupervised sentence representations by introducing a distributed sentence indicator as part of a neural language model. The downside is at test time, inference needs to be performed to compute a new vector.

In this paper we abstract away from the composition methods themselves and consider an alternative loss function that can be applied with any composition operator. We consider the following question: is there a task and a corresponding loss that will allow us to learn highly generic sentence representations? We give evidence for this by proposing a model for learning high-quality sentence vectors without a particular supervised task in mind. Using word vector learning as inspiration, we propose an objective function that abstracts the skip-gram model of mikolov2013efficient to the sentence level. That is, instead of using a word to predict its surrounding context, we instead encode a sentence to predict the sentences around it. Thus, any composition operator can be substituted as a sentence encoder and only the objective function becomes modified. Figure 1 illustrates the model. We call our model skip-thoughts and vectors induced by our model are called skip-thought vectors.

Our model depends on having a training corpus of contiguous text. We chose to use a large collection of novels, namely the BookCorpus dataset moviebook15 for training our models. These are free books written by yet unpublished authors. The dataset has books in 16 different genres, e.g., Romance (2,865 books), Fantasy (1,479), Science fiction (786), Teen (430), etc. Table 1 highlights the summary statistics of the book corpus. Along with narratives, books contain dialogue, emotion and a wide range of interaction between characters. Furthermore, with a large enough collection the training set is not biased towards any particular domain or application. Table 2 shows nearest neighbours of sentences from a model trained on the BookCorpus dataset. These results show that skip-thought vectors learn to accurately capture semantics and syntax of the sentences they encode.

We evaluate our vectors in a newly proposed setting: after learning skip-thoughts, freeze the model and use the encoder as a generic feature extractor for arbitrary tasks. In our experiments we consider 8 tasks: semantic-relatedness, paraphrase detection, image-sentence ranking and 5 standard classification benchmarks. In these experiments, we extract skip-thought vectors and train linear models to evaluate the representations directly, without any additional fine-tuning. As it turns out, skip-thoughts yield generic representations that perform robustly across all tasks considered.

One difficulty that arises with such an experimental setup is being able to construct a large enough word vocabulary to encode arbitrary sentences. For example, a sentence from a Wikipedia article might contain nouns that are highly unlikely to appear in our book vocabulary. We solve this problem by learning a mapping that transfers word representations from one model to another. Using pre-trained word2vec representations learned with a continuous bag-of-words model mikolov2013efficient , we learn a linear mapping from a word in word2vec space to a word in the encoder’s vocabulary space. The mapping is learned using all words that are shared between vocabularies. After training, any word that appears in word2vec can then get a vector in the encoder word embedding space.

Approach

𝑖1s_{i-1},s_{i},s_{i+1}) of contiguous sentences, with $s_{i}$ the $i$-th sentence of a book, the sentence $s_{i}$ is encoded and tries to reconstruct the previous sentence $s_{i-1}$ and next sentence $s_{i+1}$. In this example, the input is the sentence triplet I got back home. I could see the cat on the steps. This was strange. Unattached arrows are connected to the encoder output. Colors indicate which components share parameters. $\langle$eos$\rangle$ is the end of sentence token. # of books # of sentences # of words # of unique words mean # of words per sentence 11,038 74,004,228 984,846,357 1,316,420 13

We treat skip-thoughts in the framework of encoder-decoder models 111A preliminary version of our model was developed in the context of a computer vision application moviebook15 .. That is, an encoder maps words to a sentence vector and a decoder is used to generate the surrounding sentences. Encoder-decoder models have gained a lot of traction for neural machine translation. In this setting, an encoder is used to map e.g. an English sentence into a vector. The decoder then conditions on this vector to generate a translation for the source English sentence. Several choices of encoder-decoder pairs have been explored, including ConvNet-RNN kalchbrenner2013recurrent , RNN-RNN cho2014learning and LSTM-LSTM sutskever2014sequence . The source sentence representation can also dynamically change through the use of an attention mechanism bahdanau2014neural to take into account only the relevant words for translation at any given time. In our model, we use an RNN encoder with GRU chung2014empirical activations and an RNN decoder with a conditional GRU. This model combination is nearly identical to the RNN encoder-decoder of cho2014learning used in neural machine translation. GRU has been shown to perform as well as LSTM hochreiter1997long on sequence modelling tasks chung2014empirical while being conceptually simpler. GRU units have only 2 gates and do not require the use of a cell. While we use RNNs for our model, any encoder and decoder can be used so long as we can backpropagate through it.

Assume we are given a sentence tuple ($s_{i-1},s_{i},s_{i+1}$). Let $w^{t}_{i}$ denote the $t$-th word for sentence $s_{i}$ and let ${\bf x}^{t}_{i}$ denote its word embedding. We describe the model in three parts: the encoder, decoder and objective function.

Encoder. Let $w^{1}_{i},\ldots,w^{N}_{i}$ be the words in sentence $s_{i}$ where $N$ is the number of words in the sentence. At each time step, the encoder produces a hidden state ${\bf h}^{t}_{i}$ which can be interpreted as the representation of the sequence $w^{1}_{i},\ldots,w^{t}_{i}$. The hidden state ${\bf h}^{N}_{i}$ thus represents the full sentence. To encode a sentence, we iterate the following sequence of equations (dropping the subscript $i$):

subscript𝐖𝑟superscript𝐱𝑡subscript𝐔𝑟superscript𝐡𝑡1\displaystyle\sigma({\bf W}_{r}{\bf x}^{t}+{\bf U}_{r}{\bf h}^{t-1}) (1) $\displaystyle{\bf z}^{t}$ $\displaystyle=$ $\displaystyle\sigma({\bf W}_{z}{\bf x}^{t}+{\bf U}_{z}{\bf h}^{t-1})$ (2) $\displaystyle\bar{{\bf h}}^{t}$ $\displaystyle=$ $\displaystyle\text{tanh}({\bf W}{\bf x}^{t}+{\bf U}({\bf r}^{t}\odot{\bf h}^{t-1}))$ (3) $\displaystyle{\bf h}^{t}$ $\displaystyle=$ $\displaystyle(1-{\bf z}^{t})\odot{\bf h}^{t-1}+{\bf z}^{t}\odot\bar{{\bf h}}^{t}$ (4) where $\bar{{\bf h}}^{t}$ is the proposed state update at time $t$, ${\bf z}^{t}$ is the update gate, ${\bf r}_{t}$ is the reset gate ($\odot$) denotes a component-wise product. Both update gates takes values between zero and one.

Decoder. The decoder is a neural language model which conditions on the encoder output ${\bf h}_{i}$. The computation is similar to that of the encoder except we introduce matrices ${\bf C}_{z}$, ${\bf C}_{r}$ and ${\bf C}$ that are used to bias the update gate, reset gate and hidden state computation by the sentence vector. One decoder is used for the next sentence $s_{i+1}$ while a second decoder is used for the previous sentence $s_{i-1}$. Separate parameters are used for each decoder with the exception of the vocabulary matrix ${\bf V}$, which is the weight matrix connecting the decoder’s hidden state for computing a distribution over words. In what follows we describe the decoder for the next sentence $s_{i+1}$ although an analogous computation is used for the previous sentence $s_{i-1}$. Let ${\bf h}^{t}_{i+1}$ denote the hidden state of the decoder at time $t$. Decoding involves iterating through the following sequence of equations (dropping the subscript $i+1$):

superscriptsubscript𝐖𝑟𝑑superscript𝐱𝑡1superscriptsubscript𝐔𝑟𝑑superscript𝐡𝑡1subscript𝐂𝑟subscript𝐡𝑖\displaystyle\sigma({\bf W}_{r}^{d}{\bf x}^{t-1}+{\bf U}_{r}^{d}{\bf h}^{t-1}+{\bf C}_{r}{\bf h}_{i}) (5) $\displaystyle{\bf z}^{t}$ $\displaystyle=$ $\displaystyle\sigma({\bf W}_{z}^{d}{\bf x}^{t-1}+{\bf U}_{z}^{d}{\bf h}^{t-1}+{\bf C}_{z}{\bf h}_{i})$ (6) $\displaystyle\bar{{\bf h}}^{t}$ $\displaystyle=$ $\displaystyle\text{tanh}({\bf W}^{d}{\bf x}^{t-1}+{\bf U}^{d}({\bf r}^{t}\odot{\bf h}^{t-1})+{\bf C}{\bf h}_{i})$ (7) $\displaystyle{\bf h}^{t}_{i+1}$ $\displaystyle=$ $\displaystyle(1-{\bf z}^{t})\odot{\bf h}^{t-1}+{\bf z}^{t}\odot\bar{{\bf h}}^{t}$ (8) Given ${\bf h}^{t}_{i+1}$, the probability of word $w_{i+1}^{t}$ given the previous $t-1$ words and the encoder vector is

𝑖1𝑡superscriptsubscript𝑤𝑖1absent𝑡subscript𝐡𝑖expsubscript𝐯superscriptsubscript𝑤𝑖1𝑡subscriptsuperscript𝐡𝑡𝑖1P(w_{i+1}^{t}|w_{i+1}^{${\bf v}_{w_{i+1}^{t}}$ denotes the row of ${\bf V}$ corresponding to the word of $w_{i+1}^{t}$. An analogous computation is performed for the previous sentence $s_{i-1}$.

Objective. Given a tuple ($s_{i-1},s_{i},s_{i+1}$), the objective optimized is the sum of the log-probabilities for the forward and backward sentences conditioned on the encoder representation:

subscript𝑡log𝑃conditionalsuperscriptsubscript𝑤𝑖1𝑡superscriptsubscript𝑤𝑖1absent𝑡subscript𝐡𝑖subscript𝑡log𝑃conditionalsuperscriptsubscript𝑤𝑖1𝑡superscriptsubscript𝑤𝑖1absent𝑡subscript𝐡𝑖\sum_{t}\text{log}P(w_{i+1}^{t}|w_{i+1}^{

2 Vocabulary expansion

We now describe how to expand our encoder’s vocabulary to words it has not seen during training. Suppose we have a model that was trained to induce word representations, such as word2vec. Let $\mathcal{V}_{w2v}$ denote the word embedding space of these word representations and let $\mathcal{V}_{rnn}$ denote the RNN word embedding space. We assume the vocabulary of $\mathcal{V}_{w2v}$ is much larger than that of $\mathcal{V}_{rnn}$. Our goal is to construct a mapping $f:\mathcal{V}_{w2v}\rightarrow\mathcal{V}_{rnn}$ parameterized by a matrix ${\bf W}$ such that ${\bf v}^{\prime}={\bf W}{\bf v}$ for ${\bf v}\in\mathcal{V}_{w2v}$ and ${\bf v^{\prime}}\in\mathcal{V}_{rnn}$. Inspired by mikolov2013exploiting , which learned linear mappings between translation word spaces, we solve an un-regularized L2 linear regression loss for the matrix ${\bf W}$. Thus, any word from $\mathcal{V}_{w2v}$ can now be mapped into $\mathcal{V}_{rnn}$ for encoding sentences. Table 3 shows examples of nearest neighbour words for queries that did not appear in our training vocabulary.

We note that there are alternate strategies for solving the vocabulary problem. One alternative is to initialize the RNN embedding space to that of pre-trained word vectors. This would require a more sophisticated softmax for decoding, or clipping the vocabulary of the decoder as it would be too computationally expensive to naively decode with vocabularies of hundreds of thousands of words. An alternative strategy is to avoid words altogether and train at the character level.

Experiments

In our experiments, we evaluate the capability of our encoder as a generic feature extractor after training on the BookCorpus dataset. Our experimentation setup on each task is as follows:

Using the learned encoder as a feature extractor, extract skip-thought vectors for all sentences.

If the task involves computing scores between pairs of sentences, compute component-wise features between pairs. This is described in more detail specifically for each experiment.

Train a linear classifier on top of the extracted features, with no additional fine-tuning or backpropagation through the skip-thoughts model.

We restrict ourselves to linear classifiers for two reasons. The first is to directly evaluate the representation quality of the computed vectors. It is possible that additional performance gains can be made throughout our experiments with non-linear models but this falls out of scope of our goal. Furthermore, it allows us to better analyze the strengths and weaknesses of the learned representations. The second reason is that reproducibility now becomes very straightforward.

To induce skip-thought vectors, we train two separate models on our book corpus. One is a unidirectional encoder with 2400 dimensions, which we subsequently refer to as uni-skip. The other is a bidirectional model with forward and backward encoders of 1200 dimensions each. This model contains two encoders with different parameters: one encoder is given the sentence in correct order, while the other is given the sentence in reverse. The outputs are then concatenated to form a 2400 dimensional vector. We refer to this model as bi-skip. For training, we initialize all recurrent matricies with orthogonal initialization saxe2013exact . Non-recurrent weights are initialized from a uniform distribution in [-0.1,0.1]. Mini-batches of size 128 are used and gradients are clipped if the norm of the parameter vector exceeds 10. We used the Adam algorithm kingma2014adam for optimization. Both models were trained for roughly two weeks. As an additional experiment, we also report experimental results using a combined model, consisting of the concatenation of the vectors from uni-skip and bi-skip, resulting in a 4800 dimensional vector. Since we are using linear classifiers for evaluation, we were curious to what extent performance gains can be made by trivially increasing the vector dimensionality post-training of the skip-thought models. We refer to this model throughout as combine-skip.

After our models are trained, we then employ vocabulary expansion to map word embeddings into the RNN encoder space. The publically available CBOW word vectors are used for this purpose 222http://code.google.com/p/word2vec/. The skip-thought models are trained with a vocabulary size of 20,000 words. After removing multiple word examples from the CBOW model, this results in a vocabulary size of 930,911 words. Thus even though our skip-thoughts model was trained with only 20,000 words, after vocabulary expansion we can now successfully encode 930,911 possible words.

Since our goal is to evaluate skip-thoughts as a general feature extractor, we keep text pre-processing to a minimum. When encoding new sentences, no additional preprocessing is done other than basic tokenization. This is done to test the robustness of our vectors.

2 Semantic relatedness

Our first experiment is on the SemEval 2014 Task 1: semantic relatedness SICK dataset marelli2014semeval . Given two sentences, our goal is to produce a score of how semantically related these sentences are, based on human generated scores. Each score is the average of 10 different human annotators. Scores take values between 1 and 5. A score of 1 indicates that the sentence pair is not at all related, while a score of 5 indicates they are highly related. The dataset comes with a predefined split of 4500 training pairs, 500 development pairs and 4927 testing pairs. All sentences are derived from existing image and video annotation datasets. The evaluation metrics are Pearson’s $r$, Spearman’s $\rho$, and mean squared error.

Given the difficulty of this task, many existing systems employ a large amount of feature engineering and additional resources. Thus, we test how well our learned representations fair against heavily engineered pipelines. Recently, tai2015improved showed that learning representations with LSTM or Tree-LSTM for the task at hand is able to outperform these existing systems. We take this one step further and see how well our vectors learned from a completely different task are able to capture semantic relatedness when only a linear model is used on top to predict scores.

To represent a sentence pair, we use two features. Given two skip-thought vectors $u$ and $v$, we compute their component-wise product $u\cdot v$ and their absolute difference $|u-v|$ and concatenate them together. These two features were also used by tai2015improved . To predict a score, we use the same setup as tai2015improved . Let $r^{\top}=[1,\ldots,5]$ be an integer vector from 1 to 5. We compute a distribution $p$ as a function of prediction scores $y$ given by $p_{i}=y-\lfloor y\rfloor$ if $i=\lfloor y\rfloor+1$, $p_{i}=\lfloor y\rfloor-y+1$ if $i=\lfloor y\rfloor$ and 0 otherwise. These then become our targets for a logistic regression classifier. At test time, given new sentence pairs we first compute targets $\hat{p}$ and then compute the related score as $r^{\top}\hat{p}$. As an additional comparison, we also explored appending features derived from an image-sentence embedding model trained on COCO (see section 3.4). Given vectors $u$ and $v$, we obtain vectors $u^{\prime}$ and $v^{\prime}$ from the learned linear embedding model and compute features $u^{\prime}\cdot v^{\prime}$ and $|u^{\prime}-v^{\prime}|$. These are then concatenated to the existing features.

Table 4 (left) presents our results. First, we observe that our models are able to outperform all previous systems from the SemEval 2014 competition. This is remarkable, given the simplicity of our approach and the lack of feature engineering. It highlights that skip-thought vectors learn representations that are well suited for semantic relatedness. Our results are comparable to LSTMs whose representations are trained from scratch on this task. Only the dependency tree-LSTM of tai2015improved performs better than our results. We note that the dependency tree-LSTM relies on parsers whose training data is very expensive to collect and does not exist for all languages. We also observe using features learned from an image-sentence embedding model on COCO gives an additional performance boost, resulting in a model that performs on par with the dependency tree-LSTM. To get a feel for the model outputs, Table 5 shows example cases of test set pairs. Our model is able to accurately predict relatedness on many challenging cases. On some examples, it fails to pick up on small distinctions that drastically change a sentence meaning, such as tricks on a motorcycle versus tricking a person on a motorcycle.

3 Paraphrase detection

The next task we consider is paraphrase detection on the Microsoft Research Paraphrase Corpus dolan2004unsupervised . On this task, two sentences are given and one must predict whether or not they are paraphrases. The training set consists of 4076 sentence pairs (2753 which are positive) and the test set has 1725 pairs (1147 are positive). We compute a vector representing the pair of sentences in the same way as on the SICK dataset, using the component-wise product $u\cdot v$ and their absolute difference $|u-v|$ which are then concatenated together. We then train logistic regression on top to predict whether the sentences are paraphrases. Cross-validation is used for tuning the L2 penalty.

As in the semantic relatedness task, paraphrase detection has largely been dominated by extensive feature engineering, or a combination of feature engineering with semantic spaces. We report experiments in two settings: one using the features as above and the other incorporating basic statistics between sentence pairs, the same features used by socher2011dynamic . These are referred to as feats in our results. We isolate the results and baselines used in socher2011dynamic as well as the top published results on this task.

Table 4 (right) presents our results, from which we can observe the following: (1) skip-thoughts alone outperform recursive nets with dynamic pooling when no hand-crafted features are used, (2) when other features are used, recursive nets with dynamic pooling works better, and (3) when skip-thoughts are combined with basic pairwise statistics, it becomes competitive with the state-of-the-art which incorporate much more complicated features and hand-engineering. This is a promising result as many of the sentence pairs have very fine-grained details that signal if they are paraphrases.

4 Image-sentence ranking

We next consider the task of retrieving images and their sentence descriptions. For this experiment, we use the Microsoft COCO dataset lin2014microsoft which is the largest publicly available dataset of images with high-quality sentence descriptions. Each image is annotated with 5 captions, each from different annotators. Following previous work, we consider two tasks: image annotation and image search. For image annotation, an image is presented and sentences are ranked based on how well they describe the query image. The image search task is the reverse: given a caption, we retrieve images that are a good fit to the query. The training set comes with over 80,000 images each with 5 captions. For development and testing we use the same splits as Karpathy15 . The development and test sets each contain 1000 images and 5000 captions. Evaluation is performed using Recall@K, namely the mean number of images for which the correct caption is ranked within the top-K retrieved results (and vice-versa for sentences). We also report the median rank of the closest ground truth result from the ranked list.

The best performing results on image-sentence ranking have all used RNNs for encoding sentences, where the sentence representation is learned jointly. Recently, klein2015associating showed that by using Fisher vectors for representing sentences, linear CCA can be applied to obtain performance that is as strong as using RNNs for this task. Thus the method of klein2015associating is a strong baseline to compare our sentence representations with. For our experiments, we represent images using 4096-dimensional OxfordNet features from their 19-layer model simonyan2014very . For sentences, we simply extract skip-thought vectors for each caption. The training objective we use is a pairwise ranking loss that has been previously used by many other methods. The only difference is the scores are computed using only linear transformations of image and sentence inputs. The loss is given by:

subscript𝐱subscript𝑘max0𝛼𝑠𝐔𝐱𝐕𝐲𝑠𝐔𝐱subscript𝐕𝐲𝑘subscript𝐲subscript𝑘max0𝛼𝑠𝐕𝐲𝐔𝐱𝑠𝐕𝐲subscript𝐔𝐱𝑘\displaystyle{\sum_{\bf x}\sum_{k}\text{max}\{0,\alpha-s({\bf Ux},{\bf Vy})+s({\bf Ux},{\bf Vy}_{k})\}+\sum_{\bf y}\sum_{k}\text{max}\{0,\alpha-s({\bf Vy},{\bf Ux})+s({\bf Vy},{\bf Ux}_{k})\},} where ${\bf x}$ is an image vector, ${\bf y}$ is the skip-thought vector for the groundtruth sentence, ${\bf y}_{k}$ are vectors for constrastive (incorrect) sentences and $s(\cdot,\cdot)$ is the image-sentence score. Cosine similarity is used for scoring. The model parameters are $\{{\bf U},{\bf V}\}$ where ${\bf U}$ is the image embedding matrix and ${\bf V}$ is the sentence embedding matrix. In our experiments, we use a 1000 dimensional embedding, margin $\alpha=0.2$ and $k=50$ contrastive terms. We trained for 15 epochs and saved our model anytime the performance improved on the development set.

Table 6 illustrates our results on this task. Using skip-thought vectors for sentences, we get performance that is on par with both Karpathy15 and klein2015associating except for R@1 on image annotation, where other methods perform much better. Our results indicate that skip-thought vectors are representative enough to capture image descriptions without having to learn their representations from scratch. Combined with the results of klein2015associating , it also highlights that simple, scalable embedding techniques perform very well provided that high-quality image and sentence vectors are available.

5 Classification benchmarks

For our final quantitative experiments, we report results on several classification benchmarks which are commonly used for evaluating sentence representation learning methods.

We use 5 datasets: movie review sentiment (MR) pang2005seeing , customer product reviews (CR) hu2004mining , subjectivity/objectivity classification (SUBJ) pang2004sentimental , opinion polarity (MPQA) wiebe2005annotating and question-type classification (TREC) li2002learning . On all datasets, we simply extract skip-thought vectors and train a logistic regression classifier on top. 10-fold cross-validation is used for evaluation on the first 4 datasets, while TREC has a pre-defined train/test split. We tune the L2 penality using cross-validation (and thus use a nested cross-validation for the first 4 datasets).

On these tasks, properly tuned bag-of-words models have been shown to perform exceptionally well. In particular, the NB-SVM of wang2012baselines is a fast and robust performer on these tasks. Skip-thought vectors potentially give an alternative to these baselines being just as fast and easy to use. For an additional comparison, we also see to what effect augmenting skip-thoughts with bigram Naive Bayes (NB) features improves performance 333We use the code available at https://github.com/mesnilgr/nbsvm.

Table 7 presents our results. On most tasks, skip-thoughts performs about as well as the bag-of-words baselines but fails to improve over methods whose sentence representations are learned directly for the task at hand. This indicates that for tasks like sentiment classification, tuning the representations, even on small datasets, are likely to perform better than learning a generic unsupervised sentence vector on much bigger datasets. Finally, we observe that the skip-thoughts-NB combination is effective, particularly on MR. This results in a very strong new baseline for text classification: combine skip-thoughts with bag-of-words and train a linear model.

6 Visualizing skip-thoughts and generating stories

As a final experiment, we applied t-SNE van2008visualizing to skip-thought vectors extracted from TREC, SUBJ and SICK datasets and the visualizations are shown in Figure 2. For the SICK visualization, each point represents a sentence pair, computed using the concatenation of component-wise and absolute difference of features. Remarkably, sentence pairs that are similar to each other are embedded next to other similar pairs. Even without the use of relatedness labels, skip-thought vectors learn to accurately capture this property.

Since our decoder is a neural language model, we can also generate from it. We can perform generation by conditioning on a sentence, generating a new sentence, concatenating the generated example to the previous text and continuing. Since our model was trained on books, the generated samples reads like a novel, albeit a nonsensical one. Below is a 20 sentence sample generated by our model:

she grabbed my hand . “ come on . ” she fluttered her bag in the air . “ i think we ’re at your place . i ca n’t come get you . ” he locked himself back up . “ no . she will . ” kyrian shook his head . “ we met … that congratulations … said no . ” the sweat on their fingertips ’s deeper from what had done it all of his flesh hard did n’t fade . cassie tensed between her arms suddenly grasping him as her sudden her senses returned to its big form . her chin trembled softly as she felt something unreadable in her light . it was dark . my body shook as i lost what i knew and be betrayed and i realize just how it ended . it was n’t as if i did n’t open a vein . this was all my fault , damaged me . i should have told toby before i was screaming . i should ’ve told someone that was an accident . never helped it . how can i do this , to steal my baby ’s prints ? ”

Conclusion

We evaluated the effectiveness of skip-thought vectors as an off-the-shelf sentence representation with linear classifiers across 8 tasks. Many of the methods we compare against were only evaluated on 1 task. The fact that skip-thought vectors perform well on all tasks considered highlight the robustness of our representations.

We believe our model for learning skip-thought vectors only scratches the surface of possible objectives. Many variations have yet to be explored, including (a) deep encoders and decoders, (b) larger context windows, (c) encoding and decoding paragraphs, (d) other encoders, such as convnets. It is likely the case that more exploration of this space will result in even higher quality representations.

We thank Geoffrey Hinton for suggesting the name skip-thoughts. We also thank Felix Hill, Kelvin Xu, Kyunghyun Cho and Ilya Sutskever for valuable comments and discussion. This work was supported by NSERC, Samsung, CIFAR, Google and ONR Grant N00014-14-1-0232.

References