mSLAM: Massively multilingual joint pre-training for speech and text

Introduction

Multilingual pre-trained models have demonstrated large quality gains on a variety of multilingual Natural Language Processing (NLP) tasks (Hu et al., 2020; Ruder et al., 2021). With the emergence of multilingual pre-trained models of speech like XLSR (Conneau et al., 2020; Babu et al., 2021), similar improvements have also been observed on speech understanding tasks. One key advantage of multilingual pre-trained models is the ability to overcome data skew across languages to improve quality on low resource languages (Arivazhagan et al., 2019). By training a shared set of (usually attention-based) parameters on many languages, these models can learn crosslingually aligned representations of text or speech in a shared representation space (Kudugunta et al., 2019; Wu et al., 2019). These shared multilingual representations allow multilingual pre-trained models to use supervised data in one language to benefit lower-resource languages (Conneau & Lample, 2019). An extreme scenario of cross-lingual transfer learning is zero-shot transfer, where using supervised data to fine-tune a pre-trained model on a source language exhibits non-zero performance on a target language; without utilizing any supervision for the target language (Johnson et al., 2017; Conneau et al., 2018).

Given the convergence of architectures (Vaswani et al., 2017) and objectives (Devlin et al., 2019; Baevski et al., 2020; Chung et al., 2021) across the speech and text modalities, building a single model that could learn cross-lingual cross-modal representations of speech and text from hundreds of languages is the next natural step. Such a model can enable transfer learning across the two modalities, directly benefiting languages (and domains) with limited amounts of speech or text data. In addition, joint models of speech and text can likely enable end-to-end speech understanding tasks directly from the speech signal, including tasks like speech translation, speaker intent classification and speech language-identification, bypassing errors introduced by an intermediate automatic speech recognition (ASR) system.

While there are several potential advantages from multilingual pre-trained models of speech and text, these models suffer from interference and capacity dilution (Bapna et al., 2021). This effect has also been documented in multilingual pre-trained models of text. While lower resource languages benefit from transfer learning, with increasing multilinguality, high resource languages lose quality (Caruana, 1997; Arivazhagan et al., 2019; Conneau et al., 2019). This deterioration is typically addressed by either increasing model capacity (Arivazhagan et al., 2019; Babu et al., 2021) or incorporating approaches from multi-task learning to reduce interference by leveraging architectural or optimization improvements (Wang et al., 2020; Raffel et al., 2019).

In this work we present mSLAM, a multilingual pre-trained model of speech and text that has been pre-trained with speech from $51$ languages and text from $101$ languages. mSLAM is a multilingual extension of SLAM (Bapna et al., 2021), with the addition of a Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) on the paired speech-text data, to reduce interference and encourage stronger alignment across the two modalities.

On several downstream speech understanding tasks, including CoVoST-2 21$\rightarrow$En speech translation (Wang et al., 2021b), Fleurs speech language identification (Section 4.2) and Minds-14 speech intent classification (Gerz et al., 2021), mSLAM demonstrates significant quality improvements over equivalent models trained only on speech. On multilingual ASR tasks, including MLS-10Hr (Pratap et al., 2020), VoxPopuli (Wang et al., 2021a) and Babel (Gales et al., 2014), mSLAM matches the performance of the speech-only baseline. We also evaluate mSLAM on XNLI (Conneau et al., 2018), to understand its strengths and limitations on text tasks. We find that the addition of the CTC loss significantly improves quality on several speech and text understanding tasks, highlighting the importance of alleviating interference in multi-modal pre-trained models.

We also conduct analyses to understand the extent of multi-modal representation alignment in mSLAM. When fine-tuned with only speech translation data, mSLAM is capable of zero-shot text translation in several languages, suggesting that the model is capable of learning from data in one modality to improve quality in the other. mSLAM also benefits from multi-modal supervised data. On CoVoST-2, we jointly fine-tune mSLAM on multilingual speech translation and text translation, further improving speech translation quality by $2$ BLEU; improving over a significantly larger XLS-R (2B) model (Babu et al., 2021) and establishing a new state of the art on this dataset. Increasing mSLAM model capacity to $2B$ parameters results in further quality improvements on most downstream tasks.

Background

Multimodal pre-training: SLAM (Bapna et al., 2021) is a multimodal speech and text pretraining method, which trains a single Conformer (Gulati et al., 2020) with SpanBERT (Joshi et al., 2020) and w2v-BERT (Chung et al., 2021) self-supervised losses that leverage unlabeled monomodal data, as well as a TLM loss (Conneau & Lample, 2019; Zheng et al., 2021) and a speech-text matching loss (Li et al., 2021) that both use supervised speech recognition data. Pre-trained speech representations have been shown to be close to text (Baevski et al., 2021) and SLAM leverages this similarity for cross-modal transfer. Compared to mono-modal pre-trained models, SLAM shows improvements on speech translation, similar performance on speech recognition but degradation on text downstream tasks, exposing a transfer-interference trade-off that has been previously studied in multilingual models (Arivazhagan et al., 2019). Because SLAM focuses on English it is harder to notice cross-modal transfer, as both modalities have a large amount of unlabeled data. In many languages, speech data is scarcer than text, or vice-versa. In this scenario cross-modal transfer is more likely, similar to how high-resource languages transfer to low-resource languages in multilingual pre-training. mSLAM exploits both cross-lingual and cross-modal transfer by simultaneously training on both modalities in a large number of languages.

Multilingual pre-training: In multilingual understanding literature, models like mBERT (Devlin et al., 2019), XLM-R (Conneau & Lample, 2019) or mT5 (Xue et al., 2021b) have shown the benefit of cross-lingual transfer for improving representations of low-resource languages: on these languages, multilingual models strongly outperform monolingual pre-trained models on public benchmarks (Conneau et al., 2018, 2019; Lewis et al., 2019; Hu et al., 2020; Ruder et al., 2021). Past work has also leveraged parallel data to improve multilingual text representations, e.g. with TLM (Conneau & Lample, 2019), explicit alignment (Hu et al., 2021) or nmT5 (Kale et al., 2021). Similarly, in speech understanding, multilingual pre-trained models (Kawakami et al., 2020; Conneau et al., 2020; Babu et al., 2021) based on self-supervised losses (Oord et al., 2018; Baevski et al., 2020) improve representations of low-resource languages at the cost of reduced performance on high-resource languages. In particular, multilingual pre-trained models like XLS-R expanded the few-shot learning capability of wav2vec 2.0 (Xu et al., 2021) to many other languages, both for speech recognition and speech translation (Wang et al., 2021b). Interestingly, for speech, no lexical overlap across languages is leveraged during training, but multilingual representations still emerge from parameter sharing of the Transformer network (Wu et al., 2019). Leveraging text can potentially create connections between speech representations across languages through shared text anchor embeddings of identical character strings. Past work also leverages supervised ASR data to build multilingual representations of speech (Kannan et al., 2019; Bai et al., 2021), similar to how multilingual machine translation in NLP is used to build multilingual representations (Eriguchi et al., 2018; Siddhant et al., 2020).

Pre-training Method

Our pre-training approach builds on SLAM (Bapna et al., 2021) and extends it to the massively multilingual setting. Specifically, we build on SLAM-TLM, that combines together pre-training on speech unlabeled data with w2v-BERT (Chung et al., 2021), text with spanBERT (Joshi et al., 2020) and on paired speech-transcript data with TLM (Conneau & Lample, 2019). We skip the Speech-Text-Matching (STM) task since preliminary experiments didn’t reveal any advantages over SLAM-TLM.

mSLAM pre-training differs from SLAM on a couple of points. First, instead of using $32k$ token sentence-piece tokenization (Kudo & Richardson, 2018), we use a character vocabulary, containing $4096$ tokens spanning $101$ languages. This results in longer sequence lengths, which we cap to $512$ characters. We also increase the length of masked spans from $5$ to $20$ tokens for the spanBERT objective. Second, we apply a CTC loss (Graves et al., 2006; Graves & Jaitly, 2014) on the speech portion of the paired input, using the character-level transcript as the target. This CTC loss is applied in addition to TLM, so the input consists of a concatenated masked speech and masked text sequence, with the CTC loss applied to the speech portion of the output. We share the softmax vocabulary and parameters used for CTC with the softmax used for training the spanBERT objective with text input. We find that this CTC loss ensures stronger alignment between the speech and text representations learnt by the model, as further discussed in Sections 5 and 6.

In our $2B$ model, we increase the model dimension from $1024$ to $1408$, and the number of conformer layers from $24$ to $40$. We keep $8$ layers in the contrastive block, and $32$ in the MLM block. The peak learning rate is reduced from $6.0e-4$ to $3.6e-4$ for better training stability. Other hyper-parameters remain the same as the base $600M$ model. Note that our $2B$ model contains close $1.84B$ parameters.

2 Pre-training Data

We use three types of data for pre-training mSLAM; unlabeled speech drawn from multiple public datasets, unlabeled text from mC4 (Xue et al., 2021a) and paired speech and text transcript data from multiple sources.

We use approximately $429k$ hours of unlabeled speech data in $51$ languagesCounting languages with more than $1$ hour of speech data.. Our unlabeled speech data closely follows the pre-training data used for XLS-R (Babu et al., 2021) with one major difference: we do not use VoxLingua. As a consequence our model is pre-trained on speech from $51$ languages as compared to $128$ for XLS-R, and our pre-training set is smaller by $6.6k$ hours.

We train on $372k$ hours of speech data spanning 23 languages from VoxPopuli (Wang et al., 2021a), read speech data in 25 languages drawn from the v6.1 release of Common Voice (Ardila et al., 2019), $50k$ hours of read books data in eight European languages from Multilingual LibriSpeech (Pratap et al., 2020) and $1k$ hours of telephonic conversation data spanning $17$ African and Asian languages from BABEL (Gales et al., 2014).

2.2 Unlabeled Text Data

For pre-training with unlabeled text, we use the mC4 dataset (Xue et al., 2021b) spanning $101$ languages. We upsample lower resource languages using temperature-based sampling (Arivazhagan et al., 2019), with $T=3.0$.

2.3 Paired Speech-Transcript Data

In addition to training with unlabeled speech and text, we also use approximately $2.4k$ hours of paired speech and transcript data spanning $32$ languages, for training with the CTC and TLM alignment losses. This data is drawn from the following sources:

VoxPopuli: Approximately $1.3k$ hours of speech and transcript data spanning 14 languages. We exclude languages with less than $1$ hour of data following Wang et al. (2021a).

Multilingual LibriSpeech (MLS): We use the $10$ hour training splits of the paired data for each of the $8$ MLS languages. We exclude any paired data outside the $10$ hour splits to align with our downstream evaluations.

Babel: $1k$ hours of speech and transcript data spanning $17$ languages from the Babel ASR task.

3 Optimization and Hyperparameters

At each training step, we train mSLAM on all three types of data; each batch is composed of $2048$ sequences of unlabeled speech, $8192$ sequences of text and $256$ sequences of paired data. Our speech-only baseline, w2v-bert-51 (0.6B), sees a batch composed of $4096$ unlabeled speech sequences at every step. For our best run based on CoVoST dev performance, the speech loss has a coefficient of $1.0$, the text loss has a coefficient of $0.3$ and the paired CTC loss has a coefficient of $0.03$ (to avoid over-fitting to the small paired data). We use the Adam optimizer (Kingma & Ba, 2014) with a Transformer learning rate schedule (Vaswani et al., 2017). We use $40k$ warmup steps, linearly increasing the learning rate to $6\times 10^{-4}$, followed by inverse square root decay. We train all $600M$ models for $1.3m$ steps. The $2B$ model was pre-trained for $350k$ steps.

Downstream tasks

CoVoST 2 Speech Translation: CoVoST 2 (Wang et al., 2021b) is a multilingual speech translation (ST) dataset created by professional translation of the Common Voice speech corpus (Ardila et al., 2020). The audio consists of read speech crowd-sourced through the Mozilla Common Voice project. We evaluate on a multilingual XX-to-English task that covers translation from 21 source languages into English. The training data ranges in size from 264 hours speech for French to about 1 hour speech for Indonesian.

Multi-modal fine-tuning: Apart from fine-tuning with just ST data, we leverage the ability of mSLAM to learn from both speech and text modalities by using text translation data in addition to the CoVoST 2 ST data for multi-modal joint fine-tuning. For each CoVoSt 2 XX-to-English language pair, we use the text translation data from CoVoST 2 combined with all data from either WMT or TED Talks, as available. Specifically, we pair with WMT20 (Barrault et al., 2020) for ja, ta, WMT19 (Barrault et al., 2019) for de, ru, zh, WMT18 (Bojar et al., 2018) for et, tr, WMT17 (Bojar et al., 2017) for lv, WMT15 (Bojar et al., 2015) for fr, WMT13 (Bojar et al., 2013) and TED59 (Qi et al., 2018) for ar, fa, id, it, nl, pt, sl, sv, leaving ca and cy unpaired.

We attach a 6-layer, 512-dimension Transformer decoder to our pre-trained encoders. This decoder has 34M parameters. For ST-only fine-tuning, this model is then fine-tuned on the CoVoST 2 ST dataset. A dropout probability $0.3$ is used on the input embedding and all residual connections in the Transformer decoder to mitigate overfitting. For multi-modal fine-tuning, this model is fine-tuned on the CoVoST 2 ST dataset simultaneously with the MT dataset described above. Each training batch contains equal numbers of ST and MT examples, with a higher loss weight, $5.0$, on the MT objective. A lower dropout probability $0.1$ is used because more training data is available. These hyper-parameters were found by optimizing the w2v-BERT speech-only baseline for CoVoST 2 development BLEU.

2 Speech Classification

Fleurs-LangID: FleursDataset to be released with another publication. is a speech extension of the FLORES massively multilingual benchmark for MT (Goyal et al., 2021). Fleurs contains $2009$ sentences from the FLORES multi-way parallel evaluation set in $102$ languages. We collect read speech corresponding to these sentences, and split these utterances into train-dev-test splits with $1109$ for training (around 1.3 hours of data), $400$ for dev and $500$ for test, per-language. We collected $2.3$ utterances per sentence on average. We evaluate our pre-trained models on Speech Language Identification (LangID) on this dataset.

MINDS-14: MINDS-14 (Gerz et al., 2021) is an intent classification task from spoken data. It covers 14 intents extracted from the e-banking domain, with spoken examples in 14 language varieties. We merge monolingual datasets into a single dataset, with a $30$-$20$-$50$ train-dev-test split.

Fine-tuning setup: When fine-tuning our models on speech classification we train the multi-modal and speech encoders. Speech input is fed into the speech encoder, and the outputs from the multi-modal encoder are max-pooled together before feeding into a softmax classifier. Optionally a projection layer is applied before pooling. We tune hyper-parameters on dev performance; tuning batch sizes over $\{16,32,64\}$, learning rates over $\{2e-6,4e-6,2e-5,4e-5\}$ and projection over $\{None,model\_dim\}$. For MINDS we tune number of epochs over $\{100,300\}$ and for Fleurs over $\{5,10,20\}$. We pick the run with the best dev performance and evaluate on the test split. For MINDS-14, we report the macro-averaged accuracy over all $14$ languages.

3 Multilingual Speech Recognition

VoxPopuli: Following Wang et al. (2021a), we evaluate on the $14$ languages with more than $1$-Hr of data from the VoxPopuli ASR task.

MLS-10Hr: We report results on the $10$-Hr training split for the MLS task (Pratap et al., 2020).

Babel: Following Babu et al. (2021), we report results on $5$ languages from the Babel-ASR task.

Fine-tuning Setup: We fine-tune our pre-trained encoders with a $2$-layer LSTM (Hochreiter & Schmidhuber, 1997) as a conformer-transducer model, following Chung et al. (2021). We use a merged grapheme vocabulary based on the task-specific training set for all ASR fine-tuning experiments. We do not use language-model fusion for any experiments. For VoxPopuli and MLS we report results with multilingual fine-tuning, while we fine-tune separate models per language for Babel. Our finetuning parameters follow (Zhang et al., 2020); for the pre-trained encoder, we use a peak learning rate of $3e-4$ with $5k$ warm-up steps, while for the decoder, a peak learning rate of $1e-3$ and $1.5k$ warm-up steps. All finetuning experiments on ASR use a constant $256$ batch size. In practice, these parameters worked well across several tasks and amounts of data.

4 Text Classification

XNLI: We also evaluate mSLAM models on the XNLI sentence-pair classification task (Conneau et al., 2018) to understand its strengths and weaknesses on text understanding tasks. We evaluate our models under both the zero-shot and translate-train-all settings (Ruder et al., 2021), and compare performance against mT5 (Xue et al., 2021a).

Fine-tuning setup: We train the multi-modal and text encoders on XNLI. We tune batch sizes over $\{16,32\}$, learning rates over $\{2e-5,4e-5\}$, projection over $\{None,model\_dim\}$ and number of epochs over $\{3,5\}$.

Results

ST fine-tuning: Multilingual speech translation results are shown in Table 1. Removing all text and paired data from pre-training gives us our speech-only pre-training baseline, w2v-bert-51 (0.6B), which is already very competitive with the state-of-the-art, outperforming XLS-R (1B) (Babu et al., 2021), despite having fewer parameters and not using a pre-trained decoder. mSLAM-TLM adds text and a paired TLM objective to pre-training as described by Bapna et al. (2021), and actually leads to an average degradation in ST quality, potentially due to interference between the speech and text modalities alongside the additional pressure of massive multilinguality. Fortunately, mSLAM-CTC’s addition of a CTC component to the TLM objective, as described in Section 3, recovers w2v-bert-51 (0.6B) performance; in fact, it improves slightly, mostly on low-resource languages. As we show in Section 6, this CTC component is essential to zero-shot cross-modal behavior.

ST + MT joint fine-tuning: The picture becomes more interesting as we introduce MT (text-to-text) data during fine-tuning in the bottom four lines of Table 1. On top of the speech-only w2v-bert-51 (0.6B), adding MT data produces a modest average improvement of $+0.6$ BLEU. However, adding MT data to mSLAM-CTC, results in a larger improvement of $+1.8$ BLEU, suggesting that exposure to text during pre-training makes the encoder more amenable to using text during fine-tuning. This results in a new state-of-the art for the CoVoST 21$\rightarrow$En task, surpassing the 4$\times$ larger XLS-R (2B) by $0.3$ BLEU, enabled by large gains on high-resource languages. Increasing the capacity of mSLAM-CTC to $2B$ parameters further improves performance by $2.4$ BLEU.

2 Speech Classification

Evaluations on the MINDS-14 and Fleurs-LangID tasks are detailed in Table 2. We find that pre-training jointly with text and paired data with a TLM loss, mSLAM-TLM, improves over our speech-only baseline, w2v-bert-51 (0.6B), by $1.3\%$ and $4.6\%$ on MINDS-14 and Fleurs-LangID respectively. The addition of a CTC loss in mSLAM-CTC further improves accuracy by $2.9\%$ on MINDS-14. On Fleurs-LangID, mSLAM-CTC is worse than mSLAM-TLM by around $2.7\%$, still maintaining an accuracy improvement of $1.9\%$ over our speech-only baseline. Increasing mSLAM-CTC capacity to $2B$ parameters results in further $1.7\%$ improvement over our previous best accuracy on Fleurs, while being $0.3\%$ worse than the $600M$ model on MINDS-14.

3 Multilingual Speech Recognition

We present ASR results on VoxPopuli, Babel and MLS-10hrs in Table 3. Our speech-only pre-training baseline, w2v-bert-51 (0.6B) already outperforms XLS-R (Babu et al., 2021) on VoxPopuli and MLS-10hrs as shown in Table 3. Our mSLAM-CTC (0.6B) model slightly outperforms the speech-only baseline on VoxPopuli and slightly lags on MLS-10hrs, but both improve over published results. On Babel, our model is behind XLS-R (1B) (Babu et al., 2021); possibly due to a lack of language model fusion. mSLAM-CTC is very close to w2v-bert-51 (0.6B) and both improve over mSLAM-TLM. In conclusion, mSLAM achieves competitive ASR results without losing speech capacity across a variety of ASR tasks and languages. Increasing mSLAM-CTC capacity to $2B$ parameters results in improvements over both, the $600M$ model and our speech-only baseline.

4 Text Classification

On XNLI, similar to SLAM results on GLUE, we observe decreases in performance compared to mono-modal models due to capacity dilution (see Table 4). In the translate-train-all setting, our mSLAM-CTC (0.6B) model obtains 70.0% accuracy on average compared to 79.8% for an mT5-Base model (0.6B). However, it performs comparably to the smaller mT5-Small model (0.3B) which gets 71.3%.

On zero-shot classification, we observe a bigger drop in performance when using multi-modal pre-training compared to the mT5 models. Zero-shot classification being a test-bed for the sharing of multilingual representations, we attribute this to speech interfering with the sharing of text representations across languages. Looking more closely at per-language results, the performance drops in particular for non-European languages, e.g. Thai and Chinese where the model loses around $20\%$ accuracy. Note that the paired data used during pre-training is predominantly from European languages, and the performance of mSLAM-CTC improves significantly over mSLAM-TLM on this set of languages. We hypothesize that having in-language paired data and alignment losses could be contributing to reduced interference between speech and text for these languages, resulting in more robust representations. This is also supported by the significantly improved performance on non-European languages with the mSLAM-CTC (2B) model, where the increased capacity might alleviate some of the interference. However, there are other confounding factors in our text pre-training approach compared to standard multilingual text pre-training, including the conformer architecture and fully character-level encoder pre-training, which might be contributing to these findings. We leave the study of multilingual representation alignment in joint speech-text models to future work.

Analysis

Do we really need text pre-training or just alignment losses? mSLAM-CTC models add two improvements over the speech-only baseline: (i) TLM and CTC alignment losses over paired data, and (ii) Pre-training with large amounts of web-text. This raises the question whether our improvements are arising from (i), (ii) or a combination of the two. To answer this question we train a mSLAM-CTC model on unlabeled speech and paired speech text data, but no unlabeled text. We evaluate this model on CoVoST ST, MINDS-14 and Fleurs-LangID and present results in Table 5. We find that the performance of the mSLAM-CTC model without text falls somewhere between our speech-only model and mSLAM-CTC on MINDS-14 and Fleurs-LangID, suggesting that the additional text pre-training data is at least partially responsible for the observed improvements on these tasks. On CoVoST-2, mSLAM-CTC without text almost matches the performance of mSLAM-CTC when fine-tuning jointly with text translation data, suggesting that a majority of the improvements in this setting arise from MT data, and the alignment loss might be enough to enable the model to benefit from text supervised data for fine-tuning.

Are cross-modal representations really aligned?

We have seen benefits from adding text to pre-training alongside a CTC loss. The importance of this CTC loss suggests that some amount of cross-modal representation alignment is necessary to take advantage of speech and text data in the same model, but can we construct an experiment to clearly demonstrate this alignment?

Zero-shot performance is one strong indicator for representation alignment. To that end, we use our joint fine-tuning infrastructure to conduct CoVoST 2, 21$\rightarrow$En translation experiments where we fine-tune the mSLAM-CTC model on one modality (speech or text) and evaluate on the CoVoST 2 test set using the the other input modality.

Cross-modal results, alongside the amount of paired data available during pre-training, are shown in Table 6. For score calibration, the S$\rightarrow$S column shows a modality-matched scenario of fine tuning on speech and testing on speech, corresponding to the sixth row of Table 1. First, note that zero-shot cross-modal translation is possible: the S$\rightarrow$T column shows that fine-tuning on speech and testing on text results in translation performance above 5 BLEU for 13 of 21 languages. Furthermore, 6 of those 13 languages had no paired data available during pre-training, demonstrating the power of being both multimodal and multilingual. Most surprisingly of all, Russian (ru) has an excellent zero-shot score of 21.9 BLEU, and it not only has no paired data during pre-training, but also no paired data in its Cyrillic script, yet the mSLAM model can translate it into English.

We tested for the same behavior with w2v-bert-51 (0.6B) and mSLAM-TLM and found no evidence of zero-shot S$\rightarrow$T transfer. We also tested the impact of unlabeled text: average zero-shot BLEU is 9.4 for full mSLAM-CTC but only 6.4 without any unlabeled text during pre-training (not shown). The languages without paired data suffer disproportionately: Swedish (sv) drops from 15.2 to 0.7 BLEU and Russian (ru) drops from 21.9 to 4.2 BLEU.

There is still much work left to be done. Russian is somewhat of an outlier in terms of script sensitivity, all other cross-modal success stories are for predominantly European languages in the Latin script. Furthermore, some languages such as Turkish (tr) have paired data available during pre-training, but demonstrate limited zero-shot transfer. Finally, note that this zero-shot transfer does not work in the other direction: the T$\rightarrow$S column clearly shows that a system fine-tuned only on text cannot translate speech.

Examining zero-shot text translation outputs. While the system’s cross-modal capabilities are surprising, there is still a substantial drop in BLEU for zero-shot translation of text: compare the S$\rightarrow$S column to the S$\rightarrow$T column in Table 6. This reduced performance often manifests as repeated or empty outputs: see Table 8 for contrastive zero-shot text translation examples. This is reminiscent of oscillatory hallucinations caused by unexpected inputs (Lee et al., 2018).

Visualizing cross-modal alignment with a CTC probe.

To visualize the information available when text is input to a model fine-tuned only for speech, we create a CTC probe for mSLAM encodings. Freezing the mSLAM encoder after pre-training, we tune only the softmax parameters of a CTC decoder using a 21-language ASR objective on the CoVoST 2 data: speech is input, and the gold character-level transcription is the output. We can then decode the CoVoST 2 test set using either speech or text inputs. If speech is input, the ASR task matches the fine-tuning objective. If text is input, this represents a zero-shot character-level auto-encoding (CAE) task. We measure the success of both tasks using character-error-rate (CER).

Per-language results of this CTC probe are also shown in Table 6 as S$\rightarrow$T CAE. First, it is notable that even with a frozen encoder and a far less powerful decoder, we still see zero-shot transfer from speech to text inputs. In fact, with the exception of Turkish (tr) and Russian (ru), zero-shot CAE performance with less than 20 CER is predictive of zero-shot translation performance greater than 5 BLEU. Russian again is an interesting case, with terrible zero-shot CAE performance, but excellent zero-shot MT performance. However, the real value of such a probe is the ability to inspect the outputs. Table 7 shows randomly selected examples for both, a typical success (French, fr) and one of our more mysterious languages (Russian, ru). For French, most of the content is retained with text input, though some capitalization and punctuation is lost. Interestingly, for Russian, its Cyrillic text input results in a partial transliteration into the Latin script. This suggests that Russian is mapped into the same encoding space as Latin script languages during pre-training, which helps explain its strong cross-modal and cross-lingual transfer behavior.

Conclusion

We introduced mSLAM, a multilingual pretrained model capable of representing speech and text in a shared representation space. mSLAM is trained on unlabeled speech in $51$ languages with a w2v-BERT objective and character-level text in $101$ languages with a SpanBERT objective. In addition to unlabeled data, we train mSLAM on small amounts of paired speech-transcript data with a novel TLM+CTC objective to encourage representation sharing across the two modalities. Downstream evaluations on CoVoST 2 Speech Translation, Speech intent classification and Speech LangID demonstrate that mSLAM improves over equivalent speech-only baselines on speech understanding tasks, while maintaining similar quality on ASR. In addition to fine-tuning with labeled speech data, mSLAM can also leverage text supervision to improve the quality of end-to-end speech tasks, as demonstrated by our experiments on CoVoST 2 Speech Translation, establishing a new state of the art on this dataset. Increasing the capacity of mSLAM to $2B$ parameters further improves quality on Speech Translation, Speech Language Identification and multilingual ASR.

On XNLI sentence-pair classification, we observe cross-lingual zero-shot performance equivalent to text-only models half the size of mSLAM on (European) languages with relatively large amounts of paired data, but severe quality degradation on languages with scarce parallel data. We notice that this degradation can be addressed to some extent by increasing the capacity of the joint speech-text model.

When fine-tuned on speech translation only, mSLAM is capable of cross-modal zero-shot text translation, demonstrating strong evidence for representation alignment. Probing a frozen mSLAM encoder with a CTC head fine-tuned for ASR demonstrates high quality on text reconstruction, providing additional supporting evidence in favour of aligned speech-text representations.

The use of paired data and alignment losses results in quantitative improvements on several speech understanding tasks and reduced degradation on text understanding tasks, highlighting the need for mitigating interference in multilingual multi-modal pre-training. We hope that this work catalyzes further research towards improving and understanding universal, multi-modal pre-trained models.

References