Search code, repositories, users, issues, pull requests...

The LCM operates on an explicit higher-level semantic representation, which we name a "concept". Concepts are language- and modality-agnostic and represent a higher level idea. In this work, a concept corresponds to a sentence, and we use the SONAR embedding space, which supports up to 200 languages in text and 57 languages in speech. See the list of supported languages here.

The LCM is a sequence-to-sequence model in the concepts space trained to perform auto-regressive sentence prediction. We explore multiple approaches:

These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We include in this repository recipes to reproduce the training and finetuning of 1.6B MSE LCM and Two-tower diffusion LCM. See instructions below.

The LCM repository relies on fairseq2. If you have uv installed on your system, you can install a virtual environment with all the necessary packages by running the following commands:

Leave a Comment