(Why) Was the LLM breakthrough useful for images, audio, etc.?

The breakthrough of Large Language Models (LLMs) was transformative not just for text, but for images, audio, and video. This revolution was driven by several key innovations and paradigms that the LLM community established. First, the Transformer architecture, originally designed for text translation, proved to be a universal engine. Its self-attention mechanism is agnostic to data type, enabling its application to images and audio. For instance, the Vision Transformer (ViT) outperforms traditional CNNs by treating image patches as sequences, while audio models like the Audio Spectrogram Transformer process sound as tokenized sequences.

Tokenization became a universal interface bridging different modalities. While text naturally exists as discrete tokens, continuous data like images and audio required innovations in modality tokenization. Techniques like VQ-VAE (Vector Quantized Variational Autoencoders) allowed researchers to compress images and audio into discrete tokens, making them compatible with the next-token prediction paradigm central to LLMs. This approach underpins tools like OpenAI’s Sora, which treats video as sequences of 'spacetime tokens.'

Scaling laws and self-supervised learning, pioneered by LLMs, were equally impactful. Traditional vision models relied on labeled datasets like ImageNet, but LLMs demonstrated that models could scale effectively with self-supervised learning (SSL). Techniques like Masked Autoencoders (MAE) randomly hide image patches and reconstruct them, allowing vision models to learn from vast, unlabeled datasets. This shift mirrored the LLM paradigm of scaling with compute and data.

Another critical innovation was contrastive alignment, exemplified by CLIP (Contrastive Language-Image Pre-training). By pairing a text encoder (an LLM) with an image encoder, CLIP mapped matching text-image pairs to the same semantic space. This enabled text-to-image generators like DALL-E and Stable Diffusion to understand and generate images based on human-intelligible prompts. The alignment of models to human preferences, pioneered by Reinforcement Learning from Human Feedback (RLHF) in LLMs, extended to media generation. Models like DALL-E 3 and Midjourney v6 use RLHF and instruction-tuning for more conversational, user-friendly interfaces.

While LLMs inspired these advancements, it’s important to note that diffusion models, not autoregressive LLMs, dominate image and audio generation. However, even diffusion models have been 'LLM-ified.' For example, researchers have replaced traditional U-Net architectures with Diffusion Transformers (DiTs) to leverage the scaling laws discovered in LLMs. Stable Diffusion 3 and Sora both rely on this LLM-inspired approach.

In summary, the LLM breakthrough was instrumental for other modalities because it provided a universal blueprint: tokenize data, process it with Transformers, predict missing pieces, scale compute, and align outputs with human intent. This paradigm unified AI research, enabling breakthroughs across text, images, audio, and video.