Deep Learning
Louis
LLM Interpretability: The Role of Sparse Autoencoders
Large language models (LLMs) like GPT-4 and Claude 3 are incredibly powerful, yet they remain black boxes that are difficult to decipher. It is only natural to wonder how these models manage to generate such precise responses or even come up with creative ideas. The interpretability of these models is a major challenge, and one promising approach to exploring these mysteries relies on "Sparse Autoencoders" (SAEs). In this article, we show you how SAEs allow us to break down these LLMs into interpretable components, offering a glimpse into how these complex artificial intelligence models function.
To understand how SAEs contribute to LLM interpretability, it is essential to first understand what an autoencoder is. An autoencoder is a neural network that learns to compress and decompress its inputs. Simply imagine a neural network that takes a 100-dimensional vector as input, reduces it to 50 dimensions, and then reconstructs it back into 100 dimensions. The goal of the model is to learn how to minimize the difference between the input vector and the output vector after reconstruction.
Sparse Autoencoders distinguish themselves by adding a "sparsity" penalty to their loss function, which encourages the model to activate only a small proportion of its neurons in the hidden layers. The sparsity penalty therefore pushes the model to create an intermediate vector (activations) with as many zero values as possible. We will see later that this penalization technique makes it possible to obtain a compressed, and therefore more interpretable, representation of the intermediate activations of other, larger models like LLMs.
The idea behind using SAEs for LLM interpretability is to decompose the intermediate activations of LLMs to make them easier for humans to interpret. The activations of a language model are often opaque: each neuron can potentially encode multiple concepts at once, a phenomenon known as superposition. This complexity makes it difficult to understand what each neuron truly represents.
SAEs can be trained using the model's intermediate activations at various points in its architecture, often between two layers. We call this a feature, an interpretable component of the activations produced by training an SAE between two layers of an LLM. Each feature aims to represent an identifiable concept that can be interpreted by humans (for example: semantic concepts—"human flaws" or "price increases"—grammatical categories, or specific themes).
To summarize, the goal of using SAEs is to obtain a sparse representation (with as many zero values as possible), where each feature corresponds to an interpretable concept. These representations offer a way to break down LLMs into smaller, more explicit elements, thereby facilitating the interpretation of the responses produced by the model.
As previously explained, one of the strengths of SAEs is their ability to extract features from the intermediate activations of LLMs. The SAE training process transforms these complex activations into a set of features interpretable ones. Here is how it works:
This process, which allows for the extraction of precise concepts from LLM activations, also makes it possible to directly influence the model's behavior. By applying a technique called feature steering, it is possible to directly manipulate the activations of a feature to steer the model's responses toward a specific subject. Anthropic researchers demonstrated that by artificially amplifying the activation of the feature associated with the Golden Gate Bridge, they could prompt the Claude model to systematically include references to the bridge in its responses, even when the subject was irrelevant. For example, when asked to describe its physical form, Claude replied: "I am the Golden Gate Bridge... my physical form is the iconic bridge itself."
Recently, notable efforts have been made by companies like OpenAI and Anthropic to develop methods for decomposing language models using Sparse Autoencoders. OpenAI has successfully decomposed GPT-4 activations into 16 million features potentially interpretable features. These features include themes such as "rising prices" or "human imperfection," which correspond to concepts activated within the model.
Anthropic has focused on extracting features from Claude 3 Sonnet. They have discovered complex features, ranging from "data type signatures in computer code" to "personality traits" or "cultural biases," and even "abstract behavioral traits related to deception or manipulation." Their work highlights that Sparse Autoencoders are capable of extracting very high-level features, which not only allow us to understand how the model reacts to certain types of input but also to manipulate it, for instance, to avoid certain forms of bias or to force the appearance of specific themes.
Despite their promise, Sparse Autoencoders have limitations. One of the main problems is evaluating interpretability: how can we ensure that a feature is truly interpretable or that it corresponds to a concept understandable to humans? Today, evaluation methods still rely heavily on subjective assessment. One must manually examine the activations of features and decide if they make sense. Furthermore, although SAEs are capable of decomposing activations locally, they reveal nothing about how these features are used across the other layers of the model.
However, the outlook is encouraging. Efforts to make LLMs more explainable help build trust in these models, a crucial issue for ensuring their safe use in sensitive contexts. Improvements in SAE training, particularly through increasing the size of decoders or applying new regularization methods, could eventually extend the reach of interpretability to a wider variety of models and use cases.
Anthropic recently introduced Sparse Crosscoders, a new approach that expands the capabilities of Sparse Autoencoders by allowing the extraction of interpretable features from LLMs across multiple layers simultaneously. Unlike traditional methods that analyze each layer in isolation, Sparse Crosscoders connect activations between layers and identify common patterns that persist throughout the model. This new approach allows for a clearer interpretation of the internal mechanisms of LLMs. Furthermore, they allow for the comparison of different training versions of the same model architecture by isolating the features shared and specific features, which is particularly useful for studying the effects of fine-tuning on these models. This promising approach paves the way for more granular analyses of model evolution, strengthening transparency and safety in AI development.
Sparse Autoencoders represent a promising breakthrough in opening the black box of LLMs by decomposing their activations into interpretable components. This approach is essential for improving understanding of and trust in these AI models. Research conducted by OpenAI and Anthropic shows that these methods can be applied to increasingly large models, offering valuable insights into their internal workings. Recently, the introduction of Sparse Crosscoders by Anthropic has opened new perspectives for interpretability by allowing the extraction of interpretable features across multiple layers simultaneously. With these advancements, SAEs and Sparse Crosscoders play a central role in improving the reliability and transparency of LLMs, paving the way for trustworthy AI that is more accessible for human analysis.
[1]An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability by Adam Karvonen
[2] Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet by Anthropic
[3] Extracting Concepts from GPT-4 by OpenAI
[4] Sparse Crosscoders for Cross-Layer Features and Model Diffing by Anthropic
Lorem ipsum dolor sit amet consectetur. Ipsum penatibus viverra feugiat amet varius pharetra porta risus lectus. Turpis enim massa non tincidunt in faucibus.
