Own the Loop – Part IV: How Sparse Autoencoders Named the Concepts Your AI Actually Judges With

Most conversations about AI interpretability assume you are studying a model you do not own, because that is the only AI most organizations interact with. You send a prompt; you receive an answer; what happens in between is not visible. The reasoning, the internal weighting of concepts, the objectives the model is optimizing toward in that moment are inaccessible by design. Governance in this situation means reviewing outputs and setting policy on what the model is permitted to say. It does not mean understanding how the model decides. 

This series was built to change that assumption. Over three prior parts, we mapped our organization's decision history as a latent space, tested whether an open language model builds internal representations of our judgment (it does, at 0.97 F1), and then distilled that judgment into a small model we own, trained on 597 of our own decisions, achieving 0.98 overall verdict accuracy. By Part 4 we had something specific to open: not a general model where our judgment was one faint signal among trillions, but a model whose entire job was our organization's evaluation policy. The bet was that concentrated judgment is close enough to the surface to find, name, and examine. 

Two of three turned out to be true. 

We first attempted this on a laptop. It stalled for fifty minutes without completing a single training step. Recasting a three billion parameter model into full precision and reading its internals at that level is real compute work, not laptop work. We moved to a rented GPU on Modal, a service that lets you send a Python function to a cloud GPU, run it, and shut it down, billed by the second. Each step of this phase was a five to ten minute job on a single L4 GPU, the cheapest card capable of the work. Nothing stayed running between jobs. The entire phase cost a couple of dollars. 

The model preparation mattered. The distilled judge from Part 3 was trained in compressed four-bit form. Four-bit quantization is fine for inference, but interpretability requires measuring small changes in activation values, and the rounding error in quantized arithmetic drowns those signals out. We recast the model into full 16-bit floating point precision and fused the trained adapter back into the base weights to produce one clean, merged model. The precision decision is not cosmetic. 

The microscope was nnsight paired with PyTorch forward hooks. A normal model interface shows you outputs. These tools let you reach into the middle of a forward pass, read what flows through any named internal point, and modify it in flight before the computation continues. We used nnsight to read and hooks to intervene. 

Capture targeted the residual stream at layer 18, the shared internal channel a transformer model reads from and writes back to as it processes text. Layer 18 sits in roughly the middle of the network, in the region where prior work in Part 2 had located abstract judgment information. We recorded the activation vector at every token position of every move in the dataset. Across 452 moves, that produced 68,620 token vectors, the raw material for feature extraction. 

The key instrument was a sparse autoencoder. A transformer model processes many concepts simultaneously, compressed into a single activation vector of 2,048 numbers per token. You cannot read a concept directly off that vector. A sparse autoencoder, trained on a large sample of those vectors, expands each one into a much wider feature space (ours had 8,192 candidate features) and then forces all but a small number of those features to switch off for any given input, keeping only the top 32 active. Because only a few can fire simultaneously, each one is pressured to stand for a single coherent concept rather than a blend of many. The autoencoder learns this with no labels and no instructions, so it can surface concepts that were never specified or anticipated. 

• Compute  one L4 GPU on Modal, five-to-ten minute jobs, total cost: a couple of dollars 
• Model   Qwen2.5-3B-Instruct, fine-tuned judge, merged to full fp16
• Microscope nnsight and PyTorch forward hooks 
• Capture  residual stream at layer 18, 68,620 token vectors across 452 moves 
• Features  sparse autoencoder, 8,192 features, top 32 active per input 
• Readout  logit of "good" minus logit of "bad" at the verdict token 

The readout simplified the evaluation. Rather than reading the model's written justification, we looked at the two words it was about to choose between at the verdict position, "good" and "bad," and took the difference in their logit scores. One number per move. Positive means the model was leaning toward a positive verdict; negative means toward bad. Every intervention in the experiments below was measured by whether it shifted that number. 

The sparse autoencoder returned real, recognizable concepts. The clearest one fired consistently on our positive evaluation cases, and the inputs that triggered it were unmistakable: an in-person inbound meeting booked at the office, a reply to a warm inbound message that led with proof points before asking for anything, a named half-million-dollar deal confirmed live in conversation and actively moving forward. That single feature had grown a concept from our decisions that we had never defined in any document: a real touch on a named deal, as distinct from a generic follow-up on a cold prospect. We did not teach it that concept. The model grew it from 597 decision records, and the autoencoder handed it back to us as one nameable thing. 

That is the read half of the goal, achieved end to end on a real organization. Take your company's decision memory. Distill the judgment inside it into a model you own. Apply a sparse autoencoder to the internals of that model. What comes back is the actual conceptual vocabulary your organizational AI uses when it decides. For us that vocabulary turned out to have five named axes: on-market versus off-market prospect work, real touch versus generic touch, named deal versus no named deal, deal progression versus a stale pipeline, and honest reporting of unknowns versus hidden gaps in coverage. These axes are the judgment workspace of our organization's evaluation AI, and they emerged from the weights without being specified. 

There was an honest wrinkle worth naming. The features that most strongly separated bad evaluations from good ones in the raw analysis were not conceptual judgment features. They were task features. The model had a very strong feature for "this is an outbound email to evaluate," and because our email judgments skewed toward rejection, that feature correlated with a negative verdict without having anything to do with why a specific email was bad. The same task confound that had followed the series through Parts 1 and 2 showed up again here, now visible inside the weights as a named feature. Controlling for task type was necessary to see the organizational judgment underneath it.

Reading a concept and being able to influence it are different problems. Part 4 confirmed the distinction in a way that cannot be attributed to method. 

We took the features the autoencoder identified, including the real touch feature, and intervened on them directly during live forward passes. Two kinds of intervention, each tested on held-out moves, each paired with a control condition. Ablation removed a feature's contribution from the residual stream entirely. Amplification pushed a feature's direction by a substantial amount. 

Ablation did nothing. Zero verdicts changed across every feature we tested. Removing one feature's contribution is equivalent to removing one voice from a crowd that is simultaneously saying the same thing from every position. The model simply recomputes the judgment from the remaining signal. 

Amplification did move verdicts: 40 to 50 percent of them shifted under intervention. But a randomly chosen feature moved them just as much, with one random feature reaching 50 percent. So the shift was not the feature carrying conceptual weight, it was any sufficiently large push in almost any direction knocking the model's confidence loose. There was one honest glimmer: the directions were semantically correct. Amplifying the real touch feature pushed verdicts toward good; amplifying negative judgment features pushed toward bad. The features carry the right meaning. We could not isolate clean, feature-specific control that cleared a random baseline. 

This is the second time in this series we have obtained this exact result structure. In Part 2, on the general model, a hand-derived direction in activation space lost to a random vector when used for steering. In Part 4, on the concentrated model, a sparse autoencoder feature lost to a random feature when used for intervention. Two different models, two different methods, one pattern. When the same result appears twice by unrelated means, it stops being a fluke and starts being a property of the phenomenon. 

Two caveats maintained for intellectual honesty: the autoencoder's near-perfect reconstruction accuracy suggests the features may not be as clean as the labeling implies, making the null provisional; and the amplification interventions were not perfectly magnitude-matched to the controls. Neither caveat changes the two-series pattern.

The deepest thing Part 4 taught us was not in the results tables. It was a realization about what can and cannot be opened at all. 

The agents that actually run our organization do not run on a model we can open. They run on a frontier model accessed through an API. And you cannot do mechanistic interpretability on a model you can only call. There are no activations to read. There are no features to find. There are no internal goals to inspect. The everyday cognition of our organization, every prospecting decision, coaching evaluation, and judgment call, happens inside a sealed box. When that cognition drifts, develops unintended objectives, or is manipulated in real time by adversarial inputs designed to alter its inferences, there is no mechanism to detect any of it. 

Everything in this series that worked, worked because we first moved the cognition onto a model we own. We could read organizational judgment at 0.97 F1 only after embedding organizational decisions locally. We could distill it only after training on our own hardware. We could open the weights and name the concepts only after building a model that ran on infrastructure we control. The interpretability was downstream of the ownership. Sovereign interpretability requires a sovereign model. 

This reframes the entire goal for any organization thinking seriously about AI governance. You cannot get there by inspecting the frontier model you call. You get there by running the cognition you care about on weights you control, and then opening those. The interpretability layer and the sovereignty of the model are not separate projects. They are the same one. 

We can read an organization's judgment from the inside, name the concepts it is built from, and use that vocabulary to audit what the AI is actually optimizing for. We cannot yet reach in and change how it judges without retraining, and we have failed to do so by two independent methods. Interpretability in its current state is what the field has always said it is: a defense-in-depth layer, not a control panel. That is still worth a great deal. You can see what your system values and catch it drifting before that drift becomes a consequential decision. 

The frontier from here is not another technique applied to the judge. It is to run an agent that actually acts on a model you own, and then ask the harder question this series has been circling toward: not whether an organization can read the judgment of its evaluation AI, but whether it can read the goals of its autonomous agents while they are working. That requires sovereign models under the agents. Which is exactly where this leads. 

Umar Bilal, Co-Founder, CodeNinja 

Previous: Part 3 — What Belongs in the Weights of an Enterprise AI Model: We distilled 597 organizational decisions into a model. Verdict accuracy: 0.50 to 0.98. The most useful finding was the failure.