Own the Loop – Part II: What Activation Steering Reveals About AI Judgment

Part 1 of this series mapped 6,573 rows of agent memory as a latent space and found that the geometry of organizational decisions is readable: it recovers operating structure, surfaces influence patterns, detects synthetic data, and reveals instrumentation gaps. What embeddings cannot do is model the reasoning process itself. They describe the shape of decisions; they say nothing about what a language model does when it reads those decisions and tries to reason about them. 

That is the question Part 2 takes up. When an AI reads your organization's decision traces, does it build an internal representation of your judgment? If it does, can you locate that representation inside the network? And if you can locate it, can you do anything with it? The short version: reading turned out to be easy. Writing did not. And the way it failed is the most useful thing we learned. 

For decision-makers, the stakes of this question are not abstract. If an AI system represents your organization's standards internally, that representation shapes every downstream decision the model makes. Whether you can inspect that representation, verify it matches your actual standards, and correct it when it drifts determines whether you govern the intelligence or merely observe it. Part 2 is about what is currently possible and where the ceiling is. 

Everything ran on our own hardware. No API, because you cannot read internal model activations through an API, and no cloud, because this is our company's decision history. The model was Qwen2.5-3B-Instruct, a three billion parameter open weights language model (36 layers, internal dimension 2,048, float16 precision). The interpretability library was TransformerLens 3.5.1, which provides hooks on the residual stream at every layer. The residual stream is the internal communication channel a transformer model maintains as it reads and processes text; each layer reads from it and writes back to it, and it is the primary site for interpretability work. Probing used scikit-learn logistic regression with standard scaling, balanced class weights, and stratified 5-fold cross-validation, with macro F1 as the primary metric. The baseline for comparison was the BAAI/bge-small-en-v1.5 embedding model applied to identical texts. The dataset was 849 labeled decision traces from Engram. 

One environment note that will save a team an afternoon: Python 3.14 was the system default on our machine and it breaks the numba and torch dependency chain. Pin 3.12. Running the model in float16 carries a silent accuracy caveat in some configurations; this returns at the end of the post. 

From the memory store we built four supervised tasks. The rep task (449 examples, 5 classes) asked which team member a given decision trace was about. The reward task (182 examples, binary) asked whether our evaluator agent had scored a given move positive or negative. The mode task (129 examples, binary) asked whether an episode was explore or exploit. The market task (89 examples, 3 classes) asked whether a trace was about Pakistan, KSA, or US work, kept as a positive control because market identity is partially surface-level. 

The step that determines whether any of this means anything is label masking. If the text of a decision trace contains the team member's name, a probe predicting the team member is reading a name, not representing an abstract concept. So every surface form of every identity was replaced with a [REP] token, including abbreviations caught only by reading samples by hand. Reward examples had the score, the rationale, and the evaluator's assessment fields excluded entirely; the probe sees only the move description and its context. Mode examples had every "explore" and "exploit" token masked. A leak check then ran over the full dataset. Zero identity tokens survived. If you take one thing from this section, take the leak check. Most published probe results are decided there, and most do not say so. 

Residual stream activations were captured at every layer for every example. The first attempt used the last token's activation, the standard shortcut in probe work, because the last token is often where a model consolidates its prediction before generating output. The probes looked strong, until we compared them with the bge embedding baseline on the same masked texts. 

By our pre-registered criterion, the transformer probe must beat the embedder or the phase fails. On the last-token measure, it failed. But the comparison was methodologically inconsistent: bge mean-pools over all token positions, while the last-token probe reads a single position. Re-capturing with mean-pooled activations across all token positions changed the picture entirely. 

Same model. Same data. Opposite verdict. Rep and reward now beat the specialized embedder by meaningful margins. Shuffled label controls sit at chance, confirming the probes are reading something real. The positive control behaves as expected. If the first table had been published, the conclusion would have been wrong in a paper-shaped way. Pooling is not a detail. 

Two structural observations are worth keeping. Identity peaks at layer 5, partly because names are partially lexical. Judgment peaks at layer 17, deep in the middle of the network where abstract features typically live. And mode fails everywhere, in the transformer and in the embedder alike: whether an agent was in exploration or exploitation mode is not recoverable from its state summaries, because that distinction arrived as an external instruction rather than something the agent itself recorded. A null result found an instrumentation gap, which is exactly the right use of a null result. 

There was also a free result that deserves to stand on its own. Prompted zero-shot, with no training and no examples, the model's own good-versus-bad verdict on each decision trace agreed with our evaluator's label at an AUC of 0.886. The model does not just represent our judgment. It reproduces it, unprompted, at close to 90 percent accuracy. An off-the-shelf open weights model reading your organization's decisions is already approximating your evaluator's standards without being told what those standards are. 

If organizational judgment is linearly readable at layer 17, the next question is whether there is a stable direction in that layer's activation space that specifically carries the judgment. Something you could identify, isolate, and use to steer the model's output. 

We derived the most obvious candidate: the difference between the mean activation vectors for positive and negative examples at layer 17, computed on a training split only. Then three tests on held-out data. 

Invariance. The model was prompted to paraphrase each decision trace, preserving all factual content but changing the surface phrasing. We then measured how much each paraphrased trace's projection onto the judgment direction shifted relative to the original. The shift was 0.84 of the full class gap width. A representation that moves almost an entire class gap under rewording is not a stable concept handle. It is a surface-sensitive correlate, not a deep structural feature. 

Steering. We added the derived direction to the residual stream at layer 17 and measured how much the model's verdict logits shifted. At small amplitudes, nothing. We then swept: 8, 16, and 32 standard deviations of the projection spread, with the largest representing 42 percent of the residual stream norm, which is not a subtle nudge. Crucially, we ran a norm-matched random direction at every amplitude as a control. The random direction moved the verdict logits more than our judgment direction at every single amplitude tested, 2.92 versus 0.43 at the largest scale. And the judgment direction's own effect was sign-incoherent: nudging toward positive and nudging toward negative both pushed verdicts in the same direction. That is not steering. That is corrupting the model's internal state and watching it lose confidence. 

Patching. We swapped the complete last-token activation at layer 17 from a positive trace into a negative one. Mean effect: negative 0.02 change in verdict logits. Nothing moved. 

The three tests agree on a diagnosis. The judgment information is real and statistically present at layer 17, but it is distributed. After layer 17, the network re-reads the whole trace through attention, recomputing the judgment signal from many positions simultaneously. Any local edit made at a single position gets averaged away in that recomputation. The concept is not stored like a variable in a register. It is re-derived on demand, at many positions, with redundancy. 

There is also a mismatch in our own design worth naming explicitly: the winning probes measured mean-pooled information across all tokens, and our interventions performed surgery at a single token. We measured a crowd and operated on one person. The intervention could not have succeeded given how we designed it. 

Two honest constraints on the null result: the direction estimator was simple (class means on 182 examples), and we tested one layer at one position. The result constrains what a 3B model with linear tools shows in a specific context. It does not rule out the existence of more accessible representations under different instruments. The original global workspace interpretability research used Jacobian-based methods across all layers on substantially larger models. Sharper instruments exist. The lesson from the failure is methodological as much as it is factual. 

For the practitioner building with these models, the practical takeaway is this: an open model reading your decisions already carries an approximation of your organizational judgment. That judgment is readable with high accuracy using standard probe methods. But it lives in a distributed computation that resists local editing. The path forward is not to find a better lever inside the general model. It is to concentrate the judgment in something smaller, owned, and purpose-built, where the mechanism has nowhere to hide. That concentration is the subject of Part 3.

Mask your labels, then run an automated leak check. Probe results without one are not reviewable. Report both last-token and mean-pooled probes; if your conclusion flips between them, say so at the top, because ours did. Every steering claim needs a norm-matched random direction control at the same amplitude range; without one, you cannot distinguish steering from state corruption. A dose-response sweep distinguishes the two: incoherent sign effects at high amplitude mean corruption, not influence. Match the intervention to the measurement: a pooled probe plus a single-token surgery is a category error, and we made it so you do not have to. And publish the null. Reading without writing is a real, reportable state of knowledge, and it is where most applied interpretability actually lives. 

Umar Bilal, Co-Founder, CodeNinja 

All experiments ran locally on our own hardware. The agents that wrote the dataset were working the whole time. 

Previous: Part 1 — The Architecture of Organizational Intelligence: The shape of 6,573 agent memory rows as a latent space. 

Next: Part 3 — What Belongs in the Weights of an Enterprise AI Model: We stopped trying to grab the judgment inside a general model. We trained it into a small model we own. Overall verdict accuracy went from 0.50 to 0.98, and the most important finding was the failure.