How We Created a Reinforcement Learning Gym at CodeNinja

Most organizations running agents in production cannot produce a queryable record of what an agent tried on a given day, what score it received, who or what gave that score, and what changed as a result afterward. Ask the question plainly and the honest answer, in most places, is that the record simply does not exist in any form that can be pulled up on demand. That gap is the problem this piece is about, and it is also the gap we set out to close inside our own operations at CodeNinja. 

We had agents running against outbound research, code review, candidate screening, sales briefings, and reconciliation, among other things, and we were spending real tokens on all of it every day. What we did not have, when we actually looked, was evidence that any of it was getting better over time. It was automation. It was not learning anything. 

This piece is the account of what we built to close that gap, inside our own operations, over roughly a year, and what the resulting estate looks like now that it has grown to something we did not fully plan for at the outset. Close to forty production agent loops, running across six different parts of the business, most of them scored automatically every night and verified by a human most mornings. It is also, more usefully, an account of the mistakes made along the way, because most of what compounds now was learned by getting it wrong first.

The distinction sounds obvious once it is stated and is very easy to miss in practice, because automation and a learning system look identical from a distance. Both involve agents doing work without a person typing every step. The difference only shows up over time. Two organizations can spend an identical amount on inference in a given year and end that year in completely different places. The one that wrote down what happened on every run, scored it, and fed that score back into the next run will have a policy that has genuinely improved. The other will have a large invoice and roughly the same capabilities it had in January. 

“Token throughput is necessary and nowhere near sufficient. Structure is what converts throughput into compounding capability.” 

We had, without really noticing, built the first kind of organization. Every function had agents. None of those agents were improving on their own, because nothing about how we ran them created a feedback loop. An agent that runs the same way in December as it did in January, regardless of how many times it has run in between, is not learning. It is a very expensive way of doing the same thing repeatedly. 

The test we now use internally is blunt, and it is the same test underneath this whole piece. Ask any team running agents to show you the record of what their agent tried on a specific day, what score it got, who or what gave that score, and what changed as a result. If that record does not exist as something you can actually query, the team is running automation no matter how capable their underlying model is. If it does exist, they are running a gym, and everything else in this piece is really just about making that gym bigger without breaking it.

Once we looked closely, most of the pieces of a learning system already existed inside the company. The market, the customers, and the day to day work were already there. What was missing was three specific things. A record of what an agent actually read, decided, and did on a given run, not just what it produced. A score attached to that run, with a reason. And a memory that the next run actually reads before it acts. Almost none of that existed anywhere durable. 

Closing those three gaps meant building four things, in this order. First, a shared model of the business so two runs of the same task are actually measuring the same thing, otherwise their scores cannot be compared to each other. Second, three different places for a loop to run, ranging from a desk where a person watches every step, to a scheduled cadence someone still reviews, to full unattended execution earned only after real evidence. Third, a way to score and remember every run, using a fast automated judge, a slower human verification, and the real outcome, so the system has something to condition the next attempt on. And fourth, a habit, protected on a calendar, of a team lead reading the actual trajectory behind a run rather than only the finished output, and writing down exactly one thing that should change next time. 

The full architecture behind each of those four, the shared model, the three surfaces and their promotion gates, the reward engine and its schemas, and the way we keep custody of the resulting history, is documented in full in the companion technical reference for anyone building the equivalent inside their own organization. What matters for this piece is what changed once all four were in place, and what we still cannot see.

The single largest change anywhere in this rollout was not technical. It was to one job description, and it is the one most likely to decide whether any of this works at all in a given organization. What a team lead actually does all day. 

Told to verify the loop, most leads, understandably, review the output, which is quality control, and which they were already doing. Verifying the loop is a different act with a different product. Reviewing output asks whether this particular thing is good enough to ship, looks only at the deliverable, and produces an approval or a correction that helps this one run and nothing after it. Verifying the loop asks whether a system that produced this would produce something better next time, looks at the actual trajectory, what was read, what was decided, where it branched, compares that to the judge’s score, and produces a score plus exactly one named amendment, written into the episode where the next run can actually find it. The first does not compound. The second is the entire point. 

Four other roles changed in ways that had to be written into actual job descriptions, not just implied, because a role that changes in practice but not on paper reverts the moment things get busy. An individual contributor’s leverage now comes from how well the loops they operate are specified, not from how much they personally produce by hand. A functional manager is closer to a portfolio manager than a headcount planner, deciding which loops get promoted, which get retired, and where the next shared model needs to be built. A domain owner now owns the shared model of their domain the way they used to own a process document, with a name attached to it when that model goes stale. And an executive’s actual job, in an operating review, comes down to one recurring question. What fraction of our tokens ran inside a verified loop last month, and is that fraction rising? Almost everything else in this piece is downstream of someone senior actually caring about that number. 

The obvious objection to any of this is that verification does not scale. A lead can realistically verify a handful of loops, not a hundred. That objection is correct, and it is answered in three ways rather than one. The judge absorbs the volume, since a lead is the calibrator of the automated score, not the reviewer of every run. Verification concentrates where the actual uncertainty is, the runs the judge was least confident about, the runs where the judge and the real outcome disagreed, and any loop amended in the last week, rather than being spread evenly across everything. And promotion itself reduces the load. A loop that has earned unattended execution needs a sampled audit, not per-run review, which is exactly the property that lets an estate grow without the lead’s calendar growing at the same rate. 

The estate now runs close to forty production loops across six functional domains, on all three surfaces, with loops firing across the full day and concentrated heavily in the hours before the working day so that output is already waiting when people arrive. A single shared memory store holds per-agent and per-team namespaces. Judging happens nightly on a model different from whichever one did the work. Human verification happens daily on active loops and on a sampled basis on mature ones. The whole estate is reviewed through a live dashboard rather than a stack of individual outputs, because a human cannot open forty outputs by hand every morning, and the one number we put in front of leadership is the simplest one, the share of monthly tokens that ran inside a loop that actually got scored.

None of that shape was designed in advance. It was extracted from a year of getting things wrong, and the lessons that came out of it are the actual proof of the thesis this piece opened with. 

Build the shared model of the business before the third loop, not after it:  

We built ours late, and several early loops were simply discarded rather than migrated, because retrofitting a shared vocabulary onto agents that had each invented their own private one cost more than starting over would have. 

Feedback that is not written down does not exist:  

For a long stretch, verification happened out loud, in review, and produced no measurable improvement, because none of it landed anywhere the next run could actually read. The improvement curve began the week feedback started being persisted, not the week it started being given. 

Naming one change is harder, and more valuable, than listing five:  

This single discipline produced more improvement per unit of effort than anything else we tried. A list of changes gets averaged by the next run into nothing. 

The judge has to be a different model from the actor, and this is not a formality: 

Same-model self-scoring produced scores that were high, stable, and completely uncorrelated with anything real. A different model produced lower, more useful scores immediately, and the disagreements were the informative part. 

A loop that never fails is not being pushed hard enough:  

Every loop we found scoring consistently high for months was either measuring something that had stopped mattering, or had quietly overfit its own rubric. Deliberately revising the objective is what surfaced this, nothing else did. 

Promoting too early is the expensive mistake: 

A loop pushed to unattended execution before its specification stabilized produced wrong action at volume that took weeks to unwind. A loop held back longer than strictly necessary cost nothing but time. 

Shared memory produces returns nobody explicitly designs for:  

The clearest compounding effect in our own estate came from one loop’s discovery becoming another, unrelated loop’s advantage, without anyone deliberately transferring it. We noticed it before we designed for it. 

A dashboard is part of the architecture, not a nicety: 

Daily verification is not survivable if a human has to open forty separate outputs by hand. A rendered, always-current view of the whole estate is what keeps the ritual alive. 

Some loops should stop using a model at all:  

A recurring audit of our own estate identified loops mature and stable enough to be rewritten as ordinary deterministic code. A loop that has stopped exercising real judgment is just paying inference cost for a function call. Just as important, name explicitly which loops must never be hardened this way, because they still exercise genuine judgment, and hardening them freezes their quality permanently. 

The estate is the proof, not the pitch:  

Every attempt we made to explain this architecture in the abstract landed poorly. Showing an actual estate, the cadences, the scores, three months of one loop’s improvement curve, landed immediately. 

A few things are worth stating plainly rather than letting them be discovered the hard way, because the honest version of this architecture is more useful than the polished one. 

It is not fine-tuning:  

Improvement here happens by conditioning on scored, retained history, not by updating any model’s weights. That is a deliberate choice, not a limitation we intend to quietly work around, it is what keeps the whole system portable across models. 

It does not eliminate judge blind spots:  

A judge built on a different model than the actor reduces shared error, but does not remove it. The only real defense is anchoring to a genuine outcome wherever one exists, and that signal is slow by nature. 

It does not remove human inconsistency:  

The single highest-quality signal in the whole system is still produced by a person, on whatever day they happen to be verifying, and a tired Friday produces a different verification than a sharp Tuesday. Rubrics and exemplar banks reduce this. Nothing we have built eliminates it. 

It does not see how an agent actually reasoned, yet:  

Everything scoring a run today is behavioral. It observes outputs, not the internal path that produced them. That gap is why we have not built the interpretability layer yet, and why we designed everything else so it can be added without a rewrite. 

It does not run itself:  

A shared model nobody owns goes stale silently. A rubric nobody revises gets overfit. A lead who delegates verification or batches it monthly turns the whole thing back into automation without anyone deciding that on purpose. 

None of this is finished. The layer that would let us see how an agent actually reasoned, rather than only what it produced, is designed for and not yet built. The estate is still roughly forty loops, not the much larger number a fully mature version of this would eventually run. What is finished is the part that was actually hard to get right the first time, a shared model everyone scores against the same way, real gates between how much a human is watching, a reward signal built from three genuinely different sources instead of one convenient one, and a habit, now protected on a calendar, of writing down exactly one thing that should change after every run that gets checked. 

For the complete architecture behind this, including every schema, every promotion gate, and the full field notes from the estate, See the companion technical reference here, Your Organization Should be a Recursive Learning Gym

If you are running agents inside your own organization and want to compare notes on where your own estate actually sits on this curve, or want to see what a scored episode looks like end to end, write to us at [email protected].