RL Tasks: Where the reward went

Field notes: agentic coding evals

Two tasks from a catalog of real commits rebuilt as graded RL environments. In both, the models did recognizable, competent work and scored close to nothing — and in both, the reason turned out to have almost nothing to do with whether they understood the problem.

The setup is the same for every task in the catalog: take a real commit from a working codebase, rewind the repo to the parent, hand the agent a prose description of what to build, and grade the result against a test suite the agent never sees. The original diff is kept as the oracle. Reward is continuous — the weighted fraction of checks passed.

I went looking for tasks where the scores were strange, and read the transcripts. These two were the most instructive. Both are ordinary integration work of the kind that fills a sprint: wire up an SMS provider, wire up a transactional email provider. Neither is algorithmically hard. Both produced scores that look like total incompetence and aren't.

Task one: Send an SMS through Twilio

A Java monorepo — Spring, Maven, nine modules. The user service already delivers auth codes over email and stores them in Redis for later verification; the SMS branch falls through to a no-op. The instruction asks for a Twilio-backed SMS component: configurable credentials, sender number, message template; render the code into the template; return true on success and false on failure without throwing; persist to Redis only when delivery succeeds; leave the email path alone.

Nine models ran it, most for three trials. The scores clustered strangely:

That 0.6111 shows up seven times across four different model families. It is 11 of 18 tests. 0.6667 is 12 of 18. Models built by different labs, run months apart, converge on the same two scores — which usually means they are all failing the same specific thing.

They are. Every failing test at both plateaus carries the same assertion message:

The difference between the two plateaus is exactly one test. Everything else is identical. So: why does an accepted delivery not get reported as accepted?

The hidden variable

The test installs a fake HTTP transport into the Twilio SDK's shared client and has it answer 201 with a valid message SID. It is a careful piece of work — it doesn't guess field names, it calibrates roles by watching a probe send, and it deliberately survives the component re-initializing Twilio mid-flight. It is implementation-agnostic in every way its authors thought to make it.

Except one. Sort every run by the Twilio SDK version the model wrote into the POM:

The commit this task was derived from is from 2019. The test double was written against the 7.x client architecture. A model that reaches for a current Twilio SDK — which is what you would want a competent engineer to do, and what several of them did — gets a client the fake transport can't intercept, a delivery that never returns accepted, and seven failing tests it has no way to see.

The benchmark is not measuring whether the model can integrate Twilio. It is measuring whether the model guessed the same SDK version as a commit from 2019.

Nobody could check their work

The second thing in these transcripts is worse, and it explains why nobody caught the version problem. Neither the model that scored 1.0 nor the models that scored 0.6111 were able to run the test suite.

The agent's container is built FROM node:22-slim and installs Debian's default-jdk — a modern JDK. The repository is a 2019 Spring project pinned to Lombok 1.16.22, which does not work on modern JDKs. The verifier, separately, installs and uses JDK 17 before grading. The agent and the grader do not run the same toolchain.

gpt-5.6-sol spent roughly half its session on this. It diagnosed the problem precisely — "Lombok 1.16.22 references BaseFileObject, removed from the installed modern JDK" — tried -Dmaven.compiler.proc=none, tried opening JDK modules, and eventually gave up on Maven and compiled the sources directly with javac -proc:none against a manually built classpath just to prove the code compiled. It also lost two full steps to a mangled multi-line command that left the shell at a > continuation prompt, and two more to a pager stuck at (END). Its final validation was mvn dependency:tree — confirming the Twilio jar resolved. It never saw a test result. It scored 1.0.

grok-4.5 took the other route. In 15 steps and 80 seconds, for 13 cents, it wrote the provider, read its own files back to "confirm correctness", and concluded:

Every clause of that is true. It scored 0.6111. The self-assessment was accurate about the code and silent about the only thing that mattered, which it had no way to observe.

Task two: Send an invitation email through SendGrid

A TypeScript service. Invitations need a working email path: send through the configured provider to the invitee, identify the inviter by name, build a join link from a configured base URL, treat only an HTTP 202 as success, fail cleanly when mail credentials are missing, refuse invitations with no contact information, and keep SMS as an explicit not-yet-implemented outcome.

Five models, seventeen trials. Here is the complete result table:

A wipeout like that normally means the task is broken. It isn't: the oracle patch scores a clean 1.0, passing all 8 core checks and all 13 behavioral ones. So the environment works and the target is reachable. Something else is going on.

The floor

deepseek-v4-flash's best run — the 0.075 — passed the setup check, passed both build checks, passed 6 of 8 core checks, and passed 8 of 13 behavioral tests. Seventeen of twenty-four checks green, including two thirds of the behavioral suite. That is a substantially working feature. It scored 0.075.

The reason is thirteen lines in the grading script:

Miss a single core check and everything else you did is discarded. Your score becomes your core fraction times one tenth. The arithmetic matches every observed value exactly:

The second row is the one to sit with. That is the oracle's own code with a single behavior removed as a QC mutation — remove SendGrid's 202-acknowledgement check and nothing else — and it falls from 1.0000 to 0.0875. The catalog's health check passes this task because it only asks whether a dropped behavior lowers the score. It lowers it by 91 percentage points.

A continuous reward that is 1.0 for exactly right and ~0.08 for anything else is not a continuous reward. It's a pass/fail gate with decorative digits.

The gate is a deliberate anti-reward-hacking measure, and as that, it works: you can't farm points off structural greps without doing the task. But for RL training data the side effect is severe. A model that gets two-thirds of the behavior right receives almost the same signal as one that does nothing — 0.075 against a no-op floor of 0.0. There is nearly no gradient across the entire range where learning would happen.

And then the runner crashed

The other four models scored 0.0000 rather than a small fraction, and for a different reason. Their core checks didn't fail — they aborted:

Every one of the eight core checks died the same way, a native panic inside the test runner. The harness — correctly, and by explicit design — charges a crashed test as a failure rather than a pass. Core lands at 0/8, the floor multiplies it to zero, and four models across four labs receive an identical score of 0.0000 for four different implementations. The oracle doesn't trigger it and deepseek's run didn't either; what in the other four implementations sets it off isn't recoverable from the shipped artifacts.

What I'd take from this

Neither task is badly made. The SMS test double is more carefully written than most production test code I've read — it calibrates roles by observation specifically so it won't punish a differently-named field. The grading harness charges crashed tests as failures, detects TAP tampering, and refuses to let a compile failure quietly shrink the denominator. Somebody thought hard about both.

The failures are all one level below where the care was spent.

• Unpinned dependencies are hidden answers. The SMS task has a correct version and never says so. Any environment reconstructed from an old commit carries these, and they are invisible to the task author, who had the right version on disk the whole time.
• If the agent can't run the tests, you're grading its guess. Both toolchains were fine on their own; they just weren't the same toolchain. The single highest-value check on an environment like this is whether the agent can execute the thing that will judge it.
• A gate makes the reward binary — know if you wanted that. Clamping partial credit stops reward hacking and simultaneously deletes the gradient. For eval reporting that's defensible; for RL training data it may be self-defeating.
• Health checks that only consult the oracle can't see this. Every failure here sits in the gap between "the grader responds correctly to the reference solution" and "a competent attempt gets a sensible score." Running one real model and asking whether the score is plausible would have caught all of it.

The recurring shape is that the models mostly did the work. What they couldn't do was see the environment they were being judged in — and in both tasks, that environment, not the code, is what set the score.