Our tasks improve models
on frontier coding problems.
Long-horizon RL environments built from licensed, never-public production repositories. We screened 110 environments, routed 70 to RL and 19 to targeted SFT, and moved Qwen3.8-27B from 1/74 to 4/74 on Terminal-Bench 3.0: a +4.1 percentage-point gain.
For frontier-lab post-training · RL & SFT data procurement
One paired 74-task pass@1 sweep. Same model architecture, serving configuration, Terminus-2 harness, benchmark parameters, and fixed task set; the trained adapter changed.
Trained on private tasks. Measured on a public frontier bench.
We post-trained Qwen3.8-27B using the RL- and SFT-qualified environments from a 110-environment screen, then ran one paired pass@1 sweep on Terminal-Bench 3.0. Base and trained arms used the same architecture, serving configuration, Terminus-2 harness, benchmark parameters, and fixed task set. The trained adapter was the intended variable. Our repository- and task-overlap checks found no overlap between the training environments and Terminal-Bench 3.0; the exact checks are documented in the downloadable report.
Context only, not a rank claim: GLM is averaged over five trials per task and uses a different harness; Qwen is one paired sweep.
Each environment is positioned relative to the current policy. Reward variance routes a task to RL; a flat policy with demonstration margin routes to targeted SFT; saturated or non-teachable tasks are dropped.
The full run is re-runnable end to end: trajectories, verifiers, and adapters all ship. A two-page report with every number is yours for the asking.
Frontier models score them. Few score well.
Five released sample tasks, each swept with five frontier model configurations: four at three trials, one single-trial spot check. 3 of the 62 recorded trials fully solved a task in the August 2026 sweep, and model rank order changes from task to task. Each sample is a long-horizon feature an engineer shipped in a never-public production codebase (median 25 to 75 agent steps per trial, max 121), graded 0 to 1 over 13 to 28 ordered behavioral checks.
Your reviewers can run all of it today: send GitHub usernames and they get the sample repo with trajectories, verifiers, and QC evidence included.
A real task, end to end.
Every task is a feature or fix an engineer actually shipped, respecified as a graded environment. The verifier returns a continuous reward on [0, 1] over ordered behavioral checks, so hard tasks still produce gradient where binary pass/fail goes all-zero. The verifier is written and probed independently of the shipped implementation; the oracle is the engineer's real shipped diff and is never exposed to the agent.
$ cat instruction.md ## Task: Redis-backed payment-intent coordination Coordinate duplicate booking requests across service instances. Requests are equivalent only when vehicle, interval, and user match. For concurrent equivalent requests, perform at most one provider operation. If coordination ownership is lost mid-flight, do not report success. … 28 graded checks · live Redis beside the service · no mocks $ harbor run redis-payment-lock … 31 agent steps · 0.9M cumulative prompt tokens across steps (task medians) checks passed 19/28 (weighted) reward = 0.6786 best trial, August 2026 sweep $ cat trajectories/sweep.txt 3-trial avgs: opus-4.8 0.68 · gpt-5.6 0.45 · grok-4.5 0.44 · muse 0.10 spot check (1 trial): 0.00 no model fully solves it
task/ ├── instruction.md the ticket the agent gets, no leakage ├── environment/ Dockerfile + repo at the before-commit ├── tests/ graded checks, independent of the impl ├── solution/ real shipped diff (oracle) · hidden from agent ├── trajectories/ per model, per trial: steps, logs, reward ├── qc-report.md probe evidence, re-runnable └── task.toml harbor config, pins the commit range
Catalog-wide: oracle 1.0 · median empty-solve floor 0.046 · 99.3% of mutation probes behave as predicted. Evidence ships in every bundle.
1,000+ production codebases. None in anyone's pretraining.
Every environment is mined from licensed, never-public production code, drawn from a corpus of more than a thousand repositories under direct license and growing daily: over ten billion tokens of private code and commit history across 25+ languages and every engineering domain, from full-stack web and native mobile to infra, data engineering, ML, and security, with history back to 2008. The repositories were never intentionally publicly released, and our commit-hash checks find no task commit on public GitHub; a file-tree hash ships with every bundle so you can confirm overlap against your own training set.
Verified through commit-hash and file-tree overlap checks against public GitHub as of 11 Aug 2026. This establishes public absence under our checks, not universal proof about every model's training corpus. Token count covers source snapshots plus git history of licensed repos; historical duplicates are not deduplicated.
Ruby
C++
Vue
Terraform
C
Dart
Go
Elixir
Rust
Svelte
SolidityShare of licensed repos containing each language; repos hold several languages, so shares sum past 100%.
Real iOS and Android apps, full-stack web and e-commerce platforms, backends, data pipelines, ML and AI-research code, games, and on-chain systems. The hard-to-source distribution, not just another pile of JavaScript.
Licensed directly from the teams that built it, through the pre.dev network. No brokers, no resold scrape. The candidate task funnel is measured: 201,000 candidate code changes in test-bearing repos, 30,000 eligible candidates after source-level screening, then environment construction and verifier QC before anything ships. Supply scales with compute.
Questions post-training teams ask.
What exactly is one of your RL tasks?
A runnable engineering task lifted from a real private production codebase: repo snapshot, task spec, and a verifier that scores each attempt with continuous reward on [0, 1]. You get the container and the reward entrypoint, and it plugs into your own RL or eval infrastructure.
Does training on these actually improve a model?
We measured a controlled improvement. In one paired Terminal-Bench 3.0 sweep, Qwen3.8-27B moved from 1/74 to 4/74 at pass@1 (+4.1pp) after post-training on pre.dev environments. Base and trained arms used the same architecture, serving configuration, Terminus-2 harness, benchmark parameters, and fixed task set. The full run ships with trajectories, verifiers, adapters, and configuration for review.
Where do the codebases come from?
From private production repos: codebases licensed directly from the teams that built them, through the pre.dev network, plus internal production systems. Every task is cut from private git history. The repositories were never intentionally publicly released, and our commit-hash and file-tree checks find no task commit or tree on public GitHub; that establishes public absence under our checks, and a file-tree hash ships with every bundle so you can test overlap against your own corpus.
How do we evaluate before buying?
Book an intro call below, or email reviewer GitHub usernames to arjun@pre.dev. We deliver a runnable sample set the same day: repo snapshots, task specs, verifiers, QC bundles, and the capability sweep across frontier models, so your team can run everything end to end on your own infra before any commitment.