RL tasks
Practice problems with an automatic marker. The model attempts the work, the marker scores it, and a pass is the training signal.
Deep native tech lab
Expert tasks, built and scored here.
Ossa Labs builds expert-level reinforcement-learning tasks, coding tasks, and the benchmarks that measure them. The work is made here, for frontier models.
Inside a coding task
| Codebase | The code as it stood before the change |
|---|---|
| Instruction | A ticket the model can read |
| Verifier | A hidden marker, pass or fail |
| Reference | The change that actually shipped |
Coding tasks
A real codebase, frozen before a feature or a fix, and an instruction written the way a ticket would be written.
Benchmarks
Difficulty is a measured pass rate on frontier models. The evidence for the number stays with the task.
A task
Hard, learnable, and graded the same way every time.
A task is useful when a frontier model fails it often enough to learn from, a careful engineer could pass from the instruction, and the same code gets the same verdict on every run.
Measured
Difficulty is a pass rate on frontier models, with an interval. It is a count of attempts, kept with the evidence.
Stated in the work
Every rule the marker checks is in the instruction, or visible in the code and data the agent can read.
Same verdict
The verifier is hidden from the agent. Wrong solutions fail. A different correct design can still pass.
Where tasks come from
Private code first. One standard for every source.
The work we deliver today is feature work in private production code: the codebase as a team left it, and the change that team later shipped. Personal data is scrubbed, and the code is used under a licence from its owner.
Open-source code and synthetic environments go through the same checks. A task is named in a track only after it has been measured into that track.
Tracks
| Feature work | Build a shipped feature in an existing codebase |
|---|---|
| Bug fix | Find and fix a real regression from a short report |
| Agent safety | Keep a rule that lives in the code, such as who can see what |
| ML engineering | Train, evaluate, or fix a pipeline |
| Long-horizon | Several dependent changes, or an hour or more of agent time |
Measurement
How a task is measured
A task ships with proof. The shipped change passes, the untouched starting code fails, and frontier-model runs are kept with the files behind every number.
The change that shipped scores a pass. That proves the task can be solved.
The untouched codebase scores a fail. That proves the marker is checking the new work.
Frontier models attempt the task. Each attempt is one full try, from start to finish.
The pass rate, the interval, and the attempt record stay with the task.
Difficulty, by pass ratepass@1
0%5%33%70%100%
Little signalUnder 5%
Hard5–33%
Medium34–70%
EasyAbove 70%