How the harness actually works

This page walks through the parts that matter, with the real code from the repository. The through-line is one idea. A deterministic verifier decides whether an answer is correct, and every other piece is built so that decision cannot be gamed. The snippets below are lightly trimmed from the actual source in the evalkit and training packages.

The one rule, a verifier decides

A large amount of evaluation work uses one model to grade another model, often called LLM-as-a-judge. It is cheap to set up and it correlates with human preference on open-ended writing, but it is not reliable for code and it can be steered. Judges prefer the first answer they see, prefer longer answers, and prefer answers that sound like their own writing [2023] [2024]. For a task that has a right answer, none of that is acceptable. gradientsmith never uses a model as the grader. It runs the code and checks the result.

The reward is the public tests and nothing else

The reward function is small on purpose. It builds the verifier for the task, runs the candidate against the public tests, and returns either the fraction of checks passed or a strict pass-or-fail number. There is no model in this path and no hand-tuned scoring.

async def compute_reward(task, completion, *, binary=False) -> float:
    # reward in [0, 1], computed from PUBLIC tests only
    verifier = build_verifier(task)
    verdict = await verifier.verify(completion, task.public_tests)
    if binary:
        return 1.0 if verdict.passed else 0.0
    return verdict.pass_fraction

async def hidden_pass_rate(task, completion) -> float:
    # fraction of HIDDEN tests passed, used ONLY by the monitor,
    # never as a training reward
    verifier = build_verifier(task)
    verdict = await verifier.verify(completion, task.hidden_tests)
    return verdict.pass_fraction

The two functions look alike and that is the point. The hidden pass rate is measured with the exact same verifier, but its number is only ever read by the monitor described at the bottom of this page. Keeping the two paths in separate functions is what keeps the boundary honest.

Running untrusted code without trusting it

Model-written code has to run somewhere, and it will sometimes loop forever, allocate all the memory it can, or try to open files. The verifier runs every candidate in a fresh subprocess with hard resource limits set in the child before the program starts. The process runs in Python isolated mode, so it ignores the user environment and the current directory, and it gets its own process group so a timeout can kill the whole tree at once.

def set_limits():  # runs in the child, before exec
    resource.setrlimit(resource.RLIMIT_CPU, (cpu_s, cpu_s + 1))
    resource.setrlimit(resource.RLIMIT_AS, (memory_bytes, memory_bytes))
    resource.setrlimit(resource.RLIMIT_FSIZE, (10_485_760, 10_485_760))
    resource.setrlimit(resource.RLIMIT_NOFILE, (64, 64))
    resource.setrlimit(resource.RLIMIT_CORE, (0, 0))

proc = await asyncio.create_subprocess_exec(
    sys.executable, "-I", str(script),          # -I is isolated mode
    cwd=tmp, env={"PATH": safe_path, "HOME": tmp},
    stdin=asyncio.subprocess.DEVNULL,
    start_new_session=True,                       # own process group
    preexec_fn=set_limits,
)
# on timeout the whole group is killed, not just the parent
os.killpg(proc.pid, signal.SIGKILL)

This is a resource and fault boundary, which is the right level for solutions written by a cooperative model. It is not a defense against code written to break out, and the source says so plainly. If the tasks ever ran genuinely hostile code, the next step would be a container or a microVM. Public work on sandboxing model-generated code, such as SandboxEval, studies exactly this gap between a resource limit and a security boundary [2025].

A verdict that printing cannot corrupt

There is a subtle failure mode in execution-based grading. If the harness reads the result from standard output, a solution that prints a lot, or prints something that looks like the expected answer, can confuse the reader. The fix here is to append a small harness after the solution that writes its verdict to a file, so the verdict survives no matter what the solver prints. Serialization uses a setting that makes NaN and infinity fail rather than round-trip into a bogus value.

# appended after the solver's own code, then the whole thing runs once
results = []
for i, t in enumerate(tests):
    r = {"name": f"test_{i}", "passed": False, "expected": t["expected"]}
    try:
        out = ENTRY(*t["args"])
        # allow_nan=False makes NaN and inf raise here instead of
        # silently becoming a value that later compares unequal
        norm = json.loads(json.dumps(out, allow_nan=False))
        r["passed"] = norm == t["expected"]
    except Exception as e:
        r["passed"] = t.get("raises") == type(e).__name__
    results.append(r)

# the verdict goes to a file, a side channel the solver cannot flood
with open("__ek_result.json", "w") as f:
    json.dump(results, f, allow_nan=False)

This is the kind of detail that separates a demo from something you can trust to score thousands of rollouts. Both the NaN guard and the result-file channel exist because earlier versions of the harness were fooled by exactly those cases. The benchmarks that popularized execution-based scoring, HumanEval and MBPP, use the same core move of running the code against held tests rather than reading a self-report [2021] [2021].

Mining counterexamples the reference can defend

A solution that passes the tests you showed it has only proven that it passes the tests you showed it. The adversary looks for an input where the solution is wrong. It proposes inputs, and each proposal is checked deterministically before anything is banked. The rule for keeping a proposal is strict. The reference solution has to produce a well-defined answer on that input, and the candidate solution has to disagree with it.

async def mine_counterexample(task, solver_code, proposal):
    # 1. the reference solution defines ground truth on this input
    reference = await run_entry_point(
        task.reference_solution, task.entry_point, proposal.args)
    if not reference.ok:
        return None                 # input is undefined for a correct solver
    if not _is_finite(reference.value):
        return None                 # NaN or inf can never match by equality

    # 2. the candidate must actually disagree with the reference
    solver = await run_entry_point(
        solver_code, task.entry_point, proposal.args)
    if solver.ok and solver.value == reference.value:
        return None                 # candidate is right here, not a counterexample

    # 3. bank a hidden test whose expected value came from the reference,
    #    never from the model
    return MinedCounterexample(
        test=CodeTest(args=proposal.args, expected=reference.value),
        category=("timeout" if solver.timed_out
                  else "crash" if not solver.ok else proposal.category),
        solver_actual=solver.value, solver_error=solver.error)

The expected value always comes from the reference solution, so the adversary can never bank a test that the reference itself would fail. That single constraint is what makes the mined tests trustworthy. This is a small, self-hosted version of the counterexample-guided idea that shows up in program synthesis and in step-checked math reasoning, where a candidate is kept only when a checker confirms it [2023].

The solve, mine, steer, retry loop

The pieces above combine into a loop for a single task. The solver writes a first solution from the public prompt. The adversary tries to break it. If nothing reproduces, the solution has survived and the loop stops. If counterexamples are found, they are banked, folded into a steering prompt that shows the solver exactly which inputs it got wrong, and the solver tries again, up to a round limit. The loop records a few numbers per task that are more honest than a single pass rate.

fix_survival is the one to watch. A model can pass a specific counterexample by special-casing that exact input while quietly breaking the general case. Measuring the fix against the original hidden tests, rather than against the counterexample that triggered it, catches that. There is also a deliberate choice at the last round. The loop does not report a retry solution that never faced the adversary, so the survived flag stays truthful.

for round_index in range(self.max_rounds):
    proposal_count, mined = await _mine_round(task, adversary, solver_code, verifier)
    rounds.append(RoundRecord(round_index=round_index, mined=len(mined), ...))
    if not mined:
        survived = True                 # adversary found nothing, stop
        rounds_to_fix = round_index
        break
    if self.bank is not None:
        await self.bank.append(task.task_id, mined, mined_from_model=...)
    if round_index == self.max_rounds - 1:
        break                           # do not emit an untested retry
    solver_code = await self.solve(task, build_retry_prompt(task, solver_code, mined))

Rejection-sampling SFT

The first way to improve a model here does not use reinforcement learning at all. Sample several solutions per task, keep only the ones the verifier accepts, remove duplicates, and fine-tune on the survivors. The model teaches itself from its own correct work. The dataset builder also records the first-sample pass rate, which is the baseline the fine-tune has to beat. This is the STaR and rejection-sampling recipe used inside larger post-training pipelines [2022] [2023] [2024].

for i, completion in enumerate(completions):
    reward = await compute_reward(task, completion, binary=binary_reward)
    if i == 0:
        first_reward = reward            # tracks the baseline pass@1
    if reward < keep_threshold:
        continue                         # reject: keep only passing samples
    norm = _normalize(task, completion)  # canonical form for dedup
    if norm is None or norm in seen:
        continue
    seen.add(norm)
    examples.append(RSExample(task_id=task.task_id, prompt=prompt,
                              completion=completion, reward=reward))

GRPO, the group is the baseline

The second way is reinforcement learning with a verifiable reward, using group relative policy optimization. Ordinary policy-gradient methods need a second network, a value function, to estimate how good a state is before you can tell whether an action beat expectations. GRPO removes that network. It samples a group of answers for the same prompt and scores each answer relative to its own group. An answer that beat the group average gets a positive advantage and an answer that fell below it gets a negative one. This is the method introduced with DeepSeekMath and used to train DeepSeek-R1 [2024] [2025].

The advantage for one answer is advantage = (reward - group_mean) / (group_std + eps). The whole computation is a few lines, and it is kept pure so it can be tested on its own.

def group_advantages(rewards_by_prompt, *, normalize_std=True):
    out = []
    for prompt_id, group in rewards_by_prompt.items():
        rewards = [r for _, r in group]
        mean = statistics.fmean(rewards)
        std = statistics.pstdev(rewards) if len(rewards) > 1 else 0.0
        for completion, reward in group:
            centered = reward - mean
            advantage = centered / (std + 1e-6) if normalize_std else centered
            out.append(GroupRollout(prompt_id=prompt_id, completion=completion,
                                    reward=reward, advantage=advantage))
    return out

A group where every answer earned the same reward carries no signal, because no answer was better than its peers, so it yields all-zero advantages and the trainer skips it rather than dividing by a near-zero spread. Small correctness details like that are the difference between a stable run and one that quietly learns nothing.

Watching for reward hacking

Reward hacking is when a model raises its measured reward without getting better at the task. It is a well-documented failure of optimization, from the early survey on concrete problems in AI safety to Anthropic work showing models that learn to tamper with their own reward [2016] [2024]. The defense here follows directly from the public-only reward. Because hidden tests are never trained on, they are a clean measuring stick. If the public reward rises while the hidden pass rate falls between two checkpoints, the model is overfitting the visible signal, and the monitor raises it as a divergence.

def record(self, step, public_reward, hidden_pass_rate):
    prev = self._history[-1]
    public_delta = public_reward - prev.public_reward
    hidden_delta = hidden_pass_rate - prev.hidden_pass_rate
    # public reward up while hidden pass rate down is the signature
    # of a model gaming the visible signal
    if public_delta > self.eps and hidden_delta < -self.eps:
        return Divergence(from_step=prev.step, to_step=step,
                          public_delta=public_delta, hidden_delta=hidden_delta)
    self._history.append(...)

The monitor is a first-class training metric here, not an afterthought. Training with a verifiable reward is what recent open post-training work does at scale, and keeping a held-out check on the side is how you know the reward is still measuring the thing you care about [2024]. Read the models page for what these loops are driving, or the references for every source above.