Spending the reward signal on an open model

Evaluation is the means and this is the end. gradientsmith uses its own verifiers as the reward to fine-tune an open-weights model, so the eval harness is not only a scoreboard, it is the training signal. The target is a Qwen coder model, and the reward is wired to resist the failure mode that motivated the whole project, reward hacking.

Why a Qwen model

The post-training target is Qwen2.5-Coder-32B-Instruct, a roughly 32 billion parameter open-weights model [2024]. Three reasons make it the right target. It is open weights, so you can actually fine-tune it and own the result, unlike a closed API model. It is a strong coder to begin with, which matters because the tasks here are code and structured output, and you want a base that is already in the neighborhood so the training signal sharpens it rather than teaching it from scratch. And roughly 32 billion parameters is a practical size, big enough to be genuinely capable and small enough to fine-tune with LoRA on a single high-memory GPU rather than a cluster.

The Qwen family also shows up on the leaderboard as baselines, served through OpenRouter alongside the frontier closed models, so the before number the post-trained model has to beat is measured on the exact same verifiers with no translation needed. There is more on what the model is made of on the models page.

Rejection-sampling SFT, learn from your own wins

The first method is the simplest thing that works. Sample many solutions per task from the base model at temperature. Keep only the ones the verifier passes. Deduplicate them. Then fine-tune the model on those passing solutions with LoRA. You are teaching the model to reproduce its own successes more reliably, and because success is defined by a deterministic verifier there is no ambiguity about which samples to keep. Before any training, the harness records the base model's pass@1 under the same prompt, which is the number the fine-tune has to improve on. This is the STaR and rejection-sampling recipe that larger open post-training pipelines use as their first stage [2022] [2023] [2024].

GRPO, learn from relative advantage

The second method is reinforcement learning. GRPO groups several rollouts for the same prompt and turns each rollout's reward into an advantage relative to its group mean, with no separate learned value function, because the group itself is the baseline. A rollout that beats its peers gets a positive advantage, one that lags gets a negative one, and a group where every rollout scores the same carries no signal and is skipped. A KL penalty keeps the model from drifting too far from the base. The reward is the verifier verdict, shaped by the fraction of public tests passed. This is the method introduced with DeepSeekMath and used to train DeepSeek-R1 [2024] [2025]. The exact advantage computation is on the methods page.

The tripwire that keeps the reward honest

Here is the invariant that makes the whole thing trustworthy. The training reward uses public tests only. Hidden tests are evaluated on every checkpoint but never enter the reward. If they did, they would stop being an independent measure. Instead they become a reward-hacking detector. The monitor records public reward and hidden pass rate at each checkpoint, and the moment public reward climbs while hidden pass rate falls, it fires. That divergence is the signature of a model learning to satisfy the visible reward without solving the task, and it is logged as a first-class training metric right next to loss. Reward hacking is a well-studied failure of optimization, from the early survey on concrete safety problems to work showing models that learn to tamper with their own reward [2016] [2024].

Held-out evaluation uses a frozen task split, deterministic in a seed so it never overlaps training, and reports pass@1 with a bootstrap confidence interval, an honest generalization number rather than a training-set echo. Training with a verifiable reward and a held-out check on the side is what recent open post-training work does at scale [2024].

The runs

Two runs below. One is a healthy rejection-sampling SFT that lifts held-out pass@1. The other is a GRPO run that starts to reward-hack, with public reward rising while hidden pass rate falls. The divergence is shaded red where it appears.