Forge better models against tests they can't argue with.
gradientsmith is a platform for measuring what language models can actually do — and then using those measurements to fine-tune an open-weights model that does more. Every score comes from a deterministic verifier: code is executed against unit tests in a sandbox; structured output is checked against a schema. Never an LLM judge. That one property is what makes the rest — an adversarial test-mining loop, cost-aware routing, and reward-hacking-aware training — trustworthy.
The problem it solves
Most "which model is best" comparisons quietly rot, for one reason: the scoring isn't reproducible. If a benchmark uses a language model to grade answers, the grade drifts with the grader, and a model can learn to sound right rather than be right. The moment you try to post-train against a fuzzy metric, that gap becomes a loophole — the model optimizes the metric instead of the task. That failure has a name: reward hacking.
gradientsmith starts from one non-negotiable constraint: every primary metric comes from a verifier that is deterministic and sandboxed. Same solution in, same verdict out, every time. Everything else is built on top of that.
evalkit; that's the wrench, gradientsmith is the workshop.The pipeline, end to end
Five stages, one feedback loop. Tasks carry hidden tests the solver never sees. Verifiers score deterministically. An adversary mines new failing cases and banks them. Campaigns run thousands of rollouts across many models and route each task to the cheapest one that's good enough. And the same verifier verdict becomes the reward that fine-tunes an open model.
What it produces
A campaign yields a cost/quality picture of every model at once. These figures are from the current snapshot (frontier-calibration, 6 models):