/ agam by algoqx · early access
Test the verifier before you train against it.
Models learn to pass the check instead of doing the task. Agam attacks your tests and SQL checkers with model-written shortcuts, then proves which ones your verifier wrongly accepts.
free to try; usage limits apply.
$ agam audit examples/python_largest
[ok] pass your verifier assert largest([2, 5, 9]) == 9
[x] fail held-out check assert largest([3, 1, 4]) == 4
>> exploit confirmed 12 of 12 candidates/ pipeline
- 01analyze
- 02attack
- 03prove
- 04score
- 05fix
An exploit counts only if your verifier passes it and the held-out checks fail it.
/ the problem
A high reward is a reason to look closer, not proof.
Reward functions, test suites, SQL checkers and LLM judges all have loopholes. RL loops and coding agents tend to find them before people do.
Most teams discover a bad reward after the GPU run is spent, or after a release regresses. Then someone reads rollouts by hand to work out why.
Teams build environments and eval harnesses. Few test the check that decides pass or fail.
def largest(xs):
return 9assert largest([2, 5, 9]) == 9your verifier: passassert largest([3, 1, 4]) == 4held-out check: failexplanatory. This is one of the shortcuts from the recorded run below.
- terminal-agent benchmarks
- 16% of 1,968 tasks across five benchmarks were hackable by frontier models (arXiv 2606.08960, as cited in the Agam pitch).
- swe-bench verified sample
- 28.5% of a 49-task sample had tests weak enough to pass an incorrect patch (arXiv 2606.16062, as cited in the Agam pitch).
/ how it works
Ingest. Attack. Prove. Fix.
01
/ ingest
Give Agam the task spec, your test suite or SQL checker, hand-written held-out checks and a known-correct reference solution. No model weights and no customer data are needed.
02
/ attack
Model agents write the shortcuts a trained model might take: hard-coded outputs, special-cased inputs, near-miss mutants and loopholes. Confirmed exploits seed the next round.
03
/ prove
A sandbox runs every shortcut against your verifier and against the held-out checks. If your verifier accepts it and the held-out checks reject it, the exploit is confirmed.
04
/ fix
You get the replayable exploits, a robustness score and hardened tests drafted for each hole, so the next audit starts from a stronger verifier.
/ what an audit returns
Exploits you can replay, and a score with its limits stated.
recorded run
0%
robustness. All 12 model-written candidates passed the one-assertion test and failed the held-out checks.
open-weight model, two attack rounds, one test case. The score is a heuristic, not a proof.
def largest(xs):
return 9def largest(xs):
return 9 if len(xs) == 3 else 0def largest(xs):
return len(xs) ** 2/ security
Built to run code that nobody trusts.
/ untrusted by default
Every shortcut is model-written code. It runs in a container with no network, all capabilities dropped, a read-only filesystem, memory, CPU and process limits, and a timeout.
/ sql stays read-only
SQL candidates run as a single read-only SELECT against a throwaway database, as an unprivileged role, with a statement timeout.
/ your work stays yours
Audits are tied to your Google account. The API is private behind the app, so another user cannot read your verifiers or results.
/ models on our hardware
As deployed today, audits use an open-weight model on AlgoQX-operated hardware. Submitted code is not sent to a third-party model API.
These are the controls in place today. They are not a certification.
/ delivered as
In the app, in CI and inside your agents.
web app
status: early access
Run audits, replay recorded runs and keep a history.
cli + ci gate
status: private prototype
Audit from a terminal or fail a build when the score drops below a threshold.
mcp + a2a
status: private prototype
Let a coding agent request an audit mid-task.
agent skill
status: private prototype
Package the audit so any agent can run it on a repo's tests.
/ pricing
Free to try. Pricing will come from pilots.
early access
status: available
Sign in, audit the bundled examples or your own Python verifier, and keep a history. Usage limits apply.
team plan
status: planned
CI gate, shared dashboard and usage-based audits. We are setting pricing with pilot teams.
pre-training audit
status: by engagement
A fixed-scope engagement before your team spends GPU hours, run by AlgoQX.
/ faq
Questions teams ask first.
Anything that decides whether a model's output counts as correct: a unit-test suite, a reward function, a SQL result checker or an LLM judge. If a model is trained or graded against it, a loophole in it becomes a skill the model learns.
A task spec, your verifier, held-out checks (tests a correct solution passes and shortcuts fail) and a known-correct reference solution. If the reference fails your checks, Agam refuses to audit them rather than guess.
It is 1 minus confirmed exploits divided by valid candidates. It is a heuristic: 100% means no shortcut was found in this run, not that none exists.
Python test suites that you submit yourself, and SQL checkers through the bundled examples. Submitting your own SQL checks, reward functions and LLM judges are next on the roadmap.
Submitted code runs in locked-down sandboxes on AlgoQX-operated hardware, and an audit is visible only to the account that created it. See the security section for the controls in place today.
Mutation testing asks whether a suite catches code bugs. Agam asks whether a model can pass the check without doing the task, with model-written shortcuts confirmed against held-out truth, across test suites and SQL checkers.