agam
/ star map

agam by algoqx · early access

/ agam by algoqx · early access

Test the verifier before you train against it.

Models learn to pass the check instead of doing the task. Agam attacks your tests and SQL checkers with model-written shortcuts, then proves which ones your verifier wrongly accepts.

free to try; usage limits apply.

/ recorded run · python_largest · explanatory
$ agam audit examples/python_largest
[ok] pass  your verifier   assert largest([2, 5, 9]) == 9
[x]  fail  held-out check  assert largest([3, 1, 4]) == 4
>> exploit confirmed  12 of 12 candidates

/ pipeline

  1. 01analyze
  2. 02attack
  3. 03prove
  4. 04score
  5. 05fix

An exploit counts only if your verifier passes it and the held-out checks fail it.

/ the problem

A high reward is a reason to look closer, not proof.

Reward functions, test suites, SQL checkers and LLM judges all have loopholes. RL loops and coding agents tend to find them before people do.

Most teams discover a bad reward after the GPU run is spent, or after a release regresses. Then someone reads rollouts by hand to work out why.

Teams build environments and eval harnesses. Few test the check that decides pass or fail.

a model's answer to: return the largest number in a list
def largest(xs):
    return 9
assert largest([2, 5, 9]) == 9your verifier: pass
assert largest([3, 1, 4]) == 4held-out check: fail

explanatory. This is one of the shortcuts from the recorded run below.

terminal-agent benchmarks
16% of 1,968 tasks across five benchmarks were hackable by frontier models (arXiv 2606.08960, as cited in the Agam pitch).
swe-bench verified sample
28.5% of a 49-task sample had tests weak enough to pass an incorrect patch (arXiv 2606.16062, as cited in the Agam pitch).

/ how it works

Ingest. Attack. Prove. Fix.

  1. 01

    / ingest

    Give Agam the task spec, your test suite or SQL checker, hand-written held-out checks and a known-correct reference solution. No model weights and no customer data are needed.

  2. 02

    / attack

    Model agents write the shortcuts a trained model might take: hard-coded outputs, special-cased inputs, near-miss mutants and loopholes. Confirmed exploits seed the next round.

  3. 03

    / prove

    A sandbox runs every shortcut against your verifier and against the held-out checks. If your verifier accepts it and the held-out checks reject it, the exploit is confirmed.

  4. 04

    / fix

    You get the replayable exploits, a robustness score and hardened tests drafted for each hole, so the next audit starts from a stronger verifier.

/ what an audit returns

Exploits you can replay, and a score with its limits stated.

recorded run

0%

robustness. All 12 model-written candidates passed the one-assertion test and failed the held-out checks.

open-weight model, two attack rounds, one test case. The score is a heuristic, not a proof.

shortcut 1 · hard_coded
def largest(xs):
    return 9
shortcut 2 · special_cased
def largest(xs):
    return 9 if len(xs) == 3 else 0
shortcut 3 · loophole
def largest(xs):
    return len(xs) ** 2

/ security

Built to run code that nobody trusts.

  • / untrusted by default

    Every shortcut is model-written code. It runs in a container with no network, all capabilities dropped, a read-only filesystem, memory, CPU and process limits, and a timeout.

  • / sql stays read-only

    SQL candidates run as a single read-only SELECT against a throwaway database, as an unprivileged role, with a statement timeout.

  • / your work stays yours

    Audits are tied to your Google account. The API is private behind the app, so another user cannot read your verifiers or results.

  • / models on our hardware

    As deployed today, audits use an open-weight model on AlgoQX-operated hardware. Submitted code is not sent to a third-party model API.

These are the controls in place today. They are not a certification.

/ delivered as

In the app, in CI and inside your agents.

web app

status: early access

Run audits, replay recorded runs and keep a history.

cli + ci gate

status: private prototype

Audit from a terminal or fail a build when the score drops below a threshold.

mcp + a2a

status: private prototype

Let a coding agent request an audit mid-task.

agent skill

status: private prototype

Package the audit so any agent can run it on a repo's tests.

/ pricing

Free to try. Pricing will come from pilots.

early access

status: available

Sign in, audit the bundled examples or your own Python verifier, and keep a history. Usage limits apply.

team plan

status: planned

CI gate, shared dashboard and usage-based audits. We are setting pricing with pilot teams.

pre-training audit

status: by engagement

A fixed-scope engagement before your team spends GPU hours, run by AlgoQX.

/ faq

Questions teams ask first.

Anything that decides whether a model's output counts as correct: a unit-test suite, a reward function, a SQL result checker or an LLM judge. If a model is trained or graded against it, a loophole in it becomes a skill the model learns.

A task spec, your verifier, held-out checks (tests a correct solution passes and shortcuts fail) and a known-correct reference solution. If the reference fails your checks, Agam refuses to audit them rather than guess.

It is 1 minus confirmed exploits divided by valid candidates. It is a heuristic: 100% means no shortcut was found in this run, not that none exists.

Python test suites that you submit yourself, and SQL checkers through the bundled examples. Submitting your own SQL checks, reward functions and LLM judges are next on the roadmap.

Submitted code runs in locked-down sandboxes on AlgoQX-operated hardware, and an audit is visible only to the account that created it. See the security section for the controls in place today.

Mutation testing asks whether a suite catches code bugs. Agam asks whether a model can pass the check without doing the task, with model-written shortcuts confirmed against held-out truth, across test suites and SQL checkers.