jrice-ai/cageReproduction harness, patches and analysis tools for auditing a learned cyber-defense benchmark: captures agent traces, runs red-agent ablations, and recomputes every paper figure from released data.
This repository is the harness, the patches, the analysis tools and the ledger behind the paper What a CAGE 4 Score Measures: An Audit of a Learned Cyber-Defense Benchmark, Its Reference Defenders, and the Tables Built on Them (James M. Rice and advisors, Rochester Institute of Technology, 2026). The arXiv identifier is added here in the first commit after posting. The tag arxiv-v1 is the state every number in the posted paper was computed from.
The traces and checkpoints the paper's numbers are computed from are deposited separately, at https://doi.org/10.5281/zenodo.23681828, in four archives.
| Archive | Contents | Unpack into | sha256 |
|---|---|---|---|
traces_arms_arxiv-v1.tar.zst | the step records of every evaluation arm of the main audit, the 123 that Appendix B of the paper lists and the diagnostic arms ARMS.md names, one directory per arm, one JSON-lines file per episode | data/traces/ | db2c3d17684180e5d4ca7fb9a7f1f739b5414e62da0e8c4b9b792cd31568f456 |
traces_pilot_arxiv-v1.tar.zst | the step records of the busy-check pilot and its continuation (arms named kbf_*) | data/traces/ | ece72ca24d17bf688144b2d0276ae9f0209aa8a50f95ecdd0aab9a6f8a9c0fc0 |
checkpoints_pilot_arxiv-v1.tar.zst | the kept checkpoints and resume state of the pilot and continuation training runs | repos/cage-4-submission/checkpoints/ | dc2c8219e12a0c2931263be0003c8858039931b04b019e1bfa60c0ba6856c07e |
checkpoints_continuation_arxiv-v1.tar.zst | the checkpoints of the Wrecker continuation runs of Section 5 | repos/cage-4-submission/checkpoints/ | in SHA256SUMS |
run_logs_arxiv-v1.tar.zst | the training and evaluation logs of the pilot and continuation runs from both machines, with pilot_runs.log, which records each run's host, thread count, worker count, code digest and image | data/ | in SHA256SUMS |
SHA256SUMS in the deposit lists all four. Check an archive with sha256sum -c SHA256SUMS before unpacking.
| Path | What it is |
|---|---|
bin/run | runs a command in one of the two images with this directory mounted at /work, passing the KEEP_* and HMARL_* switches through. It expects the clone at ~/cage. |
build/ | the two Dockerfiles (Dockerfile.keep, Dockerfile.marl) and the patches applied to the upstream repositories before the images are built |
scripts/capture.py | runs KEEP's released policy on paired environment seeds and writes the step record with the agents' raw observations and the simulator's true state |
scripts/capture_hmarl.py | the same for H-MARL Expert, through the authors' evaluation path (--compat) or through the supervisory harness |
scripts/oracle.py, scripts/oracle_rules.py | the supervisory harness on KEEP, the rules O1 to O15 and the rule defenders, with the delivery switches (--wait-busy, --queue, --min-gap) |
red_agents/ | the adversaries the paper runs beyond the stock FiniteStateRedAgent. Wrecker and its Pn and Qn ladders are the paper's own (Section 2), and bin/run copies every file here into the image's SimpleAgents directory at launch |
tools_*.py | the analysis tools. tools_paired.py (paired difference, bootstrap interval, wins, sign test), tools_charge.py and tools_busy_charge.py (the Restore charge and the rest), tools_dropped.py (executed against asserted interventions), tools_cont.py, tools_ceiling.py, tools_fraction_ci.py, tools_latch_sum.py, tools_drawspread.py, tools_lagcorr.py, tools_busy_restores.py, tools_pilot_decisions.py and the rest. Each prints what it computes from a trace directory under data/traces/ |
tools_patch_*.py | the patch stack on the released code. tools_patch_keep_busy.py, tools_patch_keep_seedmode.py and tools_patch_keep_readall.py, in that order, give KEEP's trainer the switches of Section 6. tools_patch_hmarl_busy.py gives H-MARL's wrapper the HMARL_BUSY=none switch. tools_patch_oracle_truth.py adds --log-truth to the harness. Each is idempotent and refuses to run on code it does not recognize |
run_keep_pilot.sh, eval_keep_pilot.sh, pilot_status.sh, test_keep_*.sh | the pilot and continuation of Section 6, their evaluation and their tests |
run_busy_reds.sh | the released KEEP policy with its busy check on and off at the stock adversary and nine others (Section 6.1) |
queue_*.sh, read_*.sh, run_*.sh, rerun_F.sh, gate_O15.sh, pilot_O15.sh, svy_O15.sh | the campaign launchers and readers. Every arm in the paper was launched by one of these, so they are the record of each arm's exact invocation |
ARMS.md | every arm, by name, with its command, its mean and the file that holds it |
RESULTS.md | the dated log of the work, newest first |
papers/ | the artifact copy of the paper's analysis layer, without the paper's text. papers/paper1/ledger.csv is the ledger, one row per number printed in the paper, with the command that recomputes it. papers/paper1/fig/ holds the figure and table scripts. papers/paper1/check_identities.py tests the arithmetic identities every build must satisfy. The recompute_*.py and means.py scripts at papers/ are the criterion, variance and share computations the ledger calls |
Everything under data/ and repos/ is ignored by git. The traces go under data/traces/, the upstream repositories and the checkpoints under repos/.
The paper's Appendix A names three levels. They need increasingly more of the environment.
Clone, unpack the two trace archives into data/traces/, and run the recompute from the repository root. It runs every distinct command in the ledger from this directory and checks that each row's stored value appears in that command's output.
git clone https://github.com/jrice-ai/cage ~/cage && cd ~/cage && git checkout arxiv-v1
mkdir -p data/traces && tar -I zstd -xf traces_arms_arxiv-v1.tar.zst -C data/traces && tar -I zstd -xf traces_pilot_arxiv-v1.tar.zst -C data/traces
./recompute.sh
python3 papers/paper1/check_identities.py papers/paper1/ledger.csv
The ledger's commands name the analysis layer by its path on the authors' machine, ~/papers, so recompute.sh runs the check with HOME pointed at a scratch directory that links papers/ there. Nothing is written outside the repository. On a laptop the full recompute takes tens of minutes, most of it the pilot arms. Ten rows, the hyperparameters read from the released checkpoints (keys keep* and trident*, which run through bin/run), need the level-2 image and report as not found without it. check_identities.py reports the arithmetic identities (cost equals charge plus rest for every split, the four-draw and four-seed intervals against their means and sds, the H-MARL mask-off pair's zero rest). The figure and table scripts under papers/paper1/fig/ regenerate the paper's figures and tables from the same traces with CAGE=~/cage python3 papers/paper1/fig/fig_curves.py and likewise for fig_drop.py, fig_noise.py, make_tables.py, make_busyreds.py and make_arms.py.
This needs the two images, the upstream repositories at their pinned revisions, and the released checkpoints.
Clone the three upstream repositories into repos/ at the revisions the paper was computed from, which are the revisions Table 12 of the paper prints.
| Repository | Revision |
|---|---|
cage-challenge/cage-challenge-4 | 8c3c50ca54b1 |
cybermonic/cage-4-submission (KEEP, with its five released checkpoints under weights/) | 2afd652d80ce |
adityavs14/Hierarchical-MARL (H-MARL Expert, with its released checkpoint) | 6fe960f5931f |