# TEST REPORT: Agent Runaway Check v1.0.0

**Latest result (pre-launch re-run, 2026-11-02 08:00:30 GMT): 14/14 tests passed, `FAILED STEPS: 0`**, plus a run on 5 transcripts written by the real Claude Code 2.1.314 CLI. Details of each run are below; the first section is the original 30 Oct run.

## First run 2026-10-30

Run by `bash tests/run_all.sh`; output in `tests/logs/run-20261030.txt`. 2026-10-30 09:26:25 GMT. Linux, Python 3.13.5.

**Result: 12/12 tests passed, `FAILED STEPS: 0`.** (Superseded by the 14/14 runs below.)

- **Real:** the shipped script, as a module and as a CLI, against fake home folders built per test (Claude Code `settings.json` / `settings.local.json`, `~/.cursor/cli-config.json`, `hooks.json`, `.zshrc`, transcripts). One test installs the real **Agent Spend Guard** draft into a temp project and checks NO-HOOKS goes away. One runs on the test box itself (no Claude Code or Cursor installed, so: NO-HOOKS only, no transcripts).
- **Mocked:** the transcripts and settings files are hand-written in the documented formats.
- **UNTESTED in this first run:** a real Claude Code or Cursor install, and real transcripts. Both were covered by the real-app verification below. Still untested: macOS and Windows, and a live Cursor agent session (no Cursor login).

Tests: clean home (only NO-HOOKS); Claude bypass mode, nested-agent allow rules (2) and blanket Bash; Cursor unrestricted and nested-agent allow; allowlist mode is clean; aliases (a commented-out line ignored); invalid JSON; hooks present; after installing Agent Spend Guard; history (duplicate response counted once, 9 subagents -> FAN-OUT, 5 identical calls -> LOOP, an old session outside the window and a broken line ignored, 35k weighted tokens printed); exit codes and JSON; read-only (no file mtimes change); runs on this machine.

Found during development: the first version counted tool calls from a duplicated transcript line twice (6 instead of 5). Fixed so tool calls are only counted once per API response; the test now expects 5.

Free product: MIT licence, so no refund policy is included.

## Real-app verification 2026-10-30

Re-run by `bash tests/run_all.sh`. Full output is in `tests/logs/run-20261030-realapp.txt`; it started and finished at 12:01:06 GMT.

**Result: 14/14 tests passed, `FAILED STEPS: 0`** (12 before, plus 2 new).

### Checked against the real apps (Claude Code 2.1.314, cursor-agent 2026.10.22-27dcf5f)
- Both apps were installed on the test box: `claude --version` reports `2.1.314 (Claude Code)` and `cursor-agent --version` reports `2026.10.22-27dcf5f`.
- Claude Code settings paths (`~/.claude/settings.json`, `.claude/settings.json`, `.claude/settings.local.json`), keys (`permissions.defaultMode`, `permissions.allow`) and the `bypassPermissions` mode are in the installed 2.1.314 CLI. `claude --help` lists `--permission-mode` and `--dangerously-skip-permissions`. In the same real-app session, `claude doctor` loaded a project's settings files and reported no "Invalid settings" (that was the Agent Spend Guard project, not a separate Runaway Check run).
- Cursor CLI: `cli-config.json`, `cli.json`, `approvalMode`, `Shell(...)` permissions and `--yolo` appear in the installed 2026.10.22-27dcf5f bundle.
- **Run on real transcripts:** `runaway_check.py --project realcheck/proj --days 1` read 5 sessions written by the real Claude Code 2.1.314 CLI (its transcript entries are stamped `"version": "2.1.314"`) during the Agent Spend Guard real-CLI tests.
  - Those sessions used a local scripted model API (`ANTHROPIC_BASE_URL` pointed at a mock, with a dummy key). The transcript format is real, but the token counts are tiny test values, not real-world usage.
  - It flagged the retry-loop session (`max repeats 5  LOOP`) and counted 2 subagents in the spawn-cap session.
  - The total was 6k weighted tokens.

### Bugs found and fixed
1. **Cursor CLI permissions were never read.** The real CLI keeps them in `~/.cursor/cli-config.json` and the project's `.cursor/cli.json`, as shown by the installed bundle and cursor.com/docs/cli permissions.
   - Both files are now read.
   - New rule **CURSOR-SHELL-ANY** (MEDIUM) for a blanket `Shell(*)` / `Shell(*:*)` allow.
2. **Missed a real flag form.** `--permission-mode bypassPermissions` (and `=`), the spelling in `claude --help`, is now caught by SHELL-SKIP-PERMISSIONS.
3. **SHELL-CURSOR-FORCE missed real invocations.** These forms were missed: `agent -f`, `--yolo`, and a call after `;`, `|`, `&`, `(` or `=`. The regex is now `(^|[\s'"=;|&(])(cursor-agent|agent)\s[^#\n]*(--force|--yolo|-f\b)`.
   - 2 new tests: `test_cursor_project_cli_json_and_blanket_shell` and `test_real_cli_flag_aliases`.

### Not run
- A live Cursor session. That needs a Cursor login, and it isn't needed for this read-only check.
- Transcripts from real, paid Claude Code use (heavy sessions with real token counts).
- macOS and Windows.

## Pre-launch re-run 2026-11-02
`bash tests/run_all.sh`; full output in `tests/logs/run-20261102-prelaunch.txt` (08:00:30 GMT). **14/14 tests passed, `FAILED STEPS: 0`.** Counts unchanged; the realcheck transcript run gave the same output (5 sessions, LOOP flagged, 2 subagents).

## Evidence re-checked 2026-11-02 (before the site copy update)
Logs `tests/logs/run-20261030-realapp.txt` and `run-20261102-prelaunch.txt` show the realcheck run (5 sessions, 6k weighted tokens, LOOP on session 30145a08, 2 subagents on fb00ce94). The 5 JSONL files are in `~/.claude/projects/-workspace-realcheck-proj/` on the test box and carry version 2.1.314. The installed binaries report the versions above.