# TEST REPORT: Agent Spend Guard v1.0.0

Run by `bash tests/run_all.sh`; full output in `tests/logs/run-20261030.txt`. Started 2026-10-30 09:04:19 GMT, finished 09:04:22 GMT (from `date` at run time). Box: Linux, Python 3.13.5.

**Result: 4/4 steps passed, `FAILED STEPS: 0`. 42/42 tests:** 26 guard (one of them holds 41 nested-agent subtests), 8 installer, 6 usage report, 2 skill.

## What is real and what is mocked
- **Real:** `guard/spend_guard.py` as shipped, run as a separate process fed hook JSON on stdin in Claude Code's and Cursor's documented formats (PreToolUse, PostToolUse, PostToolUseFailure, SubagentStart/Stop, PostToolBatch, UserPromptSubmit; beforeShellExecution, beforeMCPExecution), with its real state files and lock. The installer writing real `.claude/settings.json` and `.cursor/hooks.json` in throwaway projects, then the installed hook command strings run through a shell.
- **Mocked:** the transcripts. Usage numbers come from hand-built JSONL files in the Claude Code transcript layout (`~/.claude/projects/<proj>/<session>.jsonl`, `subagents/agent-*.jsonl`, `message.id` + `message.usage`), not from a real Claude Code session.
- **UNTESTED: the Claude Code and Cursor apps.** Neither app is on the test box, so no real agent ran these hooks. Formats come from the official hook docs (code.claude.com/docs/en/hooks; Cursor hooks docs). Before release, install it in one real project for each tool and trigger each rule once.
- **UNTESTED:** Windows and macOS (Windows path handling is unit-tested only), Python below 3.13, usage-report accuracy against Anthropic's own meter (the weighted-token figure is a relative measure, not the plan meter).

## 1. Guard: `tests/test_guard.py` (26/26)
- SPAWN-CONCURRENCY cap and release on SubagentStop; a finished Agent tool call releases its pending slot; pending spawns expire; 6 parallel spawn hooks at once respect the cap (lock test); the older `Task` tool name also counts. SPAWN-TOTAL.
- NESTED-AGENT: 41 subtests, positive (e.g. `claude -p`, `npx -y @anthropic-ai/claude-code -p`, `cursor-agent`, `codex exec`, `$(claude -p …)`, `(cd x && agent "do")`, `C:\Tools\claude.exe -p x`) and negative (`claude --version`, `claude mcp list`, `grep claude`, `git commit -m "agent fix"`…); levels deny/ask/off; allowlist regex; Cursor shell format.
- RETRY-LOOP (3 identical failures; non-zero exit counts; a success resets). REPEAT-CALL with its 30-minute window and exempt tools, in both tools, MCP included.
- SESSION-BUDGET: warns once at 80% ("900 of 1,000"), blocks PostToolBatch and new prompts at 100%, `#spend-ok` extends by 50%; subagent transcripts counted; a half-written last line isn't counted; missing transcript is fine.
- Sessions are kept separate; denials logged; fails open by default and closed when `fail_closed: true`; `--status`.
- **Latency: median 25 ms, max 27 ms per hook call over 15 runs.**

## 2. Installer: `tests/test_installer.py` (8/8)
Fresh install of both tools; keeps existing settings and is idempotent; policy edits survive reinstall; `--uninstall` removes only our hooks; `--dry-run` writes nothing; invalid JSON stops the install; warns on Cursor CLI `approvalMode: unrestricted`; installed commands really run from a shell.

## 3. Usage report: `tests/test_report.py` (6/6)
Deduplication by `message.id` and totals; per-session rows; `--days` filter; SPIKE / FAN-OUT / LOOP / UNATTENDED flags; budget suggestion, text and CSV output; missing transcript folder.

## 4. Skill: `tests/test_skill.py` (2/2)
Frontmatter and references valid; ASCII file names; no key-shaped strings.

## Failures seen during development (fixed before the logged run)
1. NESTED-AGENT missed `npx -y @anthropic-ai/claude-code -p` and `(cd x && agent "do")`. Fixed by mapping `claude-code` to claude and splitting segments on parentheses.
2. NESTED-AGENT missed a Windows path (`C:\Tools\claude.exe`) on Linux. Fixed by splitting on both slash types.
3. One budget test expected no warning at 900/1,000; that was wrong (90% is past the 80% warning), so the test was corrected, not the code.
4. `--uninstall` left an empty `{"version":1,"hooks":{}}` in `.cursor/hooks.json`; now removed.

Nothing was published, deployed or sent.

## Real-app verification 2026-10-30

Re-run by `bash tests/run_all.sh`. Full output is in `tests/logs/run-20261030-realapp.txt`; it started at 12:00:17 GMT and finished at 12:00:24 GMT.

**Result: 5/5 steps passed, `FAILED STEPS: 0`. 49/49 tests:**
- 27 guard tests. The nested-agent test now has 57 subtests.
- 8 installer, 6 usage report, 2 skill.
- **6 inside the real Claude Code CLI** (new file `tests/test_real_claude.py`).

### Installed
- **Claude Code 2.1.314**: `npm i -g @anthropic-ai/claude-code`.
- **Cursor CLI `cursor-agent` 2026.10.22-27dcf5f**: official `curl https://cursor.com/install -fsS | bash`.

### Claude Code: real runs, PASS
- **How it ran:** `claude -p` ran with `ANTHROPIC_BASE_URL` pointing at a local scripted Messages API (`tests/real_app/mock-anthropic.mjs`) and a dummy key.
  - The CLI, its settings loader, its hook runner and its tools (Bash, Agent/Task, Workflow) are the real ones. Only the model's replies are scripted.
  - No Anthropic account was used and nothing was spent.
- **Settings load:** with the guard installed, `claude doctor` shows no "Invalid settings" section. A deliberately broken hook event name in a scratch project does produce "Invalid settings" with the list of valid events.
  - The guard's event names (`PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `SubagentStart`, `SubagentStop`, `UserPromptSubmit`, `Stop`) are all in that list.
- **Real hook payloads were captured** by a logging hook.
  - PreToolUse carries `tool_use_id` and `tool_input`.
  - Bash PostToolUse has `tool_response {stdout, stderr, interrupted…}` and **no `exit_code`**. A failed command arrives as **`PostToolUseFailure`** with `error: "Exit code 2…"`. The guard's retry-loop rule reads that event.
  - SubagentStart and SubagentStop carry `agent_id` and `agent_type`; SubagentStop also carries `agent_transcript_path`.
  - The Agent tool launches subagents asynchronously, and denials name the tool `Task`.
- **Nested agent:** a Bash call running `claude -p …` was denied. The model received `PreToolUse:Bash hook error: Agent Spend Guard [NESTED-AGENT]: …`.
- **Retry loop:** denied after 3 real failures of the same command.
- **Session budget:** 5,010 weighted tokens against a 1,000 budget, then `terminal_reason: hook_stopped`.
- **Spawn cap:** with the real Agent tool, exactly 2 subagents spawned and the third was denied.
- **Workflow fan-out:** denied. Under `-p` an "ask" becomes a deny, because nobody is there to answer.
- `usage_report.py` read the real transcripts under `~/.claude/projects/…`, including `subagents/agent-*.jsonl`.

### Bugs found and fixed
1. **The safe-subcommand lists didn't match the real CLIs.**
   - Checked against `claude --help` 2.1.314 and `cursor-agent --help` 2026.10.22.
   - `agent ls` was treated as safe, but it resumes a session; it is now guarded.
   - Added as safe:
     - claude: `auth`, `logs`, `stop`, `kill`, `rm`, `upgrade`, `plugins`, `auto-mode`, `import`.
     - cursor-agent: `status`, `whoami`, `login`, `logout`, `models`, `--list-models`, `about`, `team`, `create-chat`, `mcp`, `plugin`, `bedrock`, shell-integration commands.
   - Kept as spawning (guarded): `claude --bg`, `agents`, `attach`, `respawn`, `ultrareview`; `cursor-agent persist`, `worker`, `resume`, `generate-rule`.
   - New subtests cover each of these.
2. **New fan-out path not covered.** Claude Code 2.1.314 has a `Workflow` tool that runs multi-agent workflows.
   - New rule **WORKFLOW-FANOUT**: policy `fanout_tools: ["Workflow"]` and `fanout_tool_level: "ask"` (it can be set to deny or off).
   - Tested in the unit suite and in the real CLI.
3. **Docs claimed Cursor has no subagent hooks.** It does (`subagentStart`/`subagentStop`). The README, LISTING and SKILL now say this version uses Cursor's shell and MCP hooks only.

### Cursor: static check PASS, live run BLOCKED
- **Checked against the installed cursor-agent bundle and cursor.com/docs/agent/hooks and /permissions:**
  - Locations `.cursor/hooks.json` and `~/.cursor/hooks.json`.
  - Schema `{version: number, hooks: {…}}`.
  - `beforeShellExecution`/`beforeMCPExecution` responses `{permission: allow|deny|ask, user_message, agent_message}`.
  - `timeout` in seconds.
  - `beforeMCPExecution` `tool_input` is a JSON string.
  - The guard's install and output match all of these.
- **Also confirmed:** Cursor loads Claude-style hooks from `.claude/settings*.json` ("third-party hooks"). They need the project to be trusted, and the docs now say so.
- **BLOCKED:** a live Cursor agent run with the hooks. `cursor-agent` said "Your stored authentication is invalid… run 'agent login'", and `cursor-agent status` reports "Not logged in". This needs a Cursor account login (`agent login`, or `CURSOR_API_KEY`). It does not need a paid plan to install, but running the agent uses the account's model quota.

## Re-run 2026-11-02 (promoted from drafts/2026-10-30 to products/)
`bash tests/run_all.sh` from `products/agent-spend-guard/`; full output in `tests/logs/run-20261102.txt` (file written 2026-11-02 07:13:54 GMT).
**Result: `FAILED STEPS: 0`, 49/49 tests:** 27 guard, 8 installer, 6 usage report, 2 skill, 6 real Claude Code CLI hook runs (model replaced by a local mock; no Anthropic account used).
Unchanged limits: no live Cursor run (blocked earlier, see above); no paid model calls.

## Pre-launch re-run 2026-11-02
`bash tests/run_all.sh`; full output in `tests/logs/run-20261102-prelaunch.txt` (2026-11-02, about 08:00 GMT). **`FAILED STEPS: 0`, 49/49 tests** (27 guard, 8 installer, 6 usage report, 2 skill, 6 real Claude Code CLI hook runs with the model replaced by a local mock). Counts unchanged. Still no live Cursor run.