TEST REPORT: Agent Spend Guard v1.0.0

Run by bash tests/run_all.sh; full output in tests/logs/run-20261030.txt. Started 2026-10-30 09:04:19 GMT, finished 09:04:22 GMT (from date at run time). Box: Linux, Python 3.13.5.

Result: 4/4 steps passed, FAILED STEPS: 0. 42/42 tests: 26 guard (one of them holds 41 nested-agent subtests), 8 installer, 6 usage report, 2 skill.

What is real and what is mocked

• Real: guard/spend_guard.py as shipped, run as a separate process fed hook JSON on stdin in Claude Code's and Cursor's documented formats (PreToolUse, PostToolUse, PostToolUseFailure, SubagentStart/Stop, PostToolBatch, UserPromptSubmit; beforeShellExecution, beforeMCPExecution), with its real state files and lock. The installer writing real .claude/settings.json and .cursor/hooks.json in throwaway projects, then the installed hook command strings run through a shell.
• Mocked: the transcripts. Usage numbers come from hand-built JSONL files in the Claude Code transcript layout (~/.claude/projects/<proj>/<session>.jsonl, subagents/agent-*.jsonl, message.id + message.usage), not from a real Claude Code session.
• UNTESTED: the Claude Code and Cursor apps. Neither app is on the test box, so no real agent ran these hooks. Formats come from the official hook docs (code.claude.com/docs/en/hooks; Cursor hooks docs). Before release, install it in one real project for each tool and trigger each rule once.
• UNTESTED: Windows and macOS (Windows path handling is unit-tested only), Python below 3.13, usage-report accuracy against Anthropic's own meter (the weighted-token figure is a relative measure, not the plan meter).

1. Guard: tests/test_guard.py (26/26)

• SPAWN-CONCURRENCY cap and release on SubagentStop; a finished Agent tool call releases its pending slot; pending spawns expire; 6 parallel spawn hooks at once respect the cap (lock test); the older Task tool name also counts. SPAWN-TOTAL.
• NESTED-AGENT: 41 subtests, positive (e.g. claude -p, npx -y @anthropic-ai/claude-code -p, cursor-agent, codex exec, $(claude -p …), (cd x && agent "do"), C:\Tools\claude.exe -p x) and negative (claude --version, claude mcp list, grep claude, git commit -m "agent fix"…); levels deny/ask/off; allowlist regex; Cursor shell format.
• RETRY-LOOP (3 identical failures; non-zero exit counts; a success resets). REPEAT-CALL with its 30-minute window and exempt tools, in both tools, MCP included.
• SESSION-BUDGET: warns once at 80% ("900 of 1,000"), blocks PostToolBatch and new prompts at 100%, #spend-ok extends by 50%; subagent transcripts counted; a half-written last line isn't counted; missing transcript is fine.
• Sessions are kept separate; denials logged; fails open by default and closed when fail_closed: true; --status.
• Latency: median 25 ms, max 27 ms per hook call over 15 runs.

2. Installer: tests/test_installer.py (8/8)

Fresh install of both tools; keeps existing settings and is idempotent; policy edits survive reinstall; --uninstall removes only our hooks; --dry-run writes nothing; invalid JSON stops the install; warns on Cursor CLI approvalMode: unrestricted; installed commands really run from a shell.

3. Usage report: tests/test_report.py (6/6)

Deduplication by message.id and totals; per-session rows; --days filter; SPIKE / FAN-OUT / LOOP / UNATTENDED flags; budget suggestion, text and CSV output; missing transcript folder.

4. Skill: tests/test_skill.py (2/2)

Frontmatter and references valid; ASCII file names; no key-shaped strings.

Failures seen during development (fixed before the logged run)

1. NESTED-AGENT missed npx -y @anthropic-ai/claude-code -p and (cd x && agent "do"). Fixed by mapping claude-code to claude and splitting segments on parentheses.
2. NESTED-AGENT missed a Windows path (C:\Tools\claude.exe) on Linux. Fixed by splitting on both slash types.
3. One budget test expected no warning at 900/1,000; that was wrong (90% is past the 80% warning), so the test was corrected, not the code.
4. --uninstall left an empty {"version":1,"hooks":{}} in .cursor/hooks.json; now removed.

Nothing was published, deployed or sent.

Real-app verification 2026-10-30

Re-run by bash tests/run_all.sh. Full output is in tests/logs/run-20261030-realapp.txt; it started at 12:00:17 GMT and finished at 12:00:24 GMT.

Result: 5/5 steps passed, FAILED STEPS: 0. 49/49 tests: - 27 guard tests. The nested-agent test now has 57 subtests. - 8 installer, 6 usage report, 2 skill. - 6 inside the real Claude Code CLI (new file tests/test_real_claude.py).

Installed

• Claude Code 2.1.314: npm i -g @anthropic-ai/claude-code.
• Cursor CLI cursor-agent 2026.10.22-27dcf5f: official curl https://cursor.com/install -fsS | bash.

Claude Code: real runs, PASS

• How it ran: claude -p ran with ANTHROPIC_BASE_URL pointing at a local scripted Messages API (tests/real_app/mock-anthropic.mjs) and a dummy key.
• The CLI, its settings loader, its hook runner and its tools (Bash, Agent/Task, Workflow) are the real ones. Only the model's replies are scripted.
• No Anthropic account was used and nothing was spent.
• Settings load: with the guard installed, claude doctor shows no "Invalid settings" section. A deliberately broken hook event name in a scratch project does produce "Invalid settings" with the list of valid events.
• The guard's event names (PreToolUse, PostToolUse, PostToolUseFailure, SubagentStart, SubagentStop, UserPromptSubmit, Stop) are all in that list.
• Real hook payloads were captured by a logging hook.
• PreToolUse carries tool_use_id and tool_input.
• Bash PostToolUse has tool_response {stdout, stderr, interrupted…} and no exit_code. A failed command arrives as PostToolUseFailure with error: "Exit code 2…". The guard's retry-loop rule reads that event.
• SubagentStart and SubagentStop carry agent_id and agent_type; SubagentStop also carries agent_transcript_path.
• The Agent tool launches subagents asynchronously, and denials name the tool Task.
• Nested agent: a Bash call running claude -p … was denied. The model received PreToolUse:Bash hook error: Agent Spend Guard [NESTED-AGENT]: ….
• Retry loop: denied after 3 real failures of the same command.
• Session budget: 5,010 weighted tokens against a 1,000 budget, then terminal_reason: hook_stopped.
• Spawn cap: with the real Agent tool, exactly 2 subagents spawned and the third was denied.
• Workflow fan-out: denied. Under -p an "ask" becomes a deny, because nobody is there to answer.
• usage_report.py read the real transcripts under ~/.claude/projects/…, including subagents/agent-*.jsonl.

Bugs found and fixed

1. The safe-subcommand lists didn't match the real CLIs.
2. Checked against claude --help 2.1.314 and cursor-agent --help 2026.10.22.
3. agent ls was treated as safe, but it resumes a session; it is now guarded.
4. Added as safe:
   • claude: auth, logs, stop, kill, rm, upgrade, plugins, auto-mode, import.
   • cursor-agent: status, whoami, login, logout, models, --list-models, about, team, create-chat, mcp, plugin, bedrock, shell-integration commands.
5. Kept as spawning (guarded): claude --bg, agents, attach, respawn, ultrareview; cursor-agent persist, worker, resume, generate-rule.
6. New subtests cover each of these.
7. New fan-out path not covered. Claude Code 2.1.314 has a Workflow tool that runs multi-agent workflows.
8. New rule WORKFLOW-FANOUT: policy fanout_tools: ["Workflow"] and fanout_tool_level: "ask" (it can be set to deny or off).
9. Tested in the unit suite and in the real CLI.
10. Docs claimed Cursor has no subagent hooks. It does (subagentStart/subagentStop). The README, LISTING and SKILL now say this version uses Cursor's shell and MCP hooks only.

Cursor: static check PASS, live run BLOCKED

• Checked against the installed cursor-agent bundle and cursor.com/docs/agent/hooks and /permissions:
• Locations .cursor/hooks.json and ~/.cursor/hooks.json.
• Schema {version: number, hooks: {…}}.
• beforeShellExecution/beforeMCPExecution responses {permission: allow|deny|ask, user_message, agent_message}.
• timeout in seconds.
• beforeMCPExecution tool_input is a JSON string.
• The guard's install and output match all of these.
• Also confirmed: Cursor loads Claude-style hooks from .claude/settings*.json ("third-party hooks"). They need the project to be trusted, and the docs now say so.
• BLOCKED: a live Cursor agent run with the hooks. cursor-agent said "Your stored authentication is invalid… run 'agent login'", and cursor-agent status reports "Not logged in". This needs a Cursor account login (agent login, or CURSOR_API_KEY). It does not need a paid plan to install, but running the agent uses the account's model quota.

Re-run 2026-11-02 (promoted from drafts/2026-10-30 to products/)

bash tests/run_all.sh from products/agent-spend-guard/; full output in tests/logs/run-20261102.txt (file written 2026-11-02 07:13:54 GMT). Result: FAILED STEPS: 0, 49/49 tests: 27 guard, 8 installer, 6 usage report, 2 skill, 6 real Claude Code CLI hook runs (model replaced by a local mock; no Anthropic account used). Unchanged limits: no live Cursor run (blocked earlier, see above); no paid model calls.

Pre-launch re-run 2026-11-02

bash tests/run_all.sh; full output in tests/logs/run-20261102-prelaunch.txt (2026-11-02, about 08:00 GMT). FAILED STEPS: 0, 49/49 tests (27 guard, 8 installer, 6 usage report, 2 skill, 6 real Claude Code CLI hook runs with the model replaced by a local mock). Counts unchanged. Still no live Cursor run.