TEST REPORT: Agent Guardrails Kit v1.0.0

Run from a clean unzip of the v1.0.0 zip, as built before this report was added: agent-guardrails-kit-v1.0.0-without-report.zip, sha256 2f50f0e6f40625fce5ee9601b3436e97f5945db3cb2677bd82c8ccc80d27b63e, 24 files. The shipped agent-guardrails-kit-v1.0.0.zip is that build plus this TEST-REPORT.md; every other file is byte-identical (checked with diff -rq when the shipped zip was built).

Run by staging/new-packs-20261023/build_and_test.sh agent-guardrails-kit pre, which runs tests/run_all.sh. Started 2026-10-23 10:05:08 BST (UTC+01:00), finished 2026-10-23 10:05:59 BST (UTC+01:00) (times from date at run time). Box: Linux 6.12.94+, Python 3.13.5, git 2.47.3, Node v24.22.0 (first on PATH), n8n 2.43.7 (npm).

Result: 7/7 steps passed, FAILED STEPS: 0. - 379/379 guard decision tests - 19/19 hook-process tests - 9/9 installer tests - 15/15 skill tests - 16/16 n8n tests (4 static, 12 end-to-end) - 3/3 harness fail-fast tests - the example install: 12/12 sample actions as expected

That's 441 automated tests plus the example install.

Node version matters for the n8n tests. n8n 2.43.x needs Node 24 or later and won't start on Node 20 ("Your Node.js version 20.19.2 is currently not supported by n8n"). An earlier independent run with Node 20.19.2 first on PATH failed the end-to-end tests, but only after the harness waited for n8n. The harness now checks the Node version, a busy port, n8n crashing on start and n8n never becoming ready (AGD_N8N_START_TIMEOUT, default 120 s), and then fails every end-to-end test at once with the reason. Step 5b tests this.

What is real and what is mocked

• Real: guard/agent_guard.py as shipped, run both as a function (decision table) and as a separate process fed the same JSON that Claude Code and Cursor send on stdin. Also real: the installer writing real .claude/settings.json / .cursor/hooks.json files in throwaway projects, then those exact hook command strings run through a shell by install/selftest.py; real git repos for the incident snapshot and the staged-secret scan; and a real local n8n 2.43.7 (n8n import:credentials, import:workflow, publish:workflow, n8n start) receiving alerts signed by the real guard process.
• Not real: the Claude Code and Cursor apps. No agent app ran these hooks. The input and output formats come from each tool's hook documentation, and the Cursor ask-in-Auto-review behaviour from a Cursor staff reply (forum, 5 Oct 2026).
• Mocked: Slack. A local HTTP server stands in for the Slack incoming webhook. Nothing left the box.
• Paths in the decision table are made-up (/home/dev/proj, C:\Users\dev\proj). The guard only reads the command text, so nothing was deleted or read. Fake secrets in the tests are assembled at run time, so no key-shaped string is stored in the pack. test_pack_has_no_key_shaped_strings runs the kit's own secret_scan.py over the whole kit and requires 0 findings.

1. Guard decision table: python3 tests/test_guard.py (379/379)

tests/cases.py holds every row. Each of the 174 shell commands (146 bash/zsh, 28 PowerShell/cmd.exe) is checked twice: once in Claude Code's PreToolUse format and once in Cursor's beforeShellExecution format. That's 348 tests, each checking the decision and the rule id.

The other 31 tests: - 19 Claude Code file-tool cases (Read, Edit, Write, Grep on .env, .env.local, ~/.ssh/id_rsa vs .pub, credentials.json, *.tfstate, a Windows .env path, the guard's own files, and writes outside the project). - 6 Cursor beforeReadFile cases. - 41 everyday commands that must stay prompt-free (npm install, git pull --rebase origin main, terraform apply, ls -la ~/.ssh, rm -rf .venv && python -m venv .venv…), each in both formats, as one test. - Policy overrides: rule levels up or down, extra safe folder, extra protected branch, extra secret exception, policy-file merge. - Token masking, CLAUDE_PROJECT_DIR, Cursor multi-root workspaces. - A check that all 23 rule ids appear in the table.

The false-positive sweep found 3 problems during development, which are fixed and now in the table: grep "process.env" was read as a file name, .venv wasn't a safe folder, and rm -rf … package-lock.json asked.

2. Hook process: python3 tests/test_adapters.py (19/19)

test_all_mode ... ok                                   audit "all" logs allows too
test_blocks_mode_logs_only_blocks_and_redacts ... ok   default audit logs only ask/deny; Bearer token masked in the log
test_no_secret_no_webhook_and_dead_webhook_does_not_hang ... ok   unreachable alert URL: decision still returned (< 5 s)
test_off_mode ... ok
test_signed_webhook ... ok                             HMAC over "<ts>.<body>" checked independently in Python; no alert on allow
test_allow_is_silent_exit_0 ... ok                     Claude: allow = exit 0, no output
test_ask_json ... ok
test_bad_json_fails_closed_exit_2 ... ok               Claude: unreadable input -> exit 2 (blocks)
test_broken_policy_fails_closed ... ok                 broken policy.json -> exit 2
test_deny_json ... ok                                  hookSpecificOutput.permissionDecision "deny" + reason with rule id
test_file_tool ... ok
test_unknown_tool_allowed ... ok
test_allow ... ok                                      Cursor: {"permission":"allow"}
test_ask_becomes_deny_by_default ... ok                Cursor: ask -> deny + "confirm in chat" note to the agent
test_ask_kept_when_policy_says_so ... ok
test_bad_json_fails_closed_deny ... ok                 Cursor: unreadable input -> deny JSON
test_deny_fields ... ok
test_read_file_deny_shape ... ok                       beforeReadFile deny carries only permission + user_message
test_latency ... ok                                    hook latency over 20 runs: median 58 ms, max 86 ms

3. Installer: python3 tests/test_installer.py (9/9)

Throwaway project folders with a space in the name: - Fresh install for both tools: the files, the matcher, timeout, onFailure: block and failClosed: true are all present. - Existing permissions, other hooks and Stop hooks are kept, and a backup is written. - Re-running changes nothing and doesn't duplicate entries. - Policy edits survive a reinstall. - --uninstall removes only the kit's entries. - --dry-run writes nothing. - A settings file with comments stops the installer and stays untouched. - The installed hook commands really run: selftest 12/12 for absolute and --portable paths.

4. Skills: python3 tests/test_skills.py (15/15)

• Structure: each SKILL.md has only name and description frontmatter, the name equals the folder name (lowercase-hyphen, ASCII), the description has the expected trigger words and no angle brackets, every script it mentions exists, and there are no banned words. Every file and folder name in the pack is plain ASCII.
• safe_rm.py:
• trash → list → restore round trip, with contents checked
• 8 refusals (., the root, outside path, ../, .git, $HOME, src/*, a missing file), and nothing moves if any one is refused
• a symlink to outside moves only the link
• restore refuses to overwrite
• purge
• the guard allows python3 safe_rm.py …
• incident_snapshot.py: on a real repo after git reset --hard it reports the reflog entry and the lost staged file as a dangling blob, and the repo's file tree is unchanged (read-only). It filters audit-log rows to the last 24 h. It also handles a folder that isn't a repo.
• secret_scan.py: finds all 6 seeded kinds (Stripe live key, GitHub token, AWS key id, Anthropic key, private-key header, *_TOKEN= assignment), prints only masked values, and ignores .env.example placeholders, process.env and node_modules. --staged reports the right line numbers from git diff --cached. A clean tree exits 0 and a bad path exits 2. Run over the whole kit itself (tests included), it finds 0 secrets.

5. n8n workflow: python3 tests/test_n8n.py (16/16)

Static (4): - The builder output matches the shipped JSON. - Names and ids are unique, every connection target exists, every node is reachable from the webhook, and the workflow is inactive on import. - Raw body is on, the response comes from a node, the Crypto credential is referenced by name, there's no require(), ctEq is used, no Slack URL is shipped, and the Config defaults are as documented. - Every node type exists in the installed n8n 2.43.7. Only the type was checked, not the version number.

End-to-end in n8n 2.43.7 (12):

test_00_setup ... ok                                    import:credentials, import:workflow, publish:workflow all exit 0
test_01_real_guard_deny_reaches_slack ... ok            real guard (Claude format) denies rm -rf ~/ and its signed alert reaches mock Slack (DEL-ROOT, "blocked")
test_02_secret_masked_in_slack ... ok                   real guard (Cursor format): Bearer token absent from the Slack message
test_03_real_guard_allow_sends_nothing ... ok
test_04_wrong_secret_401 ... ok
test_05_tampered_body_401 ... ok
test_06_stale_timestamp_401 ... ok                      1000 s old
test_07_missing_headers_401 ... ok
test_08_replay_409 ... ok                               same body + headers twice in a row: 200 then 409, one Slack post
test_09_ask_not_notified_by_default ... ok
test_10_signed_garbage_400 ... ok
test_11_code_fence_in_subject_cannot_break_message ... ok

The replay test was also run once on its own with a 3-second gap (AGD_REPLAY_GAP=3) and passed.

5b. Harness fail-fast: python3 tests/test_n8n_failfast.py (3/3)

Runs tests/test_n8n.py against a fake old Node (v20.19.2), a fake n8n that exits on start, and a fake n8n that never becomes ready (5 s limit). In each case all 12 end-to-end tests fail, including test_03_real_guard_allow_sends_nothing, with the reason printed, in under 60 s. Separately, a manual check (not part of run_all) with port 5890 already taken failed all 12 at once with "port 5890 is already in use".

6. Example task (1/1)

install/install.py into a fresh sample project, then install/selftest.py: 12/12 as expected (6 sample actions × Claude Code and Cursor command lines).

Not tested

• The n8n end-to-end tests on any Node other than v24.22.0.
• Inside the real Claude Code and Cursor apps. No agent app was available on the test box, so onFailure: block, failClosed, the matcher, and how each app shows the reason are taken from their documentation and not observed.
• Windows. PowerShell and cmd.exe commands are covered by the parser tests (28 rows) only; the hook was not run on Windows, and install.py was not run with a Windows Python.
• macOS (only Linux was used).
• Concurrent replays. The replay check uses n8n workflow static data, which is saved when a run ends, so two identical alerts arriving at the same moment might both pass. Only back-to-back replays were tested.
• Real Slack, n8n Cloud, n8n 1.x, queue mode, and n8n behind a public URL or proxy.
• What happens inside scripts or binaries the agent runs (./cleanup.sh, make clean). By design the guard sees only the command text (SETUP.md, section 5).
• Real AI agents following the three skills. The skills' structure and scripts are tested, but their instructions were not run by an agent.

Known limits (by design, see SETUP.md)

• A hook is one layer: it can be bypassed by code it can't see (scripts, MCP tools, a human). Keep backups.
• Alerts make a blocked action up to 2 s slower when n8n is slow (the guard's timeout), and never change the decision.

Sources

The incidents and vendor statements this kit responds to are listed, with links and dates, in README.md under "Sources".