AWS Incident Response
Overview
Three coding agents investigated seven breached AWS environments with and without an incident-response runbook, three repetitions each — 126 graded runs.
Scenarios included
- Leaked CI key, privilege escalation and customer-data theft
- Stolen instance credentials and lateral movement through SSM
- Ransomware hidden behind an application failure
- Persistence hunting after a supposedly closed incident
Agent profiles
Per-metric comparison
Success is a weighted score, not a pass rate. Evidence rigor is scored separately from completeness. Agent time excludes waiting when available.
Success
Evidence rigor
Adherence
Tokens
Agent time
Commands
Scenarios
Overall takeaways
Methodology & grading
Seven scenarios × three agents × two arms × three repetitions: 126 runs. All tasks were inspect-only. The skill arm received the same adapted AWS incident-response runbook.
Success is the final weighted score across scenario-specific criteria, including applicable penalties; it is not a binary pass rate. Adherence measures compliance with the instructions. Evidence rigor measures unsupported or fabricated claims (1 = none, 0.5 = minor, 0 = major); it is separate from completeness and success. Scores include the reviewed overrides in the source results.
Agent time excludes time waiting for user input when available. Tokens use each runner's reported accounting, including cached input; they are not equivalent to cost. A tilde marks estimated costs, whose rates and assumptions are retained in each run's downloadable JSON. Approval was not required because these investigations did not authorize remediation.
These are comparisons of runners and their configured models, not isolated model capabilities. Claude used claude-opus-5-5 in 40 runs and claude-fable-5-1 in two. There are only three repetitions per condition; differences between arms do not establish a causal effect of the runbook.