Lessons from 126 AWS Breach Investigations with Claude Code, Codex and Cursor

With growing interest in using agents for incident response, we wanted to test how well three widely used coding agents would investigate a compromised AWS environment. We created seven AWS environments with different breaches designed to resemble real incidents. See all seven scenarios below. We then asked the agents to determine what happened and recommend a response without changing the accounts. We ran each scenario three times per agent, both with and without a runbook, for 126 runs in total. In half the runs, we supplied an adapted version of AWS incident-response guidance to see how it affected the investigations.

We tested Codex (gpt-6.1-sol), Claude Code (claude-opus-5-5) and Cursor (Composer 2.5 Fast), using the runners' default model selections at the time rather than overriding them for this comparison. All agents received shared execution rules and the same scenario tasks. This was a comparison of the agents as configured for these runs, including their models and tools; it does not isolate the capabilities of the underlying models.

Average results across all 126 runs, with and without the runbook. Explore the interactive comparisons.

All three demonstrated excellent investigative capabilities. Average success scores across the seven scenarios were 97.4% for Codex, 95.3% for Claude Code and 80.5% for Cursor. Cursor trailed the other two overall, with larger gaps in some scenarios, but still reconstructed substantial parts of the incidents. Its use of Composer 2.5 Fast may partly explain that gap, though these runs do not establish model size or age as the cause. The results were encouraging, while also showing that a strong overall score could coexist with a missed evidence source, an unsupported explanation or incomplete containment advice.

We measured:

• Success: Identifying the attack path, assessing impact, finding persistence and recommending appropriate containment or recovery.
• Instruction adherence: Following the investigation rules and boundaries.
• Evidence rigor: Supporting conclusions with evidence rather than inventing explanations.
• Cost, tokens and time: Tracking reported token use, runtime and cost estimates where available; tokens are not directly equivalent to cost.

The transcripts also revealed how each agent investigated, used tools and summarized its findings—habits that aggregate scores alone could obscure.

Claude likes narratives, Codex hedges, Cursor pushes further

Claude: When the story outruns the evidence

Claude writes the fullest narratives and chronologies, often pursuing additional investigative leads to construct a complete picture. However, it sometimes makes claims in its summary that are stronger than its own supporting evidence, even when the detailed findings preserve the relevant qualifications.

Codex: Careful conclusions, sometimes cautious recommendations

Codex creates a clear boundary between what the evidence establishes, what it supports as an inference, and what remains unknown. It sometimes hedges so cautiously that its advice, especially around containment, becomes less clear. It also constructs Python scripts that pack many AWS CLI calls into fewer tool invocations, making its investigation look smaller by command count than it actually is.

Cursor: Aggressive validation, uneven coverage

Cursor runs quickly, uses fewer reported tokens and is the most tool-heavy. It sometimes corrects errors effectively, but also misses decisive evidence or leaves its search coverage unverified. It occasionally validates claims aggressively, including reproducing an exploit despite an inspect-only instruction. This shows a willingness to test directly, but weaker adherence to the investigation boundary.

These are tendencies in this cohort, with meaningful exceptions, not fixed traits of the three agents.

“Nothing found” can mask investigation errors

An agent reporting no evidence of activity may have completed a valid search—or encountered an investigation failure. In s1, Claude reported “no user data” after its decoding step failed. In s4, Cursor suppressed errors during a regional Lambda sweep and converted failed calls into a zero count. In s6, Cursor described the retrieved logs as containing only routine execution messages without following the response’s pagination token. That does not establish that another page contained crucial evidence, but it means the search did not justify a complete-sounding conclusion.

For teams using agents in IR, the practical recommendation is to request a short coverage statement alongside the findings: which accounts, regions, identities and time windows were checked, which evidence sources were available, and which checks failed or remain incomplete. The agent should distinguish “checked successfully and found nothing” from “could not establish.” A runbook can specify the checks, but the investigation still needs evidence that they actually completed.

Review the conclusion as carefully as the investigation

All three agents showed ways that a final report could become less accurate than the evidence behind it. Claude sometimes strengthened qualified findings into absolute claims; Codex sometimes compressed away an identity transition or an observed event; Cursor sometimes assembled real facts into an unsupported benign explanation. That matters because the explanation influences the response: treating tampering as a publishing defect can lead to recommending that the modified artifact be re-signed.

Before acting on an agent’s report, review whether its main conclusions follow from the cited evidence, whether important qualifications survived the summary, and whether its recommendations depend on unresolved assumptions. Uncertainty about who operated a credential should remain explicit, but it need not prevent recommending containment of an observed compromise. These recommendations follow from the failures we observed; we have not tested whether adding these review steps reliably eliminates them.

What a runbook does and does not do

The same runbook was provided to all skill-arm runs across the seven scenarios. It compiled AWS incident-response detection and analysis guidance, with adaptations for the available tools. Further customization, such as identifying relevant data sources and explaining the applications, might improve its usefulness, but that was not tested here.

• Increased tokens and time spent. Average reported token use rose approximately 2.3× for Claude, 3.7× for Codex and 1.5× for Cursor. Average runtime also increased, but by smaller proportions.
• Higher average success scores in the skill arm. The increase was modest for Claude and Codex and larger for Cursor. Evidence rigor improved for Claude, while Codex and Cursor’s averages remained unchanged.
• Helped correct specific evidence gaps. The clearest example was s1: two of three Cursor no-skill runs missed the S3 data-event trail, leaving downloads unconfirmed. All three skill runs read that source and confirmed the downloads.
• Changed how all three agents presented conclusions. Skill-arm reports more often included explicit limits, clearer attribution and statements about what could not be determined. Some score improvements reflected better reporting rather than new discoveries.
• Did not consistently improve investigation or judgment. Cursor still missed the s7 container in both arms and reached the same deployment-defect verdict.
• May sometimes redirect attention unhelpfully. In s4, Cursor’s skill runs devoted more attention to CloudTrail and the credential report while leaving configuration coverage incomplete. Its average score was lower, although one matched pair improved. The runbook displacing useful investigation is a hypothesis from these runs, not a demonstrated causal effect.

These comparisons use three runs per agent per arm in each scenario, so they describe observed differences rather than guaranteed runbook effects.

The seven investigation scenarios

Each scenario asked the agent to investigate and recommend next steps without changing the account. Expand a scenario for its setup and evaluation focus.