AWS Operations
Browse raw run dataOverview
Three coding-agent CLIs ran 10 live AWS operations tasks, with and without a task runbook, three repetitions each — 180 graded runs.
Tasks included
- Investigate who made an unauthorized security-group change
- Identify and release unattached Elastic IP addresses
- Investigate and fix Lambda invocation errors
- Investigate and reconcile terraform configuration drift
Agent profiles
Per-metric comparison
Success
Adherence
Cmd accuracy
Tokens
Agent time
Cost
Tasks
Overall takeaways
Methodology & grading
Three coding agents — Claude Code (claude-opus-4-8), Codex CLI (gpt-5.6-sol), and Cursor CLI (Composer 2.5 Fast) — ran ten realistic AWS operations tasks against live sandbox accounts, including incident fixes, security remediations, cost cleanups, audits, and forensics.
Test conditions
Each task was run under two conditions to learn when agents benefit from a plan and when they are better left to determine their own approach:
- Skill: The agent received a detailed, task-specific plan describing how to approach the work.
- No skill: The agent received only the goal in the prompt and decided how to complete the work on its own.
Repetitions
Each agent completed three repetitions of every task under both conditions, producing 180 graded runs in total.
Grading
Runs were graded in three ways:
- A deterministic checker inspected live cloud state after each run.
- An LLM judge graded the agent's process and final report against author-defined criteria.
- Human review examined every disputed grade.
Human review and overrides
When human review changed a machine grade, the run is labeled Manual override and includes the reason. Several of those corrections became findings in their own right; see the grading-integrity note at the end.