Benchmarking Coding Agents on Real AWS Operations

As part of working with different agents and models, we wanted to test them against common cloud operations to see if there are any learnings we can draw about which agents are better suited for these actions, how we can better prompt them, and what to expect in different scenarios.

We tested 3 agent harnesses paired with models (Claude Code with Opus 4.8, Codex with GPT-5.6-sol, and Cursor with Composer 2.5 Fast) against 10 tasks in a controlled AWS environment: things like identifying unused resources and clearing them, investigating web service errors, reviewing backup reports, and attributing infrastructure changes. Each agent ran each task six times (three with a detailed runbook, three with only a short prompt) for a total of 180 runs.

Our findings were very positive, but there were still some surprises. First the positive:

Success rates: 98%, 96%, 94%

All three agents completed the vast majority of tasks fully and correctly: Cursor 98%, Claude 96%, Codex 94%. For work that spans incident response, security remediation, and cost cleanup on live infrastructure, that's a higher floor than we expected going in.

Safe on writes, and that's exactly where you'd want it

From a safety perspective, 100% of runs paused to request approval before making changes when the prompt asked for it. No agent ever mutated infrastructure without stopping first.

The flip side surprised us: every point lost in the benchmark came from read-only tasks, not from changes. The mutating tasks were essentially perfect across all agents, but on the backup-jobs review, some runs silently analyzed only the first page of results, and on the VPC cleanup review, agents misclassified resources. Agents treat writes as dangerous (they pause, they verify) but treat reads as safe, and that's exactly where the silent failures happened.

Then some of the interesting learnings:

• Cursor with Composer 2.5 Fast was the best overall. The highest success rate (98%), and by far the cheapest and fastest: about a third of Claude's cost and an average of 59 seconds of active work per task versus 110s (Codex) and 121s (Claude), with no sacrifice in accuracy. It also had the fewest command errors (5.2% of commands vs 7.0% and 9.6%).
• Codex with GPT-5.6-sol scored the highest at following instructions (95%). Notably, it was the only agent with a spotless record on the S3 public-access audit, where the key constraint (fix it through Terraform, not the CLI) was implied rather than stated. Claude reached for a quick CLI fix when unguided, and Cursor quietly narrowed the audit scope; Codex did neither.
• Claude Code wrote the clearest analysis. Its reports read like an analyst's work product, with explicit rule-outs of decoys and caveats the others never raised. In the CloudTrail investigation, it was the only agent to warn that an attacker-controlled description field on a security-group rule "is not proof of intent."
• The agents work in visibly different rhythms. Claude thinks between commands which likely contributes to its higher token spend. Codex and Cursor plan, execute, and then come back to evaluate the output.
• Cursor validates constantly. Presented with a reported issue in the Lambda troubleshooting task, Claude and Codex got to work from logs and configuration; Cursor first invoked the function with the failing payload to confirm the issue was still there (in 6 out of 6 runs), then re-invoked after fixing to prove it was gone.

Skills are temperament calibration, not knowledge injection

We tested half the runs for each task and agent with a detailed plan loaded as a skill, and the other half with only a short prompt specifying the goal.

Giving the agent a detailed plan up front added cost without changing outcomes on tasks whose workflow was already well specified. On the Lambda fix, the 503 incident, and the SSH remediation, every run succeeded in both arms, and the skill just meant more tokens. But where the task left room for interpretation, the skill made a real difference: on the deliberately vague VPC cleanup review, scores went from all over the map to a perfect sweep for all three agents.

Whether a skill helps also depends on the agent, the model, and the task, which is worth understanding and measuring rather than assuming. The backup-jobs review required pagination (we used an older AWS CLI that doesn't auto-paginate), and the agents' output filtering could hide the pagination token entirely. An explicit instruction took Codex from 1 in 3 to 3 in 3 on that trap. But Cursor never needed it (6 for 6 without), and Claude's single miss actually happened with the corrected skill in hand: it confidently asserted the CLI would paginate for it, then "verified" with a query that stripped the very token it needed to see. A skill raises consistency; it doesn't fully override a confident wrong prior.

And skills cut both ways at the sentence level. An early version of one runbook contained a wrong sentence: "you can rely on the CLI's automatic pagination," which is false for our environment. Codex followed it literally and failed every run until we removed it. Combined with the one-line log-check finding above, the lesson is that agents execute runbook text faithfully in both directions: every sentence deserves review like production code.

Conclusion

Zooming out: today's coding agents are already capable and safe enough to be genuinely useful for day-to-day cloud operations. The differences that matter between them are less about raw capability and more about temperament: how fast and cheap they run, how literally they follow instructions, how clearly they report, and how much they verify along the way. Those differences are measurable, and they're worth measuring for your own workloads before picking an agent for a job. Guardrails are still needed when making changes, and careful verification is required, especially on the read side, where the silent failures happened. But the data says we can trust agents with more of the day-to-day than we expected.

On skills specifically, our finding is that agents have enough knowledge and access to resources to figure out how to do the task, and a detailed plan doesn't add capability. What it actually does is calibrate: it reins in the aggressive agent, un-hedges the cautious one, drives uniform output, and covers known edge cases. That's genuinely valuable; it's just a different value than "teaching the agent how."

---

A note on how we graded: every run was scored by a deterministic checker that inspects the live cloud state after the run, an LLM judge grading the process and final report against per-task criteria, and a human review of every disputed grade.

You can view the full report and data at https://cloudagent.io/benchmarks/aws-ops, where you can see each task's scores, prompts, and learnings, as well as more metrics across all the runs.

We plan to expand this to include more agents and models as well as different scenarios. We'd love your feedback and input!