Overview
Configure retention policies for your CloudWatch Log Groups so old log data expires automatically and storage/cost exposures are reduced. The plan discovers all log groups, evaluates current retention and operational context, guides you through selecting which groups to change and the retention values to apply, then applies changes while honoring owner notifications, governance constraints, and rollback preferences. Every step records authoritative API metadata and produces machine-readable artifacts for auditability and downstream automation.
Execution Details
Assessment — inventory and discovery
Purpose: Build a complete, auditable inventory of CloudWatch Log Groups and surface any conditions that affect safe retention changes.
What is done:
- Enumerate all Log Groups (handling pagination) and capture API metadata (request IDs, timestamps, pagination tokens) for each call.
- Record identifying and operational fields per group: name, ARN when available, creation time, tags, storedBytes, metricFilterCount, subscriptionFilterCount and destinations, retentionInDays (explicit or null), kmsKeyId, and any returned policy excerpts or high-risk indicators.
- Detect and record anomalies or access issues (AccessDenied, throttling, truncated responses) including exact API errors and request IDs.
- Flag groups needing special handling with short rationales (active subscriptions, critical services inferred from tags or metrics, cross-team ownership or governance tags, archived/immutable concerns).
- Produce and persist a machine-readable inventory (JSON array) with per-call API metadata and summary metrics (total groups, infinite-retention count, retention distribution, total storedBytes, flagged counts) so downstream phases reference exact API responses and concurrency tokens.
Candidate identification — fetch and validate current retention
Purpose: Determine which groups are candidates for retention changes and compute impact advisories.
What is done:
- Read authoritative retentionInDays for every inventory entry and attach read API metadata (request IDs, timestamps).
- Validate observed values against supported CloudWatch retention options and flag non-standard values with exact returned values.
- Classify each group as candidate-for-change or not based on user-provided criteria or default heuristics (examples: no retention, retention longer than target, retention shorter than minimum), recording the rule used.
- For each candidate generate an impact advisory containing current and suggested retention, storedBytes, estimated bytes affected, subscription/destination status, and governance/owner tag notes.
- Collect owner/contact info from tags when present and note whether notification is recommended.
- Produce and persist a machine-readable candidate list and summary metrics (total candidates, active subscriptions, infinite-retention count, cumulative bytes and storage/expiry impact), including raw API responses for review before approval.
User selection and approval
Purpose: Capture explicit user authorization and constraints for applying retention settings.
What is done:
- Present inventory and candidate advisories to you and guide options for selecting which log groups to change.
- Collect a precise machine-readable approval record: explicit list of LogGroupNames to modify and the exact retentionInDays per group (either one default for all or a per-group mapping), validating entries against supported retention options and recording any allowed normalization.
- Collect operational preferences and constraints: whether to skip groups with active subscriptions, notification requirements and templates, maintenance windows or blackout windows, confirmation mode (per-change or bulk), and error-handling policy (stop-on-first-error vs continue-on-error).
- Record owner/contact instructions if notifications are required and capture your acknowledgment of impacts and compliance considerations.
- Persist the approval artifact with approver identity, timestamp, approved groups and values, constraints, change window, and error-handling choices.
Configuration — apply approved retention settings
Purpose: Safely apply retention updates while minimizing race conditions and recording full audit trails.
What is done:
- Re-validate each approved group exists and fetch live metadata (retentionInDays, subscriptions, storedBytes, tags) to detect drift; capture API metadata and compare to the original inventory.
- Skip or block updates if new blocker flags appear since approval (new subscription, owner restriction, access error), unless you provided an override; record skip reasons.
- Validate requested values against supported retention options, normalize if permitted (record original and normalized values), or fail the group update if normalization is disallowed.
- If owner notification is requested, send notifications and record delivery status and acknowledgements; respect wait-for-ack windows if specified.
- Prepare per-group update payloads including previous retention for rollback, obtain concurrency/version tokens or pre-update reads to reduce races, then call the service API to set retentionInDays.
- For each update capture full API responses, request IDs, timestamps, and HTTP status; log change entries with previous/new retention, approver identity, and notification records.
- On failures capture full error details and remediation recommendations, and proceed according to your error-handling policy; support automated rollback of previously applied changes if you requested it, or provide exact rollback payloads and manual steps.
- Produce a machine-readable change-result summary for each attempted group: outcome (success/failed/skipped/rolled-back), previous and new retention, API metadata, notification records, normalization decisions, and follow-up recommendations.
Validation — verify persistence and provide evidence
Purpose: Confirm intended retention values are persisted and produce verification artifacts for audit and remediation.
What is done:
- For every attempted or updated group re-retrieve retentionInDays and capture API responses, request IDs, and timestamps.
- Confirm actual retentionInDays matches the intended numeric value; if mismatched, record expected vs actual and any anomalies.
- Assemble per-group verification evidence including pre-change value, expected and actual retention, and API metadata for update and verification calls.
- Produce and persist a machine-readable verification report summarizing totals (attempted updates, successful verifications, mismatches/failures), per-group details, and recommended corrective actions.
- For groups not reaching the expected state, include precise remediation steps (reapply, check IAM, notify owner, schedule manual intervention) and mark manual actions required.
- If requested, include a simple follow-up monitoring plan (suggested re-check intervals) and attach verification evidence to the overall change-result summary for auditors and owners.
Artifacts produced
- Inventory JSON (per-log-group full metadata + API call metadata + summary metrics)
- Candidate list JSON with advisories and raw API reads
- Approval/authorization record (approver, selections, constraints, timestamps)
- Change-result summary (per-group outcomes, API metadata, notifications, rollback data)
- Verification report (evidence and remediation steps)
These artifacts enable repeatable review, safe concurrency checks, auditability, and automated or manual rollback as needed.