aws-cloudwatch

AWS CloudWatch Logs

Retrieve logs from AWS CloudWatch using the AWS CLI.

Pre-flight: verify credentials

aws sts get-caller-identity

If this fails, the CLI is not configured. Tell the user to run aws configure or aws configure sso and retry. Do not proceed until credentials are confirmed.

Add --region <region> to any command below when logs are in a non-default region.

---

Operations

List log groups

# All log groups
aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table

# Filtered by prefix
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/my-service" \
  --query 'logGroups[*].logGroupName' --output table

List recent log streams

aws logs describe-log-streams \
  --log-group-name "<LOG_GROUP>" \
  --order-by LastEventTime \
  --descending \
  --max-items 10 \
  --query 'logStreams[*].{name:logStreamName,last:lastEventTimestamp}'

Fetch log events from a stream

aws logs get-log-events \
  --log-group-name "<LOG_GROUP>" \
  --log-stream-name "<LOG_STREAM>" \
  --limit 200 \
  --query 'events[*].{time:timestamp,msg:message}'

Add --start-time <epoch-ms> and --end-time <epoch-ms> to restrict the window.

Filter log events by pattern (across a group)

aws logs filter-log-events \
  --log-group-name "<LOG_GROUP>" \
  --filter-pattern "ERROR" \
  --start-time $(date -d '1 hour ago' +%s000) \
  --end-time $(date +%s000) \
  --limit 100

Pattern syntax: "ERROR" (substring), ?"ERROR" ?"WARN" (OR), { $.level = "ERROR" } (JSON field).

CloudWatch Insights query (best for large volumes)

Step 1 — Start the query:

aws logs start-query \
  --log-group-name "<LOG_GROUP>" \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message
    | filter @message like /ERROR|Exception|FATAL/
    | sort @timestamp desc
    | limit 50'

Save the returned queryId.

Step 2 — Poll for results:

aws logs get-query-results --query-id "<QUERY_ID>"

Re-run until status is Complete. Results are in the results array.

Useful Insights query patterns:

# Top error messages by frequency
fields @message
| filter @message like /ERROR/
| stats count() as hits by @message
| sort hits desc
| limit 20

# Error rate over time
fields @timestamp
| filter @message like /ERROR/
| stats count() as errors by bin(5m)

# Latency p99 (for structured logs with a duration field)
fields @timestamp, duration
| filter ispresent(duration)
| stats pct(duration, 99) as p99 by bin(1h)

---

Epoch time helpers

# 1 hour ago (seconds)
date -d '1 hour ago' +%s

# 1 hour ago (milliseconds — needed for filter-log-events)
date -d '1 hour ago' +%s000

# Specific time to epoch
date -d '2026-05-04 14:00:00' +%s

On macOS use date -v-1H +%s instead of date -d.

---

Common log group patterns

Pattern	Service type
/aws/lambda/<env>-<service>	Lambda functions
/ecs/<cluster>/<service>	ECS tasks
/aws/apigateway/<api-id>	API Gateway access logs
/aws/rds/instance/<id>/error	RDS error logs
<service>-<env>	Custom application groups

---

IAM permissions required

Action	Permission
List log groups	logs:DescribeLogGroups
List log streams	logs:DescribeLogStreams
Fetch events	logs:GetLogEvents
Filter events	logs:FilterLogEvents
Insights queries	logs:StartQuery, logs:GetQueryResults

If you hit AccessDeniedException, tell the user which permission is missing.

---

Critical Rules

• NEVER print or log AWS credentials — if they appear in output, redact them immediately
• Always use --limit when fetching events — unbounded fetches on high-volume groups are slow and expensive
• Prefer Insights for groups > 1GB — get-log-events is inefficient at scale
• Use milliseconds for filter-log-events time flags, seconds for Insights