Skip to main content
This guide delivers CloudWatch metrics to S3 and lets the Sawmills Collector pull them using SQS notifications. The collector uses outbound AWS API connections and its own AWS identity; no inbound collector endpoint is needed.
For source fields and message handling, see CloudWatch Metrics (S3/SQS) Source.

​
Prerequisites

  • A deployed Sawmills Collector. Use v1.1120.0 or later for the documented drop/retry behavior and drop counters.
  • A pipeline with a destination that accepts metrics.
  • AWS permissions to configure S3, SQS, Firehose, CloudWatch Metric Streams, and their IAM roles.
  • Outbound connectivity from the collector to S3, SQS, and the AWS identity service used by its role, such as STS for IRSA.
The examples use account 123456789012, Region us-west-2, bucket example-cloudwatch-metrics, prefix cwmetrics/, and queue example-cloudwatch-metrics. Replace these consistently with your resources. This walkthrough keeps the resources in one account and Region.

​
Step 1: Create the bucket and queue

  1. Create an S3 bucket for the metrics, with a dedicated object prefix such as cwmetrics/.
  2. Create a standard SQS queue in the same Region as the bucket. Record its URL and ARN.
  3. Use S3-managed encryption (SSE-S3) and SQS-managed encryption (SSE-SQS) for this example. If you use customer-managed KMS keys, also configure the permissions described below.
Use a queue dedicated to these metric objects. Set queue retention to cover expected outages and a visibility timeout long enough to process the objects in a received batch.
Configure native S3 notifications to a standard queue. S3 cannot deliver these notifications directly to a FIFO queue, and the receiver drops EventBridge message bodies. See AWS notification destinations.

​
Step 2: Allow S3 to notify SQS

Add this statement to the queue’s access policy, substituting your queue ARN, bucket name, and bucket owner’s account ID. Preserve any other statements the queue needs.
In the bucket’s Properties → Event notifications, add a notification with:
  • Event types: all object creation events (s3:ObjectCreated:*), including multipart upload completion.
  • Prefix: cwmetrics/.
  • Suffix: leave empty; Firehose object keys do not need a .json extension.
  • Destination: the SQS queue from Step 1.
If the queue uses a customer-managed KMS key, its key policy must also allow S3 to use kms:GenerateDataKey and kms:Decrypt. See AWS permissions for notification destinations.

​
Step 3: Grant the collector access

Attach the following IAM policy to the role used by the collector pods that run this source. This role is separate from the roles CloudWatch and Firehose use to deliver metrics.
For EKS with IRSA, create a role whose trust policy allows your cluster’s OIDC provider and the collector’s service account to assume it. Annotate that service account with the role ARN:
Ensure the pods running the source use that service account, including the load-balancer collector if enabled. Apply the annotation through your deployment configuration and recreate the affected pods so they receive the role credentials. Follow AWS’s IRSA setup for the OIDC trust and role association. If the S3 objects or SQS messages use customer-managed KMS keys, grant the collector role kms:Decrypt on the applicable keys and allow the role in the key policies. Firehose also needs the permissions required by your bucket’s encryption configuration. The example policy above assumes SSE-S3 and SSE-SQS.

​
Step 4: Create the Firehose stream

Create an Amazon Data Firehose delivery stream with these settings: Allow Firehose to create a service role with access to write to the bucket, or select an existing role with the equivalent permissions. Keep any error-output prefix outside cwmetrics/ so error files do not enter the source. Firehose buffers records before writing objects, so account for its buffering interval when checking delivery.

​
Step 5: Create the CloudWatch Metric Stream

In CloudWatch → Metrics → Streams, create a metric stream with:
  • Custom setup with Firehose: select the delivery stream from Step 4.
  • Output format: explicitly select JSON.
  • Metrics: select the namespaces and metrics you want to send.
  • Service role: create or select a role that allows CloudWatch to write to this Firehose stream.
The metric stream and Firehose stream must be in the same account and Region. See AWS custom metric stream setup.
The source expects CloudWatch Metric Streams JSON, with one JSON object per line. Keep this format through Firehose and S3; do not convert it to OTLP, Parquet, or a JSON array.

​
Step 6: Add the source in Sawmills

Open your pipeline, add a source, and select CloudWatch Metrics. Set: Connect a destination that accepts metrics, then deploy the pipeline to the collector.
Match the bucket and literal prefix to the actual Firehose objects. A prefix of /cwmetrics/ will not match keys beginning with cwmetrics/. Objects outside the configured bucket or prefix are skipped and their notifications deleted. The queue can drain without metrics arriving.

​
Step 7: Verify delivery and configure retention

  1. Confirm that the CloudWatch Metric Stream is running and includes metrics currently being emitted.
  2. After Firehose’s buffering interval, check for new S3 objects under cwmetrics/. Inspect a sample object’s contents for newline-delimited CloudWatch JSON.
  3. Check SQS monitoring for incoming and deleted messages. A low queue depth can be normal when the collector consumes quickly; it does not prove successful ingestion.
  4. Check collector logs for Processing new S3 object, authentication/download errors, and downstream errors. Monitor the drop counters.
  5. Verify recent metrics at the pipeline’s destination, including their expected namespace and dimensions. Allow for CloudWatch delivery, Firehose buffering, and collector processing.
  6. Add an S3 lifecycle expiration rule scoped to cwmetrics/. Choose retention long enough to cover backlog and retries. The collector never deletes S3 objects, and lifecycle expiration does not wait for ingestion.
Enabling notifications does not scan existing objects. This source reads the objects named by the notifications it receives; changing a filter does not replay notifications that have already been deleted.

​
Troubleshooting