> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sawmills.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# How to send CloudWatch metrics via S3
> Set up CloudWatch Metric Streams, Firehose, S3, SQS, and collector IAM permissions to ingest AWS metrics into Sawmills.
This guide delivers CloudWatch metrics to S3 and lets the Sawmills Collector pull them using SQS notifications. The collector uses outbound AWS API connections and its own AWS identity; no inbound collector endpoint is needed.
```text theme={null}
CloudWatch Metric Stream (JSON)
→ Amazon Data Firehose (Direct PUT, S3 destination, uncompressed)
→ S3 object creation notification → SQS
→ Sawmills Collector → pipeline → metrics destinations
```
For source fields and message handling, see [CloudWatch Metrics (S3/SQS) Source](/docs/source-cloudwatch-s3).
## Prerequisites
* A deployed Sawmills Collector. Use **v1.1120.0 or later** for the documented drop/retry behavior and drop counters.
* A pipeline with a destination that accepts metrics.
* AWS permissions to configure S3, SQS, Firehose, CloudWatch Metric Streams, and their IAM roles.
* Outbound connectivity from the collector to S3, SQS, and the AWS identity service used by its role, such as STS for IRSA.
The examples use account `123456789012`, Region `us-west-2`, bucket `example-cloudwatch-metrics`, prefix `cwmetrics/`, and queue `example-cloudwatch-metrics`. Replace these consistently with your resources. This walkthrough keeps the resources in one account and Region.
## Step 1: Create the bucket and queue
1. Create an S3 bucket for the metrics, with a dedicated object prefix such as `cwmetrics/`.
2. Create a **standard SQS queue** in the same Region as the bucket. Record its URL and ARN.
3. Use S3-managed encryption (SSE-S3) and SQS-managed encryption (SSE-SQS) for this example. If you use customer-managed KMS keys, also configure the permissions described below.
Use a queue dedicated to these metric objects. Set queue retention to cover expected outages and a visibility timeout long enough to process the objects in a received batch.
Configure native S3 notifications to a standard queue. S3 cannot deliver these
notifications directly to a FIFO queue, and the receiver drops EventBridge
message bodies. See [AWS notification
destinations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html).
## Step 2: Allow S3 to notify SQS
Add this statement to the queue's access policy, substituting your queue ARN, bucket name, and bucket owner's account ID. Preserve any other statements the queue needs.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowMetricBucketNotifications",
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-west-2:123456789012:example-cloudwatch-metrics",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:s3:::example-cloudwatch-metrics"
},
"StringEquals": { "aws:SourceAccount": "123456789012" }
}
}
]
}
```
In the bucket's **Properties → Event notifications**, add a notification with:
* **Event types**: all object creation events (`s3:ObjectCreated:*`), including multipart upload completion.
* **Prefix**: `cwmetrics/`.
* **Suffix**: leave empty; Firehose object keys do not need a `.json` extension.
* **Destination**: the SQS queue from Step 1.
If the queue uses a customer-managed KMS key, its key policy must also allow S3 to use `kms:GenerateDataKey` and `kms:Decrypt`. See [AWS permissions for notification destinations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html).
## Step 3: Grant the collector access
Attach the following IAM policy to the role used by the collector pods that run this source. This role is separate from the roles CloudWatch and Firehose use to deliver metrics.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadMetricObjects",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::example-cloudwatch-metrics/cwmetrics/*"
},
{
"Sid": "ConsumeMetricNotifications",
"Effect": "Allow",
"Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage"],
"Resource": "arn:aws:sqs:us-west-2:123456789012:example-cloudwatch-metrics"
}
]
}
```
For EKS with **IRSA**, create a role whose trust policy allows your cluster's OIDC provider and the collector's service account to assume it. Annotate that service account with the role ARN:
```yaml theme={null}
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/sawmills-cloudwatch-metrics
```
Ensure the pods running the source use that service account, including the load-balancer collector if enabled. Apply the annotation through your deployment configuration and recreate the affected pods so they receive the role credentials. Follow [AWS's IRSA setup](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) for the OIDC trust and role association.
If the S3 objects or SQS messages use customer-managed KMS keys, grant the collector role `kms:Decrypt` on the applicable keys and allow the role in the key policies. Firehose also needs the permissions required by your bucket's encryption configuration. The example policy above assumes SSE-S3 and SSE-SQS.
## Step 4: Create the Firehose stream
Create an Amazon Data Firehose delivery stream with these settings:
| Setting | Value |
| :------------------------------------------ | :------------------------------------------------------------ |
| Source | **Direct PUT** |
| Destination | **Amazon S3**, using the bucket from Step 1 |
| S3 bucket prefix | `cwmetrics/` |
| Compression | **Uncompressed** (`UNCOMPRESSED`) |
| Record transformation and format conversion | Disabled; preserve the CloudWatch Metric Streams JSON records |
Allow Firehose to create a service role with access to write to the bucket, or select an existing role with the equivalent permissions. Keep any error-output prefix outside `cwmetrics/` so error files do not enter the source. Firehose buffers records before writing objects, so account for its buffering interval when checking delivery.
## Step 5: Create the CloudWatch Metric Stream
In **CloudWatch → Metrics → Streams**, create a metric stream with:
* **Custom setup with Firehose**: select the delivery stream from Step 4.
* **Output format**: explicitly select **JSON**.
* **Metrics**: select the namespaces and metrics you want to send.
* **Service role**: create or select a role that allows CloudWatch to write to this Firehose stream.
The metric stream and Firehose stream must be in the same account and Region. See [AWS custom metric stream setup](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-setup-datalake.html).
The source expects [CloudWatch Metric Streams
JSON](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-json.html),
with one JSON object per line. Keep this format through Firehose and S3; do
not convert it to OTLP, Parquet, or a JSON array.
## Step 6: Add the source in Sawmills
Open your pipeline, add a source, and select **CloudWatch Metrics**. Set:
| Field | Example |
| :------------ | :---------------------------------------------------------------------------- |
| Name | `cloudwatch-metrics` |
| Region | `us-west-2` |
| S3 Bucket | `example-cloudwatch-metrics` |
| SQS Queue URL | `https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics` |
| Key Prefix | `cwmetrics/` |
Connect a destination that accepts metrics, then deploy the pipeline to the collector.
Match the bucket and literal prefix to the actual Firehose objects. A prefix
of `/cwmetrics/` will not match keys beginning with `cwmetrics/`. Objects
outside the configured bucket or prefix are skipped and their notifications
deleted. The queue can drain without metrics arriving.
## Step 7: Verify delivery and configure retention
1. Confirm that the CloudWatch Metric Stream is running and includes metrics currently being emitted.
2. After Firehose's buffering interval, check for new S3 objects under `cwmetrics/`. Inspect a sample object's contents for newline-delimited CloudWatch JSON.
3. Check SQS monitoring for incoming and deleted messages. A low queue depth can be normal when the collector consumes quickly; it does not prove successful ingestion.
4. Check collector logs for `Processing new S3 object`, authentication/download errors, and downstream errors. Monitor the [drop counters](/docs/source-cloudwatch-s3#processing-retries-and-dropped-data).
5. Verify recent metrics at the pipeline's destination, including their expected namespace and dimensions. Allow for CloudWatch delivery, Firehose buffering, and collector processing.
6. Add an S3 lifecycle expiration rule scoped to `cwmetrics/`. Choose retention long enough to cover backlog and retries. The collector never deletes S3 objects, and lifecycle expiration does not wait for ingestion.
Enabling notifications does not scan existing objects. This source reads the objects named by the notifications it receives; changing a filter does not replay notifications that have already been deleted.
## Troubleshooting
| Symptom | Check |
| :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| No new objects in S3 | Metric stream state and metric selection; Firehose destination, write role, and delivery errors. |
| Objects arrive but no SQS notifications | Bucket event prefix, all object creation events, queue Region, queue access policy, and any KMS key policy. |
| Queue backlog grows | Collector deployment, outbound connectivity, the role used by the pods running the source, `ReceiveMessage`/`DeleteMessage` permissions, and downstream health. |
| S3 `AccessDenied` | Collector `GetObject` scope, bucket policy, and any KMS decrypt permission. Download failures retain the message for retry. |
| Queue drains but no metrics arrive | Exact bucket/prefix match, CloudWatch JSON output, uncompressed Firehose objects, a metrics-capable destination, and drop counters. Bucket/prefix skips do not increment the drop counters. |
| `not_s3_notification` or `invalid_sns_message` drops | Use native S3 notifications, directly or correctly wrapped by SNS. EventBridge bodies are unsupported. |
| `invalid_json`, `unsupported_format`, `decompress_failed`, or `decode_failed` drops | Inspect the notification or object format; remove transformations and unrelated objects from the ingestion prefix. |
| Repeated delivery or missing objects | Check processing time against SQS visibility timeout, retries, and whether lifecycle expiration deleted objects before ingestion. An S3 `NoSuchKey` record is treated as done. |