CloudWatch Metrics (S3/SQS) Source

Supported Data Types

📈 Metrics Select CloudWatch Metrics in the Sawmills source picker to ingest CloudWatch metrics delivered to Amazon S3. The collector polls an Amazon SQS queue for object notifications, downloads the objects, and decodes their CloudWatch Metric Streams JSON records into metrics.

CloudWatch Metric Stream (JSON)
  → Amazon Data Firehose (Direct PUT, S3 destination, uncompressed)
    → S3 object creation notification → SQS
      → Sawmills Collector → pipeline → metrics destinations

The collector makes outbound requests using its own AWS identity, such as an IAM role for a Kubernetes service account (IRSA). This source needs no inbound endpoint, listener port, or shared access key. The pipeline must include a destination that accepts metrics. For AWS resources, IAM policies, and verification steps, see How to send CloudWatch metrics via S3.

Configuration

Name	Required	Description
Name	Yes	Unique source name within Sawmills.
Region	Yes	AWS Region of the bucket and queue, for example `us-west-2`. It must match the Region in the queue URL.
S3 Bucket	Yes	Bucket name, for example `example-cloudwatch-metrics`. Use the name without `s3://` or a path.
SQS Queue URL	Yes	Full queue URL, for example `https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics`. Use the URL, not the queue ARN.
Key Prefix	No	Literal beginning of the object key, for example `cwmetrics/`. Empty means no prefix filter. This is not a wildcard or an S3 URI.

The encoding is fixed to CloudWatch Metric Streams JSON (cwmetrics); there is no format selector. Logs, traces, OTLP files, and arbitrary JSON are not supported by this source. The equivalent source configuration data is:

region: us-west-2
s3_bucket: example-cloudwatch-metrics
sqs_queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics
encoding: cwmetrics
s3_prefix: cwmetrics/ # Optional; omit for no prefix filter.

Objects outside the configured bucket or prefix are skipped, and their SQS notifications are deleted once all records in the message are processed or skipped. A wrong bucket or prefix can silently lose metrics even while the queue drains. Prefixes are literal: cwmetrics/ and /cwmetrics/ match different keys. Correcting the source later does not replay deleted notifications.

Use a dedicated queue for this source. An unrelated consumer, or another source with a different bucket or prefix, can consume and delete notifications before the intended source receives them.

AWS permissions

Grant the AWS role used by the collector pods running this source:

s3:GetObject on the bucket’s metric objects, scoped to the configured prefix when possible.
sqs:ReceiveMessage and sqs:DeleteMessage on the notification queue.

S3 also needs permission to publish notifications to the queue. See the setup guide for the queue policy, collector policy, and encryption considerations.

Processing, retries, and dropped data

Use Sawmills Collector v1.1120.0 or later for the behavior below and its drop counters. The receiver accepts native S3 object notifications, either directly or inside an SNS notification. EventBridge message bodies are not supported. A notification is deleted only when all its records have been processed or skipped; a retryable failure retains the whole message.

Condition	Behavior
Metrics accepted by the next pipeline component	Marks the object as processed.
Object outside the configured bucket or prefix	Skips the object.
Invalid SQS JSON	Deletes the message; reason `invalid_json`.
JSON that is not an S3 or SNS notification, including EventBridge	Deletes the message; reason `not_s3_notification`.
Invalid SNS payload or no embedded S3 event	Deletes the message; reason `invalid_sns_message`.
No matching decoder, decompression failure, or decoding failure	Skips the object; reason `unsupported_format`, `decompress_failed`, or `decode_failed`.
S3 download failure, other than `NoSuchKey`, or an error returned by the next pipeline component	Keeps the entire message for retry after the SQS visibility timeout, including records already processed or skipped.
Object no longer exists (`NoSuchKey`)	Treats the record as done.

Monitor these collector counters, grouped by their reason label:

otelcol_receiver_awss3_sqs_messages_dropped: unreadable or unsupported notifications.
otelcol_receiver_awss3_objects_dropped: unsupported object formats, decompression failures, and decoding failures.

Both counters increase only after SQS message deletion succeeds. Bucket and prefix mismatches are not included in these counters, so verify actual metrics arrival as well as queue consumption. See Monitor the Collector with Prometheus. Retries can process an object again. Even for successfully decoded objects, queue deletion only confirms that the next pipeline component accepted the data; it is not a guarantee that a remote destination has stored it. Check destination health and collector export errors as part of verification.

S3 object retention

The receiver never deletes S3 objects. Set an S3 lifecycle rule on the metrics prefix to expire objects according to your retention policy. Allow enough time for queue backlog, outages, and retries before expiration. An object that expires before the collector downloads it cannot be ingested from its notification.

Overview

Sawmills Agent

Collectors

Pipelines

Processors

Live Logs

Data Insights

Analytics Dashboard

How To Guides

Integrations

CLI

MCP

Administration

CloudWatch Metrics (S3/SQS) Source

Supported Data Types

Configuration

AWS permissions

Processing, retries, and dropped data

S3 object retention

​Supported Data Types

​Configuration

​AWS permissions

​Processing, retries, and dropped data

​S3 object retention

Supported Data Types

Configuration

AWS permissions

Processing, retries, and dropped data

S3 object retention