# Sawmills Changelog
Source: https://docs.sawmills.ai/changelog/changelog
All Sawmills product updates, new features, improvements, and bug fixes organized by version and release date for the telemetry management platform
## Changelog
Welcome to the changelog! Here, you'll find information about new features, fixes, and updates for Sawmills.
## Unreleased
### New Relic Infrastructure Agent
* **New Relic Infrastructure Agent:** New Relic sources can now receive Infrastructure agent host, process, and Kubernetes samples. Sawmills proxies the agent control-plane requests, applies pipeline processors to metric samples, and restores the native format for the [New Relic destination](/docs/destination-new-relic). See the [Infrastructure agent setup guide](/docs/guide-send-data-from-newrelic-infrastructure-agent).
## \[0.55] - 2026-10-05
### New Features
* **Issue Webhooks:** You can now send Issue events to your own webhook receivers. Receivers get Issue lifecycle events, deliveries are retried automatically, and you can test a receiver before you save it
* **Collector Worker Capacity Controls:** Collectors now support per-collector worker capacity settings, including a maximum number of worker replicas, so you can size each collector independently
### Improvements
* **Coralogix Alert Quotas:** The Coralogix alert integration now keeps your configured alert request quota settings instead of resetting them
* **More Accurate Insights:** Insights now evaluate messages per service. Services with missing totals are classified by their recent activity, which reduces inaccurate findings
### Bug Fixes
* **Stuck Collector Deployments:** Deployments that reach a Helm wait timeout now resolve to a final status. Failed deployments no longer block later deployments to the same collector
* **GCP Pub/Sub Sources:** Each GCP Pub/Sub source now gets its own receiver, so multiple Pub/Sub sources in one pipeline no longer conflict
* **S3 Recovery Settings:** Collector configurations that include S3 recovery settings now deploy correctly and keep their auto-completed deployment attributes
## \[0.54] - 2026-05-06
### New Features
* **Manual Deployment Profiles:** Added manual deployment profiles to the [Installation Guide](/docs/install-guide) and [Manual Deployment](/docs/manual-collector-deployment) guide so environments that require manual rollouts can pick direct chart, umbrella chart, or Helmsman-compatible values and download tailored Helm install artifacts instead of editing commands by hand
* **Syslog Auto Mode:** The [Syslog source](/docs/source-syslog) now supports auto mode, so it automatically detects RFC 3164 vs RFC 5424 messages instead of requiring you to configure the format upfront
* **S3 Tables Advanced Tuning:** The [S3 Tables destination](/docs/destination-s3-tables) now exposes advanced tuning options for partitioned Iceberg writes, giving you finer control over throughput and file sizing
* **Active Processors on Metrics:** The metrics view now shows which processors are actively affecting each metric, with a one-click jump into the processor for editing
### Improvements
* **CSV Enrichment Auto-Refresh:** The [Lookup processor](/docs/processor-lookup) now auto-refreshes CSV files from S3 when the source object changes — no manual reload needed
* **Drop Metrics — Conditions Only:** The [Drop Metric Datapoints processor](/docs/processor-drop-metric-datapoints) now accepts rules with conditions only (no name pattern required), making cleanup rules easier to express
* **Label Aggregation Validation:** The [Label Aggregation processor](/docs/processor-label-aggregation) now validates uniqueness in real time and prevents duplicate Label Aggregation processors in the same pipeline, so misconfigurations are caught before save
* **Processor Search by Metric Name:** Processor search now matches metric names in addition to processor names, so you can quickly find every processor touching a given metric. The Action column also shows affected metric names directly in the table
* **Insight Digest Email:** Redesigned the insight digest email for better readability and prioritization of high-impact findings
* **Default-Group Email Alerts:** Org admins now receive email alerts for the default [Ownership Group](/docs/ownership-groups) by default, so important notifications are not missed before groups are configured
* **Datadog Source Throughput:** [Datadog source](/docs/source-datadog) configurations using both logs and metrics receivers now report throughput correctly across both
* **Datadog Logs Smoke Test:** Added a Datadog logs smoke test you can trigger from the collector setup flow to confirm log ingestion end-to-end before committing a deployment
* **Collector API Key Visibility:** Collector pages now display and let you copy the collector API key directly, simplifying re-installation and rotation
* **AWS S3 Marshaler Names:** The [AWS S3 destination](/docs/destination-aws-s3) now accepts the more readable `ndjson` and `parquet` aliases for output format, alongside existing values
* **Auto Helm Upgrade on Chart Changes:** Pipeline deployments now auto-trigger a Helm upgrade when the underlying chart version changes, removing a manual step during collector upgrades
* **Collector Resource Base Name:** Manual collector deployments now document how `resourceBaseName` keeps collector Deployments, Services, headless Services, load balancer routing, and KEDA scaler addresses stable when the collector chart is embedded in an umbrella chart
* **CSV Enrichment UX:** The Lookup processor no longer flags rows with no match as errors in the UI — misses are now treated as expected behavior
### Bug Fixes
* **Live Tail Labels:** Live Tail raw-data hint label and tooltip now reflect the active telemetry type (logs, metrics, traces) instead of a static value
* **Live Tail Pipeline Selector:** Live Tail now uses the deployed pipeline selector consistently, preventing stale pipeline selections after deploys
* **Telemetry Groups:** Cleaned up the Slack channel tooltip styling on the [Ownership Groups](/docs/ownership-groups) page
* **Processor Modal Navigation:** Closing the processor modal from the metrics view now returns you to the metrics page instead of an unexpected route
* **Slack Test Notifications:** Fixed a regression where Slack test notifications could fail to send from notification settings
* **Label Aggregation Edge Cases:** Fixed glob/wildcard handling, delta metric start-time truncation, and aggregation function constraints in the [Label Aggregation processor](/docs/processor-label-aggregation)
* **Drop Metrics Wildcards:** Fixed glob wildcard matching for metric names in the [Drop Metric Datapoints processor](/docs/processor-drop-metric-datapoints)
* **Dashboard:** Fixed New Relic metric view defaults and vendor usage availability on the [dashboard](/docs/dashboard)
* **Insights Email Routing:** Fixed an issue where qualified insight status was not always trusted when routing notifications
* **KEDA Scaler Addressing:** Updated collector autoscaling guidance to avoid hardcoded namespaces for the KEDA external scaler endpoint
## \[0.53] - 2026-04-20
### New Features
* **Sawmills Agent:** Introduced the [Sawmills Agent](/docs/sawmills-agent-overview), a conversational AI assistant for telemetry investigation, automation, and team collaboration. You can ask the agent questions about your telemetry data & processors, take actions on insights, manage processors and hand off conversations to teammates. Key capabilities include [conversations](/docs/sawmills-agent-conversations), [collaboration](/docs/sawmills-agent-collaboration), [memory](/docs/sawmills-agent-memory), [skills](/docs/sawmills-agent-skills), [tasks](/docs/sawmills-agent-tasks), [notifications](/docs/sawmills-agent-notifications), and [role-based access](/docs/sawmills-agent-roles)
* **Agent Slack Integration:** Use the Sawmills Agent directly in [Slack](/docs/sawmills-agent-slack) via DMs, channel mentions, and threaded conversations — no context-switching required
* **Instrumentation Reviewer (closed beta):** Added the [Instrumentation Reviewer](/docs/sawmills-agent-instrumentation-reviewer), which automatically reviews telemetry-related code changes in GitHub PRs and GitLab MRs before merge. Catches issues like high-cardinality fields, PII exposure, and noisy logs
* **GitHub & GitLab Integrations:** Added [GitHub](/docs/github-integration) and [GitLab](/docs/gitlab-integration) integrations to connect your repositories for instrumentation review workflows
* **S3 Tables Destination:** Added [S3 Tables destination](/docs/destination-s3-tables) support, allowing you to send logs to AWS S3 Tables in Iceberg format with Snowflake query integration
* **Ownership Groups:** Added [Ownership Groups](/docs/ownership-groups) to define team-based telemetry ownership boundaries with per-group notification channels (Email, Slack, Webhook)
* **Lookup Processor:** Added the [Lookup processor](/docs/processor-lookup), which enriches logs by matching an attribute against a CSV file (from S3 or shared filesystem) and adding matching columns as new fields
* **Drop Metric Datapoints Processor:** Added the [Drop Metric Datapoints processor](/docs/processor-drop-metric-datapoints) to remove specific metric datapoints based on name patterns and label filters, helping reduce metric volume
* **Aggregate Metric Labels Processor:** Added the [Label Aggregation processor](/docs/processor-label-aggregation) to reduce metric cardinality by removing selected labels and combining values using aggregation functions
* **Sawmills CLI:** Added the [`sm` CLI tool](/docs/sawmills-cli), installable via Homebrew, for working with Sawmills from the terminal
* **Sawmills MCP Server:** Added a [hosted MCP server](/docs/sawmills-mcp) so you can connect AI coding tools like Claude Code and OpenAI Codex to Sawmills
* **Datadog Integration:** Added the [Datadog integration](/docs/datadog-integration), which validates API and Application Key permissions and enables vendor usage references in the dashboard
* **New Relic Integration:** Added the [New Relic integration](/docs/newrelic-integration), which validates User API Key permissions and enables vendor usage references in the dashboard
### Improvements
* **Analytics Dashboard:** The [dashboard](/docs/dashboard) now includes a Metrics view with time-series counts, volume, cardinality drivers, and data reduction percentages. When Datadog or New Relic integrations are configured, you also get vendor usage references showing where each metric is used in dashboards and alerts
* **Enrichment Processor:** The Lookup processor now supports [S3 as a source](/docs/processor-lookup) for CSV enrichment files, in addition to shared filesystem paths
* **Datadog Source:** The [Datadog source](/docs/source-datadog) now accepts metrics payloads sent directly via the Datadog HTTP intake API — no Datadog Agent required
* **Label Aggregation Processor:** Expanded supported metric types by vendor, now generally available (no longer alpha)
* **Splunk HEC Source:** Added source type selection support for the [Splunk HEC source](/docs/source-splunk-hec)
## \[0.52] - 2026-01-12
### New Features
* **Agent:** Added an Agent experience in the UI (including session history visibility)
* **Live Tail:** Added metrics Live Tail support (metrics display components and telemetry-type switching improvements)
* **Processors:** Added Enrichment Processor support (including CSV-based enrichment workflows)
* **Processors:** Added Drop Metrics processor
### Improvements
* **Processors:** Improved truncate attribute UX (multi-select and safer switching behavior)
* **UX/UI:** Improved attribution display in agent session history (show names consistently)
* **Stability:** Reliability improvements across collectors/pipelines validation and deployments
### Bug Fixes
* **Processors:** Fixed save/edit edge cases and attribute-key handling errors in processor configuration
* **UX/UI:** Fixed UI rendering issues and interaction bugs in the newest screens
## \[0.51] - 2025-12-31
### New Features
* **Integrations:** Added New Relic metric receiver and exporter support
* **Collectors:** Added self-managed collector state reporting (visibility into collector management mode)
### Improvements
* **Collectors:** Improved load balancer behavior and retry handling for higher reliability under load
* **Collectors:** Improved hot reload behavior across more signals and configurations
* **Deployments:** Improved validation messaging and “deployment required” logic to reduce false blockers
### Bug Fixes
* **Collectors:** Fixed deployment validation and configuration edge cases
* **Pipelines:** Fixed issues where large numbers of pipelines/processors could fail to load correctly in the UI
## \[0.50] - 2025-11-30
### New Features
* **Processors:** Added Parse Logs processor (including Grok-style parsing support)
* **Processors:** Added Logs-to-Metrics / Signal-to-Metrics processing support
* **Processors:** Added processor reordering support (UI + backend sequencing)
* **Auth:** Added user API key management and token exchange capabilities
### Improvements
* **Processors:** Added LLM-assisted generation for patterns (e.g., regex/parsing helpers) and improved validation feedback
* **Processors:** Added support for “truncate” mode in Modify Attributes (including multi-select and advanced options)
* **UX/UI:** Improved processor/insights pickers (pipeline dropdowns, hierarchy handling, clearer selection behavior)
### Bug Fixes
* **Auth:** Fixed SSO/OAuth redirect issues and sign-in edge cases
* **Processors:** Fixed create/edit ordering issues and Live Tail filtering issues
## \[0.49] - 2025-10-31
### New Features
* **Notifications:** Added Insights notifications (including Slack delivery options, such as threading)
* **Notifications:** Added webhook delivery hardening and controls
### Improvements
* **Insights:** Added caching for insight results to speed up repeated review workflows
* **Insights:** Added more granular time filtering (including 1-hour views) for faster investigations
* **Insights:** Improved qualification logic and presentation (percent-based views and clearer thresholds)
### Bug Fixes
* **Notifications:** Fixed notification formatting and delivery edge cases
* **Insights:** Fixed “NaN% / missing denominator / no-signal” display and calculation issues
## \[0.48] - 2025-09-30
### New Features
* **Integrations:** Added Coralogix receiver support (including fallback configuration)
* **Integrations:** Added Google Cloud Storage (GCS) exporter support
* **Integrations:** Added Elastic/Elasticsearch receiver support (including fallback options)
* **Integrations:** Added Sumo Logic destination support
* **Integrations:** Added Syslog as a supported data source
### Improvements
* **Collectors:** Improved hot reload reliability and deployment validation behaviors
* **Metrics:** Improved metrics pipeline/gateway stability and operational configuration
### Bug Fixes
* **Integrations:** Fixed configuration and validation issues across newly added integrations
* **UX/UI:** Fixed integration form issues and deployment-state edge cases shown in the UI
## \[0.47] - 2025-08-31
### New Features
* **Processors:** Introduced a Global Processors page for managing processors across pipelines/destinations
* **Activity Log:** Added activity logs for processor changes and system actions (where applicable)
### Improvements
* **Processors:** Added richer processor tables (type, owner, last updated, usage/impact visibility)
* **Processors:** Improved navigation between affected processors, Live Tail, and processor details
* **UX/UI:** Improved empty states and discoverability of actions (tooltips, clearer labels)
### Bug Fixes
* **Processors:** Fixed issues with processor list refresh, table rendering, and “active” state handling
* **UX/UI:** Fixed modal/view-mode bugs (scrollbars, alignment, and action button behavior)
## \[0.46] - 2025-07-31
### New Features
* **Dashboard:** Released Dashboard v2 with filters, grouping, sparklines, and improved tables
* **Dashboard:** Added deployment markers so you can correlate telemetry changes with rollouts
### Improvements
* **Dashboard:** Improved chart accuracy and readability (bucket sizing, tooltips, axis formatting)
* **Dashboard:** Increased resolution for “last 24h” views and improved Sent/Received/Reduced comparisons
* **Performance:** Reduced unnecessary UI queries and improved page responsiveness
### Bug Fixes
* **Dashboard:** Fixed chart calculation and rendering issues (including “N/A” and rounding edge cases)
* **UX/UI:** Fixed filtering UI bugs (dropdown selection, search behavior, scrolling)
## \[0.45] - 2025-06-30
### New Features
* **Insights:** Added audit/activity history in the Insights experience
* **Insights:** Added the ability to view impact by volume or by message count
* **Processors:** Introduced a redesigned Live Tail + Simulation experience while editing processors
### Improvements
* **Insights:** Updated default timeframes to focus on recent impact (24h) and improved insight status reporting
* **Processors:** Improved simulation validation and reduced “re-run” / state reset friction during editing
* **UX/UI:** General UX polish across processor editing and insight review flows
### Bug Fixes
* **Processors:** Fixed simulation and Live Tail edge cases (loading, filtering, empty states)
* **UX/UI:** Fixed form validation and layout issues across key configuration screens
## \[0.44] - 2025-04-25
### New Features
* **Pipelines:** Added Live Tail that lets you view live logs from various points in the pipeline
* **Pipelines:** Added support for saving in Parquet format via the S3 destination
* **Pipelines:** Added bypass mechanism (per pipeline destination) based on health checks
### Improvements
* **Collector:** Improved performance of Datadog-based filters
* **Pipelines:** Added OTLP authentication support
## \[0.42] - 2025-04-07
### New Features
* **Pipelines:** Added Elastic destination support
### Improvements
* **Pipelines:** Improved filter and search
## \[0.25.0] - 2025-02-20
### New Features
* **Pipelines:** Added support for Loki basic auth to Loki destination
* **Pipelines:** Added Prometheus Remote Write support - source & destination
* **Pipelines:** Added support for Coralogix Units
* **Pipelines:** Added an option to pass the X-Scope-Org-Id header for LGTM
### Improvements
* **Pipelines:** Added view mode to all parts of live pipeline
* **General:** Added support to pass org ID into URL and replace with slug
* **Collector:** Enhanced API key management with advanced features
## \[0.24.47] - 2025-02-10
### New Features
* **Pipelines:** Added an option to encode AWS S3 data into Datadog format
* **Pipelines:** Added support in attribute selector for Body (String)
* **Pipelines:** Added support in attribute selector for Severity
## \[0.23.30] - 2025-01-21
### New Features
* **Pipelines:** Added probabilistic sampler to the filter processor. The filter processor can now sample logs
* **Pipelines:** Added masking processor to mask sensitive information based on predefined patterns or custom ones
* **Pipelines:** Added the ability to deploy a pipeline to multiple collectors
* **Pipelines:** Added support for Coralogix OTLP & Coralogix format destinations
* **Collectors:** Added collector page with basic info, pipelines running, and installation instructions
* **Collectors:** Added collector status: Installed (Idle), Running (Active), Error, Disconnected
* **Collectors:** Added the ability to detach (uninstall) a pipeline from a collector
### Bug Fixes
## \[0.23] - 2025-01-10
### Bug Fixes
* **Pipelines:** Fixed issue when a user inputs incorrect Datadog API key
* **Collectors:** Fixed issue for installing Sawmills collector, added creation of namespace before adding a secret
### Improvements
* **Collectors:** Changed Helm install command to support external k8s secret
## \[0.22.17] - 2025-01-08
### New Features
* **Pipelines:** Added processor that can drop multiple attributes from Resources, Attributes, Body
* **Pipelines:** Added a processor that can insert, modify, upsert attributes across Resources, Attributes, Body
* **Pipelines:** Added batch processor to allow users to configure how to batch messages (logs, traces, metrics)
* **Pipelines:** Added Splunk HEC source and destination
* **Pipelines:** Added Metrics support for Datadog
### Improvements
* **Pipelines:** Added "Skip TLS Verification" toggle to Splunk destination
## \[0.21.11] - 2024-12-26
### Bug Fixes
* **Pipelines:** Fixed rendering issue when adding a new processor to a pipeline
## \[0.21.9] - 2024-12-24
### Improvements
* **Pipelines:** Added Next/Previous buttons to processor details panel
* **Pipelines:** UI/UX improvements for pipeline editing
## \[0.21.9] - 2024-12-10
### New Features
* **Pipelines:** Added multi-pipeline support
* **Pipelines:** Added support for Datadog metrics
### Improvements
* **Pipelines:** UI/UX improvements for pipeline editing
## \[0.21] - 2024-12-06
### New Features
* **Pipelines:** Added Live and Draft mode for pipelines
* **Pipelines:** Live mode shows telemetry data analytics
* **Collectors:** Collector page has a new installation page to add new collectors
### Bug Fixes
* **Pipelines:** When deleting a filter, the system requires a confirmation modal
* **Pipelines:** Rendering issue when adding a new processor
## \[0.19] - 2024-11-26
### New Features
* **Pipelines:** Processor page can now support many processors with search and filter
### Bug Fixes
* **Pipelines:** Processor page can now support many processors with search and filter
### Improvements
* **Collectors:** Performance improvement to the Sawmills collector that reduces the collector memory footprint
# API Keys
Source: https://docs.sawmills.ai/docs/api-keys
Create, manage, and revoke user API keys for programmatic authentication with the Sawmills platform using JWT tokens and REST API calls.
API keys provide a secure way to authenticate with the Sawmills platform. This guide covers how to create, manage, and use API keys for programmatic access to Sawmills services.
## Overview
Sawmills supports two types of API keys:
* **User API Keys**: Personal API keys that can be created by individual users for their own use
* **Organization API Keys**: System-level API keys for organization-wide access (currently only used by Sawmills itself)
This documentation focuses on **User API Keys**, which are the most commonly used for individual authentication.
## Obtaining Your JWT Token
Before creating API keys, you need to obtain a JWT token from the Sawmills UI. This token is used to authenticate your requests to the API.
### Method 1: Using Chrome DevTools
1. **Log in to Sawmills**: Open [https://app.sawmills.ai](https://app.sawmills.ai) in Chrome
2. **Open DevTools**: Press `F12` or right-click and select "Inspect"
3. **Go to Network Tab**: Click on the "Network" tab in DevTools
4. **Filter Requests**: In the filter box, type "api" to see API requests
5. **Find a Request**: Look for any request to `api.sawmills.ai` (you may need to navigate around the UI to trigger API calls)
6. **Check Headers**: Click on the request and look at the "Request Headers" section
7. **Copy the Token**: Find the `Authorization` header and copy the JWT token (the part after "Bearer ")
### Method 2: Using Application Storage
1. **Log in to Sawmills**: Open [https://app.sawmills.ai](https://app.sawmills.ai) in Chrome
2. **Open DevTools**: Press `F12` or right-click and select "Inspect"
3. **Go to Application Tab**: Click on the "Application" tab
4. **Check Local Storage**: In the left sidebar, expand "Local Storage" and click on the Sawmills domain
5. **Find Token**: Look for keys containing "token", "jwt", or "auth" and copy the JWT value
**Token Expiration**: JWT tokens expire after a certain period. If you get authentication errors, you may need to refresh the page and obtain a new token.
## Creating a User API Key
### Prerequisites
* You must be authenticated with a valid Sawmills account
* You must have appropriate permissions to create API keys in your organization
* You need a JWT token from the Sawmills UI (see [Obtaining Your JWT Token](#obtaining-your-jwt-token) below)
### API Endpoint
```http theme={null}
POST /v1/user-api-keys
```
### Request Body
```json theme={null}
{
"label": "My API Key"
}
```
**Parameters:**
* `label` (optional): A human-readable name for your API key (max 255 characters)
### Response
```json theme={null}
{
"api_key": {
"id": "123e4567-e89b-12d3-a456-426614174000",
"organization_id": "org_123",
"user_id": "user_456",
"label": "My API Key",
"status": "ACTIVE",
"created_at": "2024-01-15T10:30:00Z",
"updated_at": "2024-01-15T10:30:00Z",
"last_used_at": null,
"value": "sk_test.abc123def456.xyz789uvw012"
},
"opaque_key": "sk_test.abc123def456.xyz789uvw012"
}
```
**Important**: The `opaque_key` and `value` fields are only returned once when the API key is created. Store this value securely as it cannot be retrieved later.
### Example: Creating an API Key with cURL
```bash theme={null}
curl -X POST "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"label": "Production API Key"
}'
```
**JWT Token**: Replace `YOUR_JWT_TOKEN` with the JWT token you obtain from the Sawmills UI. See [Obtaining Your JWT Token](#obtaining-your-jwt-token) for instructions.
### Example: Creating an API Key with JavaScript
```javascript theme={null}
const response = await fetch('https://api.sawmills.ai/v1/user-api-keys', {
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_JWT_TOKEN', // Replace with JWT from UI
'Content-Type': 'application/json'
},
body: JSON.stringify({
label: 'My Application Key'
})
});
const data = await response.json();
console.log('API Key created:', data.opaque_key);
```
## Managing API Keys
### Listing Your API Keys
```http theme={null}
GET /v1/user-api-keys
```
**Query Parameters:**
* `page` (optional): Page number for pagination (default: 1)
* `limit` (optional): Number of keys per page (default: 20, max: 100)
* `include_revoked` (optional): Include revoked keys in results (default: false)
### Getting a Specific API Key
```http theme={null}
GET /v1/user-api-keys/{id}
```
### Revoking an API Key
```http theme={null}
DELETE /v1/user-api-keys/{id}
```
Revoking an API key immediately invalidates it and prevents any further use. This action cannot be undone.
## Security Best Practices
### Storage
* Store API keys in environment variables or secure key management systems
* Never commit API keys to version control
* Use different API keys for different environments (development, staging, production)
### Rotation
* Regularly rotate your API keys
* Use short-lived keys when possible
* Revoke unused or compromised keys immediately
## Error Handling
### Common Error Responses
**401 Unauthorized**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "user not authenticated"
}
```
**403 Forbidden**
```json theme={null}
{
"code": "PERMISSION_DENIED",
"message": "insufficient permissions"
}
```
**404 Not Found**
```json theme={null}
{
"code": "NOT_FOUND",
"message": "api key not found"
}
```
**400 Bad Request**
```json theme={null}
{
"code": "INVALID_ARGUMENT",
"message": "invalid request parameters"
}
```
## Troubleshooting
### API Key Not Working
1. **Check the format**: Ensure your API key follows the correct format
2. **Verify environment**: Make sure you're using the correct environment prefix
3. **Check status**: Ensure the API key is active and not revoked
4. **Verify scopes**: Confirm the API key has the required permissions
### Common Issues
**"Invalid API key format"**
* Check that the API key is correctly formatted
* Verify the key is complete and not truncated
**"API key revoked"**
* The API key has been revoked and cannot be used
* Create a new API key to continue
**"Invalid API key credentials"**
* The API key is malformed or corrupted
* Verify the key was copied correctly without extra characters
## Next Steps
Once you have created an API key, you can use it to obtain access tokens for authenticating with other Sawmills services. See [Obtaining Tokens](/docs/tokens) for more information.
# Authentication Quick Reference
Source: https://docs.sawmills.ai/docs/authentication-quick-reference
Copy-paste curl commands for creating API keys, listing active keys, revoking credentials, and exchanging keys for access tokens in Sawmills.
## API Key Commands
### Create API Key
```bash theme={null}
curl -X POST "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"label": "My Key"}'
```
**JWT Token**: Get `YOUR_JWT_TOKEN` from Chrome DevTools → Network tab → Find API request → Copy Authorization header value (after "Bearer ").
### List API Keys
```bash theme={null}
curl -X GET "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer YOUR_JWT_TOKEN"
```
### Revoke API Key
```bash theme={null}
curl -X DELETE "https://api.sawmills.ai/v1/user-api-keys/{key_id}" \
-H "Authorization: Bearer YOUR_JWT_TOKEN"
```
## Token Commands
### Exchange API Key for Token
```bash theme={null}
curl -X POST "https://api.sawmills.ai/v1/token" \
-H "Authorization: ApiKey sk_test.abc123def4.xyz789uvw0123456789abcdef" \
-H "Content-Type: application/json" \
-d '{}'
```
### Use Token for API Call
```bash theme={null}
curl -X GET "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
```
## Authentication Headers
| Purpose | Header Format | Example |
| - | - | - |
| Create/Manage API Keys | `Authorization: Bearer {jwt}` | `Authorization: Bearer eyJhbGciOiJSUzI1NiIs...` |
| Exchange for Token | `Authorization: ApiKey {key}` | `Authorization: ApiKey sk_test.abc123def4.xyz789uvw0123456789abcdef` |
| Make API Calls | `Authorization: Bearer {token}` | `Authorization: Bearer eyJhbGciOiJSUzI1NiIs...` |
## Common Error Codes
| Code | Meaning | Solution |
| - | - | - |
| `UNAUTHENTICATED` | Invalid credentials | Check API key/token validity |
| `PERMISSION_DENIED` | Insufficient permissions | Verify scopes |
| `NOT_FOUND` | Resource not found | Check if key exists |
| `INVALID_ARGUMENT` | Bad request | Validate parameters |
## Environment Variables
```bash theme={null}
## Required
export SAWMILLS_API_KEY="sk_test.abc123def4.xyz789uvw0123456789abcdef"
export SAWMILLS_BASE_URL="https://api.sawmills.ai"
# Optional
export SAWMILLS_TIMEOUT="30"
export SAWMILLS_RETRY_ATTEMPTS="3"
```
## Testing Authentication
### Test API Key
```bash theme={null}
# Test with your API key
curl -X POST "https://api.sawmills.ai/v1/token" \
-H "Authorization: ApiKey YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
-w "HTTP Status: %{http_code}\n"
# Expected: HTTP Status: 200
```
### Test Token Exchange
```bash theme={null}
# Test with valid API key
curl -X POST "https://api.sawmills.ai/v1/token" \
-H "Authorization: ApiKey $SAWMILLS_API_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
-w "HTTP Status: %{http_code}\n"
# Expected: HTTP Status: 200
```
### Test API Call
```bash theme={null}
# Test with access token
curl -X GET "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-w "HTTP Status: %{http_code}\n"
# Expected: HTTP Status: 200
```
## Troubleshooting Checklist
* [ ] API key is correctly formatted
* [ ] API key is active and not revoked
* [ ] Authorization header uses correct scheme (`ApiKey` vs `Bearer`)
* [ ] Token hasn't expired (30 minutes default)
* [ ] HTTPS is being used
* [ ] No extra whitespace in credentials
* [ ] Correct environment (test vs live)
# Building Pipelines
Source: https://docs.sawmills.ai/docs/build-pipelines
Build Sawmills pipelines end-to-end by connecting sources, processors, and destinations to control how telemetry data flows through your stack.
Sawmills' Pipelines are workflows that collect, process, and transfer data to where it's desired. Pipelines enable you to manage this flow of information by efficiently collecting, transforming, and delivering it to destination systems for monitoring, analysis, and storage.
## How Pipelines Work
A pipeline is composed of multiple connected stages:
* **Source**: The starting point where data is ingested (e.g., logs and traces from your application or metrics from cloud services).
* **Processing**: Intermediate steps where data is transformed, filtered, or enriched. Some examples of processors include adjusting timestamps, aggregating metrics, or removing unnecessary information that reduces noise and saves money on storage and/or third party ingestion and processing.
* **Destination**: The endpoint where processed data is sent, such as a monitoring platform, alerting system, or storage service.
By setting up a pipeline, you automate how data flows from start to finish.
## Steps to Build a Pipeline
1. Go to the **Pipeline** page.
2. Click **Create New Pipeline**. The system will create a new, empty pipeline.
### Step 1: Settings
1. Name the pipeline with a descriptive name that encapsolates the pipeline's purpose.
2. Optionally, add a detailed description for the pipeline.
### Step 2: Add a Source
1. Click **Add Source** on the left side.
2. Select the appropriate source. Remember, a **Source** is where data originates before reaching the Sawmills Collector.
3. Configure the Source's settings according to the provided instructions. For detailed configuration options, refer to the **Sources** documentation.
4. Review and save your changes.
### Step 3: Add a Destination
1. Click **Add Destination** on the right side.
2. Select the appropriate destination. Remember, a **Destination** is where you want to send the data, such as an observability platform or storage solution.
3. Configure the Destination's settings by following the provided instructions. For more details, check the **Destinations** documentation.
4. Review and save your changes.
> **Notes:**- You can add multiple sources and multiple destinations to a single pipeline.- Data flows from all sources to all destinations, creating a many-to-many relationship.
### Step 4: Deploy the Pipeline
Now that you have built your pipeline, deploy it to one or more collectors.
To deploy a pipeline to a Collector, click the Deployments button in the top-right corner. This will display all available collectors and their statuses.
Select all collectors or individual ones you want and then proceed with the deployment.
Deployment to a collector may take a several minutes. This delay ensures that the collector’s configuration is updated without causing any data loss.
### Step 5: Send Data to the Pipeline
To start sending data to your pipeline, direct data to the **Ingress** endpoint of the Kubernetes cluster where the Sawmills Collector is installed.
Pipelines start in **Draft** mode, once deployed, the **Live** view of the pipeline would be accessible, please use the toggle above the pipeline to move between **Live** and **Draft** mode.
### Live & Draft
* **Draft**: This is where you can edit a pipeline.
* **Live**: Once a pipeline is deployed, changes will appear on the live tab with some telemetry data statistics. You can also use **Preview Output** on supported destinations to inspect sampled telemetry as it is sent to that destination. To edit a pipeline, select the Draft tab.
Following these steps will enable you to set up a pipeline that controls and automates the flow of your telemetry data.
# Create a Processor
Source: https://docs.sawmills.ai/docs/building-processor
Add, insert, or duplicate processors in a Sawmills pipeline with step-by-step instructions for configuring filtering, masking, and enrichment rules.
### Add Processor
Sawmills allows you to configure processors to manage telemetry data efficiently by filtering, aggregating, masking, or modifying log attributes. You can do this by clicking the Add Processor button, or by clicking the + on the top right of the processors list under Processors in the Destination header. Processors created this way will appear at the **end** of the execution order.
### Insert Processor
You can also choose to "Insert Processor" by clicking that option on an existing processor's Actions menu. Processors that are created this way will appear **before** the currently selected processor in execution order.
### Duplicate Processor
Alternatively, you can select Duplicate Processor from the processor's Actions menu, which will duplcate that selected processor and it's various settings. This might be quicker than creating a processor from scratch in certain scenarios.
## Step-by-Step Configuration
Follow these steps to configure a processor:
### Step 1: Open Processor Configuration Page
1. Go to the **Processor** page.
2. Click **Create New Processor**. The system will create a new, empty processor.
3. Select **Pipeline** from the dropdown.
4. Select **Destination** from the dropdown.
5. Select **Processor Type**.
### Step 2: Explore Logs with Live Tail
* Use the Live Tail panel on the left to observe real-time log entries.
* Inspect log attributes and utilize the filter option to pinpoint specific logs.
* When you expand a log entry to view all its keys and values, a context menu appears on hover to the left of each row:
* **Copy Value** – copy just the value to your clipboard.
* **Copy Key & Value** – copy the pair in `key=value` form.
* **Show Logs with This Value** – add an equals filter for this field to the Live Tail filter bar.
* **Exclude Logs with This Value** – add a not equals filter for this field.
* **Add Filter to Processor** – add key and value to the processor panel on the right side.
### Step 3: Set Processor Parameters
* In the configuration panel to the right, define the rules and actions your processor will apply.
* Adjust processor-specific settings such as filters, attributes to modify, etc.
### Step 4: Simulate Configuration
* Once you've set up your configuration, click on the **Simulate** button.
* The system will generate simulation results based on the current log snapshot (coming from the live tail on the left side) and your processor settings.
### Step 5: Review Simulation Results
* Simulation results appear in a new panel to the right of the processor configuration.
* Logs are color-coded for clarity:
* **Red:** Logs that will be removed.
* **Blue:** Logs modified by the processor. Expand these logs to see detailed changes.
* **White:** Logs unaffected by the configuration.
### Step 6: Save Configuration
* Click the **Save** button to save your processor configuration.
* The system will save your configuration and you will be redirected to the processor list page.
Changes take effect immediately after saving.
# Collector Customization
Source: https://docs.sawmills.ai/docs/collector-customization
Customize your Sawmills Collector deployment in Kubernetes using Helm values for resource limits, environment variables, and advanced configuration.
## Overview
This guide explains how to customize your Sawmills collector deployment in Kubernetes. Follow the steps below based on whether you're installing a new collector or updating an existing one.
### NEW Sawmills Collector
1. Log in to the Sawmills dashboard.
2. Navigate to the **Collectors** section (located at the bottom tab of the left navigation menu).
3. Click the `New Collector` button and follow the instructions to install the agent with your organization API key.
### EXISTING Sawmills Collector
1. Log in to the Sawmills dashboard.
2. Navigate to **Collector Management**.
3. Click on an existing collector to access its management page.
## Customizing the Sawmills Collector Installation
### Install or Upgrade Collector Using Helm Values
To install a new automatically managed collector or update an existing one, copy the remote operator Helm command shown in the Sawmills UI. It includes the chart version and controller address for your environment:
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress= \
--set collectorName=""
```
> **Note**: For first-time installations, you must create the namespace and secret as described in [Create a New Collector](/docs/collector-installation-new).
For manually managed collectors, Sawmills generates collector chart metadata and values files instead of applying collector changes automatically. See [Manual Collector Deployment](/docs/manual-collector-deployment).
### Updating Collector Values
The remote operator deploys the collector to your Kubernetes cluster. To modify collector values, update the `managedChartsValues.sawmills-collector` section in your `values.yaml` file and then upgrade the remote operator. The legacy key `managedCharts` continues to work as an alias, but `managedChartsValues` is the preferred field going forward.
After modifying the remote operator values, the collector should automatically redeploy.
Example: To add a nodeSelector to the collector, update the `managedChartsValues.sawmills-collector` section:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
nodeSelector:
region: us-east-1
```
### Overriding the collector chart (private registries)
If you need the operator to pull the collector chart from a private registry or pin a different chart version, set `managedChartsOverrides`:
```yaml theme={null}
managedChartsOverrides:
sawmills-collector:
chartName: oci://registry.example.com/sawmills-collector
version: 1.2.3
```
Use `managedChartsOverrides` only for chart reference/version. Continue using `managedChartsValues` for regular values (replicas, image tags, tolerations, etc.).
After making changes, run the following command to update the remote operator:
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress= \
--set collectorName="" \
-f values.yaml
```
> **Important**: After adding new `nodeSelector`, `podAntiAffinity`, or similar configuration options, you must redeploy the pipeline for the changes to take effect.
### Manual Deployment Values Ownership
For manually managed collectors, keep Sawmills-generated values separate from customer-owned overrides:
* `sawmills-generated.values.yaml`: generated by Sawmills for the deployed pipelines; do not edit it manually.
* `sawmills-customer-overrides.yaml`: customer-owned overrides for scheduling, resources, annotations, and environment-specific Helm values.
Apply the generated values file before the customer override file so your customer-owned settings win when both files set the same value.
### Collector Values File Options
The following configuration options are available in your values file under the `managedChartsValues.sawmills-collector` section (or the legacy `managedCharts.sawmills-collector` alias):
```yaml theme={null}
managedChartsValues:
sawmills-collector:
# Number of replicas for the Sawmills collector
replicaCount: 3
apiSecret:
name: sawmills-secret
key: api-key
prometheusremotewrite:
endpoint: https://ingress.sawmills.ai
# Additional Environment Variables
# Example:
# - name: DD_API_KEY
# valueFrom:
# secretKeyRef:
# name: datadog
# key: api-key
extraEnv: []
# The node selector for the Sawmills collector
nodeSelector: {}
# Pod annotations for the Sawmills collector pods (e.g. for Prometheus scraping)
# podAnnotations:
# prometheus.io/scrape: "true"
# prometheus.io/path: "/metrics"
# prometheus.io/port: "metrics"
# Tolerations for the Sawmills collector pods
# Example:
# - key: "key1"
# operator: "Equal"
# value: "value1"
# effect: "NoSchedule"
tolerations: []
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
-
topologyKey: kubernetes.io/hostname
# The resources for the Sawmills collector
resources:
requests:
memory: 512Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 500m
# Telemetry Configuration
telemetry:
resources:
requests:
memory: 512Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 500m
# Prometheus port for the telemetry collector, can be used by customers to scrape metrics
prometheus:
port: 19465
# HPA configuration for the Sawmills collector
autoscaling:
enabled: false
minReplicas: 3
maxReplicas: 50
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
# Add behavior configuration for faster scaling
behavior:
scaleUp:
stabilizationWindowSeconds: 120
policies:
- type: Percent
value: 50
periodSeconds: 120
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 120
# KEDA-based autoscaling (separate from standard HPA)
keda:
enabled: false
minReplicas: 1
maxReplicas: 10
# How often KEDA checks the metrics for scaling decisions (in seconds)
pollingInterval: 30
# How long to wait after the last scaling operation before scaling down (in seconds)
cooldownPeriod: 300
# Scaling configuration for different metric sources
scaling:
# Prometheus-based scaling configuration
prometheus:
enabled: false
metricType: Value
metadata:
# Prometheus server address
serverAddress: http://prometheus:9090
# PromQL query to fetch metrics
query: "histogram_quantile(0.95, sum(rate(http_server_duration_bucket[1m])) by (le))"
# Target value to maintain (will scale up if metric exceeds this value)
threshold: "2000"
# Minimum value before scaling starts (will not scale if metric is below this value)
activationThreshold: "1000"
# External scaler configuration for OpenTelemetry metrics
# This section configures the external scaler that works with the kedaScaler component
# defined below. When enabled, it uses the OTLP metrics from the kedaScaler to make
# scaling decisions based on the configured query and target value.
external:
enabled: false
metricType: Value
metadata:
# The chart derives scalerAddress from the scaler service and namespace by default.
# Set scalerAddress only if Sawmills Support asks you to override it.
# PromQL query to fetch metrics from the OTLP scaler
query: "histogram_quantile(0.95, sum(rate(http_server_duration_bucket[2m])) by (le))"
# Target value to maintain (will scale up if metric exceeds this value)
targetValue: "2000"
# CPU-based scaling configuration
cpu:
enabled: true
# Target CPU utilization percentage to maintain
targetUtilization: 80
# Memory-based scaling configuration
memory:
enabled: true
# Target memory utilization percentage to maintain
targetUtilization: 80
# Service Configuration
service:
type: ClusterIP
headless:
enabled: true
# internalTrafficPolicy controls how traffic from within the cluster is routed
# Available values: "Cluster" (default) or "Local"
# When set to "Local", traffic from within the cluster is routed to endpoints on the same node
internalTrafficPolicy: ""
# Service annotations
# Example:
# annotations:
# service.kubernetes.io/topology-mode: "Auto" # Enables automatic topology-aware routing
# custom.annotation: "value"
annotations: {}
# Additional containers to be added to the pod
# This allows you to add sidecar containers to the collector pod.
# Each container follows the Kubernetes container spec format.
# For detailed examples, see the examples directory.
#
# Example:
# additionalContainers:
# sidecar:
# image: busybox:1.35
# command: ['sh', '-c', 'echo Hello && sleep 3600']
# resources:
# requests:
# memory: "64Mi"
# cpu: "100m"
# limits:
# memory: "128Mi"
# cpu: "200m"
additionalContainers: {}
# Additional volumes to be mounted by additional containers
# This allows you to add volumes that can be mounted by sidecar containers.
# For detailed examples, see the examples directory.
#
# Example:
# additionalVolumes:
# - name: config-volume
# configMap:
# name: sidecar-config
additionalVolumes: []
# Setup a service account for the collector, should be used to grant permissions
# to the collector to access resources (e.g. s3 buckets)
serviceAccount:
# -- Whether to create a service account for the Sawmills Collector deployment.
create: false
# -- Additional labels to add to the created service account.
additionalLabels: {}
# -- Annotations to add to the created service account.
annotations: {}
# -- The name of the service account,
# default is "default" when serviceAccount.create is false.
# default is "sa-collector-service" when serviceAccount.create is true.
name: null
# ServiceMonitor configuration for Prometheus Operator
serviceMonitor:
# Enable or disable ServiceMonitor resource creation
enabled: false
# Additional labels for the ServiceMonitor
labels:
release: prometheus
# Metrics endpoints configuration
metricsEndpoints:
- port: prometheus
interval: 15s
# Ingress Configuration
ingress:
# Type of ingress controller to use
# Available options - nginx or haproxy
type: nginx
# Nginx-specific ingress configuration
nginx:
# Enable/disable nginx ingress
enabled: false
# The ingress class name to use
# This should match your nginx ingress controller's class name
className: nginx
# The port that the ingress will forward traffic to
# This should match the port your collector is listening on
port: 4318
# Annotations for the ingress resource
# These are passed directly to the Kubernetes ingress resource
annotations:
# Specify the ingress class
kubernetes.io/ingress.class: nginx
# Disable SSL redirect if you're not using HTTPS
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Host configuration for the ingress
hosts:
- host: sawmills-collector.local # The hostname to use
paths:
- path: / # The URL path to match
pathType: Prefix
tls: []
haproxy:
# Enable/disable haproxy ingress
enabled: false
# The ingress class name to use
# This should match your haproxy ingress controller's class name
className: haproxy
# The port that the ingress will forward traffic to
# This should match the port your collector is listening on
port: 4318
# Annotations for the ingress resource
# These are passed directly to the Kubernetes ingress resource
annotations:
# Specify the ingress class
kubernetes.io/ingress.class: haproxy
# Disable SSL redirect if you're not using HTTPS
haproxy.ingress.kubernetes.io/ssl-redirect: "false"
# Host configuration for the ingress
hosts:
- host: sawmills-collector.local # The hostname to use
paths:
- path: / # The URL path to match
pathType: Prefix
# TLS configuration for secure connections
# Uncomment and configure if you want to enable TLS
tls: []
# - secretName: collector-tls
# hosts:
# - sawmills-collector.local
# KEDA external scaler configuration for OpenTelemetry integration
# This component acts as an external scaler for KEDA (Kubernetes Event-driven Autoscaling)
# that integrates with OpenTelemetry (OTel) collector. It enables dynamic scaling of your
# Sawmills collector based on OTLP metrics, providing more precise and efficient
# autoscaling capabilities compared to standard Kubernetes HPA.
#
# Note: To use this scaler, you must also enable the external scaling in the KEDA section above
# by setting `keda.scaling.external.enabled: true` and configuring the appropriate metadata.
kedaScaler:
# Enable or disable the KEDA scaler component
enabled: false
# Container image pull policy (Always, IfNotPresent, Never)
imagePullPolicy: Always
podSecurityContext: {}
probes:
# Liveness probe settings to determine if the pod is healthy
liveness:
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 5
# Readiness probe settings to determine if the pod is ready to receive traffic
readiness:
initialDelaySeconds: 5
periodSeconds: 5
# Service configuration for the KEDA scaler
service:
# Type of Kubernetes service (ClusterIP, NodePort, LoadBalancer)
type: ClusterIP
# Port for receiving OTLP data
otlpReceiverPort: 4518
# Port for KEDA external scaler communication
kedaExternalScalerPort: 4418
# Port for health check endpoints
healthcheckPort: 13134
# Port for monitoring metrics
monitoringPort: 19465
otelPrometheusPort: 19876
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 500m
memory: 128Mi
# Sawmills telemetry configuration for sending metrics to KEDA scaler
telemetryConfig:
receivers:
prometheus/keda_scaler:
config:
scrape_configs:
- job_name: keda_scaler
scrape_interval: 15s
static_configs:
- targets:
[
"${env:MY_POD_IP}:${env:KEDA_SCALER_OTEL_PROMETHEUS_PORT}",
]
processors:
filter/keda:
error_mode: ignore
metrics:
metric:
- name != "http.server.duration"
# For create a KEDA scaler query on haproxy metrics and enable prometheus receiver
# - name != "haproxy_server_http_responses_total" and name != "http.server.duration"
exporters:
otlp/keda:
# Address of the KEDA scaler service
endpoint: '{{ include "sawmills-collector.kedaScalerSvcFQDN" . }}:${env:KEDA_SCALER_OTLP_RECEIVER_PORT}'
compression: "none"
tls:
insecure: true
service:
pipelines:
metrics/telemetry_keda_scaler:
receivers: [prometheus/keda_scaler]
processors:
- memory_limiter
- batch
exporters: [routing, forward]
metrics/keda:
exporters:
- otlp/keda
processors:
- filter/keda
receivers:
- forward
# For create a KEDA scaler query enable prometheus receiver and change filter/keda filter
# - prometheus
config:
# OTLP receiver configuration for collecting metrics
receivers:
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:${env:OTLP_RECEIVER_PORT}
exporters:
kedascaler:
engine_settings:
timeout: 30s
max_samples: 500000
lookback_delta: 5m
retention_duration: 10m
monitoring_http:
endpoint: ${env:MY_POD_IP}:${env:MONITORING_PORT}
scaler_grpc:
endpoint: ${env:MY_POD_IP}:${env:KEDA_EXTERNAL_SCALER_PORT}
transport: tcp
# Processors for handling metrics data
processors:
memory_limiter:
check_interval: 1s
limit_mib: 0
limit_percentage: 95
spike_limit_mib: 0
spike_limit_percentage: 10
batch:
metadata_cardinality_limit: 0
metadata_keys: []
send_batch_max_size: 2048
send_batch_size: 1024
timeout: 1s
# Extensions for additional functionality
extensions:
cgroupruntime:
gomaxprocs:
enabled: true
gomemlimit:
enabled: true
ratio: 0.95
health_check:
endpoint: ${env:MY_POD_IP}:${env:HEALTHCHECK_PORT}
service:
extensions:
- health_check
- cgroupruntime
pipelines:
metrics:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- kedascaler
# Telemetry configuration for logging and metrics
telemetry:
resource:
pod: ${env:MY_POD_NAME}
logs:
encoding: json
error_output_paths:
- stdout
level: info
output_paths:
- stdout
metrics:
address: ${env:MY_POD_IP}:${env:OTEL_PROMETHEUS_PORT}
```
# Collector Management
Source: https://docs.sawmills.ai/docs/collector-management
View collector status, track versions, manage API key rotation, and monitor associated pipelines from the Sawmills Collector Management page.
## Overview
The **Collectors** page provides a central location to manage, monitor, and troubleshoot collectors in your system. Use this interface to track collector status, versions, and associated pipelines.
### Collectors
The table on the **Collectors** page displays the following columns:
1. **Name**\
The name of the collector.
2. **Status**\
The current status of the collector:
* **Awaiting Deployment** – The collector was created in the UI but has not been deployed yet (installation not completed).
* **Installed (Idle)** – The collector is installed without any deployed pipelines.
* **Running (Active)** – The collector is actively running with at least one deployed pipeline.
* **Degraded** – Collector pods are running, but one or more pods are not working properly.
* **Error** – The collector has encountered an issue; pipelines are not running.
* **Disconnected** – The remote operator cannot communicate with the collector.
3. **Version**\
Shows the version of the collector. If the version is unavailable, displays `N/A`.
4. **Pipelines**\
Lists pipelines associated with a collector. If no pipelines are associated, displays `N/A`.
5. **Last Deployed**\
Displays the date the collector was most recently deployed in `MM/DD/YYYY` format. If unavailable, displays `N/A`.
6. **Delete Collector**\
The trashcan image button can be used to delete a collector.
A collector with no pipelines deployed to it will display a status of
**Installed (Idle)** and will not display any version number.
## Manage an Existing Collector
1. Log in to the Sawmills dashboard.
2. Navigate to **Collector Management**.
3. Click a collector name to open its management page.
The collector page consists of three main areas:
1. **Pipelines**
2. **Installation**
3. **Rotate API Key**
### Pipelines
The **Pipelines** section lists pipelines attached to this collector:
1. **Name**\
The name of the pipeline with a link to the pipeline's management page.
2. **Last Deployed**\
Displays the date the pipeline was deployed to this collector.
3. **Detach**
Button to detach the pipeline from this collector. This button is only enabled if the collector is running and responsive.
### Installation
The **Installation** section shows the deployment instructions for this collector.
For automatically managed collectors, the UI shows the remote operator Helm command for your environment:
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress= \
--set collectorName=""
```
For manually managed collectors, the UI shows the collector chart, chart version, values format, resource base name, and generated values files. Direct chart installs also include a Helm command. Umbrella chart and Helmsman installs intentionally omit a standalone Helm command because the collector is deployed through your parent chart. See [Manual Collector Deployment](/docs/manual-collector-deployment).
### Rotate API Key
To rotate the API key used by this collector:
1. Click **Generate New Key**.
2. Update the API key secret using the kubectl patch command shown in the UI:
```bash theme={null}
kubectl patch secret sawmills-secret \
-n sawmills -p='{"stringData":{"api-key": "****************"}}'
```
> **Note**: Even though the API key is masked it will still be copied to your clipboard properly.
# Configure Invitation Method
Source: https://docs.sawmills.ai/docs/configure-invitation-method
Configure your organization's user invitation method in Sawmills, choosing between manual invitations, automatic enrollment, or automatic suggestions.
### User Invitation System
This section explains how to use the user invitation system within Sawmills platform. The system provides flexibility by offering three different invitation methods, allowing administrators to choose the best approach for their organization.
Before inviting users to your organization, follow these steps to configure your organization’s invitation method
1. **Navigate to Your Profile**:
* Click on your profile button located at the top right corner of the application.
2. **Open the Organization Profile**:
* From the drop-down menu, select "Organization Profile."\\
![]()
3. **Add a New Domain**:
* Click on the "+ Add Domain" button.
* You will be presented with three **Invitation methods** for inviting users to your organization.\\
![]()
4. **Choose invitation method**\
Choose the preferred methods to invite users to the platform:
* **No Automatic Enrollment**:
* Users can only be invited manually to the organization.
* Administrators must explicitly send invitations for each user they wish to add.
* This method ensures full control over who is invited to join the platform.
* **Automatic Invitations**:
* Users are automatically invited to join the organization when they sign up.
* This method allows users to join the organization anytime without waiting for an invitation, the org is defined by the **verified domain** the admin configures.
* Ideal for organizations that want to streamline the onboarding process.
* **Automatic Suggestions**:
* Users receive a suggestion to request to join the organization when they sign up. the org is defined by the **verified domain** the admin configures.
* After receiving the suggestion, users must be approved by an admin before they can join.
* This method offers a balance between control and convenience, allowing admins to approve new members while encouraging user engagement.
Once you choose the invitation method, don't forget to click **Save**.
1. **Edit or delete domain**
* To edit or delete a domain, click on the 3 dots on the top right part of the `verified domain`section and follow the instructions
# Analytics Dashboard
Source: https://docs.sawmills.ai/docs/dashboard
Monitor telemetry volume, measure data reduction ratios, and compare received versus forwarded log and metric usage across pipelines and destinations.
## Overview
The **Analytics Dashboard** shows how much telemetry your Sawmills collectors receive and how much they forward to each destination. Use it to measure reduction, spot high-volume sources, and validate whether filtering or aggregation changes are reducing downstream usage.
You can switch between **Logs** and **Metrics** views. Both views are scoped to one pipeline, one destination, and the selected time range.
## Requirements
The Analytics Dashboard requires Sawmills Collector version `1.577.0` or later.
The Metrics view requires Sawmills Collector version `1.780.0` or later.
* The selected pipeline must be deployed to collectors that are sending analytics data.
* If a pipeline was never deployed, or its collectors are not reporting analytics data yet, the dashboard may be empty.
* Vendor-specific usage details are shown only when a supported vendor integration is configured for the selected destination.
## Open The Dashboard
Open **Dashboard** from the main Sawmills navigation.
After the page loads, select:
1. A time range
2. A deployed pipeline
3. A destination for that pipeline
4. Either **Logs** or **Metrics**
## Configuration Options
Use the selectors at the top of the page to refine the results:
* **Time Picker**: Limits the analysis to the selected range, for example `1h` or `24h`.
* **Pipeline**: Selects one deployed pipeline. A deployed pipeline is a pipeline whose current configuration was already deployed to collectors.
* **Destination**: Selects one destination from the selected pipeline. Only destinations that support the current telemetry type are shown.
If the selected pipeline or destination does not support the current telemetry type, the view may be empty until you choose a different selection.
## Logs View
Use the **Logs** view to understand log traffic and identify the biggest contributors to log volume.
### Top-Level Metrics
In the **Logs** view, the summary cards always reflect the current pipeline, destination, time range, and **Messages / Volume** selection.
The **Messages / Volume** toggle controls which unit is used across the summary cards and chart:
* **Messages**: shows log record counts
* **Volume**: shows log data size in bytes
For the selected mode, the dashboard calculates:
1. **Received**
The total number of log messages or total log volume received by Sawmills collectors during the selected time range.
2. **Sent**
The total number of log messages or total log volume forwarded to the selected destination during the selected time range.
3. **Data Reduction (%)**
The percentage reduction between **Received** and **Sent** for the selected log mode.
4. **Received vs Sent**
A time-series chart comparing received and sent values over time using the selected log mode.
### Logs View Controls
* **Messages / Volume**
* **Messages** shows the count of log records seen during the selected time range.
* **Volume** shows the total log data size in bytes during the selected time range.
* **Group By**
Breaks the results down by a selected log attribute, for example `service.name`.
Use this to identify which services, environments, or other attributes contribute the most log traffic.
### Logs View Output
The Logs view includes:
* Summary cards for reduction, received volume, and sent volume
* A **Received vs Sent** trend chart
* A grouped breakdown table or chart based on the selected **Group By** field
## Metrics View
Use the **Metrics** view to understand metric volume, time-series growth, and cardinality drivers.
### Top-Level Metrics
In the **Metrics** view, the summary cards always reflect the current pipeline, destination, time range, and **Time Series / Volume** selection.
The **Time Series / Volume** toggle controls which unit is used across the summary cards and chart:
* **Time Series**: shows unique metric time-series count
* **Volume**: shows metrics data size in bytes
For the selected mode, the dashboard calculates:
1. **Received**
The total metric cardinality or total metrics volume received by Sawmills collectors during the selected time range.
2. **Sent**
The total metric cardinality or total metrics volume forwarded to the selected destination during the selected time range.
3. **Data Reduction (%)**
The percentage reduction between **Received** and **Sent** for the selected metrics mode.
4. **Received vs Sent**
A time-series chart comparing received and sent values over time using the selected metrics mode.
### Metrics View Controls
* **Time Series / Volume**
* **Time Series** shows how many unique metric time series were observed during the selected time range.
* **Volume** shows the total metrics data size in bytes during the selected time range.
* **Metrics / Labels**
* **Metrics** groups the analysis by metric name.
* **Labels** groups the analysis by label keys and values to help identify labels that drive cardinality or volume.
### Metrics View Output
The Metrics view includes:
* Summary cards for reduction, received volume, and sent volume
* A **Received vs Sent** trend chart
* A breakdown of metrics or labels, depending on the selected mode
* A metric details drill-down for deeper inspection
### Metric Details
Click a metric name to open **Metric Details** for the selected pipeline, destination, and time range.
Metric Details can show:
* Time-series count for that metric
* Volume for that metric
* Labels observed on that metric
* Vendor usage details and references, when available
## Vendor Usage And References
The **References** section in the Metrics view shows where a metric is used in the configured vendor platform, for example in dashboards, alerts, or queries.
**Referenced By**, **Usage Status**, **Reduction Opportunity**, and the **References** section in Metric Details are available only when a supported vendor integration is configured for the selected destination.
Use these fields as follows:
* **Referenced By**: The number of vendor assets or queries that reference the metric.
* **Usage Status**: Whether the metric appears to be actively used, unused, or has unknown usage based on vendor integration data.
* **Reduction Opportunity**: A hint that the metric may be a candidate for reduction because it has low or no detected usage.
Vendor usage coverage depends on the selected integration and the metadata available from that vendor.
## In-Context Actions
The dashboard includes shortcuts that open related workflows with the current context already selected.
1. **Live Tail**
From the Logs view, open a Live Tail session pre-filtered to a selected group-by value, for example a specific service name.
2. **Drop Metric Datapoints Processor**
From the Metrics view, open the processor creation flow with the selected metric pre-filled so you can stop forwarding that metric to the selected destination.
3. **Aggregate Metric Labels Processor**
From Metric Details, open the processor creation flow with the selected metric and selected labels pre-filled so you can reduce metric cardinality.
These actions open a pre-configured workflow. Review the generated processor before saving or deploying it.
## Limits And Notes
* Dashboard results reflect the currently selected time range, pipeline, destination, and telemetry type.
* Metrics and logs use different units depending on the selected mode, so compare values only within the same view and mode.
* Some labels cannot be aggregated. For example, labels that are not supported by the processor or labels that would change metric meaning may be excluded.
* Vendor reference data may be unavailable, partial, or delayed if the vendor integration is not configured or does not expose enough metadata.
# Enable Datadog Integration
Source: https://docs.sawmills.ai/docs/datadog-integration
Connect your Datadog API and Application Key credentials to Sawmills with step-by-step setup instructions and automatic permission validation.
## About Datadog Integration
The Datadog Integration lets Sawmills use Datadog credentials for Datadog-backed features.
When you save the Datadog configuration, Sawmills validates the credentials and checks predefined permissions before storing them.
## Step-by-Step: Configure Datadog Integration
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
### 2. Open Datadog Configuration
In the Integrations section, open **Datadog** configuration.
### 3. Enter Credentials
Provide:
* **Datadog API Key**
* **Datadog Application Key**
* **Datadog Site**
## Best Practice: Service Account + Custom Role
For production, use a dedicated Datadog **service account** (not a personal user key).
### 1. Create a service account in Datadog
Create a service account for Sawmills integration usage.
### 2. Assign roles
Assign:
* **Datadog Read Only Role** (baseline read access)
* A custom role for extra Sawmills checks, for example: **Sawmills Integration Role**
### 3. Add permissions to the custom role
Grant these Datadog permissions (typo-safe).
Required to save:
* `logs_read_config`
* `metrics_read`
* `dashboards_read`
* `monitors_read`
* `slos_read`
* `notebooks_read`
Optional but checked in UI (non-blocking):
* `audit_logs_read`
* `logs_live_tail`
* `logs_read`
* `datalogs_read_index_data`
* `usage_notifications_read`
* `usage_read`
* `logs_read_archives`
* `metrics_read`
* `timeseries_query`
### 4. Create keys for the service account
* Create an API key in Datadog.
* Create an application key for the same service account.
* Use those keys in Sawmills Datadog integration.
### 5. Save and Validate
Click **Save**. Before persisting, Sawmills runs validation and shows:
* Keys status: **Working** or **Invalid**
* Required permissions status
* Per-permission status for predefined checks
Save is blocked if keys are invalid or required permissions are missing.
## Backend Validation Flow
Validation is enforced in the backend service:
* `POST /v1/notification-integrations/datadog/validate`: validates keys and returns per-permission status.
* `POST /v1/notification-integrations`: blocks save when keys are invalid or required permissions are missing.
* `PATCH /v1/notification-integrations/{id}`: re-validates merged Datadog config before update is persisted.
## Required Permission
* `logs_read_config`
* `metrics_read`
* `dashboards_read`
* `monitors_read`
* `slos_read`
* `notebooks_read`
`notebooks_read` is required, but notebook data access still follows Datadog sharing rules. The Application Key can only read notebooks visible to the key owner (user or service account). Private or team-restricted notebooks are not readable unless that identity has access.
## Predefined Permissions Checked
* `audit_logs_read`
* `logs_live_tail`
* `logs_read`
* `datalogs_read_index_data`
* `usage_notifications_read`
* `usage_read`
* `logs_read_archives`
* `metrics_read`
* `timeseries_query`
# AWS S3 Destination
Source: https://docs.sawmills.ai/docs/destination-aws-s3
Export logs, metrics, and traces to AWS S3 buckets in Sawmills. Configure region, Role ARN, file prefix, and output format for object storage.
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Region | String | "us-east-1" | true | AWS region. |
| S3 Bucket | String | none | true | S3 bucket name. |
| Role ARN | String | none | false | The Role ARN to be assumed. |
| File Prefix | String | none | false | Prefix for the S3 key (root directory inside the bucket). |
| Output Format | String | OTLP JSON | false | Format used to produce output data (see Output Format options below). |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| REST API Endpoint | String | None | false | Overrides the endpoint, instead of constructing it from region and s3\_bucket. |
| S3 Force Path Style | Boolean | false | false | Set this to true to force the request to use path-style addressing. |
| Use SSL | Boolean | true | false | Set this to false to disable SSL when sending requests. |
| Compression | String | none | false | Should the file be compressed. |
## Output Format
The Output Format determines how data is serialized to AWS S3.
* **OTLP JSON** (default): The OpenTelemetry Protocol format represented as JSON.
* **OTLP (Protobuf)**: The OpenTelemetry Protocol format represented as Protocol Buffers. A single protobuf message is written into each object.
* **Sumo Logic (JSON)**: The Sumo Logic Installed Collector Archive format (logs only).
* **Raw payload**: Exports the log body as a string (logs only).
* **NDJSON (.json.gz)**: Newline-delimited JSON, gzipped, hourly-partitioned (`dt=YYYYMMDD/hour=HH/…`). One log record per line. Compatible with Datadog Logs Rehydration, Splunk Generic S3 input, AWS Glue / Splunk FSS3, and other archive readers. (Logs only.)
* **Parquet**: Apache Parquet files, hourly-partitioned. Compatible with Splunk FSS3, Snowflake, AWS Athena, and other analytics tools. (Logs only.)
### Snowflake Table Schema for Parquet
When using the **Parquet** output format, you can create a Snowflake table with the following schema:
```sql theme={null}
CREATE TABLE PARQUET_LOGS (
Date TIMESTAMP_LTZ,
Service VARCHAR,
Source VARCHAR,
Host VARCHAR,
Status VARCHAR,
Message VARCHAR,
Attributes OBJECT,
Tags OBJECT
);
```
### AWS Credential Configuration
Sawmills collector is running on a Kubernetes cluster and uses helm charts. To provide AWS credentials, pass `extraEnvs` in `values.yaml` as shown below:
```yaml theme={null}
extraEnvs:
- name: AWS_ACCESS_KEY_ID
value: ""
- name: AWS_SECRET_ACCESS_KEY
value: ""
```
# Coralogix Destination
Source: https://docs.sawmills.ai/docs/destination-coralogix
Forward logs, metrics, and traces to Coralogix from Sawmills. Configure API private key, application name, and multi-region domain for ingestion.
### Configuring the Coralogix Destination
The Coralogix destination forwards logs, metrics, and traces to Coralogix for analysis and monitoring. This guide details configuration using the interface components:
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier for the configuration. |
| API Private Key | String | none | true | The private key used to authenticate requests with Coralogix. |
| Application Name | String | none | true | Coralogix “Application Name” for categorizing logs and metrics. |
| Domain | String | US1 coralogix.us | true | The Coralogix region or domain where data is sent. Supported Regions: |
| | | | | - **US1**: coralogix.us |
| | | | | - **US2**: cx498.coralogix.com |
| | | | | - **AP1**: coralogix.in |
| | | | | - **AP2**: coralogixsg.com |
| | | | | - **AP3**: ap3.coralogix.com |
| | | | | - **EU1**: coralogix.com |
| | | | | - **EU2**: eu2.coralogix.com |
| Subsystem Name | String | none | false | Coralogix “Subsystem Name” for further categorization. |
| Enabled Data Types | Checkbox | Metrics, Logs, Traces | | The telemetry data types you want to export. |
| Log Data Format | Radio | Coralogix | | Log data format: |
| | | | | - **Coralogix**: Native format for full compatibility. |
| | | | | - **OTLP**: OpenTelemetry Protocol for standard interoperability. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Flush Timeout (seconds) | Int | 30 | false | When using Log Data Type (Coralogix), this is the seconds to wait per individual attempt to send data to a backend. |
| Timeout (seconds) | Int | 5 | false | When using Log Data Type (OTLP), this is the maximum time to wait for a response from the backend per send attempt. If the backend doesn’t respond within this time, the request is considered failed. |
| Application Name Attributes | List | none | false | Ordered list of resource attributes used for Coralogix Application Name. |
| Subsystem Name Attributes | List | none | false | Ordered list of resource attributes used for Coralogix Subsystem. |
# Datadog Destination
Source: https://docs.sawmills.ai/docs/destination-datadog
Export logs, metrics, and traces to Datadog from Sawmills. Configure your Datadog API key, site selector, and enabled data types for forwarding.
## Supported Data Types:
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
**Traces**: While the Datadog destination can export traces, the Sawmills Collector does not support receiving traces from a Datadog agent. If you are using a Datadog agent as your source, traces must be sent directly to Datadog. See the [Datadog Agent Configuration guide](/docs/send-data-from-datadog-agent#traces-bypass-configuration) for details.
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| API key | String | none | true | The Datadog API key to associate your agent's data with your organization. |
| Site | String | US1 - datadoghq.com | true | The site of the Datadog intake to send agent data to. |
| Enabled Data Types | Checkbox | Metrics, Logs, Traces | | The telemetry data types you want to export. |
# Debug (/dev/null) Destination
Source: https://docs.sawmills.ai/docs/destination-debug
Use the Debug (/dev/null) destination in Sawmills to discard all telemetry data during testing. Validate pipeline configurations without exporting data.
### Debug (/dev/null) Destination
The **Debug (/dev/null)** destination acts as a blackhole for your observability pipeline. It is designed for testing and debugging purposes: any data sent to this destination is discarded and **not** forwarded to any observability backend. This is useful for validating pipeline configurations, transformations, and flows without actually exporting data.
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
***
#### Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier for this destination configuration. |
***
**Note:** No data is sent to any backend when using this destination. Use it for safe testing and pipeline validation.
# Elasticsearch Destination
Source: https://docs.sawmills.ai/docs/destination-elasticsearch
Export logs to Elasticsearch from Sawmills. Configure endpoint URLs, Cloud ID, authentication via basic auth or API key, and Logstash index routing.
## Supported data types:
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Endpoint | String | none | true | The target Elasticsearch URL to which data will be sent. If multiple endpoints are added, they will be attempted in round-robin order. |
| Cloud ID | String | none | true | The Elastic Cloud ID of the Elastic Cloud Cluster to which data will be sent. |
| Enable Basic Auth | Boolean | false | true | Choose how to provide Elasticsearch credentials: via HTTP Basic Auth using a username and password or Elasticsearch API Key. Select 'None' to use your cluster's authentication configuration. |
| Username | String | none | true | Username when using basic authentication. |
| Password | String | none | true | Password when using basic authentication. |
| API Key | String | none | true | Elasticsearch API Key in "encoded" format, when using API Key authentication. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Logs Index | String | none | false | The index or data stream name to publish logs to. Logs Index should be empty unless all logs should be sent to the same index. |
| Mapping Mode | Radio | ECS | true | Determines how OpenTelemetry data is mapped to Elasticsearch index documents. The 'OTel' mapping mode requires Elasticsearch 8.12 or above. |
## Logstash Index Configuration
The Elasticsearch exporter supports dynamic index configuration through Logstash. You can specify the target index for your logs using the `elasticsearch.index` field in your Logstash configuration.
### Example Logstash Configuration
```yaml theme={null}
if [cluster] == "staging-cluster" {
mutate {
add_field => {
"elasticsearch.index" => "staging-cluster-%{+YYYY.MM.dd}"
}
}
}
```
In this example:
* The exporter will use the `elasticsearch.index` attribute to determine the target index
* The index name can include dynamic date patterns (e.g., `%{+YYYY.MM.dd}`)
* You can use any Logstash condition to set different indices based on your requirements
### Notes
* If `elasticsearch.index` is not set, the exporter will use the default index configuration
* The index name must follow Elasticsearch naming conventions
* Date patterns in the index name will be evaluated at the time of sending the logs
# Google Cloud Storage Destination
Source: https://docs.sawmills.ai/docs/destination-google-cloud-storage
Export logs to Google Cloud Storage from Sawmills in OTLP JSON format. Configure bucket name, project ID, authentication method, and compression.
The Google Cloud Storage exporter marshals telemetry data into OTLP JSON format and writes it to the specified GCS bucket.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Bucket Name | String | none | true | The name of the Google Cloud Storage bucket. |
| Project ID | String | none | false | The ID of the Google Cloud project that owns the bucket. |
| Authentication Method | Auth | Auto | false | The authentication method to use for accessing the bucket. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Granularity | Hour/Minute | Minute | false | The time partitioning granularity for objects in the bucket. |
| Compression | None/Gzip | None | false | Compression method for data before sending to GCS (e.g., gzip). |
| Folder Name | String | None | false | The folder name to store objects under in the bucket. |
| Object Prefix | String | None | false | The prefix to add to object names in the bucket. |
## Authentication
The exporter supports three types of authentication:
* **Auto**: Use the automatic authentication that GCP provides (via environment variables or metadata server). Read about [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) for more information.
* **JSON**: Use credentials JSON provided by GCP. Paste the service account key directly into the configuration.
* **File**: Specify the path to the credentials JSON file on the filesystem.
### Mounting Credentials File
To mount the credentials JSON file, configure the collector using [Helm](https://github.com/Sawmills/helm-charts). Configure this via the remote operator values YAML using the following values:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /creds/credentials.json
additionalVolumeMounts:
- name: credentials
mountPath: /creds
readOnly: true
additionalVolumes:
- name: credentials
secret:
secretName: credentials
items:
- key: credentials.json
path: credentials.json
```
# Grafana Loki Destination
Source: https://docs.sawmills.ai/docs/destination-grafana-loki
Forward logs to Grafana Loki from Sawmills with optional basic authentication, customizable default labels, and configurable endpoint URLs.
## Supported data types:
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Endpoint | String | none | true | The target URL to send Loki log streams to (e.g., `http://loki:3100/loki/api/v1/push`). |
| Enable Basic Auth | Boolean | false | true | Toggle on to require Basic Auth. When enabled, you must provide a username and password. |
| Username | String | none | false | Username when using basic authentication. |
| Password | String | none | false | Password when using basic authentication. |
| Customize Enabled Default Labels | Boolean | none | false | Select which default labels to include in logs. If omitted, all default labels are added. At least one label must be present; otherwise, logs will be dropped. |
## Authentication
Loki destination supports basic authentication, enable this by clicking on the `Enable Basic Auth` toggle and adding Username and Password
# Logz.io Destination
Source: https://docs.sawmills.ai/docs/destination-logzio
Export logs and traces to Logz.io from Sawmills. Configure your account token, region, and optional custom endpoint for development or testing.
## Supported Data Types:
📘 **Logs** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Account Token | String | none | true | Your Logz.io account token for your tracing or logs account. |
| Region | String | US East | false | Your Logz.io account region code. |
| Custom Endpoint | String | none | false | Custom endpoint, mostly used for development or testing. This will override the `region` parameter. |
| Enabled Data Types | Checkbox | Logs, Traces | | The telemetry data types you want to export. |
# New Relic Destination
Source: https://docs.sawmills.ai/docs/destination-new-relic
Export Metric API, APM time-slice, and Infrastructure agent metric data to the correct native New Relic endpoints.
## Supported Data Types
📈 **Metrics**
## Overview
The New Relic destination selects the native New Relic endpoint from the source data format.
| Data format | New Relic service |
| :- | :- |
| Dimensional metrics | Metric API |
| APM time-slice metrics | APM agent listener |
| Infrastructure agent samples | Infrastructure metrics API |
Infrastructure samples do not go to the Metric API. Sawmills rebuilds their native event and entity grouping after processors run and sends them to the Infrastructure metrics API.
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | yes | Unique destination name in the pipeline. |
| License Key | String | none | yes | New Relic ingest license key used for all native export paths. |
| Endpoint | String | `https://metric-api.newrelic.com/metric/v1` | yes | Metric API endpoint for dimensional metrics. |
| Enabled Data Types | Checkbox | Metrics | yes | Telemetry data types sent by this destination. |
The APM and Infrastructure endpoints use service defaults. The Infrastructure endpoint is `https://infra-api.newrelic.com/infra/v2/metrics/events/bulk` for US keys and `https://infra-api.eu.newrelic.com/infra/v2/metrics/events/bulk` for EU keys.
## Infrastructure account requirement
The destination license key must be the exact ingest license key used by the Infrastructure agent. The collector compares a one-way fingerprint of the source key with the destination key before it sends data.
This check prevents an identity request for one account from being combined with metric data for another account. A mismatch stops Infrastructure export before a network request.
> **Warning:** Use an ingest license key. Do not use the User API key from the
> New Relic account integration.
## Metric mapping
* Gauge metrics map to `gauge`.
* Sum metrics map to `count`, with `interval.ms` from the start and end timestamps.
* Summary metrics map to `summary`, with count, sum, minimum, and maximum.
* Histogram metrics map to `summary`, with count, sum, minimum, and maximum.
Exponential histograms are not supported.
## APM time-slice metrics
Summary metrics go to the APM agent listener when they have the New Relic time-slice marker or agent attributes.
## Infrastructure samples
The New Relic source converts Infrastructure samples to OpenTelemetry metrics. Processors can reduce the data between the source and destination. The destination then:
1. Groups fields by the saved sample and entity identity.
2. Rebuilds the native `MetricPost` batches.
3. Restores original field names and value types.
4. Sends the batches to the regional Infrastructure metrics API.
5. Forwards the required New Relic entity headers.
The raw source license key is not stored in OpenTelemetry attributes.
## Related documentation
* [New Relic Source](/docs/source-new-relic)
* [New Relic Infrastructure agent](/docs/guide-send-data-from-newrelic-infrastructure-agent)
* [New Relic Java app](/docs/guide-send-data-from-newrelic-java-app)
# OTLP Destination
Source: https://docs.sawmills.ai/docs/destination-otlp
Export logs, metrics, and traces via OTLP gRPC or HTTP from Sawmills. Configure endpoints, TLS settings, compression, and custom headers.
## Supported data types:
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## OTLP Configuration (gRPC)
| Field | Default | Required | Description | |
| :- | :- | :- | :- | - |
| Name | none | true | Unique identifier within Sawmills. | |
| Protocol | gRPC | true | Selected gRPC protocol. | |
| Endpoint | none | true | host:port to which the exporter is going to send OTLP data, using the gRPC protocol. | |
| Enabled Data Types | Checkbox | Metrics, Logs, Traces | | The telemetry data types you want to export. |
### Advanced Options (gRPC)
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Compression | Select | gzip | false | Choose the compression type. By default, gzip compression is enabled. Set to ‘None’ to disable compression. |
| Additional Headers | Key/Value Pairs | None | false | List of additional headers attached to each request. |
| Skip TLS Verification | Boolean | false | false | Enabling this setting skips verifying the server's certificate chain when sending data over gRPC. |
| Enable Insecure Connection | Boolean | false | false | Enabling this setting connects to the OTLP endpoint with an unencrypted (non-TLS) gRPC connection. |
## OTLP Configuration (HTTP)
| Field | Default | Required | Description | |
| :- | :- | :- | :- | - |
| Name | none | true | Unique identifier within Sawmills. | |
| Protocol | HTTP | true | Selected HTTP protocol. | |
| Endpoint | none | true | The target base URL to send data to. To send each signal a corresponding path will be added to this base URL, i.e. for traces "/v1/traces" will appended, for metrics "/v1/metrics" will be appended, for logs "/v1/logs" will be appended. | |
| Enabled Data Types | Checkbox | Metrics, Logs, Traces | | The telemetry data types you want to export. |
| Metrics Endpoint | none | false | The target URL to send metric data to. If this setting is present the Endpoint setting is ignored for metrics. | |
| Logs Endpoint | none | false | The target URL to send log data to. If this setting is present the Endpoint setting is ignored for logs. | |
| Traces Endpoint | none | false | The target URL to send traces data to. If this setting is present the Endpoint setting is ignored for traces. | |
### Advanced Options (HTTP)
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Timeout (seconds) | Int | 30 | false | HTTP request time limit |
| Compression | Select | gzip | false | Choose the compression type. By default, gzip compression is enabled. Set to ‘None’ to disable compression. |
| Additional Headers | Key/Value Pairs | None | false | List of additional headers attached to each request. |
| Skip TLS Verification | Boolean | false | false | Enabling this setting skips verifying the server's certificate chain when sending data over gRPC. |
# Prometheus Destination
Source: https://docs.sawmills.ai/docs/destination-prometheus
Export metrics via Prometheus Remote-Write from Sawmills. Configure endpoint URL, namespace prefix, external labels, headers, and batch size.
## Supported Data Types:
📈 **Metrics**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Endpoint | String | none | true | The target URL to send Prometheus data to. |
| Namespace | String | none | true | Prefix attached to each exported metric name. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| **Additional Headers** | none | No | Additional headers attached to each HTTP request. | |
| **External Labels** | none | No | Map of labels names and values to be attached to each metric data point.. | |
| **Timeout (seconds)** | 30 | Yes | Seconds to wait per individual attempt to send data to a backend. | |
| **Max Batch Size (bytes)** | 3000000 | Yes | Maximum size of a batch of samples to be sent to the remote write endpoint. If the batch size is larger than this value, it will be split into multiple batches.. | |
Note the following headers cannot be changed: **Content-Encoding**, **Content-Type**, **X-Prometheus-Remote-Write-Version**, and **User-Agent**..
# S3 Tables Destination
Source: https://docs.sawmills.ai/docs/destination-s3-tables
Export logs to AWS S3 Tables in Apache Iceberg format from Sawmills. Configure the table contract plus advanced batching and partition-buffering knobs.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Region | String | `us-east-1` | true | AWS region for the table bucket. |
| Table Bucket ARN | String | none | true | ARN of the existing S3 Tables table bucket. |
| Namespace | String | none | true | Namespace inside the table bucket. |
| Table Name | String | none | true | Existing table name inside the namespace. |
### Advanced tuning fields
These settings are hidden under advanced settings in the UI. For most tables, start with the defaults.
| Field | Type | Default | Allowed Range | Description |
| :- | :- | :- | :- | :- |
| `max_batch_compressed_bytes` | Integer | `134217728` | `1 MiB` to `512 MiB` | Collector-side batch size before records move into partition buffering. |
| `max_batch_age` | Duration | `5m` | `30s` to `30m` | Collector-side age cap before a batch is handed to the partition buffer. |
| `partition_buffer_compression` | String | `zstd` | `none`, `snappy`, `zstd` | Compression used for sealed in-memory partition-buffer chunks. |
| `partition_buffer_chunk_target_bytes` | Integer | `16777216` | `1 MiB` to `64 MiB` | Chunk size before the partition buffer seals and compresses buffered payloads. |
| `target_partition_file_bytes` | Integer | `134217728` | `8 MiB` to `512 MiB` | Main efficiency knob for larger parquet files and fewer Iceberg snapshots. |
| `max_partition_file_age` | Duration | `15m` | `1m` to `2h` | Freshness cap for sparse partitions. Lower it when low-volume services need faster visibility. |
| `max_open_partitions` | Integer | `10000` | `1` to `100000` | Protects memory under high service fanout by forcing older partitions to flush earlier. |
| `max_buffered_partition_bytes` | Integer | `2147483648` | `64 MiB` to `16 GiB` | Global in-memory cap across all buffered partitions. |
| `max_records_per_partition_file` | Integer | `0` | `0` or positive integer | Escape hatch for very wide rows. Keep `0` unless row width forces an explicit record cap. |
## Prerequisites
* An existing S3 Tables table. Sawmills does not create the table for you in v1.
* A Sawmills Collector instance with AWS access configured.
* S3 Tables REST control-plane access for the target table bucket.
* Table-scoped S3 Tables write access for the target table or table bucket, plus data-plane access to the managed warehouse bucket used by that table.
* If you will query the same table from Snowflake, create the Snowflake catalog integration and AWS IAM role before you depend on `AUTO_REFRESH`.
## Notes
* The current S3 Tables destination is logs-only.
* Partitioning is fixed to `service + ts_hour` for this exporter path.
* Existing tables must already match the required schema and partition contract.
* The advanced knobs trade freshness for larger files, fewer commits, and slower Iceberg metadata growth.
* Snowflake reads from the same Iceberg table through the AWS Glue catalog path after the table is provisioned correctly.
## Tuning guidance
* If metadata grows too fast or S3 Tables starts creating many tiny files, raise `target_partition_file_bytes` first.
* If exporter memory is the main problem, keep `partition_buffer_compression=zstd` and raise `partition_buffer_chunk_target_bytes` before shrinking file targets.
* If sparse services take too long to arrive, lower `max_partition_file_age` before shrinking file-size targets.
* If memory usage grows under high-cardinality service fanout, lower `max_open_partitions` or `max_buffered_partition_bytes`.
* If your rows are unusually wide, set `max_records_per_partition_file` to a positive value instead of aggressively shrinking byte targets.
## Efficiency Metrics
The collector now emits additional metrics for this destination:
* `s3tablesexporter.open_partitions`
* `s3tablesexporter.buffered_partition_bytes`
* `s3tablesexporter.compressed_buffered_partition_bytes`
* `s3tablesexporter.partition_buffer_compression_ratio`
* `s3tablesexporter.partition_buffer_compression_duration_milliseconds`
* `s3tablesexporter.partition_buffer_decompression_duration_milliseconds`
* `s3tablesexporter.flush_size`
* `s3tablesexporter.file_size`
* `s3tablesexporter.files_per_commit`
* `s3tablesexporter.flush_latency_milliseconds`
* `s3tablesexporter.upload_latency_milliseconds`
* `s3tablesexporter.commit_latency_milliseconds`
* `s3tablesexporter.commit_conflicts`
* `s3tablesexporter.forced_partition_flushes`
* `s3tablesexporter.unknown_commit_outcomes`
## Create The Table
For the current native writer, create an Iceberg table that matches the fixed Sawmills v1 contract exactly.
Recommended naming:
* Namespace: `telemetry_logs`
* Table name: `logs_service_hour_v1`
### Required schema
Use these 16 columns:
| Column | Type | Notes |
| :- | :- | :- |
| `ts` | `timestamp` | Event timestamp in microseconds. |
| `ts_raw` | `string` | Raw timestamp as text. |
| `service` | `string` | Lowercased service name. |
| `status` | `string` | Log status / severity. |
| `message_text` | `string` | Rendered message body. |
| `host` | `string` | Host name. |
| `source` | `string` | Log source. |
| `trace_id` | `string` | Trace ID when present. |
| `span_id` | `string` | Span ID when present. |
| `schema_version` | `string` | Current value is `snowflake_v1`. |
| `body_json_text` | `string` | Optional JSON body payload. |
| `attributes_hot_text` | `string` | Canonical JSON text. |
| `attributes_cold_text` | `string` | Canonical JSON text. |
| `tags_hot_text` | `string` | Canonical JSON text. |
| `tags_cold_text` | `string` | Canonical JSON text. |
| `source_file` | `string` | File / source marker. |
### Required partition spec
Partition the table by:
* `service` using the `identity` transform
* `ts` using the `hour` transform, with partition field name `ts_hour`
### Example AWS CLI create flow
The AWS CLI supports creating S3 Tables Iceberg tables directly. Save the metadata below as `s3tables-schema.json` and create the table with:
```bash theme={null}
aws s3tables create-table \
--region us-east-1 \
--table-bucket-arn arn:aws:s3tables:us-east-1:123456789012:bucket/your-table-bucket \
--namespace telemetry_logs \
--name logs_service_hour_v1 \
--format ICEBERG \
--metadata file://s3tables-schema.json
```
This command requires at least:
* `s3tables:CreateTable`
* `s3tables:PutTableData` when you pass `--metadata`
Example metadata file:
```json theme={null}
{
"iceberg": {
"schema": {
"fields": [
{ "id": 1, "name": "ts", "type": "timestamp", "required": false },
{ "id": 2, "name": "ts_raw", "type": "string", "required": false },
{ "id": 3, "name": "service", "type": "string", "required": false },
{ "id": 4, "name": "status", "type": "string", "required": false },
{ "id": 5, "name": "message_text", "type": "string", "required": false },
{ "id": 6, "name": "host", "type": "string", "required": false },
{ "id": 7, "name": "source", "type": "string", "required": false },
{ "id": 8, "name": "trace_id", "type": "string", "required": false },
{ "id": 9, "name": "span_id", "type": "string", "required": false },
{ "id": 10, "name": "schema_version", "type": "string", "required": false },
{ "id": 11, "name": "body_json_text", "type": "string", "required": false },
{ "id": 12, "name": "attributes_hot_text", "type": "string", "required": false },
{ "id": 13, "name": "attributes_cold_text", "type": "string", "required": false },
{ "id": 14, "name": "tags_hot_text", "type": "string", "required": false },
{ "id": 15, "name": "tags_cold_text", "type": "string", "required": false },
{ "id": 16, "name": "source_file", "type": "string", "required": false }
]
},
"partitionSpec": {
"specId": 0,
"fields": [
{ "sourceId": 3, "transform": "identity", "name": "service", "fieldId": 1000 },
{ "sourceId": 1, "transform": "hour", "name": "ts_hour", "fieldId": 1001 }
]
}
}
}
```
### AWS-side permissions
The collector needs:
* S3 Tables REST access to the table bucket
* table-scoped write permissions such as `s3tables:PutTableData` on the target table or table bucket
* any additional S3 Tables actions your provisioning path needs, such as `s3tables:GetTable` during validation or planning
* direct S3 access to the managed warehouse bucket behind the table
If you plan to query the same table from Snowflake through the Glue catalog path, you also need to grant the Snowflake side access to:
* the AWS Glue catalog entry for the table
* Lake Formation `DESCRIBE` and `SELECT` on the target table if Lake Formation is enforcing permissions
### Example collector IAM
The collector needs both S3 Tables control-plane access and direct S3 data-plane access to the managed warehouse bucket for the table.
Attach the AWS-managed policy:
```json theme={null}
{
"PolicyName": "AmazonS3TablesFullAccess",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3tables:*"
],
"Resource": "*"
}
]
}
```
Then add an inline policy for the managed warehouse bucket:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "WarehouseBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::"
]
},
{
"Sid": "WarehouseObjects",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::/*"
]
}
]
}
```
You can discover the managed warehouse bucket after table creation:
```bash theme={null}
aws s3tables get-table \
--region us-east-1 \
--table-bucket-arn arn:aws:s3tables:us-east-1::bucket/ \
--namespace telemetry_logs \
--name logs_service_hour_v1 \
--query '{tableARN:tableARN,metadataLocation:metadataLocation,warehouseLocation:warehouseLocation}' \
--output json
```
The `warehouseLocation` value looks like:
```text theme={null}
s3://
```
Use that bucket name in the inline IAM policy above.
## Query From Snowflake
S3 Tables uses the AWS Glue Iceberg REST catalog path. The validation path that worked end to end used:
* AWS IAM role for Snowflake SigV4 auth
* Snowflake catalog integration with `CATALOG_SOURCE = ICEBERG_REST`
* Snowflake Iceberg table object pointing at `CATALOG_TABLE_NAME` and `CATALOG_NAMESPACE`
* explicit `GRANT USAGE ON INTEGRATION`
* explicit `ALTER ICEBERG TABLE ... SET AUTO_REFRESH = TRUE`
### 1. Create the AWS IAM role Snowflake will assume
Create a role in your AWS account with permissions for Glue and, if Lake Formation is enabled, the matching Lake Formation table permissions.
Permissions policy example:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GlueRead",
"Effect": "Allow",
"Action": [
"glue:GetCatalog",
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetTableVersion",
"glue:GetTableVersions",
"glue:GetPartition",
"glue:GetPartitions"
],
"Resource": "*"
}
]
}
```
You can create the role first with a placeholder trust relationship, then tighten it after Snowflake returns the final `API_AWS_IAM_USER_ARN` and `API_AWS_EXTERNAL_ID`.
Initial trust policy example:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::user/"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": ""
}
}
}
]
}
```
### 2. Create the Snowflake catalog integration
Use `ICEBERG_REST`, not the older `GLUE` catalog integration shape.
```sql theme={null}
USE ROLE ACCOUNTADMIN;
CREATE OR REPLACE CATALOG INTEGRATION MY_S3TABLES_GLUE_REST_INT
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'telemetry_logs'
REFRESH_INTERVAL_SECONDS = 30
REST_CONFIG = (
CATALOG_URI = 'https://glue.us-east-1.amazonaws.com/iceberg'
CATALOG_API_TYPE = AWS_GLUE
WAREHOUSE = ':s3tablescatalog/'
ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
)
REST_AUTHENTICATION = (
TYPE = SIGV4
SIGV4_IAM_ROLE = 'arn:aws:iam:::role/'
SIGV4_SIGNING_REGION = 'us-east-1'
)
ENABLED = TRUE;
```
Important:
* In `REST_CONFIG`, specify `WAREHOUSE` and do **not** also specify `CATALOG_NAME`.
* The `WAREHOUSE` format is `:s3tablescatalog/`.
Immediately inspect the integration:
```sql theme={null}
DESCRIBE INTEGRATION MY_S3TABLES_GLUE_REST_INT;
```
Record these properties:
* `API_AWS_IAM_USER_ARN`
* `API_AWS_EXTERNAL_ID`
* `REST_CONFIG`
* `REST_AUTHENTICATION`
Then update the AWS IAM role trust policy so the principal and external ID exactly match the values returned by `DESCRIBE INTEGRATION`.
### 3. Grant Lake Formation permissions
If Lake Formation is enabled, grant `DESCRIBE` and `SELECT` on the table to the Snowflake AWS role you used in `SIGV4_IAM_ROLE`.
```bash theme={null}
aws lakeformation grant-permissions \
--region us-east-1 \
--principal DataLakePrincipalIdentifier=arn:aws:iam:::role/ \
--resource '{
"Table": {
"CatalogId": "",
"DatabaseName": "telemetry_logs",
"Name": "logs_service_hour_v1"
}
}' \
--permissions DESCRIBE SELECT
```
If your Snowflake environment uses a Snowflake-managed AWS IAM user for the integration handshake, grant the same permissions to that user too:
```bash theme={null}
aws lakeformation grant-permissions \
--region us-east-1 \
--principal DataLakePrincipalIdentifier=arn:aws:iam:::user/ \
--resource '{
"Table": {
"CatalogId": "",
"DatabaseName": "telemetry_logs",
"Name": "logs_service_hour_v1"
}
}' \
--permissions DESCRIBE SELECT
```
### 4. Create the Snowflake Iceberg table object
Create the database, schema, and query role first if they do not already exist:
```sql theme={null}
USE ROLE ACCOUNTADMIN;
CREATE DATABASE IF NOT EXISTS OBSERVABILITY;
CREATE SCHEMA IF NOT EXISTS OBSERVABILITY.ICEBERG;
CREATE ROLE IF NOT EXISTS OBSERVABILITY_ICEBERG_READER;
GRANT USAGE ON DATABASE OBSERVABILITY TO ROLE OBSERVABILITY_ICEBERG_READER;
GRANT USAGE ON SCHEMA OBSERVABILITY.ICEBERG TO ROLE OBSERVABILITY_ICEBERG_READER;
```
Then create the Iceberg table object:
```sql theme={null}
USE ROLE ACCOUNTADMIN;
USE DATABASE OBSERVABILITY;
USE SCHEMA ICEBERG;
CREATE OR REPLACE ICEBERG TABLE LOGS_SERVICE_HOUR_V1
CATALOG = 'MY_S3TABLES_GLUE_REST_INT'
CATALOG_TABLE_NAME = 'logs_service_hour_v1'
CATALOG_NAMESPACE = 'telemetry_logs';
```
If another role will own or query this table, grant it integration usage explicitly:
```sql theme={null}
GRANT USAGE ON INTEGRATION MY_S3TABLES_GLUE_REST_INT TO ROLE OBSERVABILITY_ICEBERG_READER;
GRANT SELECT ON TABLE OBSERVABILITY.ICEBERG.LOGS_SERVICE_HOUR_V1 TO ROLE OBSERVABILITY_ICEBERG_READER;
```
Then switch to the role that will operate on the table and enable refresh:
```sql theme={null}
USE ROLE OBSERVABILITY_ICEBERG_READER;
ALTER ICEBERG TABLE OBSERVABILITY.ICEBERG.LOGS_SERVICE_HOUR_V1
SET AUTO_REFRESH = TRUE;
```
Why document it this way:
* the table object can exist while `AUTO_REFRESH` is still off
* the reader can stop on an integration-permission issue
* the exact fixes were `GRANT USAGE ON INTEGRATION ...` and `ALTER ICEBERG TABLE ... SET AUTO_REFRESH = TRUE`
### 5. Validate Snowflake refresh health
Run this before debugging the collector:
```sql theme={null}
SELECT SYSTEM$AUTO_REFRESH_STATUS(
'OBSERVABILITY.ICEBERG.LOGS_SERVICE_HOUR_V1'
);
```
Read these fields carefully:
* `executionState`
* `currentSnapshotId`
* `lastSnapshotTime`
* `pendingSnapshotCount`
* `invalidExecutionStateReason`
Healthy example:
* `executionState = RUNNING`
Common failure patterns:
* `GENERALIZED_PIPE_STOPPED`
* `invalidExecutionStateReason` mentioning integration access control
* `pendingSnapshotCount` growing faster than `lastSnapshotTime` advances
### 6. Query it
Example:
```sql theme={null}
USE ROLE OBSERVABILITY_ICEBERG_READER;
SELECT
service,
message_text,
schema_version,
ts_raw
FROM OBSERVABILITY.ICEBERG.LOGS_SERVICE_HOUR_V1
WHERE service = 'your_service_name'
ORDER BY ts DESC
LIMIT 100;
```
If this query fails while the collector is writing successfully, re-check:
* the table schema matches the 16-column contract above
* the partition spec is `service + hour(ts)`
* `AUTO_REFRESH` is enabled on the Snowflake table object
* `GRANT USAGE ON INTEGRATION ...` was applied to the role you are using
* Lake Formation grants are present
* the Snowflake catalog integration trust relationship is valid
* `SYSTEM$AUTO_REFRESH_STATUS(...)` is healthy
### 7. Production operating notes
For a production table, the main operational risk is usually Snowflake refresh lag rather than collector write correctness.
Watch these regularly:
* `SYSTEM$AUTO_REFRESH_STATUS(...)`
* `executionState`
* `lastSnapshotTime`
* `pendingSnapshotCount`
If `executionState` is healthy but `pendingSnapshotCount` keeps growing, Snowflake is falling behind the table's snapshot churn. In that case:
* reduce commit frequency from writers if possible
* avoid unnecessary tiny commits
* verify the Snowflake reader role still has `USAGE` on the integration
* verify Lake Formation access has not drifted
If `executionState` is not healthy, inspect `invalidExecutionStateReason` first. That field is usually the fastest path to the real issue.
## AWS Credential Configuration
Sawmills Collector runs on Kubernetes. Prefer IAM-based access, such as IRSA on EKS, so the collector can reach both the S3 Tables API and the managed warehouse bucket.
If you use a pod role or node role, make sure that role includes:
* `AmazonS3TablesFullAccess`
* the inline managed-warehouse-bucket policy shown above
Only use static credentials if your environment cannot provide a workload identity.
```yaml theme={null}
extraEnvs:
- name: AWS_ACCESS_KEY_ID
value: ""
- name: AWS_SECRET_ACCESS_KEY
value: ""
```
# Snowflake Destination
Source: https://docs.sawmills.ai/docs/destination-snowflake
Export logs, metrics, and traces to Snowflake from Sawmills. Configure account ID, credentials, data warehouse, database, schema, and table settings.
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
### Snowflake Destination Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | none | true | Unique identifier within Sawmills. | |
| Account ID | String | none | true | The account identifier number of the Snowflake account to which telemetry data will be sent. |
| Username | String | none | true | The username for the account used by the Destination to authenticate with Snowflake. |
| Password | String | none | true | The password for the account used by the Destination to authenticate with Snowflake. |
| Role | String | none | false | The Snowflake role the exporter should use to obtain the correct permissions; specify this only if the user's default role does not have the necessary permissions. |
| Data Warehouse | String | none | true | The Snowflake data warehouse that should be used for storing telemetry data. |
| Database Name | String | telemetry | false | The Snowflake database where the destination will store telemetry data; it will create the database if it doesn't exist. |
| Schema Name | String | logs | false | Schema in which to store the log table. |
| Database Table | String | data | false | Table in which to store the log table. |
## Prerequisites
* A Snowflake account with a database, warehouse, and user.
* A Sawmills Collector instance.
* Network connectivity between the source and destination clusters.
### Snowflake Setup Script
The following SQL script creates the necessary user, role, and permissions in Snowflake. Each section performs specific setup tasks:
```sql theme={null}
begin;
/* Set variables for script. Replace the values in {brackets} with your own values */
set var_user = '{user}'; -- Username for the Sawmills service account
set var_password = '{password}'; -- Secure password for the service account
set var_role = '{role}'; -- Name of the role to be created (e.g., 'SAWMILLS_ROLE')
set var_database = '{database}'; -- Database where telemetry will be stored
set var_warehouse = '{warehouse}'; -- Warehouse to be used for data processing
/* Create and configure Sawmills role and user */
use role securityadmin; -- Switch to security admin to manage roles and users
create role if not exists identifier($var_role);
grant role identifier($var_role) to role SYSADMIN;
create user if not exists identifier($var_user)
password = $var_password
default_role = $var_role
default_warehouse = $var_warehouse;
grant role identifier($var_role) to user identifier($var_user);
/* Grant necessary permissions */
use role accountadmin; -- Switch to account admin for granting warehouse access
grant USAGE -- Allow the role to use the warehouse
on warehouse identifier($var_warehouse)
to role identifier($var_role);
grant CREATE SCHEMA, MONITOR, USAGE -- Grant database-level permissions
on database identifier($var_database)
to role identifier($var_role);
commit;
```
This script performs the following actions:
1. **Variable Setup**: Defines variables for all the configurable values needed in the script.
2. **Role Creation**:
* Switches to the `securityadmin` role for security-related operations
* Creates a new role for Sawmills if it doesn't exist
* Grants the new role to `SYSADMIN` for management purposes
3. **User Creation**:
* Creates a new user account for Sawmills
* Sets the default role and warehouse for the user
* Associates the newly created role with the user
4. **Permission Setup**:
* Switches to `accountadmin` for granting warehouse access
* Grants warehouse usage permissions to the role
* Grants necessary database permissions (CREATE SCHEMA, MONITOR, USAGE) to the role
After running this script, your Snowflake environment will be properly be configured for use with Sawmills.
## How to get the account ID:
1. Login to your Snowflake account.
2. Click on the user menu in the bottom-left corner and select **Account**.
3. Hover over the **Account** you want to use and copy the **Account Identifier**.
4. Replace the `.` with `-` in the account identifier.
# Splunk HEC Destination
Source: https://docs.sawmills.ai/docs/destination-splunk-hec
Export logs, metrics, and traces to Splunk via the HTTP Event Collector protocol from Sawmills. Configure endpoint URL, token, and index settings.
## Supported Data Types:
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
### Splunk HEC Destination Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Endpoint | String | none | true | The Splunk HTTP Event Collector (HEC) endpoint where telemetry data is sent. Format: `https://splunk:8088/services/collector`. Consider separating to address, port, and path. |
| Account Token | String | none | true | The authentication token required for sending data to Splunk. |
| Index | String | none | false | Optional name of the Splunk index targeted. |
| Enabled Data Types | Checkbox | Metrics, Logs, Traces | | The telemetry data types you want to export. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Source | String | none | false | Default Splunk source assigned to all events sent through this destination. |
| Source Type | String | none | false | Default Splunk source type assigned to all events sent through this destination. |
| Skip TLS Verification | Boolean | false | false | Enabling this setting skips verifying the server's certificate chain when sending data over HTTPS. |
### Setting Source and Source Type
By default, sources like Fluent Forward do not set Splunk `source` and `sourcetype`. You can set fallback values for all events through the `Source` and `Source Type` destination options above.
To set them dynamically per-event, use a [Modify Attribute Processor](/docs/processor-modify-attribute) to set `Resource.com\.splunk\.source` and `Resource.com\.splunk\.sourcetype` — either to a static value or from an existing log attribute. When present, these resource attributes override the static destination values.
# Sumo Logic Destination
Source: https://docs.sawmills.ai/docs/destination-sumologic
Export logs to Sumo Logic from Sawmills via an OTLP/HTTP source endpoint. Configure log format, compression, batch size, and advanced options.
## Supported Data Types:
📘 **Logs**
## Configuration
### Sumo Logic Destination Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Endpoint | String | none | true | The Sumo Logic OTLP/HTTP source endpoint where telemetry data is sent (e.g., `https://endpoint4.collection.sumologic.com/receiver/v1/otlp/`). |
| Log Format | String | OTLP | true | Format to use when sending logs to Sumo Logic. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Compression | String | gzip | false | The type of compression applied to the data before sending to Sumo Logic. |
| Sticky session support | Boolean | false | false | Defines if sticky session support is enabled. |
| Max request body size (bytes) | Number | 1048576 | false | Maximum HTTP request body size in bytes before compression. |
| Timeout (seconds) | Number | 30 | false | Timeout in seconds for every attempt to send data to the backend. |
# Getting Started with Sawmills
Source: https://docs.sawmills.ai/docs/getting-started
Introduction to Sawmills, an AI-powered telemetry management platform. Learn the core concepts of Collectors, Pipelines, Processors, and Destinations.
This page gives you a quick, practical orientation to Sawmills: what it is, what problems it solves, and the core building blocks you’ll use to configure and operate the Sawmills telemetry platform.
***
## What is Sawmills.ai?
Sawmills.ai is an AI-powered telemetry (logs, metrics, traces) management platform built on open standards (OpenTelemetry) that helps engineering teams regain control of observability data before it becomes an expensive, noisy, and fragile firehose. Instead of treating logs, metrics, and traces as “ship everything and pray,” Sawmills continuously analyzes telemetry as it flows through your pipelines, identifies waste and risk (for example, verbose logging, duplicate attributes, or high-cardinality metrics), and helps you apply optimizations that reduce cost while preserving the visibility you actually need.
Sawmills sits in front of your existing observability tools and works with them rather than replacing them. You can design and manage pipelines powered by the OpenTelemetry Collector, route data to multiple destinations, and keep things vendor-agnostic.
After you understand the concepts below, you’ll be ready to deploy Collectors, define pipelines, and start tightening control over volume, quality, routing, and cost.
***
## Key Concepts
This section introduces the key concepts essential to understanding how Sawmills works. These components are the building blocks of all Sawmills deployments and collectively define how data flows through the system.
***
## 1. Collector
The central component of Sawmills is the `Sawmills Collector`. It is based on OpenTelemetry’s [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). It is responsible for gathering, processing, and exporting telemetry logs, metrics, and traces. The Collector decouples generation and analysis of observability data, allowing teams to manage data collection, transformation, and forwarding in a streamlined, scalable manner.
### Key Collector Features
* Vendor-agnostic: Integrates with many sources and destinations, reducing vendor lock-in.
* Scalable: Supports distributed deployments to handle high-throughput telemetry data.
* Flexible: Enables custom processing, transformation, and enrichment before data is sent downstream.
In Sawmills, the Collector is configured through `pipelines` that specify how telemetry data flows from sources to destinations.
***
## 2. Remote Operator
The `Sawmills Remote Operator` is the control plane component that manages Collector behavior in the field. It coordinates configuration deployments, reports on collector health and held acessing live data stream.
***
## 3. Pipelines
A `Pipeline` is a sequence of sources, processors, and destinations that define how data moves through Sawmills. Each component of a pipeline can be configured to specify how telemetry data is received, transformed, enriched, and forwarded.
A pipeline can run on many Collectors, and Collectors can run many pipelines.
### Sources
Sources are the entry points for telemetry data in Sawmills. They specify the origin of your data (application logs, infrastructure metrics, traces, and other telemetry signals). Under the hood, sources typically map to OpenTelemetry Collector receivers, supporting a variety of protocols and formats.
Common source patterns:
* Log sources: Ingest logs from applications, servers, or services.
* Metric sources: Collect metrics from infrastructure or applications.
* Trace sources: Gather traces from distributed applications.
### Destinations
Destinations are endpoints where telemetry data is sent after processing. Under the hood, destinations typically map to OpenTelemetry Collector exporters. Sawmills supports many destination types, providing flexibility in routing data to the tools and storage systems your teams already use.
Common destination patterns:
* Monitoring tools: Datadog, Prometheus, New Relic, and other providers of real-time analysis.
* Logging and storage systems: Elasticsearch or object storage (for example, Amazon S3) for longer-term retention.
* Analytics platforms: Systems used for aggregation, reporting, and analysis.
### Processors
Processors define how telemetry data is transformed as it flows through the pipeline (for example, filtering, enrichment, aggregation, routing logic). See the `Processors` section below for deeper detail.
***
## 4. Processors
`Processors` modify and/or enrich telemetry data as it flows through a pipeline. They apply transformations that clean, filter, aggregate, or add context to the data, making it more meaningful and easier (and cheaper) to use downstream.
### Processor Functions
Here are 5 common examples:
* Filtering: Remove unwanted data to reduce noise and cost.
* Transformation: Rename/copy/drop/normalize fields and attributes to standardize data.
* Sampling: Keep a representative subset of high-volume telemetry while preserving key signals.
* Routing: Send different subsets of data to different destinations based on rules.
* Redaction/masking: Remove or obfuscate sensitive values before exporting data.
These are just examples—there are many other processors and functions available depending on the signal type (logs, metrics, traces) and the tranformation type.
By configuring processors, you can create custom processing rules that streamline data and tailor it to your specific needs.
In a pipeline, data flows from all sources to all destinations.
***
## 5. Live-tail
`Live-tail` provides a real-time streaming view of your telemetry as it flows through the pipeline. It is designed for rapid debugging and validation, allowing teams to confirm that data is arriving, being transformed, and routed correctly.
### Live-tail Use Cases
* Pipeline validation: Verify that a new source or processor is emitting the expected output.
* Troubleshooting: Spot anomalies, errors, or unexpected attributes in near real time.
* Ad-hoc exploration: Filter and inspect current data without waiting for downstream storage or indexing.
***
## 6. Processor Simulation
`Processor simulation` lets you test processor logic against sample payloads before applying changes to production data. This helps teams iterate on filters, transforms, or enrichments with confidence.
### Why Simulation Matters
* Safe experimentation: Try transformations without impacting live traffic.
* Faster iteration: Validate outputs immediately and refine logic.
* Predictable outcomes: Understand how new rules affect logs, metrics, and traces.
***
## 7. Dashboard
The `Dashboard` is the operational command center for Sawmills. It provides visibility into pipeline health, data volume, and processing effectiveness, all in a single view.
### Dashboard Highlights
* Pipeline status: See which pipelines are active and where they run.
* Throughput metrics: Monitor data ingestion and export rates.
* Value provided: see total data reduced by time and by attribute.
***
## 8. Insights
`Insights` turn raw telemetry operations into actionable intelligence. They help you understand how data moves, where it is enriched or filtered, and what impact processors have on downstream costs and quality.
### Insights Benefits
* Data optimization: Detect noisy sources and opportunities to reduce volume.
* Quality improvements: Track how enrichment changes data usefulness.
***
## 9. OTLP in the Collector
Sawmills uses `OTLP (OpenTelemetry Protocol)` to standardize telemetry data as it passes through the Collector. OTLP defines a clear data model for logs, metrics, and traces, which the Collector uses internally while processing pipeline traffic. For details, see the [OTLP data model in the OpenTelemetry docs](https://opentelemetry.io/docs/specs/otlp/).
### OTLP Structure by Signal
* Logs: Log Records are grouped in `ResourceLogs`, and each log record includes attributes, timestamps, severity, and optional trace context for correlation.
* Metrics: Metric data is organized in `ResourceMetrics`, containing metric instruments (for example, counters, gauges, histograms) and their data points.
* Traces: Traces are represented as `ResourceSpans`, which contain spans with links, events, and attributes that describe distributed operations.
This common OTLP model allows Sawmills processors to apply consistent transformations and enrichments across all telemetry signals while preserving relationships between logs, metrics, and traces.
***
## 10. Deployment Model for Processors and Pipelines
Sawmills separates processor delivery from pipeline deployment to keep operational changes fast and safe:
* Deploy processors without collector redeployment: Processor logic can be added or updated centrally and delivered to Collectors without rebuilding or redeploying the Collector binary.
* Pipeline changes still require deployment: Changes to pipeline structure (for example, adding a new source, destination, or reordering processors) require a pipeline deployment so the Collector loads the updated configuration.
This separation lets teams iterate quickly on processor logic while keeping structural pipeline changes deliberate and controlled.
***
## Summary
* Collector: The core component that manages data collection, processing, and forwarding.
* Remote Operator: Control plane for managing Collector configuration.
* Pipelines: Define the flow of telemetry data through sources, processors, and destinations.
* Processors: Transform and enrich data as it moves through a pipeline.
* Live-tail: Real-time streaming visibility into active telemetry traffic.
* Processor simulation: Safe testing of processor logic before live deployment.
* Dashboard: Central view of pipeline health, throughput, and operations.
* Insights: Analytics about data quality, volume, and processing impact.
* OTLP model: Standardized internal structure for logs, metrics, and traces.
* Deployment model: Rapid processor updates with deliberate pipeline deployments.
# Enable GitHub Integration
Source: https://docs.sawmills.ai/docs/github-integration
Install the Sawmills GitHub App and connect your repositories so the Instrumentation Reviewer can analyze telemetry changes in pull requests.
## About GitHub Integration
The GitHub Integration connects your GitHub repositories to Sawmills Telemetry Reviewer.
After the integration is enabled, Sawmills can use the installed GitHub App to access the repositories you selected for instrumentation review workflows.
See [Instrumentation Reviewer](/docs/sawmills-agent-instrumentation-reviewer).
## Step-by-Step: Configure GitHub Integration
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
### 2. Connect GitHub App
In the Integrations section, click **Connect GitHub App**.
### 3. Install the GitHub App
GitHub will open the installation flow. Choose the GitHub organization or account you want to connect, then select the repositories that should be available to Sawmills Telemetry Reviewer.
### 4. Return to Sawmills
After the GitHub App installation is complete, GitHub redirects you back to the Sawmills integrations page.
### 5. Verify the Integration
Confirm that the GitHub integration appears as connected. If needed, you can reconnect the app later to change the selected repositories.
## What the Integration Is Used For
The GitHub integration is used by Sawmills Telemetry Reviewer to access repository code for instrumentation review workflows.
The integration stores the GitHub App installation and uses short-lived installation tokens when reviewer workflows need repository access.
## Requested GitHub App Permissions
Sawmills Telemetry Reviewer currently requests these GitHub App permissions:
* **Contents: Read**
* **Pull requests: Write**
* **Issues: Write**
* **Checks: Read**
* **Statuses: Read**
These permissions are defined in the GitHub App manifest used by the integration.
## Why These Permissions Are Needed
These permissions let Telemetry Reviewer read pull request changes and repository context, publish review results back to GitHub, and support workflow actions such as comment-driven follow-up and check-run reruns.
# Enable GitLab Integration
Source: https://docs.sawmills.ai/docs/gitlab-integration
Connect your GitLab projects or groups to Sawmills so the Instrumentation Reviewer can analyze telemetry changes in merge requests via webhook.
## About GitLab Integration
The GitLab Integration connects your GitLab project or group to Sawmills Telemetry Reviewer.
After the integration is enabled, Sawmills can use the authorized GitLab access and webhook to support instrumentation review workflows.
See [Instrumentation Reviewer](/docs/sawmills-agent-instrumentation-reviewer).
## Step-by-Step: Configure GitLab Integration
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
### 2. Start GitLab Setup
In the Integrations section, click **Connect GitLab**.
### 3. Enter Integration Details
Provide:
* **Integration name**
* **Scope type**: project or group
* **Scope path**
Use a full GitLab project path such as `group/project` for project-level setup.
### 4. Continue to GitLab
Click **Continue to GitLab** to start the authorization flow.
### 5. Authorize and Return
Authorize the connection in GitLab. After authorization completes, GitLab redirects you back to the Sawmills integrations page and Sawmills finishes creating the integration.
### 6. Verify the Integration
Confirm that the GitLab integration appears as connected for the selected project or group.
## What the Integration Is Used For
The GitLab integration is used by Sawmills Telemetry Reviewer to access repository code for instrumentation review workflows.
For GitLab, Sawmills creates a webhook for the selected project or group and stores the authorized access needed for reviewer operations.
## Requested GitLab Token Access
For GitLab, Sawmills Telemetry Reviewer needs:
* A token for the selected GitLab host
* Access to the selected **project** or **group**
* **Maintainer** access or higher on the relevant project scope
* A token that can call the GitLab API for both read and write operations
In practice, this means using a GitLab token with API access for the selected scope.
## Why This Access Is Needed
This access lets Telemetry Reviewer read merge request changes and repository context, publish review results back to GitLab, create the webhook used to trigger reviewer runs, and support note-driven commands such as rerun or cancel.
# Monitor Sawmills with Prometheus
Source: https://docs.sawmills.ai/docs/guide-how-to-monitor-with-prometheus
Scrape Sawmills Collector metrics with any Prometheus-compatible scraper. Includes endpoint, key metrics to watch, and sample alert rules.
The Sawmills Collector exposes OpenTelemetry internal metrics in Prometheus format. Any Prometheus-compatible scraper (Prometheus, VictoriaMetrics, Grafana Agent, Datadog Agent, etc.) can collect them. This guide lists the endpoint, the metrics worth watching, and sample alert rules. A kube-prometheus-stack example is included at the end.
## Scrape Endpoint
Scrape every collector pod at:
* **Port:** `19465`
* **Path:** `/metrics`
* **Recommended interval:** `15s`
## Collector Configuration
The Prometheus port is set by the Helm chart — no collector-config edits are needed. To override the default, set the value under `managedChartsValues.sawmills-collector` in your remote-operator `values.yaml` (see [Updating Collector Values](/docs/collector-customization#updating-collector-values)):
```yaml theme={null}
managedChartsValues:
sawmills-collector:
telemetry:
prometheus:
port: 19465 # default
```
## Metrics to Monitor
The collector emits per-signal counters with the suffix `_log_records_total`, `_metric_points_total`, or `_spans_total`. Where this guide shows ``, substitute the signal you ingest. All counters carry the `_total` suffix in OpenTelemetry's Prometheus exposition.
1. **Liveness**
* `up{job=""}` — scrape target reachable (the `job` label depends on your scraper config)
* `otelcol_process_uptime_total` — monotonic uptime; resets on restart
2. **Resource Usage**
* `otelcol_process_cpu_seconds_total` — CPU rate per pod
* `otelcol_process_memory_rss` — resident memory per pod
* `otelcol_process_runtime_heap_alloc_bytes` — Go heap (use to detect leaks)
3. **Ingestion (Receiver Side)**
* `otelcol_receiver_accepted__total` — successful ingest, broken down by `receiver`
* `otelcol_receiver_refused__total` — input-side rejections; common causes: malformed data, incompatible protocol versions, rate limiting
4. **Egress (Exporter Side)**
* `otelcol_exporter_sent__total` — successful sends, broken down by `exporter`
* `otelcol_exporter_send_failed__total` — downstream send errors; common causes: network issues, authentication failures, backend overload
* `otelcol_exporter_enqueue_failed__total` — drops at the queue boundary (data lost before it could be sent)
5. **Backpressure**
* `otelcol_exporter_queue_size` / `otelcol_exporter_queue_capacity` — pair these for utilization ratio. Sustained high utilization indicates the collector cannot keep up with the destination.
## Label Conventions
Sawmills emits processor instances with the format `/`. Real `receiver` and `exporter` label values look like:
* `receiver="otlp/collector-backend/otlp/72cfa7f8-00ae-40b4-a4eb-dc6f59932c29"`
* `exporter="datadog/ec03b655-369f-4273-b255-8ed7dd33366f"`
* `exporter="awss3/sampling-destination-1876f24b-3bdb-4fe5-93e1-fa36cb89e864"`
When writing alert rules, use regex matchers (`exporter=~"datadog/.*"`) rather than literal equality. The `otelcol_exporter_queue_size` and `_queue_capacity` series also carry a `data_type` label (`logs`, `metrics`, `traces`) — if you have multiple signals exported, group or filter on it.
## A Note on Lazy Emission
OpenTelemetry only exposes a counter after its first non-zero observation. `otelcol_exporter_send_failed__total` and `otelcol_exporter_enqueue_failed__total` will be entirely absent from the scrape until at least one failure has occurred — this is normal, not a misconfiguration. Alerts on these metrics evaluate as "no data" rather than zero, so use `absent_over_time` or `OR vector(0)` if you need to distinguish "healthy" from "metric missing because never failed".
## Sample Alert Rules
The following are starting points and should be adjusted based on your collector configuration, data volumes, and tolerance for refusals. Replace `` with the actual `job` label your scraper assigns to the collector — open `/targets` in your Prometheus UI and copy the value shown on the collector rows.
```yaml theme={null}
groups:
- name: SawmillsCollector
rules:
- alert: SawmillsCollectorDown
expr: absent(up{job=""}) or up{job=""} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Sawmills collector scrape target down or missing"
description: "Prometheus has been unable to scrape the collector for 2 minutes (or the target has disappeared from service discovery)."
- alert: SawmillsReceiverRefusingLogs
expr: sum by (receiver) (rate(otelcol_receiver_refused_log_records_total{job="",receiver=~"otlp/.*"}[5m])) > 100
for: 15m
labels:
severity: warning
annotations:
summary: "Sawmills collector refusing logs at receiver {{ $labels.receiver }}"
description: "The receiver is refusing logs, possibly due to high load or malformed data."
- alert: SawmillsExporterSendFailingLogs
expr: sum by (exporter) (rate(otelcol_exporter_send_failed_log_records_total{job=""}[5m])) > 100
for: 15m
labels:
severity: critical
annotations:
summary: "Sawmills exporter {{ $labels.exporter }} failing to send logs to destination"
description: "The exporter is failing to send logs, possibly due to backend connectivity issues."
- alert: SawmillsExporterEnqueueFailingLogs
expr: sum by (exporter) (rate(otelcol_exporter_enqueue_failed_log_records_total{job=""}[5m])) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Sawmills exporter {{ $labels.exporter }} dropping logs at the export queue"
description: "Records are being dropped before they can be sent — the exporter queue is full or rejecting writes. This is data loss."
- alert: SawmillsExporterQueueSaturated
expr: max by (exporter, data_type) (otelcol_exporter_queue_size{job=""} / clamp_min(otelcol_exporter_queue_capacity{job=""}, 1)) > 0.8
for: 10m
labels:
severity: warning
annotations:
summary: "Sawmills exporter {{ $labels.exporter }} queue near capacity ({{ $labels.data_type }})"
description: "Queue utilization above 80% sustained — the destination is slower than ingest. Enqueue failures (data loss) follow if this persists."
```
## Appendix: Kube-Prometheus-Stack Example
If you run kube-prometheus-stack with Prometheus Operator, this ServiceMonitor wires up scraping directly. Adjust the `release:` label to match your Prometheus's `serviceMonitorSelector`.
### ServiceMonitor
```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sawmills-collector-monitor
namespace: sawmills
labels:
release: prometheus # Match your Prometheus serviceMonitorSelector
spec:
selector:
matchLabels:
app.kubernetes.io/instance:
namespaceSelector:
matchNames:
- sawmills
endpoints:
- port: prometheus
interval: 15s
```
Apply:
```bash theme={null}
kubectl apply -f sawmills-servicemonitor.yaml
```
### Verification
Port-forward Prometheus and check `/targets` for the `sawmills-collector-monitor` scrape pool:
```bash theme={null}
kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n
# http://localhost:9090/targets
```
For more information about OpenTelemetry collector metrics, visit the [official documentation](https://opentelemetry.io/docs/collector/internal-telemetry/).
# Autoscale Sawmills Collector with KEDA
Source: https://docs.sawmills.ai/docs/guide-how-to-setup-keda
Configure KEDA event-driven autoscaling for the Sawmills Collector in Kubernetes using Prometheus, OTLP external scaler, CPU, or memory metrics.
This guide explains how to configure [KEDA](https://keda.sh/) (Kubernetes Event-driven Autoscaling) to automatically scale your Sawmills Collector based on real-time metrics. KEDA enables dynamic, event-driven scaling, ensuring your collector adapts to changing workloads efficiently.
## Prerequisites
Before you begin, ensure you have:
* The Sawmills Collector deployed and running in the `sawmills` namespace
* KEDA installed in your Kubernetes cluster ([KEDA installation guide](https://keda.sh/docs/latest/deploy/))
* (Optional) Prometheus deployed if you want to use Prometheus-based scaling
## Enabling KEDA Autoscaling for Sawmills
KEDA can be enabled and configured via the Sawmills Collector Helm chart. For remote-operator installs, the configuration is managed under the `managedChartsValues.sawmills-collector.keda` section in your `values.yaml` file. The legacy `managedCharts` key still works as an alias.
### Step 1: Enable KEDA in Helm Values
Add or update the following section in your `values.yaml` of the remote operator:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
keda:
enabled: true
minReplicas: 1
maxReplicas: 10
pollingInterval: 30 # How often KEDA checks metrics (seconds)
cooldownPeriod: 300 # Wait time after last scaling before scaling down (seconds)
scaling:
prometheus:
enabled: false # Set true to use Prometheus-based scaling
metadata:
serverAddress: http://prometheus:9090
query: "histogram_quantile(0.95, sum(rate(http_server_duration_bucket[1m])) by (le))"
threshold: "2000"
activationThreshold: "1000"
external:
enabled: false # Set true to use OTLP-based external scaler (the Sawmills based solution)
metadata:
query: "histogram_quantile(0.95, sum(rate(http_server_duration_bucket[1m])) by (le))"
targetValue: "2000"
cpu:
enabled: true
targetUtilization: 80
memory:
enabled: true
targetUtilization: 80
```
* **minReplicas**/**maxReplicas**: Minimum and maximum number of collector pods.
* **pollingInterval**: How often KEDA checks metrics for scaling decisions.
* **cooldownPeriod**: How long to wait after scaling before scaling down.
* **scaling**: Configure one or more scaling sources (Prometheus, external, CPU, memory).
### Step 2: Apply the Configuration
After updating your `values.yaml`, upgrade your Sawmills Collector release:
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version \
--namespace sawmills \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress=https://controller.ue1.prod.plat.sm-svc.com \
--set collectorName="my-collector" \
-f values.yaml
```
## KEDA Scaling Options
You can configure KEDA to scale based on different metrics sources:
### 1. Prometheus-based Scaling
Enable `scaling.prometheus.enabled: true` and provide the Prometheus server address and PromQL query. Example:
```yaml theme={null}
scaling:
prometheus:
enabled: true
metadata:
serverAddress: http://prometheus:9090
query: "sum(rate(otelcol_receiver_accepted_spans[1m]))"
threshold: "1000"
activationThreshold: "500"
```
### 2. External Scaler (OTLP) Scaling
Enable `scaling.external.enabled: true` to use the KEDA OTLP external scaler. Example:
```yaml theme={null}
scaling:
external:
enabled: true
metadata:
query: "sum(rate(otelcol_receiver_accepted_spans[1m]))"
targetValue: "1000"
```
### 3. CPU/Memory-based Scaling
Enable `scaling.cpu.enabled` and/or `scaling.memory.enabled` to scale based on resource utilization:
```yaml theme={null}
scaling:
cpu:
enabled: true
targetUtilization: 80
memory:
enabled: true
targetUtilization: 80
```
## Example: Full KEDA Configuration
Here is a complete example for enabling KEDA with multiple scaling sources:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
keda:
enabled: true
minReplicas: 2
maxReplicas: 20
pollingInterval: 30
cooldownPeriod: 300
scaling:
prometheus:
enabled: true
metadata:
serverAddress: http://prometheus:9090
query: "sum(rate(otelcol_receiver_accepted_spans[1m]))"
threshold: "2000"
activationThreshold: "1000"
external:
enabled: false
metadata: {}
cpu:
enabled: true
targetUtilization: 75
memory:
enabled: false
```
## KEDA Scaler Component for OpenTelemetry
The Sawmills Helm chart includes an optional KEDA scaler component for advanced OTLP-based scaling. To enable it, set:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
kedaScaler:
enabled: true
# Additional configuration as needed
```
Refer to the Helm chart documentation for advanced configuration options.
## Verifying KEDA Autoscaling
1. **Check KEDA ScaledObject**
```bash theme={null}
kubectl get scaledobject -n sawmills
# Should show a ScaledObject for your collector
```
2. **Check Collector Pod Scaling**
```bash theme={null}
kubectl get pods -n sawmills -l app.kubernetes.io/instance=
# Observe the number of pods scaling up/down based on load
```
3. **Check KEDA Operator Logs**
```bash theme={null}
kubectl logs -n keda -l app=keda-operator
# Look for scaling decisions and errors
```
## Troubleshooting KEDA Autoscaling
* **ScaledObject Not Created**: Ensure `keda.enabled: true` and Helm upgrade completed successfully.
* **Pods Not Scaling**: Check metric queries, thresholds, and KEDA operator logs for errors.
* **Prometheus/External Metrics Not Detected**: Verify Prometheus/external scaler endpoints are reachable and queries return expected results.
* **Resource-based Scaling Not Working**: Ensure CPU/memory requests and limits are set on the collector pods.
* **Existing HPA Still Owns Scaling**: If you enabled KEDA on an existing collector, check for an old HPA targeting the same collector workload. KEDA creates and manages its own HPA through the ScaledObject.
## Best Practices for KEDA with Sawmills
* Start with conservative scaling thresholds and adjust based on observed workloads.
* Use multiple scaling sources (e.g., CPU and Prometheus) for robust autoscaling.
* Monitor KEDA and collector logs for scaling anomalies.
* Test scaling behavior under simulated load before deploying to production.
* For advanced scenarios, leverage the external scaler for custom OTLP metrics.
## References
* [KEDA Documentation](https://keda.sh/docs/)
* [Sawmills Collector Helm Chart](https://github.com/Sawmills/helm-charts)
* [KEDA Scalers Reference](https://keda.sh/docs/latest/scalers/)
For further assistance with KEDA configuration for Sawmills, contact the Sawmills team or consult the [Collector Customization Guide](./collector-customization).
# How to rehydrate logs to OpenSearch from S3
Source: https://docs.sawmills.ai/docs/guide-rehydrate-logs-to-opensearch-from-s3
Archive logs to a customer-owned S3 bucket via Sawmills and re-ingest them into OpenSearch on demand using Amazon OpenSearch Ingestion or OpenSearch Data Prepper.
This guide walks through setting up a "send a copy to S3, index it later in OpenSearch when needed" flow. Sawmills writes a full-fidelity archive copy to your own S3 bucket while live traffic continues to your OpenSearch destination. When you need older logs in OpenSearch, Amazon OpenSearch Ingestion or self-managed OpenSearch Data Prepper reads the S3 objects and writes them into an OpenSearch index.
## Prerequisites
* A Sawmills pipeline already shipping logs to an [Elasticsearch / OpenSearch destination](/docs/destination-elasticsearch).
* An AWS account where you can create an S3 bucket and grant read access to OpenSearch Ingestion or Data Prepper.
* An Amazon OpenSearch Service domain or an OpenSearch cluster.
* Admin access to create an OpenSearch Ingestion pipeline, or a self-managed OpenSearch Data Prepper deployment.
* A target rehydration index, such as `rehydrated-logs`, `rehydrated-logs-%{yyyy.MM.dd}`, or another index naming pattern that fits your retention policy.
***
## 1. Add the AWS S3 destination in Sawmills
This is the archive side. Sawmills writes a copy of every log record to S3 alongside the live OpenSearch feed.
1. Open your pipeline in Sawmills and click **+ Add Destination**.
2. Pick **AWS S3**.
3. Fill in the form:
| Field | Value |
| :- | :- |
| Name | e.g., `opensearch-rehydration-archive` |
| Region | The AWS region of your bucket, e.g., `us-west-2` |
| S3 Bucket | The bucket name you created, e.g., `acme-opensearch-archive` |
| S3 Prefix | A path inside the bucket, e.g., `opensearch/archive` |
| Role ARN | Leave blank for static IAM keys, or set if you use cross-account assume-role |
| Output Format | **`NDJSON (.json.gz)`** |
When you pick **Output Format = `NDJSON (.json.gz)`**, Sawmills automatically:
* Writes one JSON object per line.
* Compresses the object with gzip.
* Partitions objects by hour with keys like `dt=YYYYMMDD/hour=HH/archive_HHMMSS.NNNN..json.gz`.
* Locks **Enabled Data Types** to **Logs only**.
If your live OpenSearch destination has destination-specific processors that change field names, index names, or other document shape, decide whether the archive should store:
* **Raw pipeline logs**: simpler, but rehydrated documents may not exactly match the live OpenSearch destination.
* **OpenSearch-shaped logs**: add equivalent processors to the S3 archive destination so rehydrated documents match the live destination more closely.
For the AWS credentials the collector uses to **write** to S3, follow the credentials section in the [AWS S3 destination reference](/docs/destination-aws-s3).
4. Click **Save**, then deploy the pipeline.
After deploy, list the bucket to confirm objects are arriving:
```bash theme={null}
aws s3 ls s3://acme-opensearch-archive/opensearch/archive/dt=$(date -u +%Y%m%d)/ --recursive
```
You should see hourly-partitioned `archive_*.json.gz` files appearing within a minute or two.
***
## 2. Grant read access to the archive
OpenSearch Ingestion and Data Prepper need read access to the S3 bucket. If you use S3 event notifications, they also need access to the SQS queue.
For an Amazon OpenSearch Ingestion pipeline, configure the pipeline role with permissions to read the archive bucket and write to the OpenSearch sink. If you specify `sts_role_arn` in the pipeline configuration, use the same pipeline role in each component that declares it. AWS documents the S3 source role requirements in [Using an OpenSearch Ingestion pipeline with Amazon S3](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-s3.html).
A minimal S3 read policy looks like this:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListSawmillsArchiveBucket",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::acme-opensearch-archive"
},
{
"Sid": "ReadSawmillsArchiveObjects",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::acme-opensearch-archive/*"
}
]
}
```
If you use SQS notifications, add permissions such as `sqs:ReceiveMessage`, `sqs:DeleteMessage`, and `sqs:ChangeMessageVisibility` on the queue used by the S3 source.
***
## 3. Choose scan or SQS mode
OpenSearch supports two common ways to read S3 archive objects:
| Mode | Use When | Notes |
| :- | :- | :- |
| Scheduled or one-time S3 scan | You want to rehydrate a known historical prefix or time window. | Good first choice for an on-demand rehydration run. |
| S3 to SQS notifications | You want low-latency ingestion as new archive objects arrive. | Better for continuously tailing the archive. Requires S3 event notifications and SQS. |
OpenSearch Data Prepper's [`s3` source](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sources/s3/) supports both SQS notification processing and S3 scan processing. Amazon OpenSearch Ingestion also supports S3-SQS processing and scheduled scans.
For rehydrating a specific window, start with a narrow prefix, such as:
```text theme={null}
opensearch/archive/dt=20260504/hour=10/
```
The `dt` and `hour` folders represent when the archive object was written by the collector, in UTC. Choose `start_time` / `end_time` values that line up with those UTC partitions. Self-managed OpenSearch Data Prepper expects ISO `LocalDateTime` values without a timezone suffix, such as `2026-05-04T00:00:00`. Amazon OpenSearch Ingestion scheduled scans use UTC instants with a `Z` suffix, such as `2026-05-04T00:00:00.000Z`. If you need event-time filtering inside a large archive prefix, apply additional Data Prepper processors or OpenSearch queries after ingestion.
***
## 4. Create an OpenSearch Ingestion or Data Prepper pipeline
The pipeline reads the Sawmills archive from S3, parses each JSON line, and indexes the result into OpenSearch.
The self-managed Data Prepper example below uses S3 scan mode for an on-demand rehydration run:
```yaml theme={null}
version: "2"
sawmills-opensearch-rehydrate:
source:
s3:
codec:
newline:
compression: gzip
aws:
region: us-west-2
sts_role_arn: arn:aws:iam::123456789012:role/OpenSearchIngestionPipelineRole
scan:
start_time: 2026-05-04T00:00:00
end_time: 2026-05-04T01:00:00
buckets:
- bucket:
name: acme-opensearch-archive
filter:
include_prefix:
- opensearch/archive/dt=20260504/hour=00/
processor:
- parse_json:
sink:
- opensearch:
hosts:
- https://search-example-domain.us-west-2.es.amazonaws.com
index: rehydrated-logs-%{yyyy.MM.dd}
aws:
region: us-west-2
sts_role_arn: arn:aws:iam::123456789012:role/OpenSearchIngestionPipelineRole
```
The important parts are:
* `codec.newline`: reads each NDJSON line as one event.
* `compression: gzip`: reads the `.json.gz` archive objects written by Sawmills.
* `parse_json`: parses each JSON line into fields before indexing.
* `include_prefix`: limits the run to the archive prefix you want to rehydrate.
* `index`: writes into a dedicated rehydration index, avoiding accidental duplicates in the live index. The `%{yyyy.MM.dd}` placeholder resolves to **ingestion time** by default, so all rehydrated records land in the index for the day you ran the job. To group by original event time instead, add a `date` processor that maps your event timestamp field into `@timestamp` before the sink, or use a static index name such as `rehydrated-logs`.
OpenSearch Ingestion supports a subset of Data Prepper plugins and options. Validate the pipeline configuration against the OpenSearch Ingestion version and AWS account limits you are using.
***
## 5. Verify rehydration in OpenSearch
After the pipeline starts, verify that documents were indexed into the target rehydration index:
```http theme={null}
GET rehydrated-logs-*/_search
{
"size": 20,
"query": {
"match_all": {}
}
}
```
Search for a specific service or host:
```http theme={null}
GET rehydrated-logs-*/_search
{
"size": 20,
"query": {
"match": {
"service": "my-service"
}
}
}
```
If you see zero events:
* Confirm S3 objects exist under the exact prefix used by the pipeline.
* Confirm the pipeline role can read the bucket and, if configured, the SQS queue.
* Confirm the OpenSearch sink can write to the target index.
* Check OpenSearch Ingestion or Data Prepper logs for S3 read errors, JSON parse errors, and OpenSearch bulk indexing failures.
* Try a narrower S3 prefix first, then broaden after the first successful run.
***
## Operational guidance
Create an index template before rehydration if you need strict mappings for fields such as timestamps, keywords, and nested attributes. Without a template, OpenSearch dynamic mappings may infer types differently than your live index.
Keep the S3 prefix narrow for backfills. A daily prefix can contain many objects, and OpenSearch Ingestion/Data Prepper will read and index everything that matches the scan.
Delete or expire the rehydration index when the investigation is complete. The S3 archive remains the long-term source of truth.
# How to rehydrate logs to Splunk from S3
Source: https://docs.sawmills.ai/docs/guide-rehydrate-logs-to-splunk-from-s3
Archive logs to a customer-owned S3 bucket via Sawmills and re-ingest them into Splunk on demand using the Splunk Add-on for AWS Generic S3 input.
This guide walks through setting up a "send a copy to S3, query it later in Splunk when needed" flow. The shape is identical to the [Datadog Logs Rehydration](https://docs.datadoghq.com/logs/log_configuration/rehydrating/) pattern: the Sawmills collector dual-routes logs — live traffic continues to your Splunk HEC destination for alerting and dashboards, while a full-fidelity copy lands in your own S3 bucket as hourly-partitioned `.json.gz`. When you need older data in Splunk, the Splunk Add-on for AWS pulls those S3 objects back into a Splunk index.
This is the simplest way to support Splunk customers who want to reduce live-ingest cost without losing access to historical logs.
Sawmills writes the same archive format that Datadog's Logs Rehydration consumes (newline-delimited JSON, gzipped, hourly partitions). The format works with Splunk Cloud and Splunk Enterprise; the only thing that changes per-customer is which Splunk read mechanism you wire up at the end.
## Prerequisites
* A Sawmills pipeline already shipping to a [Splunk HEC destination](/docs/destination-splunk-hec).
* An AWS account where you can create an S3 bucket and an IAM user.
* Splunk Cloud or Splunk Enterprise admin access.
***
## 1. Add the AWS S3 destination in Sawmills
This is the archive side — Sawmills will write a copy of every log record to your S3 bucket alongside the live Splunk feed.
1. Open your pipeline in Sawmills and click **+ Add Destination**.
2. Pick **AWS S3**.
3. Fill in the form:
| Field | Value |
| :- | :- |
| Name | e.g., `splunk-rehydration-archive` |
| Region | The AWS region of your bucket (e.g., `us-east-1`) |
| S3 Bucket | The bucket name you created (e.g., `acme-splunk-archive`) |
| S3 Prefix | A path inside the bucket, e.g., `/splunk/archive` |
| Role ARN | Leave blank for static IAM keys, or set if you use cross-account assume-role |
| Output Format | **`NDJSON (.json.gz)`** |
When you pick **Output Format = `NDJSON (.json.gz)`**, Sawmills automatically:
* Writes one JSON object per line (newline-delimited JSON), gzipped — the format Splunk's Generic S3 input handles cleanly with no parsing config.
* Partitions objects by hour with a key like `dt=YYYYMMDD/hour=HH/archive_HHMMSS.NNNN..json.gz`.
* Locks **Enabled Data Types** to **Logs only**.
For the AWS credentials the collector uses to **write** to S3, follow the credentials section in the [AWS S3 destination reference](/docs/destination-aws-s3).
4. Click **Save**, then deploy the pipeline.
After deploy, list the bucket to confirm objects are arriving:
```bash theme={null}
aws s3 ls s3://acme-splunk-archive/splunk/archive/dt=$(date -u +%Y%m%d)/ --recursive
```
You should see hourly-partitioned `archive_*.json.gz` files appearing within a minute or two.
***
## 2. Create an IAM user for Splunk to read the archive
Splunk needs read access to the S3 bucket. The Splunk Add-on for AWS only accepts long-term IAM access keys (not STS temporary credentials), so create a dedicated IAM user with read-only access.
The policy below grants Splunk just enough to (a) discover the bucket in its UI dropdown and (b) read objects from your archive bucket — nothing else.
```bash theme={null}
aws iam create-user --user-name splunk-archive-reader
aws iam put-user-policy --user-name splunk-archive-reader \
--policy-name S3ArchiveReadOnly \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListBucketsForSplunkUI",
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
"Resource": "*"
},
{
"Sid": "ReadArchiveBucket",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::acme-splunk-archive",
"arn:aws:s3:::acme-splunk-archive/*"
]
}
]
}'
aws iam create-access-key --user-name splunk-archive-reader
```
The last command prints `AccessKeyId` (starts with `AKIA…`) and `SecretAccessKey`. Save both — AWS only shows the secret once. Store them in a secret manager.
The `ListAllMyBuckets` permission lets Splunk see *every* bucket name in the account, but data access is still scoped to the archive bucket only. If that's not acceptable, you can omit `ListAllMyBuckets` and type the bucket name manually in the Splunk input form instead of selecting it from the dropdown.
***
## 3. Install the Splunk Add-on for AWS
In Splunk:
1. Go to **Apps → Find more apps**.
2. Search for **Splunk Add-on for Amazon Web Services**.
3. Click **Install** and follow the prompts. (On Splunk Cloud you may need to log in with your splunk.com account when prompted.)
***
## 4. Add the AWS account in the add-on
1. Go to **Apps → Splunk Add-on for AWS → Configuration → Account → Add**.
2. Fill in:
| Field | Value |
| :- | :- |
| Name | A label (e.g., `acme-archive`) |
| Key ID | The `AKIA…` access key from step 2 |
| Secret Key | The matching secret access key |
| Region Category | **Global** (unless your bucket is in GovCloud or China) |
3. Click **Add**. Splunk validates the credentials before saving.
***
## 5. Create the Generic S3 input
This is the read side — Splunk crawls the archive bucket and ingests files into a chosen index.
1. Go to **Apps → Splunk Add-on for AWS → Inputs → Create New Input → Custom Data Type → Generic S3**.
2. Fill in:
| Field | Value |
| :- | :- |
| Name | e.g., `splunk-rehydration-archive` |
| AWS Account | The account you added in step 4 |
| Assume Role | Leave empty |
| AWS Region | Your bucket's region |
| S3 Bucket | The archive bucket (e.g., `acme-splunk-archive`) |
| S3 Key Prefix | A narrow prefix for the first run, e.g., `splunk/archive/dt=20260504/` — broaden later |
| Polling Interval | `30` (default) |
| Initial Scan Datetime | A timestamp slightly before the oldest object you want to ingest, e.g., `2026-05-04T00:00:00Z` |
| Sourcetype | `_json` (auto-extracts JSON fields with no parsing config) |
| Index | The Splunk index to ingest into (e.g., `main`) |
3. Click **Add**. The input begins polling; the first scan typically starts within a minute.
***
## 6. Verify rehydration in Splunk Search
Open **Search & Reporting** and run:
```spl theme={null}
index=main sourcetype=_json earliest=-1h
| head 20
```
Within a couple of minutes you should see records with auto-extracted fields: `date`, `service`, `host`, `message`, `tags{}`, and any `attributes.*` keys you had on the originating logs. To confirm a specific service or host:
```spl theme={null}
index=main sourcetype=_json service="my-service" earliest=-1h
| stats count by host
```
If you see zero events:
* Check **Apps → Splunk Add-on for AWS → Inputs**, click the input name, and look for status / errors.
* Search the add-on's internal logs:
```spl theme={null}
index=_internal source=*splunk_ta_aws* log_level=ERROR earliest=-1h
```
* Verify the IAM keys are still valid by checking the Configuration → Account tab.
***
## Other ways to read S3 archives from Splunk
The Generic S3 input is the most universal option — it works on every Splunk tier, needs no special licensing, and accepts the format Sawmills writes. The trade-off is that it physically ingests S3 objects into a Splunk index, counting against your daily ingest license.
A few alternatives, in case they fit your environment better:
* **Federated Search for Amazon S3 (FSS3)** — *Splunk Cloud on AWS only.* Queries the archive in place via an AWS Glue table, never ingests it. Closest UX to Datadog Logs Rehydration. Requires a Data Scan Unit license entitlement and a Glue table over the bucket. Best for ad-hoc forensic searches over large archives where you don't want to pay for ingest.
* **Splunk Cloud Data Manager → "Promote data from AWS S3"** — *Splunk Cloud on AWS only.* A first-party "true rehydration" feature that ingests a chosen prefix and time range into a dedicated infinite-retention index in one click. Per-job, not continuous.
* **SQS-based S3 input** — *same Splunk Add-on for AWS, different input type.* Replaces polling with S3 → SNS → SQS event notifications, giving you live-tail latency on new objects. Splunk publishes [an open-source crawler](https://github.com/splunk/historical-sqs-based-s3-for-splunk) that emits synthetic SQS messages so you can backfill historical objects through the same path. Recommended over the Generic S3 input for ongoing tail at scale.
* **AWS Lambda → Splunk HEC** — for environments where neither the Splunk Add-on nor FSS3 is available. S3 → SQS → Lambda function that POSTs to the HEC endpoint. Most code you'd own, but it works against any Splunk instance that exposes HEC.
Pick the Generic S3 input first to validate the pipeline end-to-end. You can always layer FSS3 or SQS-based input on top of the same archive bucket later — the Sawmills-side configuration doesn't change.
# Route the New Relic Infrastructure agent through Sawmills
Source: https://docs.sawmills.ai/docs/guide-send-data-from-newrelic-infrastructure-agent
Configure the New Relic Infrastructure agent and nri-bundle Helm chart to send host and Kubernetes samples through a Sawmills Collector.
This guide routes New Relic Infrastructure agent traffic through Sawmills. Sawmills processes the metric samples and sends them to New Relic in their native format. It proxies identity, command, inventory, and security requests to the correct New Relic service.
```text theme={null}
Infrastructure agent
-> Sawmills Collector
-> processors -> New Relic Infrastructure data endpoint
-> identity, commands, inventory, and security -> New Relic service endpoints
```
## Prerequisites
* Sawmills Collector `v1.1097.0` or later
* A pipeline with a [New Relic source](/docs/source-new-relic) and [New Relic destination](/docs/destination-new-relic)
* The same New Relic ingest license key in the Infrastructure agent and New Relic destination
* Network access from the agent pods to the Sawmills HTTPS endpoint
* `newrelic-infrastructure` Helm chart `4.0.1` or later
The validated target is `nri-bundle` chart `7.0.4`, which contains `newrelic-infrastructure` chart and `nri-kubernetes` image `4.0.1`.
## 1. Configure the pipeline
Add a New Relic source and a New Relic destination to the same pipeline. Infrastructure-agent support is enabled by default on Sawmills Collector `v1.1097.0` or later. There is no source option to enable or disable this support.
Keep the default advanced paths and upstream URLs for the first test. The destination **License Key** must be the exact ingest license key that the Infrastructure agent uses. Deploy the pipeline and collector.
## 2. Copy the correct collector endpoint
Open the collector details. In **New Relic ingestion endpoints**, copy the URL with type **Infrastructure agent**.
The URL has this format:
```text theme={null}
https://.collectors.sawmills.ai:
```
Use the displayed port. Do not add a path to this URL. Do not use the **APM agent** or **SDK** URL.
## 3. Configure nri-bundle
For the `nri-bundle` chart, add the three base URLs under `newrelic-infrastructure.common.agentConfig`. This shared configuration applies to the kubelet, KSM, and control-plane agent instances.
```yaml theme={null}
newrelic-infrastructure:
common:
agentConfig:
collector_url: "https://.collectors.sawmills.ai:"
identity_url: "https://.collectors.sawmills.ai:"
command_channel_url: "https://.collectors.sawmills.ai:"
```
Keep the existing `global.licenseKey` or custom license secret configuration. Do not put the license key in this routing file.
Apply the values to the existing release:
```bash theme={null}
helm upgrade newrelic/nri-bundle \
--namespace \
--version 7.0.4 \
--reuse-values \
--values sawmills-newrelic-routing.yaml
```
Use the chart version that is already approved for your staging environment. Version `7.0.4` is the validated target.
For more chart settings, see the [New Relic Infrastructure chart values](https://github.com/newrelic/nri-kubernetes/blob/newrelic-infrastructure-4.0.1/charts/newrelic-infrastructure/values.yaml).
## Direct agent environment variables
If you do not use `nri-bundle`, set these variables on every Infrastructure agent container:
```yaml theme={null}
env:
- name: NRIA_COLLECTOR_URL
value: "https://.collectors.sawmills.ai:"
- name: NRIA_IDENTITY_URL
value: "https://.collectors.sawmills.ai:"
- name: NRIA_COMMAND_CHANNEL_URL
value: "https://.collectors.sawmills.ai:"
```
All three variables use the same base URL. The agent adds these paths:
* `/infra/v2/metrics/events/bulk`
* `/inventory/deltas`
* `/identity/v1/connect`
* `/agent_commands/v1/commands`
> **Warning:** If you set only `NRIA_COLLECTOR_URL`, identity or command traffic
> can bypass Sawmills. Set all three variables.
## 4. Verify the rollout
Wait for all New Relic Infrastructure workloads to become ready. The release can include a kubelet DaemonSet, a control-plane DaemonSet, and a KSM Deployment.
```bash theme={null}
kubectl --namespace get pods
kubectl --namespace rollout status daemonset/
kubectl --namespace rollout status deployment/
```
Check one agent log from each workload type. The kubelet pod uses the `agent` container. The KSM and control-plane pods use the `forwarder` container.
```bash theme={null}
kubectl --namespace logs --container agent --since=10m
kubectl --namespace logs --container forwarder --since=10m
kubectl --namespace logs --container forwarder --since=10m
```
The successful request flow has these results:
| Request | Expected result |
| :- | :- |
| Identity connection | HTTP `200` |
| Command polling | HTTP `200` |
| Metric samples | HTTP `202` from Sawmills |
| Inventory deltas | HTTP `202` from New Relic |
The tested agent did not send `/security/v1/connect`. If your version sends this request, confirm that the collector proxies it without an error.
## 5. Verify data in New Relic
Run these NRQL queries after at least two reporting cycles:
```sql theme={null}
SELECT count(*) FROM SystemSample SINCE 10 minutes ago
```
```sql theme={null}
SELECT count(*) FROM K8sPodSample SINCE 10 minutes ago
```
Also verify:
* Expected hosts appear in **Infrastructure > Hosts**.
* Expected cluster, node, pod, and container entities appear in the Kubernetes view.
* Event types and important fields match the direct-to-New-Relic baseline.
* Agent and collector logs do not contain `401`, `402`, a full queue, or a continuous retry loop.
## 6. Test a processor change
First verify the route with no data reduction. Then add one processor change at a time.
1. Apply a sample rule to one high-volume, non-critical event type.
2. Apply a field rule to one non-identity numeric field.
3. Deploy the pipeline.
4. Confirm that the expected hosts and Kubernetes entities remain.
5. Confirm that only the selected samples or fields changed.
6. Record the data-volume reduction.
Do not remove entity keys, entity IDs, GUIDs, event types, or timestamps. See [Filter processor](/docs/processor-filter) and [Drop Metric Datapoints processor](/docs/processor-drop-metric-datapoints).
## Region rules
Keep the Sawmills endpoint in all three agent URL settings. The collector selects the New Relic US or EU service hosts from the license key.
* A US license uses the US identity, command, inventory, and Infrastructure data hosts.
* An EU license uses the EU service hosts.
* The source agent key and destination key must belong to the same account and region.
* A custom advanced endpoint overrides automatic selection for that service.
## Troubleshooting
| Symptom | Cause and action |
| :- | :- |
| Identity or commands bypass Sawmills | Set all three URL settings to the Infrastructure agent endpoint. |
| HTTP `400` | Confirm that the agent sends a non-empty `X-License-Key` header and valid JSON. |
| HTTP `401` or `403` | Confirm that the ingest license key is active. |
| HTTP `402` | Check the New Relic account and billing state. |
| HTTP `413` | The compressed request is larger than the 10 MiB receiver limit. |
| HTTP `502` | Check collector access to the selected New Relic regional service. |
| Export key mismatch | Use the exact same ingest license key in the agent and New Relic destination. |
| Hosts appear but Kubernetes entities do not | Check the KSM and kubelet agent instances, not only one workload type. |
| Continuous retry or full queue | Stop the test, restore the previous Helm release, and inspect the first upstream error. |
## Roll back
Use `helm rollback` to restore the release that sent data directly to New Relic:
```bash theme={null}
helm rollback --namespace
```
Then confirm that the three Sawmills URL overrides are not present in the active agent configuration.
## Related documentation
* [New Relic Source](/docs/source-new-relic)
* [New Relic Destination](/docs/destination-new-relic)
* [How to send data to the Sawmills collector](/docs/send-data-to-collector)
# How to send data from a New Relic Java app
Source: https://docs.sawmills.ai/docs/guide-send-data-from-newrelic-java-app
Route telemetry from a Java application using the New Relic agent and Telemetry SDK through the Sawmills Collector with environment variable configuration.
This guide shows how to route both New Relic metric paths from a Java app through Sawmills.
For collector endpoint exposure, first complete [How to send data to the Sawmills collector](/docs/send-data-to-collector).
> **Note:** This guide covers the Java APM language agent and the Telemetry SDK.
> It does not cover the New Relic Infrastructure agent. For Infrastructure
> setup, see [New Relic Infrastructure
> agent](/docs/guide-send-data-from-newrelic-infrastructure-agent).
## Prerequisites
* A running **Sawmills Collector** with a [New Relic source](/docs/source-new-relic) configured
* A **New Relic license key**
* A Java app instrumented with the New Relic Java agent and/or Telemetry SDK
## Environment Variables
Set both variables to route all telemetry through Sawmills. If you only set one, part of your data bypasses Sawmills.
| Variable | Controls | Format | Example |
| :- | :- | :- | :- |
| `NEW_RELIC_HOST` | Java agent (timeslice metrics) | `host:port` | `.collectors.sawmills.ai:10000` |
| `NEW_RELIC_METRICS_HOST` | Telemetry SDK (dimensional metrics) | `https://host:port` | `https://.collectors.sawmills.ai:10000` |
Your `` is shown in the Sawmills UI under your pipeline's collector details.
Setting only one of these variables causes partial routing — some telemetry
will go directly to New Relic instead of through Sawmills.
## Configure the Java app
```bash theme={null}
export NEW_RELIC_LICENSE_KEY=""
export NEW_RELIC_HOST=".collectors.sawmills.ai:10000"
export NEW_RELIC_METRICS_HOST="https://.collectors.sawmills.ai:10000"
```
For Kubernetes, inject both values from a ConfigMap:
```yaml theme={null}
env:
- name: NEW_RELIC_LICENSE_KEY
valueFrom:
secretKeyRef:
name: newrelic-license
key: license_key
- name: NEW_RELIC_HOST
valueFrom:
configMapKeyRef:
name: newrelic-config
key: NEW_RELIC_HOST
- name: NEW_RELIC_METRICS_HOST
valueFrom:
configMapKeyRef:
name: newrelic-config
key: NEW_RELIC_METRICS_HOST
```
Restart the deployment after changes:
```bash theme={null}
kubectl rollout restart deployment/
```
## Verify
1. Check application logs for connection to your Sawmills collector endpoint.
2. Validate metrics in New Relic:
```sql theme={null}
SELECT count(*) FROM Metric WHERE metricName LIKE 'system.%' SINCE 5 minutes ago
```
## Troubleshooting
| Symptom | Fix |
| :- | :- |
| Only some metrics go through Sawmills | Set **both** `NEW_RELIC_HOST` and `NEW_RELIC_METRICS_HOST` |
| Dimensional metrics fail | Use a base URL only for `NEW_RELIC_METRICS_HOST` — don't append `/metric/v1` |
| No data arrives | Verify DNS reachability to `.collectors.sawmills.ai` from the app |
| Changes not taking effect | Restart the deployment after updating env vars |
## Related Docs
* [New Relic Source](/docs/source-new-relic)
* [New Relic Infrastructure agent](/docs/guide-send-data-from-newrelic-infrastructure-agent)
* [How to send data to the Sawmills collector](/docs/send-data-to-collector)
* [New Relic Destination](/docs/destination-new-relic)
# How to send data from a New Relic Python app
Source: https://docs.sawmills.ai/docs/guide-send-data-from-newrelic-python-app
Route telemetry from a Python application using the New Relic agent through the Sawmills Collector with a single environment variable.
This guide shows how to route New Relic telemetry from a Python app through Sawmills.
For collector endpoint exposure, first complete [How to send data to the Sawmills collector](/docs/send-data-to-collector).
> **Note:** This guide covers the New Relic Python APM language agent (and,
> optionally, the standalone Telemetry SDK). It does not cover the New Relic
> Infrastructure agent. For Infrastructure setup, see [New Relic Infrastructure
> agent](/docs/guide-send-data-from-newrelic-infrastructure-agent).
## Prerequisites
* A running **Sawmills Collector** with a [New Relic source](/docs/source-new-relic) configured
* A **New Relic license key**
* A Python app instrumented with the New Relic Python agent (`newrelic`)
## Environment Variables
Unlike the Java agent — which splits timeslice and dimensional metrics across
two hosts — the New Relic **Python** agent sends its telemetry to a single
host, so one variable, `NEW_RELIC_HOST`, points the agent at Sawmills.
| Variable | Controls | Format | Example |
| :- | :- | :- | :- |
| `NEW_RELIC_HOST` | Python APM agent connection (timeslice metrics) | `host` | `.collectors.sawmills.ai` |
| `NEW_RELIC_LICENSE_KEY` | Authentication to New Relic | key | `eu01xx...NRAL` |
Your `` is shown in the Sawmills UI under your pipeline's collector details.
The New Relic source processes the APM agent's **timeslice metrics**
(`metric_data`) and forwards them to New Relic. Other APM payloads the agent
sends over this connection — transaction traces, custom/analytics events,
error data, and span events — are acknowledged by the collector but **not**
forwarded. Route through Sawmills for APM metrics; if you depend on traces,
events, or errors in New Relic, those are not preserved on this path today.
`NEW_RELIC_HOST` for the Python agent is a **hostname only** — do not include
a scheme (`https://`) or a `:port` suffix. The Python agent does **not** read
`NEW_RELIC_METRICS_HOST` (that variable is Java/Telemetry-SDK only); setting
it here has no effect.
### Port
The Sawmills collector receives the New Relic protocols on a dedicated port
(the source's default is `14275`; your collector may differ). Find the exact
value under **New Relic ingestion endpoints → APM agent** in the collector
details in the Sawmills UI — copy the `host:port` shown there.
The Python agent takes the host and port as **separate** settings and defaults
to `443` when no port is given, so set the port explicitly to match your
collector endpoint. In `newrelic.ini`:
```ini theme={null}
[newrelic]
host = .collectors.sawmills.ai
port = 14275
```
Or via environment variable:
```bash theme={null}
export NEW_RELIC_PORT="14275"
```
## Configure the Python app
```bash theme={null}
export NEW_RELIC_LICENSE_KEY=""
export NEW_RELIC_HOST=".collectors.sawmills.ai"
# Port shown for your collector's New Relic APM ingestion endpoint (default 14275):
export NEW_RELIC_PORT="14275"
```
Start the app under the New Relic admin wrapper so the agent is initialized:
```bash theme={null}
NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python your_app.py
```
For Kubernetes, inject the values from a ConfigMap and Secret:
```yaml theme={null}
env:
- name: NEW_RELIC_LICENSE_KEY
valueFrom:
secretKeyRef:
name: newrelic-license
key: license_key
- name: NEW_RELIC_HOST
valueFrom:
configMapKeyRef:
name: newrelic-config
key: NEW_RELIC_HOST
# Port of your collector's New Relic APM ingestion endpoint (default 14275):
- name: NEW_RELIC_PORT
valueFrom:
configMapKeyRef:
name: newrelic-config
key: NEW_RELIC_PORT
```
Restart the deployment after changes:
```bash theme={null}
kubectl rollout restart deployment/
```
## Telemetry SDK (optional)
If your app uses the standalone **New Relic Telemetry SDK for Python**
(`newrelic-telemetry-sdk`) to send dimensional metrics directly to the Metric
API — separate from the APM agent — point its client host at the collector when
you construct it:
```python theme={null}
from newrelic_telemetry_sdk import MetricClient
client = MetricClient(
license_key="",
host=".collectors.sawmills.ai",
port=14275, # your collector's New Relic endpoint port — the SDK defaults to 443
)
```
Most Python APM users do not use the Telemetry SDK; skip this section if you
only instrument with the `newrelic` agent.
## Verify
1. Check application logs for the agent connecting to your Sawmills collector
endpoint (look for a line reporting the reporting host on agent startup).
2. Validate data in New Relic. The New Relic source forwards only the APM
agent's `metric_data` (timeslice metrics), which New Relic stores as
`Metric` — so query the `Metric` type, not `Transaction`:
```sql theme={null}
SELECT count(*) FROM Metric WHERE appName = '' SINCE 5 minutes ago
```
## Troubleshooting
| Symptom | Fix |
| :- | :- |
| No data arrives | Verify DNS reachability to `.collectors.sawmills.ai` from the app |
| Agent can't connect / TLS error | Ensure `NEW_RELIC_HOST` is a **hostname only** (no `https://`, no `:port`) |
| Connecting on the wrong port | Set `port` in `newrelic.ini` or `NEW_RELIC_PORT` — not a `:port` suffix on `NEW_RELIC_HOST` |
| Agent never initializes | Start the app with `newrelic-admin run-program`, or `import newrelic.agent; initialize()` |
| Changes not taking effect | Restart the deployment after updating env vars |
## Related Docs
* [New Relic Source](/docs/source-new-relic)
* [New Relic Java App](/docs/guide-send-data-from-newrelic-java-app)
* [New Relic Infrastructure agent](/docs/guide-send-data-from-newrelic-infrastructure-agent)
* [How to send data to the Sawmills collector](/docs/send-data-to-collector)
* [New Relic Destination](/docs/destination-new-relic)
# How to send logs to VictoriaLogs
Source: https://docs.sawmills.ai/docs/guide-send-logs-to-victorialogs
Configure Sawmills to send logs to managed or self-hosted VictoriaLogs using the OTLP HTTP destination, with cloud and Kubernetes setup steps.
This guide explains how to configure Sawmills to send logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) using the OTLP HTTP destination. VictoriaLogs natively supports OTLP log ingestion over HTTP, so no custom destination type is needed — you can use the existing **OTLP** destination in Sawmills.
VictoriaLogs only supports **OTLP over HTTP** (not gRPC). Make sure to select **HTTP** as the protocol when creating the destination.
## Prerequisites
* A running VictoriaLogs instance (managed or self-hosted)
* A Sawmills pipeline with a log source configured (e.g., OTLP, Loki, Datadog)
## VictoriaLogs Cloud (Managed)
### 1. Get your VictoriaLogs Cloud credentials
1. Log in to [VictoriaMetrics Cloud](https://console.victoriametrics.cloud)
2. Create a **VictoriaLogs** deployment (type: "Vlogs single")
3. Note the **Access endpoint** from the deployment overview (e.g., `https://gw-c21-1a.cloud.victoriametrics.com`)
4. Go to the **Access Tokens** tab and copy your deployment access token
### 2. Create an OTLP destination in Sawmills
Navigate to your pipeline, click **+ Add Destination**, select **OTLP**, and fill in:
| Field | Value |
| :- | :- |
| Name | e.g., `victorialogs-cloud` |
| Protocol | **HTTP** |
| Endpoint | `https:///insert/opentelemetry` |
| Enable Basic Auth | **Off** |
| Enabled Data Types | **Logs** only (uncheck Metrics and Traces) |
| Logs Endpoint | Leave empty (Sawmills appends `/v1/logs` to the base endpoint automatically) |
VictoriaLogs is a log storage backend. Uncheck **Metrics** and **Traces** under Enabled Data Types — VictoriaLogs does not accept these signals.
### 3. Configure authentication
Expand **Advanced Options** and under **Additional Headers**, click **+ Add Header**:
| Name | Value |
| :- | :- |
| `Authorization` | `Bearer ` |
Leave the other advanced settings at their defaults:
* **Timeout**: 30 seconds
* **Compression**: gzip
### 4. Deploy and verify
1. Click **Save**, then deploy the pipeline
2. Send test logs through your pipeline source
3. Verify logs arrive in VictoriaLogs using the **Explore** tab in VictoriaMetrics Cloud, or query via the API:
```bash theme={null}
curl -s \
"https:///select/logsql/query" \
-H "Authorization: Bearer " \
--data-urlencode 'query=*' \
--data-urlencode 'limit=10'
```
## VictoriaLogs Self-Hosted
### 1. Identify your VictoriaLogs endpoint
The default OTLP ingestion endpoint for self-hosted VictoriaLogs is:
```
http://:9428/insert/opentelemetry
```
The default port is **9428**. Sawmills appends `/v1/logs` automatically when sending log data.
If VictoriaLogs runs in the same Kubernetes cluster as the Sawmills collector, use the in-cluster DNS:
```
http://victorialogs..svc.cluster.local:9428/insert/opentelemetry
```
### 2. Create an OTLP destination in Sawmills
Navigate to your pipeline, click **+ Add Destination**, select **OTLP**, and fill in:
| Field | Value |
| :- | :- |
| Name | e.g., `victorialogs-onprem` |
| Protocol | **HTTP** |
| Endpoint | `http://victorialogs.sawmills.svc.cluster.local:9428/insert/opentelemetry` |
| Enable Basic Auth | **Off** |
| Enabled Data Types | **Logs** only (uncheck Metrics and Traces) |
| Logs Endpoint | Leave empty |
No authentication headers are needed for self-hosted VictoriaLogs (unless you have placed an auth proxy like `vmauth` in front of it).
### 3. Deploy and verify
1. Click **Save**, then deploy the pipeline
2. Query VictoriaLogs to confirm logs are arriving:
```bash theme={null}
curl -s "http://:9428/select/logsql/query" \
--data-urlencode 'query=* | stats count() as total'
```
## Multi-Tenancy
VictoriaLogs supports multi-tenancy via `AccountID` and `ProjectID` HTTP headers. Tenants are created implicitly on first write — no server-side setup required.
### When to use multi-tenancy
* Isolate logs by team, environment, or business unit within a shared VictoriaLogs instance
* Separate infrastructure logs from application logs
* Provide per-tenant query isolation (each tenant can only see its own data)
### How to configure
Create a separate OTLP destination for each tenant, all pointing to the **same** VictoriaLogs endpoint. Under **Advanced Options → Additional Headers**, set:
| Name | Value |
| :- | :- |
| `AccountID` | Tenant ID (uint32, e.g., `1`, `2`, `3`) |
**Example setup with two tenants:**
**Destination 1 — "VL Infra Logs"**
| Field | Value |
| :- | :- |
| Endpoint | `http://victorialogs:9428/insert/opentelemetry` |
| Enabled Data Types | Logs only |
| Additional Header | `AccountID` = `1` |
**Destination 2 — "VL App Logs"**
| Field | Value |
| :- | :- |
| Endpoint | `http://victorialogs:9428/insert/opentelemetry` |
| Enabled Data Types | Logs only |
| Additional Header | `AccountID` = `2` |
Then use Sawmills **filter rules** on each destination to route the right logs to each tenant (e.g., filter by `service.name`, `k8s.namespace.name`, or other attributes).
### Querying a specific tenant
Pass the same `AccountID` header when querying to isolate reads:
```bash theme={null}
# Query infra logs (tenant 1)
curl -s "http://:9428/select/logsql/query" \
-H "AccountID: 1" \
--data-urlencode 'query=*' \
--data-urlencode 'limit=10'
# Query app logs (tenant 2)
curl -s "http://:9428/select/logsql/query" \
-H "AccountID: 2" \
--data-urlencode 'query=*' \
--data-urlencode 'limit=10'
```
If no `AccountID` header is set, logs are stored and queried under the default tenant `(0, 0)`. `ProjectID` is an optional second dimension for finer grouping within an account.
### Multi-tenancy with VictoriaLogs Cloud
VictoriaMetrics Cloud uses **deployment-level isolation** — each deployment is a separate tenant. Multi-tenancy via `AccountID`/`ProjectID` headers is typically not needed. Create separate Cloud deployments if you need tenant isolation.
## How it works
The data flow through Sawmills is:
```
Your applications → Sawmills Collector → VictoriaLogs
(OTLP HTTP export)
```
Sawmills uses the standard OpenTelemetry Collector `otlphttp` exporter to send logs. VictoriaLogs receives them at its `/insert/opentelemetry/v1/logs` endpoint using protobuf encoding.
**What gets preserved:**
* All OpenTelemetry resource attributes (e.g., `service.name`, `host.name`, `k8s.*`)
* Log severity and body
* Scope name and version
* Sawmills-injected metadata (`sawmills.pipeline_id`, `sawmills.pipeline_name`, `sawmills.collector_id`)
**Log streams:** VictoriaLogs automatically uses all OTel resource attributes as log stream identifiers. To customize which fields define a log stream, add the `VL-Stream-Fields` header (comma-separated field names) under Additional Headers.
## Troubleshooting
| Symptom | Cause | Fix |
| :- | :- | :- |
| HTTP 401 Unauthenticated | Invalid or missing bearer token (Cloud) | Verify the `Authorization` header value matches your VictoriaLogs access token |
| HTTP 400 "json encoding isn't supported" | Endpoint misconfiguration | Ensure you are using the OTLP destination (protobuf), not sending raw JSON |
| Logs not appearing | Wrong endpoint path | Verify the endpoint ends with `/insert/opentelemetry` (Sawmills appends `/v1/logs` automatically) |
| Logs in wrong tenant | Missing or wrong `AccountID` header | Check the Additional Headers section in your destination config |
| High cardinality warning in VL | Too many unique stream combinations | Add a `VL-Stream-Fields` header to limit which attributes define log streams |
| Export timeout | Network or firewall issue | Check that the collector can reach the VictoriaLogs endpoint; verify TLS settings |
# Insights
Source: https://docs.sawmills.ai/docs/insights
Automatically detect telemetry data quality and cost issues in your pipelines. Review impact, view affected logs, and apply recommended optimizations.
## Overview
The **Insights** page surfaces telemetry data issues detected automatically in your environment to help reduce observability costs and improve data quality. These issues are evaluated when the page is loaded, based on internal heuristics that detect high message volume or data volume over time.
## Insights Page
A **timeframe selector** at the top left allows switching between:
* **Last 24 hours**
* **Last 7 days**
When changing the timeframe the system would evaluate the issues based on the selected timeframe
Currently evaulation is looking at log issues that generate more then **5% of total log volume** for the respected timeframe
### Insights table
Each row in the insights table represents a specific data issue, with the following fields:
* **Status**
* `Detected`: Issue is active and requires attention
* `Actioned`: A processor was deployed to mitigate the issue
Actioned insights will be visible in the system until the issue is resolved.
* **Name** – A brief summary of the issue
* **Service** – The emitting service or data source
* **Type** – Classification (e.g., `Third-party log`, `Error/Warn burst`)
* **Destination** – The observability platform receiving this data (e.g., `Datadog`, `Loki`, `Elastic`, etc)
* **Pipeline** – The telemetry pipeline that carried this data
* **Impact** – A sparkline graph indicating data volume over the selected timeframe
* **Total Impact** – Total amount of data (GiB) of messages associated with this issue, and the percentge from the total volume over the time period selected (24H or 7 days)
### Archive Tab
The Insights page includes two tabs:
* **Current** – Active issues that may require intervention
* **Archived** – Insights that have been deferred or dismissed
Insights can be moved between the two views as needed. Click the archive button at the end of each row to archive an insight.
***
## Insight Details Page
Clicking an insight name in the insights table opens the **Insight Details Page**, with detailed information and resolution options.
### Timeframe Selector
Select **24H** or **7 days** to scope the data shown.
### Live Tail View
Click this to to see the actual log lines currently being sent that have this issue.
This view should help get understand the excat log message that generates the issue.
### Archive
You may archive an insight if no action is desired immediately.
### Summary
Shows the following:
* **Service**
* **Pipeline**
* **Destination**
* **Type**
* **Impact** (volume)
### Conditions
Displays the specific patterns, log content, or metadata attributes contributing to the issue.
### Usage Graph
A bar chart visualizing the issue's behavior over time, with a toggle to switch between:
* **Volume over time**
* **Message count over time**
### Recommended Actions
This section offers automated suggestions to address the issue:
* **Filter** – Drop noisy or unnecessary messages
* **Aggregate** – Reduce data granularity through summarization
Each action has a **Preview** button:
* Clicking **Preview** opens a new tab in the pipeline editor
* A preconfigured processor is created for the selected resolution
* You may customize the processor before deployment
After saving a processor, the pipeline must be redeployed for the change to take effect.
### Activity Log
Chronological log of all actions taken on this insight:
* Action type
* User who performed the action
* Timestamp
# Installation Guide
Source: https://docs.sawmills.ai/docs/install-guide
Deploy the Sawmills OpenTelemetry Collector into your Kubernetes cluster using Helm. Covers prerequisites, namespace setup, and the installation wizard.
To start working with Sawmills, you’ll install the Sawmills OpenTelemetry (OTel) Collector distribution. Installation deploys a remote operator into your Kubernetes cluster, which then manages the Sawmills Collector.
A Sawmills account and access to the product dashboard is required to proceed
with installation.
## Prerequisites
* **Kubernetes Cluster**: Access to the target cluster.
* **kubectl**: Installed and configured to interact with the cluster.
* **Helm 3+**: Installed and configured.
* **Permissions**: Ability to create namespaces, secrets, and deploy Helm charts.
## Install the Sawmills Collector
1. Log in to the Sawmills dashboard.
2. Go to the **Collectors** section (located at the bottom tab of the left navigation menu).
3. Click **New Collector** to open the installation wizard.
If you use Kubectl to check Sawmills collector pods, initially you will only
see the Sawmills operator running, as the collector service is started when a
pipeline is deployed. See [How to deploy a pipeline](/docs/build-pipelines).
### Step 1: Configure the Collector
* **Name**: Provide a unique, descriptive name for the collector.
* **Namespace**: The Kubernetes namespace where the collector will be installed.
The name should follow your organization’s naming conventions.
#### Advanced Options
* **Require manual deployment for pipeline changes**: Enable this only if your environment prevents Sawmills from applying collector updates automatically. When enabled, Sawmills generates chart metadata and values files for each deployment, and your team applies them through your Helm release process. See [Manual Collector Deployment](/docs/manual-collector-deployment).
Click **Next** to proceed.
### Step 2: Install the Collector
This step provides the commands needed to deploy the remote operator, which manages the collector in your cluster.
1. **Create a Namespace** (only if it does not already exist)
```bash theme={null}
kubectl create namespace sawmills
```
2. **Create a Secret** with your organization API key (only if it does not already exist)
```bash theme={null}
kubectl create secret generic sawmills-secret \
-n sawmills --from-literal api-key=****************
```
> **Warning:** If you use a different secret name or key, update `apiKeyExistingSecret` and `apiKeySecretKey` in `values.yaml`, plus `managedChartsValues.sawmills-collector.apiSecret.name` and `managedChartsValues.sawmills-collector.apiSecret.key`.
3. **Install or Upgrade the Remote Operator** using Helm
Copy the command shown in the Sawmills UI. It includes the remote operator chart version and controller address for your environment.
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version \
--create-namespace \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress= \
--set collectorName=""
```
Q: What if Helm cannot pull the Remote Operator chart from public ECR?
A: Authenticate Helm to Amazon ECR Public, then rerun the install command:
```bash theme={null}
aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
```
#### Updating Collector Values
The remote operator deploys the collector into the cluster. Collector values are passed as a values file to the remote operator Helm chart.
To change collector configuration for an automatically managed collector, update the `managedChartsValues.sawmills-collector` section in `values.yaml` and upgrade the remote operator. The legacy `managedCharts` key continues to work as an alias, but `managedChartsValues` is preferred.
For manually managed collectors, download the generated values artifacts from Sawmills and apply them through your Helm workflow.
For details, see [Collector Customization](/docs/collector-customization).
### Step 3: Validate the Collector
Wait for the install to complete and all pods to enter a `Running` state.
Validate the collector by clicking **Test Installation**. This confirms the collector is deployed and communicating back to Sawmills.
When validation passes, you’ll see success messages and a link to create or
assign a pipeline via **Pipelines Editor**. If you created the collector but
haven’t completed installation yet, its status will show as **Awaiting
Deployment** on the Collectors page.
#### Next Actions
* **Pipelines Editor**: Create a new pipeline or assign an existing pipeline to your collector.
* **Done**: Finish without creating a pipeline.
* **Previous**: Go back to change settings.
## Recommended Configuration
### Remote Operator Resources
By default, 3 replicas of the remote operator are deployed with the following resource requests and limits:
```yaml theme={null}
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 500m
```
### Autoscaling
Autoscaling is enabled by default with a minimum of 3 and a maximum of 10 collector instances.
### Capacity Planning
For detailed guidance on sizing and scaling, see [Capacity Planning](/docs/installation/capacity-planning).
### KEDA Autoscaling
We recommend using KEDA for autoscaling. KEDA supports more flexible scaling rules and metrics. Enable it by setting `keda.enabled` to `true` in `values.yaml`:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
keda:
enabled: true
```
See [KEDA Autoscaling](/docs/keda-scaler) for more information.
### Anti-Affinity
Anti-affinity is enabled by default to ensure collector pods are spread across nodes.
# Capacity Planning
Source: https://docs.sawmills.ai/docs/installation/capacity-planning
Plan Kubernetes resource requirements for your Sawmills Collector deployment with recommended CPU, memory, and instance types for AWS, Azure, and GCP.
## Overview
When deploying a Sawmills Collector, it is crucial to plan for capacity to ensure optimal performance and reliability. Proper capacity planning helps prevent bottlenecks, ensures consistent data processing, and enables cost-effective scaling as your telemetry volume grows.
The Sawmills Collector is architected for horizontal scaling to handle enterprise-scale data volumes efficiently. Under default configuration with 4 CPU cores and no processors, a single collector instance can process approximately **14 million logs per day**, providing a solid foundation for most production workloads.
## Kubernetes Resource Requirements
We recommend allocating at least **4 vCPUs and 3 GiB memory per node as a kubernetes limit with a minimum (kubernetes request) of 1 vCPU and 1 GiB memory**.
### AWS Instance Types
The collector runs best on Graviton-based instances, preferably: **M7g, M6g**
We also support amd64/x86\_64 architectures (less performant), such as: **M7i, M6i, M6a**
### Azure Instance Types
For Azure deployments, we recommend the following instance types based on workload requirements:
* **D4s\_v5**: 4 vCPUs, 16 GB RAM - Ideal for moderate telemetry volumes
* **D8s\_v5**: 8 vCPUs, 32 GB RAM - Suitable for high telemetry volumes
### Google Cloud Platform (GCP) Instance Types
For GCP deployments, we recommend the following machine types:
* **n2-standard-4**: 4 vCPUs, 16 GB RAM - Ideal for moderate telemetry volumes
* **n2-standard-8**: 8 vCPUs, 32 GB RAM - Suitable for high telemetry volumes
### Collector Sizing Table
You can reference this table as a starting point for collector system requirements. The resource recommendations do not consider multiple exporters or processors.
| Telemetry Throughput | Logs / second | Collectors |
| - | - | - |
| 5 GiB/m | 130,000 | 1 |
| 10 GiB/m | 260,000 | 2 |
| 20 GiB/m | 520,000 | 4 |
| 40 GiB/m | 1,040,000 | 8 |
| 80 GiB/m | 2,080,000 | 16 |
| 160 GiB/m | 4,160,000 | 32 |
| 320 GiB/m | 8,320,000 | 64 |
| 640 GiB/m | 16,640,000 | 128 |
| 1,280 GiB/m | 33,280,000 | 256 |
| 2,560 GiB/m | 66,560,000 | 512 |
## Fault Tolerance and High Availability
**Over-provisioning is essential** for maintaining service continuity. We recommend maintaining **20-30% additional capacity** beyond your baseline requirements to ensure your collector fleet can handle:
* **Node failures**: When one or more collector instances fail unexpectedly
* **Maintenance windows**: Planned downtime for updates, patches, or infrastructure changes
* **Traffic spikes**: Sudden increases in telemetry volume during peak usage periods
* **Rolling deployments**: Zero-downtime updates that temporarily reduce available capacity
This buffer ensures that your remaining collectors can seamlessly absorb the workload without dropping data or experiencing performance degradation.
When dealing with a fixed number of collectors, you can scale their CPU and memory vertically in order to increase throughput. See collector sizing table above.
# KEDA External Scaler
Source: https://docs.sawmills.ai/docs/keda-scaler
Understand the Sawmills KEDA External Scaler architecture that bridges OpenTelemetry metrics with Kubernetes event-driven autoscaling for dynamic workloads.
## KEDA External Scaler
The Sawmills implementation provides a powerful bridge between OpenTelemetry (OTel) and KEDA, enabling dynamic scaling based on OTel metrics. This implementation offers a flexible and efficient way to scale your applications based on custom metrics collected through OpenTelemetry.
## Architecture Overview
The implementation consists of three main components:
1. **Receiver**: Implements the OTLP Receiver specification, running a gRPC server (default port 4317) that receives and processes incoming metrics.
2. **In-Memory Metric Storage**: A high-performance in-memory storage system that maintains recent metric measurements for each metric vector. Key features include:
* Automatic cleanup of stale measurements
* Label-based querying support
* Optimized for real-time metric processing
3. **External Scaler**: Implements the KEDA External Scaler contract, running a gRPC server (default port 4318) that queries the metric storage and communicates with the KEDA operator.
## Installation
### Prerequisites
* Kubernetes cluster
* KEDA installed
* Helm 3.x
### Step 1: Install KEDA
```bash theme={null}
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm upgrade -i keda kedacore/keda --namespace keda --create-namespace
```
### Step 2: Configure Sawmills Collector with KEDA Scaler
Create a `values.yaml` file with the following configuration to enable KEDA scaler and external scaling:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
# Enable KEDA-based autoscaling
keda:
enabled: true
minReplicas: 1
maxReplicas: 10
pollingInterval: 30
cooldownPeriod: 300
scaling:
external:
enabled: true
metricType: Value
metadata:
query: "histogram_quantile(0.95, sum(rate(http_server_duration_bucket[2m])) by (le))"
targetValue: "2000"
# Enable KEDA scaler component
kedaScaler:
enabled: true
```
> **Note:** The chart derives the KEDA scaler service address from the rendered scaler service name and the Helm release namespace. Avoid hardcoding a namespace in `scalerAddress`, especially when deploying through an umbrella chart or using `resourceBaseName`.
# Live Tail
Source: https://docs.sawmills.ai/docs/live-tail
Stream logs in real time through your Sawmills pipeline. Compare events before and after each processor to validate filters and troubleshoot issues.
## Overview
Live Tail lets you watch sampled telemetry flow through your pipeline in real time. Use it to inspect data at key points in the pipeline, including before processing, after processor blocks, and from supported destination output previews.
Live Tail shows a **sample** of incoming events to reduce bandwidth and keep the UI responsive.
## Open Live Tail
Live tail can be accessed in many parts of the application
* Top level navigation menu - Live Tail tab
* Inside a pipeline (Live tab), from pipeline paths and supported destination output previews
* Building a processor
* Deployed processor page
When accessed from the top navigation menu, Live Tail displays telemetry data from the midpoint of the pipeline, prior to processing by any configured processors.
## Destination Output Preview
When viewing a specific pipeline in **Live** mode, click a supported destination and select **Preview Output**. This opens a sampled stream for that destination, so you can inspect telemetry after destination-specific handling.
Use destination output preview when you need to confirm what a backend destination receives. Some destinations, such as Datadog, Elasticsearch, or other integrations, can reshape telemetry before export. Output preview helps you compare the destination output with the data seen earlier in the pipeline.
## Live Stream View
When Live Tail opens, a sampled stream begins scrolling.
* Each row contains the `timestamp` and `severity` of the event.
* Expand a row to see the full structured payload. Expanding **pauses** the stream; collapsing **resumes** it.
* Use the **Pause / Start** button in the upper‑right corner to control the stream manually.
## Filtering the Stream
1. Click **Add Condition** in the filter section.
2. Choose a field (attribute, resource, body, or severity).
3. Select an operator (equals, contains, regex, etc.).
4. Enter a value and press **Enter**.
Live Tail automatically re‑evaluates the stream whenever filters change. Multiple conditions are combined with **AND** logic by default. Select **Match any** to use **OR**.
Filters are evaluated **client‑side** on the sampled stream; they don’t change what the collector ingests.
## Expanded Log View Options
When you expand a log entry to view all its keys and values, a context menu appears on hover to the left of each row:
1. **Copy Value** – copy just the value to your clipboard.
2. **Copy Key & Value** – copy the pair in `key=value` form.
3. **Generate Regex** - generate and copy a regex pattern matching the value (may take \~1s)
4. **Generate Grok Pattern** - generate and copy a grok pattern matching the value (may take \~1s)
5. **Show Logs with This Value** – add an *equals* filter for this field to the Live Tail filter bar.
6. **Exclude Logs with This Value** – add a *not equals* filter for this field.
To copy the *entire* log line as formatted JSON, click the **Copy JSON** icon in the upper‑right corner of the expanded view.
To debug and understand which processors will impact a specific log line., click on **Processors Impacting This Log** Button from the upper‑right corner
**Processors Impacting This Log** is available only from the top level navigation live-tail
# Load Balancer Setup
Source: https://docs.sawmills.ai/docs/load-balancer-setup
Add an optional load balancing layer with queue-based buffering between telemetry sources and Sawmills Collector backends for improved resilience and scaling.
## Configuring a Load Balancer layer
The Sawmills collector supports an optional load balancing layer that sits between your telemetry sources and the backend collector processing layer. This architecture provides enhanced resilience, scalability, and efficient resource utilization by decoupling data ingestion from data processing.
## Architecture Overview
The load balancing layer introduces a two-tier architecture:
1. **Load Balancer Layer**: Receives telemetry data from all sources (Datadog Agent, Fluent, OTLP, etc.), stores it in a queue (in-memory or storage-based), and distributes it to backend collectors.
2. **Backend Collector Layer**: Processes telemetry data, applies processors (filtering, masking, aggregation, etc.), and forwards processed data to destinations (Datadog, Splunk, S3, etc.).
```text theme={null}
Telemetry Sources -> Load Balancer (Queue) -> Backend Collectors -> Destinations
```
## Benefits
### 1. **Improved Resilience and Fault Tolerance**
* **Queue Buffering**: The load balancer queue acts as a buffer, preventing data loss during backend collector restarts, deployments, or temporary failures.
* **Decoupled Operations**: Sources continue sending data even when backend collectors are being updated or scaled, ensuring zero downtime deployments.
### 2. **Enhanced Scalability with KEDA Integration**
When combined with the KEDA scaler, the load balancing layer enables intelligent, metric-driven autoscaling:
* **Queue-Based Scaling**: KEDA monitors the queue size and utilization metrics from the load balancer
* **Proactive Scaling**: Backend collectors scale up before the queue becomes full, preventing backpressure
* **Cost Optimization**: Scale down when queue utilization is low, reducing resource consumption during low-traffic periods
* **Dynamic Response**: Automatically adjusts to traffic patterns without manual intervention
### 3. **Better Resource Utilization**
* **Load Distribution**: Evenly distributes telemetry data across multiple backend collector instances
* **Independent Scaling**: Load balancer and backend collectors can be scaled independently based on different metrics
* **Resource Efficiency**: Backend collectors can focus on processing without handling direct client connections
### 4. **Operational Flexibility**
* **Zero-Downtime Updates**: Update backend collectors without interrupting data ingestion
* **Traffic Spikes Handling**: Queue absorbs sudden traffic spikes, giving backend collectors time to scale up
* **Monitoring and Observability**: Clear separation of concerns makes it easier to monitor and troubleshoot each layer
### How It Works with KEDA
The KEDA scaler monitors the load balancer's queue metrics using Prometheus queries. When queue utilization exceeds the target threshold, KEDA scales up the backend collector replicas. When utilization drops below a lower threshold, it scales down to optimize costs.
The scaler queries metrics like:
* `otelcol_exporter_queue_size`: Current number of items in the queue
* `otelcol_exporter_queue_capacity`: Maximum queue capacity
Based on these metrics, KEDA makes scaling decisions to maintain optimal queue utilization, ensuring the backend collectors can process data efficiently without queue overflow.
## Configuration
### Basic Setup
To enable the load balancing layer, add the following configuration to your `values.yaml`:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
loadBalancer:
enabled: true
```
### Setup with KEDA Autoscaling
For optimal performance, combine the load balancer with KEDA autoscaling. This configuration enables queue-based scaling of the backend collector layer:
```yaml theme={null}
managedChartsValues:
sawmills-collector:
# Enable load balancing layer
loadBalancer:
enabled: true
# Configure KEDA-based autoscaling
keda:
enabled: true
minReplicas: 1
maxReplicas: 10
pollingInterval: 30
cooldownPeriod: 300
scaling:
external:
enabled: true
metricType: AverageValue
metadata:
query: sum(otelcol_exporter_queue_size{exporter=~"loadbalancing/collector-loadbalancer.*"})
targetValue: "10"
# Enable KEDA scaler component
kedaScaler:
enabled: true
```
### Configuration Parameters
#### Load Balancer
* `loadBalancer.enabled`: Enable or disable the load balancing layer (default: `false`)
#### KEDA Configuration
* `keda.enabled`: Enable KEDA autoscaling (default: `false`)
* `keda.minReplicas`: Minimum number of backend collector replicas (default: `1`)
* `keda.maxReplicas`: Maximum number of backend collector replicas (default: `10`)
* `keda.pollingInterval`: Interval in seconds for KEDA to check metrics (default: `30`)
* `keda.cooldownPeriod`: Cooldown period in seconds before scaling down (default: `300`)
#### External Scaler Metadata
* `scalerAddress`: Address of the KEDA OTLP scaler service. The chart default derives this from the generated scaler service name and release namespace.
* `query`: PromQL query that monitors queue utilization metrics
* `targetValue`: Target queue utilization percentage to maintain
* `metricType`: Type of metric. Use `AverageValue` for backend collector scaling from load balancer queue metrics.
This ensures the backend collector layer maintains optimal capacity to process queued telemetry data without overflow.
## Service Names and Routing
If you set `resourceBaseName`, the chart uses that value as the Kubernetes base name. If you leave it empty, the chart uses the Helm release name.
With the default base name `sawmills-collector`, load balancer mode creates these resources:
| Resource | Name | Use |
| :- | :- | :- |
| Source-facing service | `sawmills-collector` | Send telemetry sources here when the load balancer is enabled |
| Load balancer workload | `sawmills-collector-lb` | Receives telemetry and forwards to backend collectors |
| Backend collector service | `sawmills-collector-backend` | Service for backend collector pods |
| Backend collector headless service | `sawmills-collector-backend-headless` | Used by the load balancer to route to backend collector pods |
| Load balancer headless service | `sawmills-collector-headless` | Used for load balancer pod discovery when enabled |
| KEDA scaler service | `sawmills-collector-keda-otel-scaler` | Used by KEDA and collector self-telemetry |
When load balancer mode is enabled, configure Datadog Agent, OTLP sources, Fluent, Vector, and other telemetry sources to send data to the source-facing service. Do not route source traffic directly to the backend headless service.
## Prerequisites
Before setting up the load balancer with KEDA:
1. **Kubernetes Cluster**: A running Kubernetes cluster
2. **KEDA Installed**: KEDA must be installed in your cluster (see [KEDA External Scaler](/docs/keda-scaler) for installation instructions)
3. **Helm 3.x**: For deploying the Sawmills collector
## Next Steps
* Learn more about KEDA configuration in the [KEDA External Scaler](/docs/keda-scaler) documentation
* Configure telemetry sources to send data to the load balancer endpoint
* Monitor queue metrics and scaling behavior through your observability platform
* Adjust scaling thresholds based on your traffic patterns and requirements
# Manual Collector Deployment
Source: https://docs.sawmills.ai/docs/manual-collector-deployment
Deploy customer-managed Sawmills Collectors with generated values files, direct Helm installs, umbrella charts, and Helmsman-compatible values.
Use manual deployment when Sawmills should generate collector configuration, but your team applies the Helm changes in your own release process. This is common when the Sawmills collector chart is embedded in an umbrella chart, monochart, or Helmsman-managed deployment.
> **Note:** Automatic collector management and manual deployment use different workflows. With automatic management, the Sawmills remote operator applies collector upgrades. With manual deployment, Sawmills provides the chart version and values artifacts, and your deployment system applies them.
## Choose the Values Format
Set the manual deployment format in the collector setup flow or on the collector page.
| Format | Use this when | What Sawmills provides |
| :- | :- | :- |
| Direct chart | You install `sawmills-collector-chart` as its own Helm release | Chart, version, release, namespace, values files, and a Helm command |
| Umbrella chart | You embed the Sawmills collector chart as a dependency in a larger chart | Chart, version, values root key, resource base name, and values files |
| Umbrella chart with Helmsman | You embed the chart and Helmsman processes the values files | Same as umbrella chart, with `${...}` expressions escaped for Helmsman |
For umbrella and Helmsman formats, add the Sawmills collector chart dependency to your chart and apply the generated values files through your existing release process. The UI intentionally does not show a standalone Helm command for those formats because the release name, namespace, and install command belong to your parent chart.
## Generated Files
Each manual deployment creates values artifacts you can download from the deployment UI.
| File | Owner | Purpose |
| :- | :- | :- |
| `sawmills-generated.values.yaml` | Sawmills | Generated collector configuration for the deployed pipelines. Do not edit this file manually. |
| `sawmills-customer-overrides.yaml` | Customer | Customer-owned Helm overrides such as scheduling, resources, service annotations, and extra environment variables. Keep this file after the generated file when applying values. |
For direct chart installs, apply the generated file first and the customer override file second:
```bash theme={null}
helm upgrade --install sawmills-collector \
oci://public.ecr.aws/s7a5m1b4/sawmills-collector-chart \
--version \
--namespace \
-f sawmills-generated.values.yaml \
-f sawmills-customer-overrides.yaml \
--wait \
--timeout 10m
```
For umbrella chart installs, place the Sawmills collector chart under the dependency key shown as **Values root** in the UI. For example, if the values root is `sawmillsCollector`, generated values are nested like this:
```yaml theme={null}
sawmillsCollector:
resourceBaseName: sawmills-collector
# Sawmills-generated collector values continue here.
```
## Values Root Key
The values root key is the top-level key used by your umbrella chart for the embedded Sawmills collector dependency. Direct chart installs do not use a values root key.
Use the same key in:
* your umbrella chart dependency alias
* `sawmills-generated.values.yaml`
* `sawmills-customer-overrides.yaml`
* any parent chart values that override Sawmills collector settings
## Resource Base Name
`resourceBaseName` controls the Kubernetes base name used by the collector resources. Set it when your parent chart release name differs from the collector resource names you want Sawmills to manage.
The default base name is `sawmills-collector`. With that default:
| Mode | Resource | Name |
| :- | :- | :- |
| No load balancer | Collector service | `sawmills-collector` |
| No load balancer | Collector headless service | `sawmills-collector-headless` |
| Load balancer enabled | Source-facing load balancer service | `sawmills-collector` |
| Load balancer enabled | Load balancer workload | `sawmills-collector-lb` |
| Load balancer enabled | Backend collector service | `sawmills-collector-backend` |
| Load balancer enabled | Backend collector headless service | `sawmills-collector-backend-headless` |
| KEDA external scaler | KEDA scaler service | `sawmills-collector-keda-otel-scaler` |
> **Warning:** Changing `resourceBaseName` on an existing release changes rendered Kubernetes resource names. Use it on a fresh install or plan a rename migration.
## S3 Config Files
Manual deployments can write Sawmills-generated collector config files in two modes:
| Mode | Behavior |
| :- | :- |
| Changing S3 config files | Each deployment gets new encrypted S3 config file paths. Use this when your release process expects immutable generated inputs. |
| Static S3 config files | Sawmills reuses the same S3 config file paths after the first apply. Use this when your release process needs stable paths. |
## KEDA and Load Balancer Routing
If load balancing is enabled, telemetry sources should send data to the source-facing load balancer service, not directly to backend collectors. The load balancer routes to backend collectors through the backend headless service.
If KEDA external scaling is enabled, the chart derives the scaler address from the release namespace and generated scaler service name. You normally do not need to hardcode `scalerAddress`; keep the chart default unless Sawmills Support asks you to override it.
When using Helmsman, choose **Umbrella chart with Helmsman** so Sawmills escapes `${...}` expressions before Helmsman processes the values file.
# Mills Slack installation and demo
Source: https://docs.sawmills.ai/docs/mills-slack-demo
Screenshots of Mills authorization, channel setup, and issue investigations in Slack.
Mills brings production issue investigations into Slack: recorded signals,
investigation findings, an evolving timeline, and downloadable context for coding
agents. The Slack app is currently named **Sawmills**; the agent is **Mills**.
## 1. Start in Mills
Sign in at [app.agentmills.ai](https://app.agentmills.ai), select your organization,
then open **Integrations** and connect Slack. Choose whether Mills may automatically
join public notification channels. With automatic joining disabled, invite Mills
to each notification channel yourself.
![Mills Connect Slack authorization options]()
## 2. Authorize your Slack workspace
Slack opens a consent screen. Select the workspace where you want to install
Mills, review the permissions and linked terms, then choose **Allow**. Return to
Mills to finish the connection.
The screenshot shows the consent screen for an existing internal installation.
Use your own workspace when installing.
![Slack OAuth workspace selection and consent screen]()
## 3. Configure and validate the channel
Open the Slack account configuration, choose a notification channel, and save.
Use **Validate** to check the token and channel membership. Use **Send test
message** to confirm delivery. Configure escalation and resolution notifications
and the minimum severity appropriate for your team.
The captured connection passed validation. Delivery is a separate check; this
capture does not show a successful test-message delivery.
![Configured Slack account with successful token and channel validation]()
Reconnect refreshes access to the existing workspace. To switch workspaces,
disconnect the current account first, then connect Slack again. This changes
the notification destination for the selected Mills organization.
## 4. Investigate issues from Slack
An issue notification includes its severity, recorded signals, and an **Open
issue** link to the full investigation in Mills. The investigation timeline records
changes as the issue develops. The thread's **agent-context.json** attachment
provides investigation context for a coding agent.
The following image is a sanitized, illustrative example of the Slack issue
layout. Its text and timestamps have been replaced; it is not a live delivery
record or customer telemetry.
![Sanitized illustrative Mills issue notification with timeline and agent context]()
## 5. Continue the conversation
Following **Open issue** opens the investigation in Mills. This screenshot shows
an existing Acme demo issue with its findings, triggered monitors, and timeline.
It is historical demo data, not evidence of a currently running investigation.
![Historical Acme demo issue findings, monitors, and investigation timeline]()
Mention **@Mills** in a channel it has joined, or send a direct message. Ask for an
issue summary, then continue in the same thread to preserve conversation context.
See [using Mills in Slack](/docs/sawmills-agent-slack) for conversation guidance.
For setup or review assistance, contact [support@sawmills.ai](mailto:support@sawmills.ai).
# Network Requirements
Source: https://docs.sawmills.ai/docs/network-requirments
Required FQDNs, firewall rules, and DNS configuration needed to allow outbound access from the Sawmills Collector and agent to Sawmills cloud services.
## Network Requirements
To ensure proper functionality of the Sawmills agent, please configure your network to allow outbound access to the following Fully Qualified Domain Names (FQDNs). This is essential for communication between the agent and our services.
## Required FQDNs
Make sure your network allows access to these domains:
* `api.sawmills.ai`
* `app.sawmills.ai`
* `controller.sawmills.ai`
* `metrics.sawmills.ai`
* `ingress.sawmills.ai`
* `controller.ue1.prod.plat.sm-svc.com`
## Network Configuration Tips
1. **Firewall Rules**: Update your firewall settings to permit outbound traffic to these domains over the required ports (typically HTTP/HTTPS on port 80/443).
2. **Proxy Servers**: If your network uses a proxy server, ensure these domains are added to the proxy allowlist.
3. **DNS Resolution**: Verify that these FQDNs resolve correctly from within your Kubernetes cluster.
# Enable New Relic Integration
Source: https://docs.sawmills.ai/docs/newrelic-integration
Connect your New Relic User API Key and Account ID to Sawmills for metrics usage features with step-by-step setup and automatic permission validation.
## About New Relic Integration
The New Relic Integration lets Sawmills use New Relic credentials for metrics usage features.
When you save the New Relic configuration, Sawmills validates the credentials and checks predefined permissions before storing them.
## Step-by-Step: Configure New Relic Integration
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
### 2. Open New Relic Configuration
In the Integrations section, click on **Add New Relic Account**.
### 3. Enter Credentials
Provide:
* **New Relic User API Key**
* **New Relic Account ID**
* **New Relic Account Region**
## Set up New Relic Dedicated User
For production, use a dedicated New Relic **user account** (acting as a service account, not a personal user key).
### 1. Create a user in New Relic
Create a dedicated user for Sawmills integration usage. A **Basic user** type is usually sufficient since it provides access to the API and querying capabilities without incurring billable costs like a Core or Full Platform user.
**Optional: Restrict to Read-Only Access**
If you want to strictly enforce the principle of least privilege, you can choose to restrict this user to read-only access. In New Relic, you do this by creating a **Group** for the user, and then creating an **Access Grant** that links the new group to your target **Account** with the built-in **Read Only** role.
### 2. Create keys for the user
* Create a **User API Key** in New Relic for this dedicated user.
* Find your **Account ID**.
* Use those credentials in the Sawmills New Relic integration along with your account's region.
### 3. Save and Validate
Click **Save**. Before persisting, Sawmills runs validation and shows:
* User key, account and region status: **Working** or **Invalid**
* Required permissions status
* Per-permission status for predefined checks
Save is blocked if credentials are invalid or required permissions are missing.
## Backend Validation Flow
Validation is enforced in the backend service:
* `POST /v1/notification-integrations/newrelic/validate`: validates credentials and returns per-permission status.
* `POST /v1/notification-integrations`: blocks save when credentials are invalid or required permissions are missing.
* `PATCH /v1/notification-integrations/{id}`: re-validates merged New Relic config before update is persisted.
## Required Permissions
* `alerts`
* `entitySearch`
# OKTA Workforce
Source: https://docs.sawmills.ai/docs/okta-workforce
Configure Okta Workforce as a SAML 2.0 identity provider for single sign-on into Sawmills, including app creation and attribute mapping steps.
Currently Sawmills does not expose a user interface for OKTA config. You will need to work with Sawmills support to finish the OKTA configuration.
#### Create a new enterprise application in Okta
To create a new enterprise application in Okta:
1. Navigate to Okta and sign in.
2. In the Okta dashboard, select Admin in the top right corner.
3. In the navigation sidebar, select the Applications dropdown and select Applications.
4. Select the Create App Integration button.
5. In the Create a new app integration modal, select the SAML 2.0 option and select the Next button.
6. Once redirected to the Create SAML Integration page, fill in the General Settings fields. An App name is required.
7. Select the Next button to continue.
#### Configure your service provider
Once you have moved forward from the General Settings instructions, you will be presented with the Configure SAML page.
To configure your service provider (Sawmills), you must add these two fields to your IdP's application:
* Single sign-on URL - This is your application's URL that your IdP will redirect your users back to after they have authenticated in your IdP.
* Audience URI (SP Entity ID) - This is a unique identifier for your SAML connection that your IdP application needs.
You will need to fill out the appropriate values for these fields. Currently Sawmills does not expose these via the UI, please ask Sawmills support to securely send this to you.
Once you have these, please paste the Single sign-on URL and the Audience URI (SP Entity ID) values into their respective fields.
#### Map Okta claims to Sawmills attributes
Mapping the claims in your IdP to the attributes in Sawmills ensures that the data from your IdP is correctly mapped to the data in Sawmills.
| Sawmills attribute | Okta claim |
| - | - |
| mail | user.email |
| firstName | user.firstName |
| lastName | user.lastName |
1. In the Okta dashboard, find the Attribute Statements (optional) section.
2. For the Name field, enter `mail`.
3. For the Value field, choose `user.email `from the dropdown.
4. Scroll to the bottom of the page and select the Next button to continue.
5. You will be redirected to the Feedback page. Fill out the feedback however you would like and select the Finish button to complete the setup.
#### Assign selected user or group in Okta
You need to assign your users/user groups to your enterprise application.
1. In the Okta dashboard, select the Assignments tab.
2. Select the Assign dropdown. You can either select Assign to people or Assign to groups.
3. In the search field, enter the user or group of users that you want to assign to the enterprise application.
4. Select the Assign button next to the user or group that you want to assign.
5. Select the Done button to complete the assignment.
#### Configure your identity provider
Once you have completed the setup in Okta, you will be redirected to the application instances page with the Sign On tab selected.
1. Under Sign on methods, copy the Metadata URL.
2. Securely send this Sawmills support team
# Configure Outbound Proxy for Remote Operator and Collector
Source: https://docs.sawmills.ai/docs/outbound-proxy
Configure HTTP/HTTPS outbound proxy settings for the Sawmills Remote Operator and Collector using Helm values for HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.
The Remote Operator maintains a bidirectional gRPC session with `controller.sawmills.ai` and can emit Prometheus remote-write traffic. The Sawmills Collector sends telemetry data to Sawmills services. Some customers must route every outbound internet connection through an egress proxy. This guide explains how to enable that flow while keeping TLS endpoints pointed at Sawmills services.
## Prerequisites
* Remote Operator chart version `2.0.9` or later (proxy values are not present in older releases). See [GitHub releases](https://github.com/Sawmills/helm-charts/tags) for the latest version.
* Sawmills Collector chart version `2.8.3` or later with proxy support. See [GitHub releases](https://github.com/Sawmills/helm-charts/tags) for the latest version.
* An HTTP/HTTPS CONNECT proxy reachable from the cluster, including credentials if the proxy requires authentication.
* Access to update Helm values (either `values.yaml` or CLI `--set` flags).
## Configure proxy values via Helm
Both charts expose a `proxy` block that maps directly to the pod environment variables `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`. The configuration structure is identical for both components.
### Remote Operator configuration
Populate the proxy fields in your Remote Operator values file:
```yaml theme={null}
proxy:
http: http://user:pass@corp-proxy.example.com:3128
https: http://user:pass@corp-proxy.example.com:3128
noProxy:
- kubernetes.default.svc
- 10.0.0.0/8
- 169.254.169.254
```
Then upgrade the release:
```bash theme={null}
helm upgrade --install sawmills-remote-operator oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--namespace sawmills \
--values values.yaml
```
The operator keeps the TLS Server Name set to `controller.sawmills.ai`, so the proxy only tunnels the TCP stream while the certificate remains valid.
### Sawmills Collector configuration
For the Sawmills Collector, configure the proxy in the same way. If you're managing the collector via the Remote Operator, add the proxy configuration under `managedChartsValues` (the legacy key `managedCharts` still works as an alias):
```yaml theme={null}
managedChartsValues:
sawmills-collector:
proxy:
http: http://user:pass@corp-proxy.example.com:3128
https: http://user:pass@corp-proxy.example.com:3128
noProxy:
- kubernetes.default.svc
- 10.0.0.0/8
- 169.254.169.254
```
If you're installing the collector chart directly, add the proxy block to the collector's values file:
```yaml theme={null}
proxy:
http: http://user:pass@corp-proxy.example.com:3128
https: http://user:pass@corp-proxy.example.com:3128
noProxy:
- kubernetes.default.svc
- 10.0.0.0/8
- 169.254.169.254
```
The collector keeps TLS Server Names set to Sawmills service endpoints, so the proxy only tunnels the TCP stream while certificates remain valid.
### CLI-only configuration
You can also pass the values without editing a file. For the Remote Operator:
```bash theme={null}
helm upgrade --install sawmills-remote-operator oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--namespace sawmills \
--set proxy.http="http://$USER:$HOSTNAME@proxy.corp:32281" \
--set proxy.https="http://$USER:$HOSTNAME@proxy.corp:32281" \
--set proxy.noProxy[0]="kubernetes.default.svc" \
--set proxy.noProxy[1]="10.4.0.0/16"
```
For the Sawmills Collector (when managed by Remote Operator):
```bash theme={null}
helm upgrade --install sawmills-remote-operator oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--namespace sawmills \
--set managedChartsValues.sawmills-collector.proxy.http="http://$USER:$HOSTNAME@proxy.corp:32281" \
--set managedChartsValues.sawmills-collector.proxy.https="http://$USER:$HOSTNAME@proxy.corp:32281" \
--set managedChartsValues.sawmills-collector.proxy.noProxy[0]="kubernetes.default.svc" \
--set managedChartsValues.sawmills-collector.proxy.noProxy[1]="10.4.0.0/16"
```
### NO\_PROXY formatting rules
`proxy.noProxy` accepts either:
* A YAML list (recommended) that will be joined with commas, or
* A raw comma-separated string.
Include every cluster-local CIDR, service DNS name, or metadata endpoint you want to bypass the proxy. Both components only proxy traffic bound for Sawmills services; Kubernetes API calls stay inside the cluster regardless of this list.
## Observing proxy configuration
### Remote Operator
When proxy variables are detected, the Remote Operator logs a single informational line similar to:
```
info component=remote-operator http_proxy=http://redacted:redacted@corp-proxy:3128 no_proxy=kubernetes.default.svc,10.0.0.0/8 msg="Detected outbound proxy configuration for Sawmills traffic"
```
### Sawmills Collector
The Sawmills Collector uses the proxy environment variables automatically when present. To verify the configuration, check that the environment variables are set in the pod:
```bash theme={null}
kubectl exec -n sawmills deployment/sawmills-collector -- env | grep -i proxy
```
Credentials are redacted in logs. If you do not see the expected behavior, verify that the Helm values rendered into the pod (use `kubectl describe deployment` or `kubectl exec env`).
# Ownership Groups
Source: https://docs.sawmills.ai/docs/ownership-groups
Define Ownership Groups with attribute-based matching rules to assign telemetry data to teams and route notifications to the right Slack channels or emails.
## Overview
`Ownership Groups` let you define which team owns which telemetry, using attribute-based rules for logs and metrics.
Groups are useful when you want to define ownership of telemetry data. The core idea is to help the team that generates the data manage it effectively.
After a group is defined (including its communication channels), that team can focus on the insights, processors, and telemetry volume relevant to the data they own.
At a high level, each group includes:
* A name and optional description
* One or more ownership rules per signal type
* Optional notification channels (Email, Slack, Webhook)
***
## How matching works
### Ordered evaluation
Groups are evaluated in top-to-bottom order from the Groups page. Order matters.
For each telemetry item:
1. Sawmills checks groups in their configured order
2. The first matching group is selected
3. Matching stops after the first match
This means telemetry is assigned to a single group based on rule order.
### Default Group (fallback)
Sawmills includes a built-in `Default Group`.
* It has no matcher conditions
* It cannot be edited or deleted
* It acts as a fallback view for telemetry that does not match another group rule
* By default, organization admins are added to its email notifications and can remove themselves later from the group settings
***
## Defining ownership rules
When you create or edit a group, you define rules under `Ownership Rules`.
* `Any rule matches`: If any rule in the selected signal type matches, telemetry is assigned to the group
* Rules are combined with OR across rule groups
* Conditions inside a rule can include multiple attribute predicates
Use rules that reflect stable ownership boundaries, for example:
* `service.name = payments-api` and `env = prod`
* `k8s.namespace.name = checkout`
* `team = growth`
Start with high-confidence attributes that are consistently present in your telemetry.
***
## Supported signal types
In the current UI, ownership rules are configured for:
* Logs
* Metrics
You can preview rule behavior in Live Tail while editing a group.
***
## Notifications by group
Each group can have dedicated notification targets:
* Email recipients
* Slack channels
* Webhook endpoint
This helps route data-quality and operational ownership to the right team.
***
## Best practices
* Keep groups mutually understandable: use clear names like `Core API`, `Data Platform`, `Customer Success Apps`
* Put more specific groups higher in the order and broader catch-all groups lower
* Revisit order after adding new teams, services, or telemetry conventions
* Keep rule count small at first, then refine based on real traffic patterns
* Use the preview panel to verify matches before saving changes
***
## Common pitfalls
* **Overlapping rules without ordering intent**: overlap is allowed, but earlier groups win
* **Overly broad first rules**: broad rules at the top can absorb traffic that should belong to later groups
* **Unstable attributes**: avoid ownership keys that change frequently or are missing in parts of your data
# Sawmills Pipeline Destinations
Source: https://docs.sawmills.ai/docs/pipeline-destinations
Browse all supported Sawmills pipeline destinations including AWS S3, Datadog, Elasticsearch, Splunk HEC, Snowflake, Prometheus, and more.
| | |
| - | - |
| [AWS S3](/docs/destination-aws-s3) | AWS S3 Destination Configuration |
| [Coralogix](/docs/destination-coralogix) | Coralogix Destination Configuration |
| [Datadog](/docs/destination-datadog) | Datadog Destination Configuration |
| [Debug (/dev/null)](/docs/destination-debug) | Debug (/dev/null) Destination Configuration |
| [Elasticsearch](/docs/destination-elasticsearch) | Elasticsearch Destination Configuration |
| [Grafana Loki](/docs/destination-grafana-loki) | Grafana Loki Destination Configuration |
| [OTLP](/docs/destination-otlp) | OTLP Destination Configuration |
| [Logz.io](/docs/destination-logzio) | Logz.io Destination Configuration |
| [New Relic](/docs/destination-new-relic) | New Relic Destination Configuration |
| [Snowflake](/docs/destination-snowflake) | Snowflake Destination Configuration |
| [Splunk HEC](/docs/destination-splunk-hec) | Splunk HEC Destination Configuration |
| [Prometheus](/docs/destination-prometheus) | Prometheus Destination Configuration |
| [S3 Tables](/docs/destination-s3-tables) | S3 Tables Destination Configuration |
| [Sumo Logic](/docs/destination-sumologic) | Sumo Logic Destination Configuration |
# Manage an Existing Pipeline
Source: https://docs.sawmills.ai/docs/pipeline-manage
Monitor and manage an existing Sawmills pipeline, including live and draft modes, flow visualization, activity logs, and deployment to collectors.
## Overview
### Manage a Sawmills Pipeline
1. Log in to the Sawmills dashboard.
2. Navigate to **Pipeline Mangement**.
3. Click on an existing pipeline to navigate to that pipeline's management page.
The Pipeline page consists of three main areas:
1. **Pipeline Toolbar**
* **Pipeline Name**
Name of this Pipeline
* **Live/Draft**
Switches between "Edit" and "Live View" modes.
* **Settings**
Opens a dialog where you can either edit or delete a Pipeline 's Name and optional Description or Delete the Pipeline
* **Activity Log**
Opens a dialog where you can view an Audit Trail of all activity from a human in regards to this Pipeline.
Each entry has a button on the right that displays changes made to this pipeline.
* **Deployement**
Enables you to deploy this Pipeline to one or more Controllers
2. **Pipeline Flow**
Presents a visual depiction of the pipeline (Sources, Processors and Destinations). When in "Live" mode, it is read only. When in "Draft" mode, you can edit the Pipeline's Source, Processors and Destinations.
### Previewing Destination Output
In **Live** mode, supported destinations include a **Preview Output** button in the destination drawer. Use it to inspect sampled telemetry for a specific destination after the pipeline has applied destination-specific handling.
This is useful when a supported destination such as Datadog or Elasticsearch changes how telemetry is shaped before export. Instead of only checking the data earlier in the pipeline, you can preview what is being sent for that destination.
To preview destination output:
1. Set the Live/Draft toggle to **Live**.
2. Click the destination in the pipeline flow.
3. Click **Preview Output** in the destination drawer.
The preview opens a sampled stream for that destination so you can compare the output with the data seen earlier in the pipeline.
### Editing a Pipeline Source
#### To edit a Pipeline's Source
1. Set the Live/Draft Toggle to `Draft`
2. Open the pipeline's source dialog
* When adding a new pipeline source, click on the \`+' icon located under a last Source in the diagram.
* When editing an existing pipeline source, click on the source icon itself.
For specifics on a Pipeline Source's configuration, refer to the source type's configurations. For a list of supported sources, refer to [Pipeline Sources](/docs/pipeline-sources)
### Editing a Pipeline Destination
#### To edit a Pipeline's Destination
1. Set the Live/Draft Toggle to `Draft`
2. Open the pipeline's destination dialog
* When adding a new pipeline destination, click on the \`+' icon located under a last destination in the diagram.
* When editing an existing pipeline destination, click on the destination icon itself.
For specifics on a Pipeline Destination's configuration, refer to the destination type's configurations. For a list of supported destinations, refer to [Pipeline Destinations](/docs/pipeline-destinations)
After you save your updates, you must redeploy the pipeline to all associated collectors for your changes to take effect.
# Pipeline Management
Source: https://docs.sawmills.ai/docs/pipeline-management
View all pipelines in your Sawmills organization, check deployment status, see associated collectors, and navigate to individual pipeline management pages.
## Overview
The **Pipelines** page provides a central location to view pipelines defined in your system. Use this interface to track associated collectors and to navigate to individual pipeline's management pages.
### Pipelines
The table on the **Pipelines** page displays the following columns:
1. **Name**\
The name of the pipeline. Pipeline details can be viewed by clicking on a specific pipeline's name link.
2. **Collectors**\
Lists Collectors that are currently configured to run each pipeline.
3. **Owner**: Shows the owner/creator of a pipeline.
4. **Last Deployed**\
Displays the date the pipeline was most recently deployed in `MM/DD/YYYY` format. If unavailable, displays `-`.
Click `New Pipeline` to create a new Pipeine.
# Sawmills Pipeline Sources
Source: https://docs.sawmills.ai/docs/pipeline-sources
Browse all supported Sawmills pipeline sources including Datadog, OTLP, Elasticsearch, Fluent Forward, Splunk HEC, Syslog, Prometheus, and more.
| | |
| - | - |
| [CloudWatch Metrics (S3/SQS)](/docs/source-cloudwatch-s3) | CloudWatch Metrics via S3 and SQS Source Configuration |
| [CloudWatch Metric Streams (Firehose)](/docs/source-cloudwatch-firehose) | CloudWatch Metric Streams Source Configuration |
| [Datadog](/docs/source-datadog) | Datadog Source Configuration |
| [Elasticsearch](/docs/source-elasticsearch) | Elasticsearch Source Configuration |
| [Fluent Forward](/docs/source-fluent-forward) | Fluent Forward Source Configuration |
| [Grafana Loki](/docs/source-grafana-loki) | Grafana Loki] Source Configuration |
| [New Relic](/docs/source-new-relic) | New Relic Source Configuration |
| [OTLP](/docs/source-otlp) | OTLP Source Configuration |
| [Prometheus](/docs/source-prometheus) | Prometheus Source Configuration |
| [TCP Log ](/docs/source-tcplog) | TCP Log Source Configuration |
| [Splunk HEC ](/docs/source-splunk-hec) | Splunk HEC Source Configuration |
| [Syslog ](/docs/source-syslog) | Syslog Source Configuration |
# Aggregation Processor
Source: https://docs.sawmills.ai/docs/processor-aggregation
Group and summarize similar log events over a configurable time window using the Sawmills Aggregation Processor to reduce noise and storage costs.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Aggregation Processor
The Sawmills Aggregation Processor allows you to group similar log events together over a specified time window, reducing noise and storage costs by collapsing repeated events into summary records.
### Configuration Components
#### 1. Name
* **Description**: A unique identifier for your aggregation processor. Choose a descriptive name that helps you easily identify this processor in your pipeline.
#### 2. Conditions (Optional)
* **Description**: Specify attribute-based conditions to determine which events should be processed.
* **Functionality**: Only aggregate events that match (or don't match) these conditions.
* **Logic Options**:
* **Match all (AND)**: All conditions must be met for the event to be aggregated.
* **Match any (OR)**: At least one condition must be met for the event to be aggregated.
#### 3. Interval (seconds)
* **Description**: The time window (in seconds) over which matching records are aggregated.
* **Functionality**: Events occurring within this time window will be grouped together if they're considered similar.
#### 4. Dedup Threshold
* **Description**: The number of identical events required before collapsing them into one summary record.
* **Functionality**: Only when this threshold is reached will events be aggregated into a summary record.
### Advanced Options
#### 1. Aggregate Max Size
* **Description**: Maximum number of distinct event patterns that can be tracked simultaneously for aggregation.
* **Functionality**: When this limit is reached, the processor will flush all currently tracked aggregations, even if their time intervals haven't elapsed or deduplication thresholds haven't been met.
#### 2. Timezone
* **Description**: The timezone to apply to the timestamp on the new summary record.
* **Functionality**: Ensures that timestamp information on aggregated records uses a consistent timezone.
#### 3. Masks
* **Description**: Regex patterns for stripping or masking dynamic content before deduplication.
* **Functionality**: Helps identify similar events even when they contain variable elements like timestamps, IDs, or other dynamic content.
* **Components**:
* **Name**: An identifier for the mask pattern.
* **Pattern**: The regular expression pattern used to identify the dynamic content to mask.
### Aggregation Attributes
When an event group meets the dedup threshold (i.e., more than one matching log is found within the interval), the processor enriches the summary record with the following attributes:
| Attribute | Type | Description |
| - | - | - |
| `sawmills.log_count` | Integer | The total number of similar events that were collapsed into this summary record. |
| `sawmills.first_seen` | String (RFC 3339) | Timestamp of the earliest event in the group, formatted in the configured timezone. |
| `sawmills.last_seen` | String (RFC 3339) | Timestamp of the latest event in the group, formatted in the configured timezone. |
These attributes are nested under a `sawmills` key in the log record attributes. For example:
```json theme={null}
{
"sawmills": {
"log_count": 5,
"first_seen": "2025-04-10T10:19:52Z",
"last_seen": "2025-04-10T10:21:07Z"
}
}
```
Events that do not meet the dedup threshold are forwarded as-is, without any aggregation attributes.
### Use Cases
* **Error Aggregation**: Collapse repeated error messages into a single summary, showing the count and time range.
* **Noise Reduction**: Reduce the volume of high-frequency, low-value logs while preserving information about their occurrence.
* **Alerts De-duplication**: Prevent alert fatigue by combining similar alerts that occur within the specified time window.
### Implementation Notes
* Set an appropriate interval based on your application's log patterns and monitoring needs.
* Use masks to improve grouping accuracy by removing variable parts of log messages.
* The dedup threshold helps prevent premature aggregation of events that may not be part of a true pattern.
# Drop Attributes Processor
Source: https://docs.sawmills.ai/docs/processor-drop-attribute
Remove specific attributes from log data using the Sawmills Drop Attributes Processor to reduce payload size or strip sensitive information.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Drop Attributes Processor for Logs
The Sawmills Drop Attributes Processor removes specified attributes from telemetry data, reducing size, excluding irrelevant attributes, or removing sensitive information.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Attribute Filters
* **Conditions**: Specify conditions to filter events before processing. Events must satisfy all conditions (AND) or at least one (OR) based on the selected logic.
Each condition follows this sequence:
1. **Choose the condition type:**
* **Log Level (Severity)**
* **Body as String**
2. **Select a comparison operator:**
* **Equals** / **Not Equals**
3. **Provide a value:**
* **Log Level:** Select from a dropdown (INFO, WARN, ERROR, etc.).
* **Body as String:** Enter a free-text value for matching.
#### 3. Attribute Keys
* **Description**: List of attribute keys to be removed. Specify attributes directly or manually input them if not listed in the dropdown.
# Drop Metric Datapoints Processor
Source: https://docs.sawmills.ai/docs/processor-drop-metric-datapoints
Remove specific metric datapoints by name and optional label filters using the Sawmills Drop Metric Datapoints Processor to reduce metric volume.
## Supported Data Types
📊 **Metrics**
### Supported Metric Types
| Metric Type | Support Level | Notes |
| - | - | - |
| Gauge | ✅ Full | |
| Sum (Counter) | ✅ Full | |
| Histogram | ✅ Full | |
| Summary | ✅ Full | |
| ExponentialHistogram | ✅ Full | |
### Configuring the Sawmills Drop Metric Datapoints Processor
The Sawmills Drop Metric Datapoints Processor removes specific metric datapoints based on metric names and optional filters. Use this processor to reduce metric volume by dropping unwanted datapoints before they reach your destination.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Metric Names
* **Description**: List of metric names to target for dropping. Supports wildcard patterns.
* **Examples**:
* `http.server.request.duration` - matches exact metric name
* `http.server.*` - matches all metrics starting with `http.server.`
* `*.request.*` - matches any metric containing `.request.`
* **Functionality**: All datapoints from metrics matching these patterns will be dropped unless narrowed by datapoint filters.
#### 3. Datapoint Filters (Optional)
* **Description**: Conditions to narrow which datapoints are dropped. Without filters, all datapoints for the selected metrics are dropped.
* **Logic Options**:
* **Match all (AND)**: All conditions must be met for the datapoint to be dropped.
* **Match any (OR)**: At least one condition must be met for the datapoint to be dropped.
Each condition follows this sequence:
1. **Choose the attribute key:**
* Select from available metric attributes/labels
2. **Select a comparison operator:**
* **Equals** / **Not Equals**
* **Contains** / **Not Contains**
* **Regex** / **Not Regex**
3. **Provide a value:**
* Enter the value to match against
### Use Cases
* **Cost Optimization**: Drop high-cardinality metrics that provide limited value.
* **Noise Reduction**: Remove metrics from test or development environments.
* **Selective Filtering**: Keep only datapoints with specific label values by dropping everything else.
* **Cardinality Control**: Drop datapoints with specific high-cardinality label values.
### Implementation Notes
* Metric name patterns support wildcards (`*`) for flexible matching.
* When no datapoint filters are specified, all datapoints for matching metrics are dropped.
* Combine metric name patterns with datapoint filters for precise control over which datapoints to remove.
# Filter Processor
Source: https://docs.sawmills.ai/docs/processor-filter
Selectively include, exclude, or sample logs based on attribute conditions using the Sawmills Filter Processor to reduce noise and control volume.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Filter Processor for Logs
The Sawmills Filter Processor allows you to selectively include, exclude and sample logs based on specific criteria. This is particularly useful for reducing noise by excluding irrelevant logs or only including logs that meet specific criteria.\
**Note:** The `selected attribute` is searched across `Attributes`, `Resources`, and `Body`
This guide covers how to configure filters for logs using an interface with the following components:
### Configuration Components
#### 1. Name
* **Description**: This is the identifier for your filter. A unique, descriptive name helps you easily identify each filter, especially in interfaces where multiple filters may be applied.
#### 2. Attribute Filters
* **Conditions**: Specify conditions to filter events before processing. Events must satisfy all conditions (AND) or at least one (OR) based on the selected logic.
Each condition follows this sequence:
1. **Choose the condition type:**
* **Log Level (Severity)**
* **Body as String**
2. **Select a comparison operator:**
* **Equals** / **Not Equals**
3. **Provide a value:**
* **Log Level:** Select from a dropdown (INFO, WARN, ERROR, etc.).
* **Body as String:** Enter a free-text value for matching.
#### 3. Action
* **Description**: Determines whether logs that meet the specified conditions are included, excluded, or sampled.
* **Options**:
* **Exclude**: Discard logs that match the specified conditions.
* **Include**: Keep only the logs that match the specified conditions, excluding all others.
* **Sample**: Retain only x% of logs that meet the specified conditions, excluding all others.
* **Use Cases**:
* Use **Exclude** to remove logs from ingestion based on certain criteria, like low-severity logs.
* Use **Include** to capture only high-priority logs, filtering out less critical information.
* Use **Sample** to capture a percentage of logs, reducing storage and logging costs.
#### 4. Conditions
* **Description**: Conditions define the criteria for log filtering. Each condition consists of key-value pairs specifying the attributes that a log entry must have to be included, excluded or sampled.
* **Multiple Conditions**:
Enables you to define multiple conditions. When defining multiple conditions the following applies:
* **Match all (AND)**: All conditions must be met for the action to apply.
* **Match any (OR)**: Only one of the conditions needs to be met for the action to apply.
# Aggregate Metric Labels Processor
Source: https://docs.sawmills.ai/docs/processor-label-aggregation
Reduce metric cardinality by aggregating datapoints across specified labels for gauge and sum metric types using the Sawmills Label Aggregation Processor.
## Supported Data Types
📊 **Metrics**
### Supported Metric Types
| Metric Type / Vendor | Datadog | New Relic |
| - | - | - |
| Gauge | ✅ Supported | ✅ Supported |
| Sum (Counter) | ✅ Supported | ✅ Supported |
| Histogram | ❌ Not Supported | ❌ Not Supported |
| Summary | ❌ Not Supported | ❌ Not Supported |
| ExponentialHistogram | ✅ Supported | ❌ Not Supported |
If you are using the [Datadog Agent](/docs/send-data-from-datadog-agent) as a source, label aggregation on
histogram-derived metrics may produce misleading results.
### Configuring the Sawmills Aggregate Metric Labels Processor
The Sawmills Aggregate Metric Labels Processor reduces metric cardinality by aggregating datapoints across specified labels. It removes selected labels and combines metric values using an aggregation function, resulting in fewer unique time series.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Metric Names
* **Description**: List of metrics to apply label aggregation to. Select from available metrics in your pipeline.
* **Functionality**: Only the specified metrics will have their labels aggregated. All selected metrics use the same aggregation function.
#### 3. Aggregation Function
* **Description**: The mathematical function used to combine datapoint values when labels are removed.
* **Options**:
* **Sum**: Add all values together
* **Average**: Calculate the mean of all values
* **Max**: Keep the maximum value
* **Min**: Keep the minimum value
* **Use Cases**:
* Use **Sum** for counters and cumulative metrics (e.g., request counts, bytes transferred)
* Use **Average** for gauge-type metrics (e.g., CPU utilization, memory percentage)
* Use **Max** for capturing peak values (e.g., maximum latency, peak connections)
* Use **Min** for capturing floor values (e.g., minimum available capacity)
#### 4. Labels to Aggregate
* **Description**: Select which labels to remove and aggregate across. Available labels are fetched based on the selected metrics.
* **Functionality**: Datapoints that differ only by these labels will be combined into a single datapoint using the selected aggregation function.
* **Select All**: Option to select all available labels for maximum cardinality reduction.
### Use Cases
* **Cardinality Reduction**: Remove high-cardinality labels like `pod_name`, `instance_id`, or `container_id` to reduce the number of unique time series.
* **Cost Optimization**: Lower storage and query costs by reducing metric volume while preserving aggregate insights.
* **Environment Rollup**: Aggregate metrics across replicas or instances to get environment-level totals.
* **Simplified Dashboards**: Create cleaner visualizations by removing unnecessary dimensional detail.
### Example
**Before aggregation:**
```
http_requests_total{service="api", instance="pod-1"} = 100
http_requests_total{service="api", instance="pod-2"} = 150
http_requests_total{service="api", instance="pod-3"} = 120
```
**After aggregating label `instance` with Sum:**
```
http_requests_total{service="api"} = 370
```
### Implementation Notes
* Label aggregation operates on 5-second windows to batch and combine datapoints.
* Choose the aggregation function that matches your metric's semantic meaning (counters → sum, gauges → avg).
* Removing all labels results in a single aggregated value per metric name.
# Lookup Processor
Source: https://docs.sawmills.ai/docs/processor-lookup
Enrich log records by matching attributes against a CSV file using the Sawmills Lookup Processor. Supports dot notation and EFS-mounted lookup files.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Lookup Processor for Logs
The Sawmills Lookup Processor enriches log records with data from a CSV file by matching a log attribute value against a CSV column and adding all other CSV columns to the log record. This is useful for adding contextual information, such as service metadata or environment details, to your logs.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Attribute Filters
* **Conditions**: Specify conditions to filter events before processing. Events must satisfy all conditions (AND) or at least one (OR) based on the selected logic.
Each condition follows this sequence:
1. **Choose the condition type:**
* **Log Level (Severity)**
* **Body as String**
2. **Select a comparison operator:**
* **Equals** / **Not Equals**
3. **Provide a value:**
* **Log Level:** Select from a dropdown (INFO, WARN, ERROR, etc.).
* **Body as String:** Enter a free-text value for matching.
#### 3. Source Type
* **Shared Path**: Absolute path to a CSV file accessible by the collector. The file must be readable by the collector process.
* **Example**: `/etc/sawmills/lookup.csv`
* **Requirements**: Must be an absolute path and end with `.csv` extension.
* **S3 Remote Path**: URI pointing to a CSV file stored in an S3 bucket. The collector downloads the file at initialization.
* **S3 Path**: Full S3 URI to the CSV file (e.g., `s3://my-bucket/enrichment/lookup.csv`).
* **Region**: Required. Select the AWS region where the S3 bucket is located (e.g., `us-east-1`).
* **Requirements**: The collector's service account must have `s3:GetObject` permission on the target bucket. On EKS, this is typically granted via [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
#### 4. Lookup Key
* **Log Attribute**: Select the log attribute whose value will be used to lookup matching rows in the CSV file. Supports nested attribute paths using dot notation (e.g., `sawmills.service`).
* **CSV Column**: The CSV header name used as the lookup key. Must match the file header exactly. All CSV columns except this lookup column will be added to the log record.
#### 5. Enriched Fields
* **Target Scope**: Where the enriched fields will be written.
* **Resources**: Adds fields to resource attributes
* **Attributes**: Adds fields to log record attributes
* **Body**: Adds fields to the log body. Fields are only written if the body is already a structured object. Plain-text bodies are left unchanged.
* **Prefix**: Optional namespace added in front of all enriched fields. For example, a prefix of `enrichment` will write fields as `enrichment.`. Trailing dots are automatically removed. Slashes (`/`) and backslashes (`\`, except for escaping dots) are not accepted.
#### 6. Field Conflicts
* **On Conflict**: Controls what happens if a target field already exists on the record.
* **Skip existing fields**: Leaves existing fields unchanged and skips enrichment for those fields.
* **Override existing fields**: Replaces existing field values with values from the CSV file.
***
## Dot Notation for Nested Attributes
The Lookup Processor supports dot notation for accessing nested attributes in both lookup keys and enrichment prefixes. This allows you to work with structured data where attributes are organized in nested maps.
### Lookup Key with Dot Notation
When specifying a lookup key with dot notation, the processor traverses nested maps to find the value. For example, if you specify `sawmills.service` as the lookup key:
* The processor looks for `attributes["sawmills"]` (which must be a map).
* Then looks for `attributes["sawmills"]["service"]` to get the value.
* This value is used to match against the CSV lookup column.
**Example**: If a log record has `sawmills.service = "foo"`, the processor will:
1. Access `attributes["sawmills"]` (a map).
2. Read `attributes["sawmills"]["service"]` which equals `"foo"`.
3. Use `"foo"` to lookup matching rows in the CSV file.
### Enrichment Prefix with Dot Notation
When specifying an enrichment prefix with dot notation, the processor creates nested maps to organize the enriched fields. For example, if you specify `enrichment.bar` as the prefix:
* The processor creates `attributes["enrichment"]` (a map) if it doesn't exist
* Then creates `attributes["enrichment"]["bar"]` (a map) if it doesn't exist
* All enriched key-value pairs from the CSV are inserted under `attributes["enrichment"]["bar"]`
**Example**: With prefix `enrichment.bar` and CSV columns `region` and `environment`, the enriched data will be structured as:
* `attributes["enrichment"]["bar"]["region"]` = CSV value for region
* `attributes["enrichment"]["bar"]["environment"]` = CSV value for environment
### Escaping Dots
If you need to reference a flat attribute key that contains a literal dot (not a nested path), escape it with a backslash: `sawmills\.service` will look for `attributes["sawmills.service"]` as a single key rather than traversing nested maps.
***
## Processor Operations
The Lookup Processor operates in the following sequence for each log record:
1. **Condition Evaluation**: If conditions are configured, the processor evaluates them. Logs that don't match the conditions are skipped.
2. **Lookup Key Extraction**: The processor extracts the lookup key value from the specified log attribute (or resource attribute/body) using the configured lookup key path.
3. **CSV Lookup**: The extracted lookup key value is used to search the CSV file's lookup column for a matching row.
4. **Enrichment**: If a match is found, all CSV columns (except the lookup column) are added to the log record at the specified target scope with the optional prefix.
5. **Conflict Handling**: If target fields already exist, the processor applies the configured conflict resolution strategy (skip or overwrite).
The processor processes logs individually and continues processing even if individual logs fail to enrich, ensuring high availability and fault tolerance.
***
## CSV Structure Requirements and Limitations
### Requirements
* **Header Row**: The CSV file must have a header row as the first line containing column names.
* **Lookup Key Column**: One column must match the configured CSV column name exactly (case-sensitive).
* **Column Count Consistency**: All data rows must have the same number of columns as the header row.
* **File Format**: The file must be a valid CSV file with proper formatting.
### Limitations
* **Maximum Columns**: 50 columns per CSV file
* **Maximum Rows**: 5,000 rows per CSV file
* **Duplicate Keys**: If multiple rows have the same lookup key value, the first matching row is used (subsequent duplicates are ignored).
* **Empty Values**: Empty CSV cell values are allowed but are skipped during enrichment (not added to log records).
### CSV File Example
```csv theme={null}
service_name,region,environment,owner
checkout,us-east-1,production,team-a
payments,us-west-2,staging,team-b
auth,eu-central-1,production,team-c
```
In this example:
* `service_name` is the lookup key column.
* `region`, `environment`, and `owner` will be added as enrichment attributes.
* The file has 4 columns and 3 data rows (within limits).
***
## Edge Cases and Processor Behavior
### Lookup Key Not Found in Log
**Scenario**: The log record doesn't contain the specified lookup key attribute.
**Behavior**: The processor skips enrichment for that log record and continues processing. No error is raised, and the log passes through unchanged.
### CSV Lookup Key Not Found
**Scenario**: The lookup key value from the log doesn't match any row in the CSV file.
**Behavior**: The processor skips enrichment for that log record and continues processing. No error is raised, and the log passes through unchanged.
### Empty CSV Values
**Scenario**: A CSV cell contains an empty value.
**Behavior**: Empty values are skipped during enrichment. The attribute is not added to the log record.
### Duplicate Lookup Keys in CSV
**Scenario**: Multiple rows in the CSV file have the same lookup key value.
**Behavior**: The first matching row is used. Subsequent rows with the same key are ignored.
### Body Target with Non-Map Body
**Scenario**: Enrichment target is set to "Body" but the log body is plain text (not a structured object).
**Behavior**: The processor skips enrichment for that log record. Enrichment only works when the body is already a structured map/object.
### Prefix Path Conflicts
**Scenario**: The enrichment prefix path conflicts with an existing non-map value (e.g., `enrichment` exists as a string but prefix is `enrichment.bar`).
**Behavior**: Depends on the conflict resolution setting:
* **Skip**: Processor skips enrichment and logs a debug message
* **Overwrite**: Processor creates a new map at the conflicting path, replacing the existing value
### Condition Evaluation Errors
**Scenario**: An error occurs while evaluating conditions for a log record.
**Behavior**: The processor logs a debug message and skips that log record (does not enrich it).
### Target Attribute Access Errors
**Scenario**: The processor cannot access the specified target (attributes, resource, or body).
**Behavior**: The processor skips enrichment for that log record, logs a debug message, and continues processing other logs.
***
## Shared Path Mount
The Lookup Processor requires access to CSV files via a shared path that is accessible by all collector pod instances. The shared path must be an absolute path on the filesystem where the collector is running.
### Path Requirements
* **Absolute Path**: Must start with `/` (e.g., `/mnt/efs/lookup.csv`).
* **File Extension**: Must end with `.csv` extension.
* **Readable**: The collector process must have read permissions for the file.
* **Persistent**: The file should be available across collector pod restarts.
The CSV file is loaded once during processor initialization. If the file changes, the collector must be restarted to reload the updated CSV data.
For setting up the shared path mount using EFS, see the [EFS Volume Setup](#efs-volume-setup-for-sawmills-collector) section below.
***
***
## EFS Volume Setup for Sawmills Collector
This guide explains how to configure EFS (Elastic File System) volume mounts for the Sawmills Collector using the remote-operator Helm chart.
### Overview
The Lookup Processor requires access to lookup files stored on an EFS volume. This setup mounts the EFS filesystem to `/mnt/efs` in the collector pods.
### Prerequisites
Before installing the Helm chart with EFS support, you must create the StorageClass and PersistentVolumeClaim in your cluster.
#### 1. Create the StorageClass
```bash theme={null}
kubectl apply -f - < **Note:** Replace `fileSystemId` with your actual EFS filesystem ID.
#### 2. Create the PersistentVolumeClaim
```bash theme={null}
kubectl apply -f - <
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-rwx-enrichment-sawmills
resources:
requests:
storage: 500Mi
EOF
```
#### 3. Verify Resources
```bash theme={null}
## Check StorageClass
kubectl get storageclass efs-rwx-enrichment-sawmills
# Check PVC
kubectl get pvc enrichment-files -n
```
### Helm Installation
#### Option A: Using `--set` flags
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version 2.0.18 \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress=https://controller.ue1.prod.plat.sm-svc.com \
--set collectorName="my-colllector" \
--set image.tag="0.189.0" \
--set remoteWriteUrl="https://ingress.sawmills.ai" \
--set selfManaged=false \
--set 'managedChartsValues.sawmills-collector.additionalVolumes[0].name=enrichment-files' \
--set 'managedChartsValues.sawmills-collector.additionalVolumes[0].persistentVolumeClaim.claimName=enrichment-files' \
--set 'managedChartsValues.sawmills-collector.additionalVolumeMounts[0].name=enrichment-files' \
--set 'managedChartsValues.sawmills-collector.additionalVolumeMounts[0].mountPath=/mnt/efs'
```
#### Option B: Using a Values File (Recommended)
Create a values file `values-efs.yaml`:
```yaml theme={null}
# values-efs.yaml
managedChartsValues:
sawmills-collector:
additionalVolumes:
- name: enrichment-files
persistentVolumeClaim:
claimName: enrichment-files
additionalVolumeMounts:
- name: enrichment-files
mountPath: /mnt/efs
readOnly: true # Optional: set to true if files should be read-only
```
Then run:
```bash theme={null}
helm upgrade --install sawmills-remote-operator \
oci://public.ecr.aws/s7a5m1b4/sawmills-remote-operator-chart \
--version 2.0.18 \
--namespace \
--set apiKeyExistingSecret=sawmills-secret \
--set operatorAddress=https://controller.ue1.prod.plat.sm-svc.com \
--set collectorName="my-colllector" \
--set image.tag="0.189.0" \
--set remoteWriteUrl="https://ingress.sawmills.ai" \
--set selfManaged=false \
-f values-efs.yaml
```
### Verification
After deployment, verify the EFS mount is working:
```bash theme={null}
# Check if pods are running
kubectl get pods -n -l app.kubernetes.io/name=sawmills-collector
# Verify the volume mount in a pod
kubectl exec -it deployment/sawmills-collector -n -c main-collector -- ls -la /mnt/efs
# Check pod describe for volume mounts
kubectl describe pod -n -l app.kubernetes.io/name=sawmills-collector | grep -A5 "Mounts:"
```
### Lookup Processor Configuration
Once the EFS volume is mounted, configure the `csvenrichmentprocessor` in your collector config to use the mounted files:
```yaml theme={null}
processors:
csvenrichmentprocessor:
source:
file:
path: /mnt/efs/enrichment-files-service.csv
lookup:
key: sawmills.service # Supports nested paths (e.g., attributes["sawmills"]["service"])
target: attributes
csv_key_column: service_name
enrichment:
target: attributes
prefix: "" # Optional prefix for enriched attributes
on_conflict: overwrite # or "skip"
```
### Troubleshooting
#### PVC Not Bound
```bash theme={null}
kubectl describe pvc enrichment-files -n
```
Check that:
* The StorageClass exists
* The EFS CSI driver is installed
* The fileSystemId is correct
#### Mount Permission Denied
Ensure the EFS access point has correct permissions:
* `directoryPerms: "0777"` in StorageClass
* Security groups allow NFS traffic (port 2049)
#### File Not Found in Pod
```bash theme={null}
# List files in the EFS mount
kubectl exec -it deployment/sawmills-collector -n -c main-collector -- ls -la /mnt/efs
# Check if the file exists
kubectl exec -it deployment/sawmills-collector -n -c main-collector -- cat /mnt/efs/enrichment-files-service.csv | head -5
```
### Architecture
```
+-------------------------------------------------------------+
| Kubernetes Cluster |
| |
| +------------------+ +------------------------------+ |
| | StorageClass | | EFS (AWS) | |
| | efs-rwx-... |---->| fs-062ce7e15a1f260c0 | |
| +------------------+ | /k8s/sawmills-production- | |
| | | enrichment/ | |
| | +------------------------------+ |
| | | |
| v | |
| +------------------+ | |
| | PVC | | |
| | enrichment-files | | |
| +------------------+ | |
| | | |
| | | |
| v v |
| +-------------------------------------------------------+ |
| | Sawmills Collector Pod | |
| | +--------------------------------------------------+ | |
| | | main-collector container | | |
| | | | | |
| | | /mnt/efs/ <--- EFS Volume Mount | | |
| | | +-- enrichment-files-service.csv | | |
| | | | | |
| | | csvenrichmentprocessor reads from: | | |
| | | /mnt/efs/enrichment-files-service.csv | | |
| | +--------------------------------------------------+ | |
| +-------------------------------------------------------+ |
| |
+-------------------------------------------------------------+
```
# Processor Management
Source: https://docs.sawmills.ai/docs/processor-management
View, create, edit, delete, and monitor all processors across your Sawmills organization from the centralized Processor Management interface.
## Processor Management
Sawmills provides comprehensive processor management capabilities to help you efficiently organize, monitor, and maintain your processors.
### Overview
Processors are the core components that transform, filter, and route your telemetry data. The Processor Management interface allows you to:
* View all processors across your organization
* Create, edit, and delete processors
* Turn on or off processors
* Monitor processor performance and health
### Accessing Processor Management
1. Navigate to the **Processors** section in the main menu
2. Click on **Processor Management** to access the management dashboard
### Processor Dashboard
The Processor Management dashboard provides:
#### Processor List View
* **Assigned Destination & Pipeline**: Destination and pipelines using this processor
* **Status**: Current operational status (Active, Inactive)
* **Name**: Processor identifier and display name
* **Action**: The processor type (Filter, Aggregation, Mask, etc.)
* **Filter**: Filter rules for the processor
* **Filterd Events**: Number of events filtered by the processor
* **Owner**: Owner of the processor
* **Last Updated**: Timestamp of the most recent changes
#### Management Actions
* **Edit**: Modify processor configuration
* **Duplicate**: Create a copy of the processor
* **Activity Log**: View the activity log for the processor
* **Delete**: Remove the processor (with confirmation)
### Best Practices
* **Naming Convention**: Use descriptive, consistent names
* **Documentation**: Document processor purpose and configuration
* **Testing**: Test processors in development before production
# Mask Sensitive Data/PII Processor
Source: https://docs.sawmills.ai/docs/processor-mask
Identify and mask sensitive data like credit cards, SSNs, emails, and IP addresses in telemetry streams using the Sawmills Mask Processor with regex patterns.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Mask Processor
The Sawmills Mask Processor enables you to identify and mask sensitive data within telemetry streams. This ensures data privacy and compliance with regulations.
### Mask Processor Features
#### Predefined Sensitive Data Types
The processor supports predefined sensitive data types. Users can select the following fields using checkboxes in the configuration interface:
**Note:** This processor searches across `Attributes`, `Resources`, and `Body`
#### Fields (Optional)
Select which predefined fields you want to mask within your data. These filters cover the most common types of personal information.
* **Credit Card Number**
* **Date of Birth (DOB)**
* **Email Addresse**
* **IPv4 Addresse**
* **IPv6 Addresse**
* **MAC Addresses**
* **Phone Number**
* **Social Security Number (SSN)**
#### Custom Rules (Optional)
Users can define additional filtering options in the **Custom Rules** section. This section allows users to create their own patterns using:
* **Contains**: Simple string patterns.
* **Matches Regex**: Complex patterns for specific formats not covered by the default filters.
### Example Custom Rules
Below are example regular expressions that can be used in the **Custom Rules** section:
* **Phone Numbers**:
* International: `(?:\+?\d{1,4}[\s-]?)?(?:\(?\d{1,4}\)?[\s-]?)?\d{1,4}[\s-]?\d{1,4}(?:[\s-]?\d{1,9})?`
* **Other Personal Numbers (non SSN)**:
* Canada (SIN): `\b\d{3}-\d{3}-\d{3}\b`
* France (INSEE): `\b\d{1}\s?\d{2}\s?\d{2}\s?\d{2}\s?\d{3}\s?\d{3}\b`
* Sweden (Personnummer): `\b(?:\d{6}|\d{8})-\d{4}\b`
* South Korea (Resident Registration Number): `\b\d{6}-\d{7}\b`
* Finland (Personal Identity Code): `\b\d{6}[-+A]\d{3}[0-9A-Za-z]\b`
* Germany (Tax ID): `\b\d{2}\s?\d{3}\s?\d{3}\s?\d{3}\b`
* **Date of Birth (DOB)**:
* `YYYY-MM-DD`: `\b(19|20)\d{2}[-/](0[1-9]|1[0-2])[-/](0[1-9]|[12]\d|3[01])\b`
* `DD/MM/YYYY or MM/DD/YYYY`: `\b(0[1-9]|[12]\d|3[01])[-/](0[1-9]|1[0-2])[-/](19|20)\d{2}\b`
* **Credit Card Numbers (Extended)**:
* (Visa, Mastercard, American Express, Diners Club): `\b(?:4\d{3}|5[1-5]\d{2}|6(?:011|5\d{2})|3[47]\d{2}|3(?:0[0-5]|[68]\d)|2(?:2[2-9]|[3-7]\d)\d{2})[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{1,4}\b`
***
### Implementation Notes
* Regular expressions provided above are pre-tested but should be verified for edge cases.
* Masking should occur prior to data storage or transmission to ensure compliance.
* Ensure performance and accuracy by limiting custom regex patterns to those absolutely necessary.
# Modify Attribute Processor
Source: https://docs.sawmills.ai/docs/processor-modify-attribute
Update, transform, standardize, enrich, or truncate log attribute values using the Sawmills Modify Attribute Processor with flexible action types.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Modify Attribute Processor for Logs
The Sawmills Modify Attribute Processor enables you to update or transform log attributes based on specified rules. This is useful for standardizing log data, enriching it with additional context, or correcting erroneous values.
### Main Use Cases
* **Standardize fields**: Normalize attribute names and values to a consistent schema.
* **Enrich logs**: Add derived or static context like environment, service, or region.
* **Fix or redact values**: Correct malformed values or replace sensitive data.
* **Optimize payloads**: Truncate large string fields to reduce size.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Attribute Filters
* **Conditions**: Specify conditions to filter events before processing. Events must satisfy all conditions (AND) or at least one (OR) based on the selected logic.
Each condition follows this sequence:
1. **Choose the condition type:**
* **Log Level (Severity)**
* **Body as String**
2. **Select a comparison operator:**
* **Log Level (Severity):** **Equals** / **Not Equals**
* **Body as String:** **Equals**, **Not Equals**, **Contains**, **Not Contains**, **Starts With**, **Ends With**, **Regex**, **Not Regex**
3. **Provide a value:**
* **Log Level:** Select from a dropdown (INFO, WARN, ERROR, etc.).
* **Body as String:** Enter a free-text value for matching.
#### 3. Actions
* **Description**: A modify attribute processor can run one or more actions in order. Each action targets a specific path and applies a transformation.
* **Options**:
* **Insert**: Adds a new attribute with a specified value.
* **Update**: Modifies the value of an existing attribute.
* **Upsert**: Adds a new attribute if it doesn’t exist or updates the value of an attribute if it does.
* **Truncate**: Trims string values to a maximum length.
#### 4. Action Details
* **Insert / Update / Upsert**
* **Attribute Key**: Path to the attribute to modify. Must start with **Attributes.**, **Body.**, or **Resources.**
* **Value Source**: Defines where the new value comes from:
* **Static Value**: Enter a literal value.
* **From Attribute**: Copy the value from another attribute path (must start with **Attributes.**, **Body.**, or **Resources.**).
* **Truncate**
* **Attributes**: One or more attribute paths to truncate. Paths must start with **Attributes.**, **Body.**, or **Resources.**
* **BodyString** is also supported to truncate the raw body as string.
* You can use a wildcard for all immediate children, e.g. **Attributes.\*** or **Resources.\*** (wildcard is not recursive).
* **Max Length**: Required. Must be a positive integer. Truncation applies only to string fields.
For **Insert / Update / Upsert**, depending on the **Value Source**, an attribute's value will be set either by:
* A static value defined in the field **Attribute Value** or
* An existing attibute value from an existing attribute with the key defined in **Attribute Key**.
# Parse Log Processor
Source: https://docs.sawmills.ai/docs/processor-parse-logs
Extract structured fields from unstructured log messages using the Sawmills Parse Log Processor with regex patterns and named capture groups.
## Supported Data Types
📘 **Logs**
### Configuring the Sawmills Parse Log Processor
The Sawmills Parse Log Processor enables you to extracts structured data from unstructured log messages using regex patterns with named capture groups. This enables you to parse your logs into more readable and queryable fields.
### Configuration Components
#### 1. Name
* **Description**: Identifier for your processor. Use a unique and descriptive name to differentiate between multiple processors.
#### 2. Attribute Filters
* **Conditions**: Specify conditions to filter events before processing. Events must satisfy all conditions (AND) or at least one (OR) based on the selected logic.
Each condition follows this sequence:
1. **Choose the condition type:**
* **Log Level (Severity)**
* **Body as String**
2. **Select a comparison operator:**
* **Equals** / **Not Equals**
3. **Provide a value:**
* **Log Level:** Select from a dropdown (INFO, WARN, ERROR, etc.).
* **Body as String:** Enter a free-text value for matching.
#### 3. Source Field
* **Description**: Defines which field to parse the data from.
#### 4. Extraction Pattern
* **Regex**: Use `(?P...)` syntax to define a regex with at least one named capture group. The name determines the new attribute name.
* **Grok**: Use `${PATTERN:name}` syntax to define fields. The name after the colon determines the new attribute name.
Select a log line in Live Tail and use Generate Regex/Grok Pattern to auto-generate this pattern.
If the extracted named group matches an existing key, the existing value will be replaced by the parsed value.
#### 5. Target Location
* **Description**: Specifies where to store the extracted data.
* **Attributes**: Stores extracted fields in Attributes.
* **Body**: Stores extracted fields in log body (creates or updates map structure)
Extracting to Resource Attributes is not supported.
#### 6. Target Key (optional)
* **Description**: Specify a key to store all extracted fields as a nested map under that key
* If **omitted**, extracted fields are merged directly into the target location.
* If **provided**, extracted fields become a nested map under the given key.
### Insertion Behavior
* **Root Level (No Target Key):** Extracted fields will be upserted into current attributes map or body map if body is structured.
* **Sub Level (Target Key Specified):** Extracted fields are stored under the specified key as a nested map.
If the body is unstructured and the target is the body (with or without target key), the extracted fields will replace the body entirely.
# Processor Reordering
Source: https://docs.sawmills.ai/docs/processor-reordering
Change the execution order of processors within a Sawmills pipeline using drag-and-drop or move buttons to optimize cost, latency, and data quality.
## Processor Reordering Feature Documentation
## Overview
Normally when a processor is created, it gets added to the end of the execution order. The Processor Reordering feature allows users to change the execution order of processors within a destination pipeline. Processors are executed sequentially in the order they appear, so reordering them can significantly impact data processing behavior.
> ℹ️ **Why Reordering Matters**\
> Reordering lets you optimize cost and/or latency without redeploying collectors. Changes both persist immediately and propagate to live traffic.
### Key Concepts
* **Processors** execute in sequential order (top to bottom)
* **Robust Algorithm** Only changes processors you've moved, maintains stable ordering
* **Real-time Updates** changes apply immediately to live data
* **Audit Logging** tracks all reordering operations with before/after states
***
## User Guide
### How to Reorder Processors (UI)
#### 1. Open the Reorder Drawer
* Navigate to the Processors list
* Find the destination you want to reorder processors for
* Click the "Reorder" button for that destination **or** you can click Reorder Processor from a processor's Action menu
* A drawer will slide in from the right showing all processors for that destination
#### 2. Reorder Using Drag & Drop
* **Drag Handle**: Click and hold the ⋮⋮ icon on the left of any processor
* **Drag**: Move the processor up or down to the desired position. The page will scroll automatically if you drag near the top or bottom of the list.
* **Drop**: Release to place it in the new position
#### 3. Reorder Using Move Buttons
* **Move to Top** ⇈ - Jump to first position
* **Move Up** ⇑ - Move up one position
* **Move Down** ⇓ - Move down one position
* **Move to Bottom** ⇊ - Jump to last position
* **Visual Feedback**: The row will pulse/highlight when successfully moved
#### 4. Save Changes
* **Cancel** button - Discards all changes and closes drawer
* **Save** button - Opens confirmation modal
* **Confirm** - Applies changes immediately to live data
* Changes are visible immediately across the platform
***
## Activity Log
Every reorder triggers an activity log entry you can review in the Audit Log UI.
What you’ll see:
* **Before order:** full list of processors in their original positions.
* **After order:** the updated list immediately after you save.
* **Moves summary:** only the processors that changed position, showing the old and new index.
* **Metadata:** pipeline, destination, who made the change, and when it happened.
# Sawmills Processors Types
Source: https://docs.sawmills.ai/docs/processors
Browse all available Sawmills processor types including Aggregation, Filter, Mask, Drop Attribute, Lookup, Parse Log, Logs to Metrics, and more.
| | |
| - | - |
| [Aggregation](/docs/processor-aggregation) | Aggregation Processor Configuration |
| [Filter](/docs/processor-filter) | Filter Processor Configuration |
| [Mask](/docs/processor-mask) | Mask Processor Configuration |
| [Drop Attribute](/docs/processor-drop-attribute) | Drop Attribute Processor Configuration |
| [Drop Metric Datapoints](/docs/processor-drop-metric-datapoints) | Drop Metric Datapoints Processor Configuration |
| [Aggregate Metric Labels](/docs/processor-label-aggregation) | Aggregate Metric Labels Processor Configuration |
| [Modify Attribute](/docs/processor-modify-attribute) | Modify Attribute Processor Configuration |
| [Parse Log](/docs/processor-parse-logs) | Parse Log Processor Configuration |
| [Logs to Metrics](/docs/processor-logstometrics) | Logs To Metrics Processor Configuration |
| [Lookup](/docs/processor-lookup) | Lookup Processor Configuration |
# SAML
Source: https://docs.sawmills.ai/docs/saml
Enable enterprise single sign-on for Sawmills using the SAML protocol. Supports both SP-initiated and IdP-initiated authentication with your identity provider.
Sawmills.ai offers Enterprise Single Sign-On (SSO) through the SAML protocol, allowing seamless integration with your Identity Provider (IdP). This enables users to sign in effortlessly using their IdP credentials, with their user data automatically synchronized with Sawmills.ai.
Sawmills.ai supports two methods for initiating a Single Sign-On (SSO) flow via the SAML protocol: Service Provider (SP)-initiated and Identity Provider (IdP)-initiated. This guide explains the differences between the two.
**SP-Initiated Flow (Recommended)**
* User Begins Authentication on Sawmills.ai (SP): The user starts the authentication process from Sawmills.ai by providing their email address.
* Redirection to Identity Provider (IdP): The user is redirected to your SAML Identity Provider to authenticate.
Access Granted Upon Successful Authentication: After successful authentication, the user is redirected back to Sawmills.ai and gains access to their account.
**IdP-Initiated Flow**
* User Initiates Authentication on IdP: The user starts the authentication process from the SAML Identity Provider by selecting Sawmills.ai as the application they wish to access.
* Redirection to Sawmills.ai: The user is redirected to Sawmills.ai and gains access to their account upon arrival.
# Collaboration
Source: https://docs.sawmills.ai/docs/sawmills-agent-collaboration
Share Sawmills Agent conversations with teammates for incident handoffs, collaborative investigations, and asynchronous follow-up on telemetry issues.
## Overview
Sawmills Agent supports collaborative work by allowing conversations to be shared and continued with teammates. This is useful for handoffs, incident follow-up, and team reviews.
## Shared Conversations
A shared conversation keeps the same message history, context, and results in one place. This allows teammates to review what has already been asked and continue the conversation without starting over.
## When to Share a Conversation
Share a conversation when:
* you want another teammate to review the findings
* you are handing off an investigation
* the same issue needs input from multiple people
* you want to keep discussion and results attached to one investigation
## Typical Collaboration Flow
1. Start an investigation in a conversation.
2. Share the conversation with the relevant teammate or team.
3. Continue follow-up questions in the same conversation.
4. Use the shared history as the record of the investigation.
## Best Practices
* keep one topic per shared conversation
* use clear follow-up questions so new readers can catch up quickly
* ask Sawmills Agent for a summary before handing work to someone else
* continue the same conversation during a handoff instead of creating a duplicate conversation
## Notes
Conversation sharing and access depend on your workspace configuration.
# Conversations
Source: https://docs.sawmills.ai/docs/sawmills-agent-conversations
Start and continue Sawmills Agent conversations to investigate telemetry issues, ask follow-up questions, and keep related context in one thread.
## Overview
Conversations are the main way to work with Sawmills Agent. A conversation keeps your question, follow-up messages, files, and results together so you can continue the same investigation over time.
## What You Can Do in a Conversation
Use a conversation to:
* ask a new question
* continue a previous investigation
* attach files or other supporting context
* request a summary or next step
* keep related work in a single history
## Conversation Flow
A typical conversation looks like this:
1. Start with a question or problem statement.
2. Review the response from Sawmills Agent.
3. Add follow-up questions to narrow the answer or request a summary.
4. Continue the same conversation until you reach a useful outcome.
In some cases, Sawmills Agent may ask a follow-up question before returning a final answer.
## Attachments
Conversations can include file attachments. Use attachments when the investigation depends on content that is easier to share as a file than as a message.
Examples include:
* logs or samples
* screenshots
* supporting documents
When you attach a file, continue the same conversation so the follow-up questions stay connected to that context.
## Conversation Titles
Conversations can have titles to make them easier to find later. Titles are useful when you return to an investigation or share it with a teammate.
## When to Start a New Conversation
Start a new conversation when:
* the topic has changed significantly
* you are switching to a different service or incident
* the previous context is no longer useful
If the topic is still related, continue the same conversation instead of starting over.
## Best Practices
* start with a clear question
* include the service, timeframe, or environment early
* keep related follow-up questions together
* ask for a short summary when you are ready to share results
# Instrumentation Reviewer
Source: https://docs.sawmills.ai/docs/sawmills-agent-instrumentation-reviewer
Automatically review telemetry-related code changes in GitHub pull requests and GitLab merge requests to catch observability issues before production.
## Overview
Instrumentation Reviewer reviews telemetry-related code changes in pull requests and merge requests before they are merged.
It is built to catch instrumentation problems early, while the change is still in code review, instead of after the change has already affected dashboards, alerts, traces, logs, or telemetry cost.
## What It Reviews
Instrumentation Reviewer focuses on code changes that affect logs, metrics, traces, structured fields, telemetry routing, and related observability behavior.
It is designed to help teams catch issues such as:
* high-cardinality fields or labels
* PII exposure in telemetry
* metric correctness problems
* spammy or overly noisy logs
* incorrect severity levels
* wrong telemetry pillar selection
* telemetry regressions caused by removed or degraded signal quality
* telemetry schema consistency problems
## Why Teams Use It
Instrumentation quality often drifts over time, especially when many teams ship quickly across many services.
Common outcomes of that drift include:
* inconsistent instrumentation patterns between services
* dashboards and alerts that become harder to trust
* noisy or low-value telemetry
* hidden operational cleanup work for platform, DevOps, and SRE teams
* unnecessary telemetry growth and less predictable observability cost
Instrumentation Reviewer helps move those checks earlier into the development workflow.
## How It Works
Instrumentation Reviewer runs as an SCM-driven review workflow.
At a high level it:
1. Detects a relevant pull request or merge request event.
2. Fetches the PR or MR metadata, changed files, and repository context.
3. Clones the repository so it can inspect the changed code in context.
4. Runs objective-based telemetry review logic over the change.
5. Synthesizes the results into a customer-safe review.
6. Publishes the output back to GitHub or GitLab.
The reviewer is designed around real code review context, not isolated snippets. It uses both the diff and surrounding repository code to understand the instrumentation change.
## Review Workflow
Instrumentation Reviewer currently uses a staged workflow:
* **Gather context**: fetch PR or MR metadata, changed files, comments, and repository checkout
* **Objective execution**: run telemetry-focused review objectives against the change
* **Synthesis**: merge, deduplicate, and normalize findings
* **Postprocess and publish**: convert findings into SCM-ready review output
This lets the reviewer keep internal analysis artifacts separate from the final customer-facing review.
## GitHub Behavior
On GitHub, Instrumentation Reviewer is driven by the GitHub App integration.
It supports:
* automatic review runs on pull request events
* review publishing as PR reviews and inline review comments
* review-comment replies for follow-up interactions
* GitHub Checks integration for review lifecycle status
* rerun from the Checks UI
* comment-driven interactions such as `@ cancel`
GitHub setup and permissions are documented in [Enable GitHub Integration](/docs/github-integration).
## GitLab Behavior
On GitLab, Instrumentation Reviewer is driven by a dedicated GitLab integration and webhook per configured scope.
It supports:
* automatic review runs on merge request events
* inline merge request discussions when diff position data is available
* summary merge request notes as the review-level output
* note-driven commands such as `/sawmills review`
* cancellation commands such as `/sawmills cancel`
GitLab setup and access requirements are documented in [Enable GitLab Integration](/docs/gitlab-integration).
## What Gets Published Back
Instrumentation Reviewer publishes the result back to the source control system rather than only storing it internally.
Depending on provider and event, that can include:
* a review summary
* inline comments or inline discussions on changed code
* follow-up replies in existing review threads
* GitHub check-run status and annotations
This keeps the telemetry feedback in the same review workflow where engineers are already making code decisions.
## Code Findings in Sawmills Agent
The Sawmills Agent includes a **Code Findings** view for Instrumentation Reviewer analytics.
Use this view to understand review activity across connected GitHub and GitLab repositories without leaving Sawmills. It summarizes recent reviewer output and links findings back to the affected pull requests.
To open it:
1. Open **Agent** in Sawmills.
2. Select the **Code Findings** tab.
The Code Findings view shows:
* **PRs Reviewed**: pull requests reviewed during the selected time range
* **PRs With Findings**: reviewed pull requests where the reviewer found active issues
* **Resolved Findings**: findings that were resolved across reviewed pull requests
* **Finding Severity**: active finding counts grouped by severity
* **Findings by Type**: finding categories such as high cardinality, metric correctness, sensitive data or PII, spammy logs, telemetry regression, and wrong severity
You can change the time range and filter by finding type. The details table can be viewed by pull request or by repository.
In **Pull Requests** view, the table includes the PR number, repository, PR status, active findings, resolved findings, and finding types. PR numbers link back to the source control provider when a PR URL is available.
In **Repositories** view, the table groups findings by repository and shows reviewed PR counts, active findings, resolved findings, and finding types.
If no GitHub or GitLab integration is connected, Code Findings shows a setup prompt. If no issues were found in the selected time range, it shows an empty state instead of a findings table.
## Trigger Model
Instrumentation Reviewer can run both automatically and interactively.
Examples include:
* pull request open, reopen, synchronize, and ready-for-review events on GitHub
* merge request open, reopen, and update events on GitLab
* GitHub comment-driven reruns or cancellation
* GitHub Checks rerun actions
* GitLab note-driven review or cancellation commands
In beta, trigger behavior and exact command phrasing may continue to evolve.
## Access and Integrations
Instrumentation Reviewer depends on source control integrations because it needs to read code changes and publish the results back to the review surface.
To use it, your workspace needs the relevant SCM integration:
* [Enable GitHub Integration](/docs/github-integration)
* [Enable GitLab Integration](/docs/gitlab-integration)
Those integrations provide the reviewer with:
* access to PR or MR metadata
* access to changed files and repository context
* access to publish review results back to the SCM
* webhook-driven review triggering
## Best Fit Use Cases
Instrumentation Reviewer is especially useful when teams want to:
* catch telemetry problems before merge
* enforce instrumentation standards across many repositories
* reduce telemetry noise and cardinality growth
* improve consistency across logs, metrics, and traces
* keep telemetry review inside the normal PR or MR workflow
* give platform and observability teams stronger guardrails without creating a central bottleneck
# Memory
Source: https://docs.sawmills.ai/docs/sawmills-agent-memory
Learn how Sawmills Agent retains conversation context to support follow-up questions, longer investigations, and continuity across related queries.
## Overview
Memory in Sawmills Agent means conversation context. It allows the agent to use earlier messages in the same conversation so you do not need to repeat the full situation in every prompt.
## What Memory Helps With
Memory is useful when you want to:
* ask follow-up questions
* narrow down a previous answer
* continue a longer investigation
* refer back to earlier findings or attached files
## What Carries Forward in a Conversation
In a conversation, Sawmills Agent can use context such as:
* previous user messages
* previous agent responses
* files and supporting material shared in that conversation
* earlier conclusions reached in the same investigation
## Best Way to Use Memory
To get the most value from memory:
* keep related questions in the same conversation
* restate the scope only when it changes
* start a new conversation when you change topics entirely
## Limits
Memory should not be treated as a replacement for a clear prompt. If a request is too broad or the context changes significantly, provide a short restatement of what you need.
If an answer looks off, add more detail in the same conversation before starting over.
# Notifications
Source: https://docs.sawmills.ai/docs/sawmills-agent-notifications
Deliver Sawmills Agent results and task outputs to users or teams via supported notification channels including Slack, email, and webhook integrations.
## Overview
Notifications allow Sawmills Agent results to be sent to the right person or team instead of staying only in the original conversation.
Notifications are especially useful for scheduled work, summaries, and updates that need to reach other people.
## Common Notification Uses
Use notifications when you want to:
* send a result to yourself
* share a summary with a team
* deliver the output of a task
* post an update into Slack
## Supported Channels
Supported notification channels depend on your workspace configuration. Common examples include:
* Slack direct messages
* Slack channels
* email summaries
## Notifications with Tasks
Tasks and notifications are often used together. For example, a recurring task can generate a regular update and deliver it to the appropriate person or team.
## Best Practices
* send only the information people need
* use short summaries for recurring updates
* choose the destination that matches the audience
* review the first few runs before relying on an automated workflow
## Notes
Notification channels and destinations depend on your workspace configuration and enabled integrations.
# Sawmills Agent
Source: https://docs.sawmills.ai/docs/sawmills-agent-overview
Sawmills Agent is a conversational AI interface for investigating telemetry, automating recurring work, and collaborating with your team.
## Overview
Sawmills Agent provides a conversational way to work with telemetry in Sawmills. Use it to investigate telemetry, automate recurring work, and share results with your team.
Sawmills Agent can be available in the Sawmills product and, when enabled for your workspace, in Slack.
## Main Capabilities
### 1. Conversations
Use Sawmills Agent to ask questions, continue an investigation over time, and keep related work in a single conversation. Conversations can include follow-up questions, files, and titles.
See [Conversations](/docs/sawmills-agent-conversations).
### 2. Collaboration
You can share a conversation, continue it with teammates, and use the same conversation history during handoffs or incident work.
See [Collaboration](/docs/sawmills-agent-collaboration).
### 3. Memory
Sawmills Agent uses conversation context so you do not need to restate everything in every message. This is useful for follow-up questions and longer investigations.
See [Memory](/docs/sawmills-agent-memory).
### 4. Skills
Skills give Sawmills Agent a predefined way to handle common tasks. They help make repeated workflows more consistent across your workspace.
See [Skills](/docs/sawmills-agent-skills).
### 5. Tasks
Tasks let you run agent work in the background. Use them for one-time, manual, or recurring work such as regular summaries and scheduled checks.
See [Tasks](/docs/sawmills-agent-tasks).
### 6. Notifications
Tasks and workflows can send updates through supported notification channels so results can reach the right person or team.
See [Notifications](/docs/sawmills-agent-notifications).
### 7. Roles and Access
What Sawmills Agent can do depends on your role and workspace configuration. In many workspaces, members can use the agent for investigations while administrators can also manage workspace-level behavior.
See [Roles and Access](/docs/sawmills-agent-roles).
### 8. Slack
When Slack is enabled, Sawmills Agent can be used in direct messages, channel mentions, and threads.
See [Slack](/docs/sawmills-agent-slack).
### 9. Instrumentation Reviewer
Instrumentation Reviewer helps teams improve telemetry quality across logs, metrics, and traces. It can review instrumentation for consistency, reduce noisy or low-value telemetry, support shared standards across services, and help control observability costs before issues spread across production systems.
The Agent includes a **Code Findings** tab where teams can review Instrumentation Reviewer analytics across connected GitHub and GitLab repositories.
See [Instrumentation Reviewer](/docs/sawmills-agent-instrumentation-reviewer).
This capability is currently in closed beta.
## Best Use Cases
Sawmills Agent works best when you want to:
* investigate unusual telemetry behavior
* summarize findings for teammates
* automate recurring analysis
* share results in Sawmills or Slack
## Getting Started
If you are new to the feature, start with [Conversations](/docs/sawmills-agent-conversations) and [Slack](/docs/sawmills-agent-slack).
# Roles and Access
Source: https://docs.sawmills.ai/docs/sawmills-agent-roles
Learn how Member and Administrator roles control which Sawmills Agent capabilities are available, including skill creation and task management permissions.
## Overview
What Sawmills Agent can do depends on your role and your workspace configuration. In many workspaces, members can use Sawmills Agent for day-to-day investigations while administrators can also manage workspace-level behavior.
## Common Access Pattern
### Members
Members can commonly:
* start and continue conversations
* use available shared conversations
* ask follow-up questions
* use the agent in Slack when enabled
* use skills and workflows that are already available to them
### Administrators
Administrators can commonly do everything members can do, and may also be able to:
* manage custom skills
* manage task automation
* manage notification behavior
* configure other workspace-level agent capabilities
## Read and Write Actions
Some Sawmills Agent actions only read information. Other actions create, update, or send something. Workspaces often allow broader access to read actions and more limited access to write actions.
## Notes
Exact permissions depend on how your Sawmills workspace is configured.
# Skills
Source: https://docs.sawmills.ai/docs/sawmills-agent-skills
Define and use Skills to give Sawmills Agent predefined, repeatable workflows for recurring investigations, standard summaries, and common tasks.
## Overview
Skills give Sawmills Agent a predefined way to handle common tasks. A skill can guide how the agent approaches a request, which steps it follows, and how it structures the result.
## Why Skills Matter
Skills help make repeated work more consistent across a workspace. They are useful when a team wants the agent to follow the same process for similar requests.
Examples include:
* recurring investigation patterns
* standard summaries
* team-specific ways of working
## Using Skills
When a skill applies to a request, Sawmills Agent can use it to follow a more consistent workflow than a general conversation alone.
From a user point of view, this means:
* more predictable results for common tasks
* fewer repeated instructions
* easier reuse of proven workflows
## Managing Skills
In workspaces where custom skills are enabled, administrators can manage skills for their organization.
Common management actions include:
* creating a new skill
* updating an existing skill
* reviewing previous versions
* rolling back to an earlier version
## Notes
Skill availability and management access depend on your workspace configuration.
# Use Sawmills Agent in Slack
Source: https://docs.sawmills.ai/docs/sawmills-agent-slack
Interact with Sawmills Agent directly in Slack using direct messages, channel mentions, and threaded conversations for telemetry investigations.
## Overview
Sawmills Agent in Slack lets you investigate telemetry, continue existing conversations, and share results with teammates without leaving Slack.
You can use Sawmills Agent in a direct message, by mentioning it in a channel, or inside a thread.
## Ways to Use Sawmills Agent in Slack
### Direct messages
Use a direct message when you want to explore an issue privately, draft a summary, or narrow down a problem before sharing it with others.
### Channel mentions
Use a channel mention when you want other people to follow the conversation as it happens.
### Threads
Use a thread to keep one investigation in one place. This is the best option for incident work, follow-up questions, and team handoffs.
## Best Practices
To get better results in Slack:
* keep one topic per thread
* include the service, timeframe, or environment early
* ask follow-up questions in the same conversation
* ask for a short summary when you are ready to share findings
Example prompts:
* "Why did ingest volume spike after the deploy?"
* "Summarize the biggest telemetry changes for checkout today."
* "Write a short update for this alert thread."
## Conversation Context in Slack
Sawmills Agent uses the current Slack conversation as context. This means follow-up questions work best when they stay in the same direct message or thread.
If you start a new thread or move to a different channel, restate the main context so the investigation stays focused.
## When Slack Works Best
Slack is especially useful for:
* incident follow-up
* sharing investigation progress
* drafting a team update
* continuing a conversation with teammates in the same thread
## Notes
Slack availability depends on your workspace configuration.
If Sawmills Agent cannot answer or perform a requested action, add more context or continue the conversation in the same thread before starting over.
# Tasks
Source: https://docs.sawmills.ai/docs/sawmills-agent-tasks
Create and manage Sawmills Agent tasks that run in the background on demand, on a schedule, or at a recurring interval for automated telemetry workflows.
## Overview
Tasks let Sawmills Agent run work in the background. Use tasks when a request should run later, run regularly, or continue without waiting in an active conversation.
## Common Uses for Tasks
Tasks are useful for:
* recurring summaries
* scheduled checks
* one-time follow-up work
* manual runs that you want to repeat later
## Task Types
Sawmills Agent tasks can be used in several ways:
* **Manual**: created for later use and run when needed
* **One-time**: runs once at a specific time
* **Interval**: runs on a repeating interval
* **Recurring schedule**: runs on a regular schedule
## Managing Tasks
Depending on your access, you can:
* create a task
* edit the task prompt or schedule
* enable or disable a task
* run a task immediately
* review task run history
## Best Practices
* use a clear task name
* keep the prompt focused on one outcome
* choose a schedule that matches the pace of the work
* review early runs before relying on a recurring task
## Notes
Tasks work well together with [Notifications](/docs/sawmills-agent-notifications) when results need to be delivered to a person or team.
# Sawmills CLI
Source: https://docs.sawmills.ai/docs/sawmills-cli
Install the Sawmills sm CLI via Homebrew, authenticate with your account, and explore the full command reference for managing collectors and pipelines.
This page documents the `sm` binary shipped by Sawmills.
Every command accepts these global flags:
| Flag | Meaning |
| - | - |
| `--json` | Emit JSON output instead of human-readable tables or prose. |
| `--debug` | Print richer debug/error output. |
| `-h`, `--help` | Show command help. |
At the root level, `sm -V` and `sm --version` print the installed CLI version.
## Install the CLI
### Homebrew
This is the primary install path for macOS and Linux:
```bash theme={null}
brew tap sawmills/tap https://github.com/Sawmills/homebrew-tap
brew install sawmills/tap/sm
sm --version
```
## Uninstall the CLI
```bash theme={null}
brew uninstall sm
```
## Login and organization selection
The CLI uses a browser-based OAuth PKCE flow.
```bash theme={null}
sm login
```
Confirm your authentication:
```bash theme={null}
sm auth status
```
Expected output:
```bash theme={null}
Checking API access...
Profile: prod
Status: Authenticated
```
# Sawmills MCP
Source: https://docs.sawmills.ai/docs/sawmills-mcp
Connect the hosted Sawmills MCP server to AI coding agents like Codex and Claude Code for AI-assisted telemetry management and pipeline configuration.
This page documents connecting agent CLIs to the hosted Sawmills MCP server.
## Install the MCP server
### Install in Codex
```bash theme={null}
codex mcp add sawmills --url https://mcp.sawmills.ai/mcp
```
### Install in Claude Code
```bash theme={null}
claude mcp add --scope user --transport http sawmills https://mcp.sawmills.ai/mcp
```
Run /mcp in Claude Code session to go through the authentication process.
## Uninstall the MCP server
### Uninstall from Codex
```bash theme={null}
codex mcp remove sawmills
```
### Uninstall from Claude Code
```bash theme={null}
claude mcp remove sawmills
```
# How to send data from Datadog agent
Source: https://docs.sawmills.ai/docs/send-data-from-datadog-agent
Configure the Datadog Agent to send logs and metrics to the Sawmills Collector using Helm or environment variables with single or dual shipping.
This guide provides instructions for configuring a Datadog agent to send telemetry data to the Sawmills Collector installed on a different Kubernetes cluster.
For instructions on configuring external access to the Sawmills Collector, please refer to [Configure the Sawmills Collector for External Access](/docs/send-data-to-collector).
***
## Prerequisites
* A **Datadog agent** installed in the source cluster.
* The **Sawmills Collector** deployed and accessible in the destination Kubernetes cluster.
* **Network connectivity** between the source and destination clusters.
> **Note:** Use the collector endpoint from your deployment. For in-cluster traffic, the endpoint usually looks like `..svc.cluster.local:`. If the Sawmills collector load balancer is enabled, use the source-facing load balancer service.
***
## Supported Data Types
| Data Type | Supported | Notes |
| :- | :- | :- |
| Logs | Yes | All log formats |
| Metrics | Yes | Gauges, counts, rates, histograms, and distributions |
| Traces | No | Must be sent directly to Datadog (see [Traces](#traces-bypass-configuration)) |
**Traces are not supported through the Sawmills Collector.** If your Datadog
agent sends traces through Sawmills, that trace data will be lost. Make sure
APM/tracing is configured to send directly to Datadog. See [Traces Bypass
Configuration](#traces-bypass-configuration) for details.
***
## Logs Configuration
### Configuration Options
There are several ways to configure the Datadog agent to send log data to the Sawmills Collector. Choose the method that best fits your deployment:
* **Option 1**: Send only logs to Sawmills Collector (single shipping) - Configuration file method
* **Option 2**: Send only logs to Sawmills Collector (single shipping) - Environment variables method
* **Option 3**: Send logs to both Datadog and Sawmills Collector (dual shipping) - Environment variables method
* **Option 4**: Send logs to both Datadog and Sawmills Collector (dual shipping) - Configuration file method
### Configure the Datadog Agent for Logs
Before updating endpoints, make sure the Datadog Agent is collecting logs. The examples below include the required flags (`logs_enabled: true` or `DD_LOGS_ENABLED=true`) so the Agent actually ships log data to the Sawmills Collector.
#### Helm Chart Configuration
If you installed the Datadog agent using a Helm chart, add the following to your `values.yaml`:
```yaml theme={null}
datadog:
logs:
enabled: true
containerCollectAll: true # optional, collects all container logs by default
agents:
useConfigMap: true
customAgentConfig:
logs_config:
use_http: true
force_use_http: true
logs_dd_url: ..svc.cluster.local:
logs_no_ssl: true
```
#### Configuration File Method
**Option 1**: Configure sending only logs to Sawmills Collector (Single Shipping)
This approach sends only logs to the Sawmills Collector while keeping metrics and traces going to Datadog.
```yaml theme={null}
logs_enabled: true
logs_config:
use_http: true
force_use_http: true
logs_dd_url: ..svc.cluster.local:
logs_no_ssl: true
```
**Option 2**: Configure sending only logs to Sawmills Collector (Environment Variables Method)
This approach uses environment variables to configure single shipping, which is useful for containerized deployments or when you prefer environment-based configuration.
Set the following environment variables in your Datadog agent deployment:
```yaml theme={null}
- name: "DD_LOGS_ENABLED"
value: "true"
- name: "DD_LOGS_CONFIG_FORCE_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_LOGS_DD_URL"
value: "..svc.cluster.local:"
- name: "DD_LOGS_CONFIG_LOGS_NO_SSL"
value: "true"
```
**Option 3**: Configure dual shipping (Environment Variables Method)
This approach allows you to send logs to both Datadog and Sawmills Collector simultaneously using environment variables.
Set the following environment variables in your Datadog agent deployment:
```yaml theme={null}
- name: "DD_LOGS_ENABLED"
value: "true"
- name: "DD_LOGS_CONFIG_FORCE_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_LOGS_NO_SSL"
value: "true"
- name: "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS"
value: '[{"api_key": "ignore", "host": "..svc.cluster.local", "port": , "is_reliable": true}]'
```
**Option 4**: Configure dual shipping (Configuration File Method)
This approach allows you to send logs to both Datadog and Sawmills Collector simultaneously.
Add the following `additional_endpoints` section to your configuration:
```yaml theme={null}
logs_enabled: true
logs_config:
use_http: true # ensures both primary and additional endpoints use HTTP transport
force_use_http: true # this is important to force the agent to use http
logs_no_ssl: true
additional_endpoints:
- api_key: ignore
host: ..svc.cluster.local
port:
use_ssl: false
```
> **Warning:** Different Datadog agent versions might have slightly different ways of configuring the agent. If you can't manage to disable HTTPS, try using `dev_mode_no_ssl: true` under logs\_config.
> **Note:** Adjust `..svc.cluster.local:` to match the domain and port you configured in the Sawmills Collector's external access setup.
***
## Metrics Configuration
The Sawmills Collector supports Datadog metric series ingestion for **gauges**, **counts**, **rates**, and **histograms** through the Datadog metrics API-compatible endpoints. The `DD_DD_URL` environment variable controls where the agent sends metric data.
The Datadog distributions endpoint `/api/v1/distribution_points` is not
currently supported by the Sawmills Collector.
### Configuration Options
* **Option 1**: Send only metrics to Sawmills Collector (single shipping)
* **Option 2**: Send metrics to both Datadog and Sawmills Collector (dual shipping)
### Option 1: Single Shipping (Environment Variables)
This approach redirects all metric data to the Sawmills Collector. Metrics will no longer be sent directly to Datadog.
```yaml theme={null}
- name: "DD_DD_URL"
value: "http://..svc.cluster.local:"
- name: "DD_USE_V2_API_SERIES"
value: "true"
```
Using a configuration file:
```yaml theme={null}
dd_url: http://..svc.cluster.local:
use_v2_api.series: true
```
### Option 2: Dual Shipping (Environment Variables)
This approach sends metrics to both Datadog and the Sawmills Collector simultaneously.
```yaml theme={null}
- name: "DD_USE_V2_API_SERIES"
value: "true"
- name: "DD_ADDITIONAL_ENDPOINTS"
value: '{"http://..svc.cluster.local:": ["ignore"]}'
```
Using a configuration file:
```yaml theme={null}
use_v2_api.series: true
additional_endpoints:
"http://..svc.cluster.local:":
- ignore
```
The default port for the Datadog metrics source in the Sawmills Collector is
**8126**. Make sure the port in your configuration matches the port configured
in your [Datadog Source](/docs/source-datadog).
### Helm Chart Configuration (Metrics)
To configure metrics forwarding via Helm, add the following to your `values.yaml`:
```yaml theme={null}
datadog:
dd_url: http://..svc.cluster.local:
agents:
containers:
agent:
env:
- name: "DD_USE_V2_API_SERIES"
value: "true"
```
For dual shipping with Helm:
```yaml theme={null}
agents:
containers:
agent:
env:
- name: "DD_USE_V2_API_SERIES"
value: "true"
- name: "DD_ADDITIONAL_ENDPOINTS"
value: '{"http://..svc.cluster.local:": ["ignore"]}'
```
***
## Traces Bypass Configuration
**Traces are not supported through the Sawmills Collector.** You must ensure
that APM/trace data continues to be sent directly to Datadog. If trace data is
routed through Sawmills, it will be dropped and **you will lose trace data**.
When configuring the Datadog agent to send logs and/or metrics through Sawmills, make sure traces are **not** redirected. The APM endpoint is controlled independently from logs and metrics:
* **Do not** set `DD_APM_DD_URL` to the Sawmills Collector address.
* **Do not** set `DD_TRACE_DD_URL` to the Sawmills Collector address.
* If you are using `DD_DD_URL` for metrics single shipping, APM will continue to send traces directly to Datadog as long as `DD_APM_DD_URL` is not explicitly overridden.
To explicitly ensure traces go directly to Datadog, you can set:
```yaml theme={null}
- name: "DD_APM_ENABLED"
value: "true"
- name: "DD_APM_DD_URL"
value: "https://trace.agent.datadoghq.com"
```
Replace `datadoghq.com` with your Datadog site (e.g., `us5.datadoghq.com`,
`datadoghq.eu`) if you are not on the US1 site.
***
## Combined Configuration Example
Below is a complete example that sends **both logs and metrics** through the Sawmills Collector while keeping **traces going directly to Datadog**:
### Environment Variables
```yaml theme={null}
# Logs → Sawmills Collector
- name: "DD_LOGS_ENABLED"
value: "true"
- name: "DD_LOGS_CONFIG_FORCE_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_USE_HTTP"
value: "true"
- name: "DD_LOGS_CONFIG_LOGS_DD_URL"
value: "..svc.cluster.local:"
- name: "DD_LOGS_CONFIG_LOGS_NO_SSL"
value: "true"
# Metrics → Sawmills Collector
- name: "DD_DD_URL"
value: "http://..svc.cluster.local:"
- name: "DD_USE_V2_API_SERIES"
value: "true"
# Traces → Datadog directly (do NOT route through Sawmills)
- name: "DD_APM_ENABLED"
value: "true"
- name: "DD_APM_DD_URL"
value: "https://trace.agent.datadoghq.com"
```
### Configuration File
```yaml theme={null}
# Logs → Sawmills Collector
logs_enabled: true
logs_config:
use_http: true
force_use_http: true
logs_dd_url: ..svc.cluster.local:
logs_no_ssl: true
# Metrics → Sawmills Collector
dd_url: http://..svc.cluster.local:
use_v2_api.series: true
# Traces → Datadog directly (do NOT route through Sawmills)
apm_config:
enabled: true
dd_url: https://trace.agent.datadoghq.com
```
Replace `` and `` with the ports configured in your Sawmills Collector's [Datadog Source](/docs/source-datadog). The default logs port is **10518** and the default metrics port is **8126**.
***
## Datadog Serverless Forwarder (AWS Lambda)
If you are using the [Datadog Serverless Forwarder](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring) (`dd-serverless-forwarder`) to ship AWS logs to Datadog, you can route those logs through the Sawmills Collector by editing these environment variables on the Lambda function:
| Environment Variable | Description |
| :- | :- |
| `DD_URL` | The Lambda-reachable Sawmills Collector endpoint (e.g., an internal NLB DNS name like `internal-collector-abc123.us-east-1.elb.amazonaws.com`) |
| `DD_PORT` | The Sawmills Collector port (e.g., `10518` for logs) |
| `DD_NO_SSL` | Set to `true` if SSL is not required |
| `DD_API_URL` | The Datadog API endpoint used by the Forwarder for API key validation (e.g., `https://api.datadoghq.com` for US1). Keep this pointed at Datadog, not the Sawmills Collector. |
Once these variables are set, all logs forwarded by the Lambda function will flow through the Sawmills Collector.
**Important:** The Datadog Forwarder validates the Datadog API key during Lambda initialization, before it sends logs to `/api/v2/logs`. If the Lambda runs in a private VPC, make sure it has outbound HTTPS (`tcp/443`) access to the Datadog API endpoint configured in `DD_API_URL`. If that validation request is blocked, the Lambda can time out during init and no log submissions will reach the Sawmills Collector.
Make sure the Sawmills Collector is reachable from the AWS Lambda function's
network (e.g., via an internal NLB, PrivateLink, VPC peering with resolvable
private DNS, or a public endpoint). Kubernetes service DNS names such as
`..svc.cluster.local` are usually only
resolvable inside the cluster. Refer to [Configure the Sawmills Collector for
External Access](/docs/send-data-to-collector) for details.
***
## Restart the Datadog Agent
After saving the configuration changes, restart the Datadog agent to apply them:
```bash theme={null}
sudo systemctl restart datadog-agent
```
## Verify Data Flow
To confirm the data flow from the Datadog agent to the Sawmills Collector:
1. **View the Datadog agent logs** to confirm there are no errors in connecting to the Sawmills Collector:
```bash theme={null}
sudo tail -f /var/log/datadog/agent.log
```
2. **Check the Sawmills Collector logs** in the destination cluster to ensure it is receiving data from the Datadog agent.
3. **Check your backend or monitoring system** (e.g., Prometheus, Grafana) to ensure the Sawmills Collector is forwarding data as expected.
## Configuration File Location
If you installed the Datadog agent using a different method, the configuration file is usually located at `/etc/datadog-agent/datadog.yaml` on your Datadog agent host.
## Troubleshooting
* **Connection Issues**: If the Datadog agent can't reach the Sawmills Collector, verify network connectivity and firewall settings.
* **Configuration Errors**: Double-check the Datadog agent's configuration for correct endpoints, ports, and protocols.
* **External Accessibility**: Ensure that your Ingress or NodePort service for the Sawmills Collector is accessible from the source cluster.
* **Missing Metrics**: Ensure `DD_USE_V2_API_SERIES` is set to `true`. The Sawmills Collector requires the v2 series API format.
* **Missing Traces**: Traces are not supported through the Sawmills Collector. Verify that `DD_APM_DD_URL` is not pointing to the Sawmills Collector.
# How to send data to the Sawmills collector?
Source: https://docs.sawmills.ai/docs/send-data-to-collector
Send telemetry data to the Sawmills Collector from inside or outside your Kubernetes cluster using internal services or external ingress endpoints.
This guide explains how to send telemetry data (logs, metrics, traces) to the Sawmills Collector running as a deployment within a Kubernetes (K8s) cluster. You can choose to send data from within the cluster (internal) or from outside the cluster (external).
## 1. Sending Telemetry Data Internally (Within the Cluster)
When sending telemetry data from applications or services within the Kubernetes cluster, you can leverage the cluster’s internal DNS to route data to the Sawmills Collector service.
1. **Send Data to the Sawmills Collector’s Internal DNS Address**\
Use the Sawmills Collector service DNS address within the cluster:
`..svc.cluster.local`.
Kubernetes automatically manages this DNS for services, allowing pods to communicate with the Sawmills Collector by using this DNS name.
> **Note:** If the collector load balancer is enabled, send telemetry sources to the source-facing load balancer service. With the default resource base name, that service is `sawmills-collector`. Without the load balancer, send telemetry sources to the regular collector service.
2. **Using Headless Services for Direct Pod Communication**\
For scenarios where you need direct communication with specific collector pods (e.g., load balancing, service discovery), you can use a headless service. For example:
`..svc.cluster.local`.
A headless service (ClusterIP: None) allows direct pod-to-pod communication and is particularly useful when you need to bypass the default load balancing behavior of Kubernetes services.
3. **Set the Collector’s Port**\
The Sawmills Collector typically listens on a specific port (e.g., 4317 for OTLP/gRPC). Ensure your application is configured to send data to the correct port on the collector.
4. **Verify Data Flow**\
Check that data is successfully flowing from your applications to the Sawmills Collector. The collector should receive, process, and potentially forward this data to a specified backend (e.g., Prometheus, Grafana) for further analysis.
## 2. Sending Telemetry Data Externally (Outside the Cluster)
If your data source is external to the Kubernetes cluster, you’ll need to expose the Sawmills Collector service to external traffic.
### Options to Expose the Collector:
1. **NodePort Service**: A simpler method, mainly for testing purposes, exposing the service on a node’s IP address.
Creating a NodePort Service
You can use the following YAML to create a NodePort service that exposes the Sawmills Collector to external traffic:
Use the same selector labels as the source-facing collector service you want to expose. Do not select only by release instance, because load balancer mode uses the same release instance label on both load balancer and backend collector pods.
```yaml theme={null}
apiVersion: v1
kind: Service
metadata:
name: sawmills-collector-nodeport
spec:
type: NodePort
selector:
app.kubernetes.io/instance:
app.kubernetes.io/name:
ports:
- protocol: TCP
port: 80
targetPort: 8080
```
2. **Kubernetes Ingress**: Suitable for production, an ingress provides flexibility in routing external traffic to the collector service.
* Prerequisites
* An Ingress controller must be installed in your cluster (e.g., NGINX Ingress Controller or Traefik).
* A DNS entry pointing to your Ingress controller’s IP (or Load Balancer IP if you’re on a cloud provider).
* Example creates an Ingress resource that directs traffic from a specific external path (e.g., `/sawmills`) to the Sawmills Collector service.
```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sawmills-collector-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: example.com # Replace with your domain name
http:
paths:
- path: /sawmills # Path for external access
pathType: Prefix
backend:
service:
name:
port:
number: 8080 # The port on which the Sawmills Collector listens internally
# Optional: TLS configuration for secure HTTPS access
tls:
- hosts:
- example.com # Replace with your domain name
secretName: sawmills-tls # A pre-created TLS secret for HTTPS
```
# How to send logs from Vector
Source: https://docs.sawmills.ai/docs/send-logs-from-vector
Configure Vector to send logs to the Sawmills Collector using the OTLP source and opentelemetry sink, with single-shipping and dual-shipping examples.
This guide explains how to send logs from Vector to the Sawmills Collector.
It uses the **OTLP** source in Sawmills and the **opentelemetry** sink in Vector.
For instructions on configuring external access to the Sawmills Collector, see [Configure the Sawmills Collector for External Access](/docs/send-data-to-collector).
***
## Prerequisites
* **Vector** is already collecting the logs you want to send.
* The **Sawmills Collector** is deployed and reachable from the machine or cluster where Vector is running.
* An **OTLP** source is configured in Sawmills.
* You know the **hostname** and **port** of the Sawmills Collector endpoint.
***
## Supported Data Types
| Data Type | Supported | Notes |
| :- | :- | :- |
| Logs | Yes | Recommended protocol for this guide |
| Metrics | No | This guide covers logs only |
| Traces | No | This guide covers logs only |
This guide is for **logs only**. If you want to send metrics or traces from
Vector, use a different Sawmills source such as [OTLP](/docs/source-otlp).
***
## Recommended Setup
The clearest setup is:
* **Sawmills source**: [OTLP Source](/docs/source-otlp)
* **Vector sink**: `opentelemetry`
* **Protocol**: OTLP over HTTP
This gives you the standard OTLP path and avoids the TCP receiver path.
***
## What You Need to Change in Vector
Add a new sink that sends logs to Sawmills, or update an existing sink if you want Sawmills to become the only destination.
These are the fields that matter:
* `type: opentelemetry`
* `inputs`: the source or transform that already contains your logs
* `protocol.type: http`
* `protocol.uri`: your Sawmills OTLP HTTP endpoint
* `protocol.encoding.codec: otlp`
Optional:
* use `https://` in `protocol.uri` if your Sawmills endpoint expects TLS
In Sawmills, the default HTTP port for the [OTLP Source](/docs/source-otlp) is
**4318** unless you changed it.
***
## Sawmills Configuration
Before changing Vector, make sure Sawmills has an **OTLP** source with HTTP enabled, listening on the address and port you plan to use.
If your Vector instance is outside the Kubernetes cluster, use the externally exposed hostname and port instead of the internal service address.
***
## Vector Configuration Options
* **Option 1**: Send logs only to Sawmills
* **Option 2**: Send logs to both the current destination and Sawmills
***
## Option 1: Send Logs Only to Sawmills
Use this when Sawmills should be the only destination for these logs.
### YAML Example
```yaml theme={null}
sinks:
sawmills_logs:
type: opentelemetry
inputs:
- my_logs
protocol:
type: http
uri: http://..svc.cluster.local:4318/v1/logs
encoding:
codec: otlp
```
### TOML Example
```toml theme={null}
[sinks.sawmills_logs]
type = "opentelemetry"
inputs = ["my_logs"]
[sinks.sawmills_logs.protocol]
type = "http"
uri = "http://..svc.cluster.local:4318/v1/logs"
[sinks.sawmills_logs.protocol.encoding]
codec = "otlp"
```
### What to Replace
* Replace `my_logs` with the source or transform that currently feeds your log sink.
* Replace `..svc.cluster.local` with your Sawmills collector endpoint. If the collector load balancer is enabled, use the source-facing load balancer service.
* Replace `4318` with your OTLP HTTP port if it is different.
***
## Option 2: Dual Shipping
Use this when you want Vector to keep sending logs to the current destination and also send a copy to Sawmills.
In Vector, dual shipping usually means **adding one more sink** that points at the same `inputs`.
### Example
If you already have a sink like this:
```yaml theme={null}
sinks:
existing_logs_sink:
type: some_existing_sink
inputs:
- my_logs
```
Add this second sink:
```yaml theme={null}
sinks:
sawmills_logs:
type: opentelemetry
inputs:
- my_logs
protocol:
type: http
uri: http://..svc.cluster.local:4318/v1/logs
encoding:
codec: otlp
```
The important part is that both sinks use the same `inputs`.
If you want the smallest possible change, do **not** replace the existing
sink. Just add `sawmills_logs` as a second sink.
***
## TLS Configuration
If your Sawmills endpoint is exposed with TLS, use an `https://` OTLP endpoint.
### YAML Example
```yaml theme={null}
sinks:
sawmills_logs:
type: opentelemetry
inputs:
- my_logs
protocol:
type: http
uri: https://logs.example.com:4318/v1/logs
encoding:
codec: otlp
```
If your environment requires custom CA certificates or stricter verification settings, configure the additional TLS options in the Vector sink accordingly.
***
## Verification
After updating the Vector configuration:
1. Restart or reload Vector.
2. Check the Vector logs for connection or healthcheck errors.
3. Confirm that logs appear in Sawmills.
Useful checks:
```bash theme={null}
vector validate /etc/vector/vector.yaml
```
```bash theme={null}
journalctl -u vector -f
```
If Vector runs in Kubernetes, inspect the Vector pod logs instead.
***
## Troubleshooting
### No logs arriving in Sawmills
* Confirm the Sawmills **OTLP** source is enabled and HTTP is listening on the expected port.
* Confirm the `protocol.uri` in Vector points to the correct hostname, port, and `/v1/logs` path.
* Confirm network rules allow traffic from Vector to the Sawmills endpoint.
### Logs arrive, but fields are missing
* Use OTLP and keep the `opentelemetry` sink pointed at the final log event stream you want to ship.
* Make sure the fields you need still exist in the event before it reaches the `sawmills_logs` sink.
### TLS connection fails
* Confirm the endpoint expects HTTPS.
* Confirm Vector has the correct CA certificates if you use a private certificate authority.
* If your Sawmills endpoint is plain HTTP, use `http://...` instead of `https://...`.
### Vector config is valid, but the wrong logs are sent
* Check the `inputs` value on the `sawmills_logs` sink.
* Point it at the exact source or transform that contains the logs you want to ship.
***
## Summary
To send logs from Vector to Sawmills, the only required Vector change is usually to add a sink like this:
```yaml theme={null}
sinks:
sawmills_logs:
type: opentelemetry
inputs:
- my_logs
protocol:
type: http
uri: http://..svc.cluster.local:4318/v1/logs
encoding:
codec: otlp
```
Replace only these values:
* `inputs`
* `protocol.uri`
* `port`
* `http` vs `https`
# How to send CloudWatch metrics via Firehose
Source: https://docs.sawmills.ai/docs/send-metrics-via-cloudwatch-firehose
Configure Amazon CloudWatch Metric Streams and Amazon Data Firehose to push AWS metrics to the Sawmills Collector over an HTTP endpoint.
This guide configures Amazon CloudWatch Metric Streams to push metrics through Amazon Data Firehose into the Sawmills Collector. No agent runs in your account — CloudWatch streams metrics to Firehose, and Firehose posts them to an HTTPS endpoint fronting your collector.
For general external-access setup, see [Configure the Sawmills Collector for External Access](/docs/send-data-to-collector) and [Load Balancer Setup](/docs/load-balancer-setup).
***
## Prerequisites
* The **Sawmills Collector** deployed with the **load balancer** enabled (the Firehose source binds on the load-balancer collector).
* A **Network Load Balancer** (or equivalent) you control, with a **publicly-trusted (ACM) TLS certificate** on a domain you own. Firehose rejects self-signed certificates.
* Permission to create CloudWatch Metric Streams and Firehose delivery streams in your AWS account.
***
## Supported Data Types
| Data Type | Supported | Notes |
| :- | :- | :- |
| Metrics | Yes | CloudWatch Metric Streams, **JSON** output format (`cwmetrics`) |
| Logs | No | Use a dedicated logs source |
| Traces | No | — |
***
## Step 1 — Add the CloudWatch Metric Streams source in Sawmills
Create a **CloudWatch Metric Streams (Firehose)** source on your pipeline (see [source configuration](/docs/source-cloudwatch-firehose)). Set an **Access Key** — a strong shared secret you will also put on the Firehose stream. Note the source **port** (default `10433`). Deploy the pipeline so the collector begins listening.
## Step 2 — Expose the collector over HTTPS
Point your Network Load Balancer at the collector's firehose port (`10433` by default) and attach your ACM certificate to the NLB's TLS listener. The NLB terminates TLS (SSL offload) and forwards plaintext to the collector. Record the public HTTPS URL (for example, `https://firehose.example.com`).
Firehose's HTTP endpoint destination only accepts an `https://` URL backed by a CA-signed certificate. A self-signed certificate will fail the TLS handshake.
## Step 3 — Create the Firehose delivery stream
Create an **Amazon Data Firehose** stream with an **HTTP endpoint** destination:
* **HTTP endpoint URL**: the NLB URL from Step 2.
* **Access key**: the same value you set on the Sawmills source in Step 1.
* **Content encoding**: `NONE`. The collector's firehose receiver reads the HTTP request body as JSON; set content encoding to `NONE` (disabled) so records reach the `cwmetrics` decoder.
* Do **not** attach a record-transformation Lambda — the collector expects the raw CloudWatch Metric Streams records.
## Step 4 — Create the CloudWatch Metric Stream
Create a **CloudWatch Metric Stream** with:
* **Output format**: `JSON`.
* **Destination**: the Firehose stream from Step 3.
* The namespaces / metrics you want to forward.
```bash theme={null}
aws cloudwatch put-metric-stream \
--name sawmills-metric-stream \
--firehose-arn \
--role-arn \
--output-format json \
--include-filters Namespace=AWS/EC2 Namespace=AWS/Lambda
```
## Verify Data Flow
* In AWS, confirm the Firehose metric **`DeliveryToHttpEndpoint.Success`** is non-zero with no 4xx/5xx to the endpoint.
* In Sawmills, open the pipeline's metrics view and confirm your CloudWatch metrics are arriving (allow \~1–2 minutes for the Firehose buffer).
## Troubleshooting
| Symptom | Cause / Fix |
| :- | :- |
| Firehose TLS handshake failures | The NLB certificate is not CA-signed / not trusted. Use an ACM certificate on a real domain. |
| Firehose HTTP 401 / access denied | The access key on the Firehose stream does not match the source's Access Key. |
| No metrics in Sawmills, Firehose success | Confirm the metric stream **Output format is JSON** and no transform Lambda is attached. |
| `DeliveryToHttpEndpoint` failing | The NLB is not reaching the collector's firehose port, or the collector is not deployed with the source. |
# How to send CloudWatch metrics via S3
Source: https://docs.sawmills.ai/docs/send-metrics-via-cloudwatch-s3
Set up CloudWatch Metric Streams, Firehose, S3, SQS, and collector IAM permissions to ingest AWS metrics into Sawmills.
This guide delivers CloudWatch metrics to S3 and lets the Sawmills Collector pull them using SQS notifications. The collector uses outbound AWS API connections and its own AWS identity; no inbound collector endpoint is needed.
```text theme={null}
CloudWatch Metric Stream (JSON)
→ Amazon Data Firehose (Direct PUT, S3 destination, uncompressed)
→ S3 object creation notification → SQS
→ Sawmills Collector → pipeline → metrics destinations
```
For source fields and message handling, see [CloudWatch Metrics (S3/SQS) Source](/docs/source-cloudwatch-s3).
## Prerequisites
* A deployed Sawmills Collector. Use **v1.1120.0 or later** for the documented drop/retry behavior and drop counters.
* A pipeline with a destination that accepts metrics.
* AWS permissions to configure S3, SQS, Firehose, CloudWatch Metric Streams, and their IAM roles.
* Outbound connectivity from the collector to S3, SQS, and the AWS identity service used by its role, such as STS for IRSA.
The examples use account `123456789012`, Region `us-west-2`, bucket `example-cloudwatch-metrics`, prefix `cwmetrics/`, and queue `example-cloudwatch-metrics`. Replace these consistently with your resources. This walkthrough keeps the resources in one account and Region.
## Step 1: Create the bucket and queue
1. Create an S3 bucket for the metrics, with a dedicated object prefix such as `cwmetrics/`.
2. Create a **standard SQS queue** in the same Region as the bucket. Record its URL and ARN.
3. Use S3-managed encryption (SSE-S3) and SQS-managed encryption (SSE-SQS) for this example. If you use customer-managed KMS keys, also configure the permissions described below.
Use a queue dedicated to these metric objects. Set queue retention to cover expected outages and a visibility timeout long enough to process the objects in a received batch.
Configure native S3 notifications to a standard queue. S3 cannot deliver these
notifications directly to a FIFO queue, and the receiver drops EventBridge
message bodies. See [AWS notification
destinations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html).
## Step 2: Allow S3 to notify SQS
Add this statement to the queue's access policy, substituting your queue ARN, bucket name, and bucket owner's account ID. Preserve any other statements the queue needs.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowMetricBucketNotifications",
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-west-2:123456789012:example-cloudwatch-metrics",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:s3:::example-cloudwatch-metrics"
},
"StringEquals": { "aws:SourceAccount": "123456789012" }
}
}
]
}
```
In the bucket's **Properties → Event notifications**, add a notification with:
* **Event types**: all object creation events (`s3:ObjectCreated:*`), including multipart upload completion.
* **Prefix**: `cwmetrics/`.
* **Suffix**: leave empty; Firehose object keys do not need a `.json` extension.
* **Destination**: the SQS queue from Step 1.
If the queue uses a customer-managed KMS key, its key policy must also allow S3 to use `kms:GenerateDataKey` and `kms:Decrypt`. See [AWS permissions for notification destinations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html).
## Step 3: Grant the collector access
Attach the following IAM policy to the role used by the collector pods that run this source. This role is separate from the roles CloudWatch and Firehose use to deliver metrics.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadMetricObjects",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::example-cloudwatch-metrics/cwmetrics/*"
},
{
"Sid": "ConsumeMetricNotifications",
"Effect": "Allow",
"Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage"],
"Resource": "arn:aws:sqs:us-west-2:123456789012:example-cloudwatch-metrics"
}
]
}
```
For EKS with **IRSA**, create a role whose trust policy allows your cluster's OIDC provider and the collector's service account to assume it. Annotate that service account with the role ARN:
```yaml theme={null}
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/sawmills-cloudwatch-metrics
```
Ensure the pods running the source use that service account, including the load-balancer collector if enabled. Apply the annotation through your deployment configuration and recreate the affected pods so they receive the role credentials. Follow [AWS's IRSA setup](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) for the OIDC trust and role association.
If the S3 objects or SQS messages use customer-managed KMS keys, grant the collector role `kms:Decrypt` on the applicable keys and allow the role in the key policies. Firehose also needs the permissions required by your bucket's encryption configuration. The example policy above assumes SSE-S3 and SSE-SQS.
## Step 4: Create the Firehose stream
Create an Amazon Data Firehose delivery stream with these settings:
| Setting | Value |
| :- | :- |
| Source | **Direct PUT** |
| Destination | **Amazon S3**, using the bucket from Step 1 |
| S3 bucket prefix | `cwmetrics/` |
| Compression | **Uncompressed** (`UNCOMPRESSED`) |
| Record transformation and format conversion | Disabled; preserve the CloudWatch Metric Streams JSON records |
Allow Firehose to create a service role with access to write to the bucket, or select an existing role with the equivalent permissions. Keep any error-output prefix outside `cwmetrics/` so error files do not enter the source. Firehose buffers records before writing objects, so account for its buffering interval when checking delivery.
## Step 5: Create the CloudWatch Metric Stream
In **CloudWatch → Metrics → Streams**, create a metric stream with:
* **Custom setup with Firehose**: select the delivery stream from Step 4.
* **Output format**: explicitly select **JSON**.
* **Metrics**: select the namespaces and metrics you want to send.
* **Service role**: create or select a role that allows CloudWatch to write to this Firehose stream.
The metric stream and Firehose stream must be in the same account and Region. See [AWS custom metric stream setup](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-setup-datalake.html).
The source expects [CloudWatch Metric Streams
JSON](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-json.html),
with one JSON object per line. Keep this format through Firehose and S3; do
not convert it to OTLP, Parquet, or a JSON array.
## Step 6: Add the source in Sawmills
Open your pipeline, add a source, and select **CloudWatch Metrics**. Set:
| Field | Example |
| :- | :- |
| Name | `cloudwatch-metrics` |
| Region | `us-west-2` |
| S3 Bucket | `example-cloudwatch-metrics` |
| SQS Queue URL | `https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics` |
| Key Prefix | `cwmetrics/` |
Connect a destination that accepts metrics, then deploy the pipeline to the collector.
Match the bucket and literal prefix to the actual Firehose objects. A prefix
of `/cwmetrics/` will not match keys beginning with `cwmetrics/`. Objects
outside the configured bucket or prefix are skipped and their notifications
deleted. The queue can drain without metrics arriving.
## Step 7: Verify delivery and configure retention
1. Confirm that the CloudWatch Metric Stream is running and includes metrics currently being emitted.
2. After Firehose's buffering interval, check for new S3 objects under `cwmetrics/`. Inspect a sample object's contents for newline-delimited CloudWatch JSON.
3. Check SQS monitoring for incoming and deleted messages. A low queue depth can be normal when the collector consumes quickly; it does not prove successful ingestion.
4. Check collector logs for `Processing new S3 object`, authentication/download errors, and downstream errors. Monitor the [drop counters](/docs/source-cloudwatch-s3#processing-retries-and-dropped-data).
5. Verify recent metrics at the pipeline's destination, including their expected namespace and dimensions. Allow for CloudWatch delivery, Firehose buffering, and collector processing.
6. Add an S3 lifecycle expiration rule scoped to `cwmetrics/`. Choose retention long enough to cover backlog and retries. The collector never deletes S3 objects, and lifecycle expiration does not wait for ingestion.
Enabling notifications does not scan existing objects. This source reads the objects named by the notifications it receives; changing a filter does not replay notifications that have already been deleted.
## Troubleshooting
| Symptom | Check |
| :- | :- |
| No new objects in S3 | Metric stream state and metric selection; Firehose destination, write role, and delivery errors. |
| Objects arrive but no SQS notifications | Bucket event prefix, all object creation events, queue Region, queue access policy, and any KMS key policy. |
| Queue backlog grows | Collector deployment, outbound connectivity, the role used by the pods running the source, `ReceiveMessage`/`DeleteMessage` permissions, and downstream health. |
| S3 `AccessDenied` | Collector `GetObject` scope, bucket policy, and any KMS decrypt permission. Download failures retain the message for retry. |
| Queue drains but no metrics arrive | Exact bucket/prefix match, CloudWatch JSON output, uncompressed Firehose objects, a metrics-capable destination, and drop counters. Bucket/prefix skips do not increment the drop counters. |
| `not_s3_notification` or `invalid_sns_message` drops | Use native S3 notifications, directly or correctly wrapped by SNS. EventBridge bodies are unsupported. |
| `invalid_json`, `unsupported_format`, `decompress_failed`, or `decode_failed` drops | Inspect the notification or object format; remove transformations and unrelated objects from the ingestion prefix. |
| Repeated delivery or missing objects | Check processing time against SQS visibility timeout, retries, and whether lifecycle expiration deleted objects before ingestion. An S3 `NoSuchKey` record is treated as done. |
# Setup fallback endpoint
Source: https://docs.sawmills.ai/docs/setup-fallback-endpoint-for-sources
Configure fallback endpoints on pipeline sources so an HAProxy sidecar automatically reroutes telemetry data when the primary collector is unavailable.
Setting up fallback endpoints for your **sources** ensures that no logs are lost when your collector becomes unavailable or unhealthy. This is achieved by configuring a fallback endpoint that will receive your logs when the primary collector is not operational.
## How It Works
When you configure a fallback endpoint, the following components are automatically set up:
1. An HAProxy sidecar container is deployed alongside your collector pod
2. The HAProxy continuously monitors the health of your collector through periodic health checks and observing any response errors from the collector
3. If the collector is deemed unhealthy, HAProxy automatically routes all incoming data to the fallback endpoint
4. Once the collector is healthy again, HAProxy resumes sending data to the primary collector
This automatic failover mechanism ensures continuous log collection without any data loss.
## Configuration
Fallback endpoints are configured at the source level, allowing you to specify different fallback endpoints for different sources based on your requirements.
### Setting Up a Fallback Endpoint
1. Navigate to your pipeline in the UI
2. Choose the source you want to configure
3. Open the "Advanced Settings" section
4. Choose the Fallback Endpoint URL for this source
![]()
Fallback endpoints are supported for the following sources:
* **Coralogix**
* **Datadog**
* **Elastic**
* **Grafana Loki**
When configuring a custom fallback endpoint for a URL-based source (for
example, Elastic or Grafana Loki), the endpoint URL should be with the
protocol and without the path part (e.g.
[http://loki-gateway.default.svc.cluster.local).\{](http://loki-gateway.default.svc.cluster.local\).\{)" "}
For **Coralogix** sources, select the Coralogix region that should receive
fallback traffic. You can choose a supported region from the dropdown or enter a
custom hostname. The UI formats custom hostnames with the Coralogix `ingress.`
prefix and `:443` port.
### Configuring Custom Values
For some sources, such as Datadog, it's possible to control some of the HAProxy configurations such as timeouts.
Those advanced configurations can be setup via the values your helm chart.
To modify values, update the `managedChartsValues.sawmills-collector.haproxy` section in your `values.yaml` file and then upgrade the remote operator.
```yaml theme={null}
managedChartsValues:
sawmills-collector:
haproxy:
fallback_config:
server:
fall: 4
interval: null
rise: null
timeout:
client: null
connect: 5000
server: null
```
* **Connect Timeout**
* *Description*: The maximum time to wait for a connection attempt for the collector and fallback (in milliseconds, default: 5000).
* *Type*: number
* **Server Timeout**
* *Description*: The maximum inactivity time on the server side for the collector and fallback endpoint (in milliseconds, default: 5000).
* *Type*: number
* **Client Timeout**
* *Description*: The maximum inactivity time on the client side for the collector and fallback endpoint (in milliseconds, default: 5000).
* *Type*: number
* **Server Interval**
* *Description*: The interval between health checks of the collector (in milliseconds, default: 2000).
* *Type*: number
* **Server Rise**
* *Description*: The number of successful health checks to the collector before marking it up (default: 10).
* *Type*: number
* **Server Fall**
* *Description*: The number of failed health checks to the collector before marking it down (default: 1).
* *Type*: number
## Benefits
* **Zero Data Loss**: Ensures all logs are captured even when the primary collector is down
* **Automatic Failover**: Seamless switching between primary and fallback endpoints
* **Per-Source Configuration**: Flexibility to configure different fallback endpoints for different sources
Fallback endpoints are currently only available for **Coralogix**,
**Datadog**, **Elastic**, and **Grafana Loki** sources. If you would like to
add a different source, please contact Sawmills support.
# Enable Slack Integration
Source: https://docs.sawmills.ai/docs/slack-integration
Connect Sawmills to your Slack workspace to receive Insight notifications and automated action updates directly in your team's Slack channels.
## About Slack Integration
The Slack Integration allows you to keep up to date with new Insights for your data pipelines.
Configuring the integration will make sure you'll know about recent insights and their impact on your system.
You will also get a updates about Insight actions taken by yourself and your team, such as following the Insight recommendation or archiving the Insight.
## Step-by-Step: Configure Slack Integration
Follow these steps to configure your Slack Integration:
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
![]()
### 2. Connect to Slack
In the Integrations section, click the **Connect to Slack** button and follow the authentication flow.
### 3. Open Slack Configuration
After completing the authentication, locate and open the **Configure** slack drawer.
### 4. Select the notifications channel
Enter the slack channel you want to get notifications in.
### 5. Test and Save
Click the test button to make sure the integration work and save your changes.
# CloudWatch Metric Streams (Firehose) Source
Source: https://docs.sawmills.ai/docs/source-cloudwatch-firehose
Configure the CloudWatch Metric Streams source in Sawmills to receive AWS CloudWatch metrics pushed through Amazon Data Firehose over an HTTP endpoint.
## Supported Data Types
📈 **Metrics**
This source ingests **Amazon CloudWatch Metric Streams** delivered through **Amazon Data Firehose** to an HTTP endpoint on the Sawmills Collector. Firehose posts records to the collector; the collector decodes the CloudWatch Metric Streams format and forwards the metrics through your pipeline.
Only the CloudWatch Metric Streams **JSON** output format is supported (decoded via the `cwmetrics` encoding). Configure your metric stream with the `JSON` output format.
## How it works
```
CloudWatch Metric Streams
→ Amazon Data Firehose (HTTP endpoint destination)
→ Network Load Balancer (TLS termination / SSL offload)
→ Sawmills Collector (awsFirehose source, cwmetrics)
→ pipeline → destinations
```
Firehose's HTTP endpoint destination requires an **HTTPS** URL with a publicly-trusted (CA-signed) certificate. Terminate TLS at a Network Load Balancer you control and forward plaintext to the collector's firehose port. Firehose authenticates with an **access key** sent in the `X-Amz-Firehose-Access-Key` header, which the collector validates.
For the full AWS setup walkthrough, see [How to send CloudWatch metrics via Firehose](/docs/send-metrics-via-cloudwatch-firehose).
## Configuration
| Name | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills |
| Address | String | `${env:MY_POD_IP}` | false | Address the source binds to for the Firehose HTTP endpoint |
| Port | Int | 10433 | false | Port the source listens on for Firehose requests (target this port from your load balancer) |
| Access Key | String | none | true | Shared secret checked against the Firehose `X-Amz-Firehose-Access-Key` header |
| Encoding | String | none | true | Record encoding — must be **`cwmetrics`** for CloudWatch Metric Streams |
The pod's IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod's status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod's IP at runtime.
TLS is terminated at your load balancer (SSL offload), so the collector listens over plaintext HTTP on the port above. Do not expose this port publicly without the load balancer in front of it.
# CloudWatch Metrics (S3/SQS) Source
Source: https://docs.sawmills.ai/docs/source-cloudwatch-s3
Receive CloudWatch Metric Streams JSON from S3 through SQS notifications, using the Sawmills Collector's AWS identity.
## Supported Data Types
📈 **Metrics**
Select **CloudWatch Metrics** in the Sawmills source picker to ingest CloudWatch metrics delivered to Amazon S3. The collector polls an Amazon SQS queue for object notifications, downloads the objects, and decodes their CloudWatch Metric Streams JSON records into metrics.
```text theme={null}
CloudWatch Metric Stream (JSON)
→ Amazon Data Firehose (Direct PUT, S3 destination, uncompressed)
→ S3 object creation notification → SQS
→ Sawmills Collector → pipeline → metrics destinations
```
The collector makes outbound requests using its own AWS identity, such as an IAM role for a Kubernetes service account (IRSA). This source needs no inbound endpoint, listener port, or shared access key. The pipeline must include a [destination that accepts metrics](/docs/pipeline-destinations).
For AWS resources, IAM policies, and verification steps, see [How to send CloudWatch metrics via S3](/docs/send-metrics-via-cloudwatch-s3).
## Configuration
| Name | Required | Description |
| :- | :- | :- |
| Name | Yes | Unique source name within Sawmills. |
| Region | Yes | AWS Region of the bucket and queue, for example `us-west-2`. It must match the Region in the queue URL. |
| S3 Bucket | Yes | Bucket name, for example `example-cloudwatch-metrics`. Use the name without `s3://` or a path. |
| SQS Queue URL | Yes | Full queue URL, for example `https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics`. Use the URL, not the queue ARN. |
| Key Prefix | No | Literal beginning of the object key, for example `cwmetrics/`. Empty means no prefix filter. This is not a wildcard or an S3 URI. |
The encoding is fixed to **CloudWatch Metric Streams JSON** (`cwmetrics`); there is no format selector. Logs, traces, OTLP files, and arbitrary JSON are not supported by this source.
The equivalent source configuration data is:
```yaml theme={null}
region: us-west-2
s3_bucket: example-cloudwatch-metrics
sqs_queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/example-cloudwatch-metrics
encoding: cwmetrics
s3_prefix: cwmetrics/ # Optional; omit for no prefix filter.
```
Objects outside the configured bucket or prefix are skipped, and their SQS
notifications are deleted once all records in the message are processed or
skipped. A wrong bucket or prefix can silently lose metrics even while the
queue drains. Prefixes are literal: `cwmetrics/` and `/cwmetrics/` match
different keys. Correcting the source later does not replay deleted
notifications.
Use a dedicated queue for this source. An unrelated consumer, or another source with a different bucket or prefix, can consume and delete notifications before the intended source receives them.
## AWS permissions
Grant the AWS role used by the collector pods running this source:
* `s3:GetObject` on the bucket's metric objects, scoped to the configured prefix when possible.
* `sqs:ReceiveMessage` and `sqs:DeleteMessage` on the notification queue.
S3 also needs permission to publish notifications to the queue. See the [setup guide](/docs/send-metrics-via-cloudwatch-s3#step-2-allow-s3-to-notify-sqs) for the queue policy, collector policy, and encryption considerations.
## Processing, retries, and dropped data
Use **Sawmills Collector v1.1120.0 or later** for the behavior below and its drop counters.
The receiver accepts native S3 object notifications, either directly or inside an SNS notification. EventBridge message bodies are not supported. A notification is deleted only when all its records have been processed or skipped; a retryable failure retains the whole message.
| Condition | Behavior |
| :- | :- |
| Metrics accepted by the next pipeline component | Marks the object as processed. |
| Object outside the configured bucket or prefix | Skips the object. |
| Invalid SQS JSON | Deletes the message; reason `invalid_json`. |
| JSON that is not an S3 or SNS notification, including EventBridge | Deletes the message; reason `not_s3_notification`. |
| Invalid SNS payload or no embedded S3 event | Deletes the message; reason `invalid_sns_message`. |
| No matching decoder, decompression failure, or decoding failure | Skips the object; reason `unsupported_format`, `decompress_failed`, or `decode_failed`. |
| S3 download failure, other than `NoSuchKey`, or an error returned by the next pipeline component | Keeps the entire message for retry after the SQS visibility timeout, including records already processed or skipped. |
| Object no longer exists (`NoSuchKey`) | Treats the record as done. |
Monitor these collector counters, grouped by their `reason` label:
* `otelcol_receiver_awss3_sqs_messages_dropped`: unreadable or unsupported notifications.
* `otelcol_receiver_awss3_objects_dropped`: unsupported object formats, decompression failures, and decoding failures.
Both counters increase only after SQS message deletion succeeds. Bucket and prefix mismatches are not included in these counters, so verify actual metrics arrival as well as queue consumption. See [Monitor the Collector with Prometheus](/docs/guide-how-to-monitor-with-prometheus).
Retries can process an object again. Even for successfully decoded objects, queue deletion only confirms that the next pipeline component accepted the data; it is not a guarantee that a remote destination has stored it. Check destination health and collector export errors as part of verification.
## S3 object retention
The receiver **never deletes S3 objects**. Set an S3 lifecycle rule on the metrics prefix to expire objects according to your retention policy. Allow enough time for queue backlog, outages, and retries before expiration. An object that expires before the collector downloads it cannot be ingested from its notification.
# Datadog Source
Source: https://docs.sawmills.ai/docs/source-datadog
Configure the Datadog source in Sawmills to receive logs and metrics from the Datadog Agent via HTTP intake endpoints. Traces are not supported.
## Supported Data Types
📘 **Logs** | 📈 **Metrics**
Traces are **not** supported through the Datadog source. Traces must be sent directly to Datadog. See the [Datadog Agent Configuration guide](/docs/send-data-from-datadog-agent#traces-bypass-configuration) for details.
## Supported Metric Types
The Datadog source accepts standard Datadog metric series types:
| Metric Type | Description |
| :- | :- |
| Gauge | Represents the current value of a metric at a point in time |
| Count | Tracks the number of events over a flush interval |
| Rate | Tracks the per-second rate of events over a flush interval |
| Histogram | Agent-side aggregated percentiles, averages, and counts |
The Datadog distributions endpoint `/api/v1/distribution_points` is not currently supported.
## Supported Ingestion Paths
The Datadog source accepts Datadog HTTP intake requests directly. You can send Datadog metrics payloads to the Sawmills Collector without running a Datadog Agent.
Supported metrics endpoints:
* `/api/v1/series`
* `/api/v2/series`
* `/api/v1/check_run`
* `/api/v1/sketches`
* `/api/beta/sketches`
This means you can either:
* send Datadog HTTP metrics payloads directly to the Sawmills Collector from your own service or job
* configure a Datadog Agent to send metrics to the Sawmills Collector
For Datadog Agent configuration, see [How to send data from Datadog agent](/docs/send-data-from-datadog-agent).
### Direct API Example
```bash theme={null}
curl -X POST http://:8126/api/v2/series \
-H 'Content-Type: application/json' \
-d '{
"series": [{
"metric": "example.metric",
"type": 3,
"points": [{"timestamp": 1710000000, "value": 1.0}],
"resources": [{"name": "example-host", "type": "host"}],
"tags": ["env:test"]
}]
}'
```
## Configuration
| Name | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills |
| Logs Address | String | `${env:MY_POD_IP}` | true | Address the Datadog source should listen to for logs |
| Logs Port | Int | 10518 | true | Port the Datadog source should listen to for logs |
| Metrics Address | String | `${env:MY_POD_IP}` | true | Address to which the Datadog Metrics source should bind. Defaults to `${env:MY_POD_IP}`. |
| Metrics Port | Int | 8126 | true | Port to which the Datadog Metrics source should bind. |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
For instructions on configuring the Datadog agent to send data to this source, see [How to send data from Datadog agent](/docs/send-data-from-datadog-agent).
# Elasticsearch Source
Source: https://docs.sawmills.ai/docs/source-elasticsearch
Configure the Elasticsearch source in Sawmills to receive log data from Elasticsearch clusters and Logstash with optional fallback endpoint support.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) to receive Elasticsearch data. |
| Port | Int | 9200 | true | The port on which Elasticsearch data is received. |
### Advanced Options
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Fallback Endpoint | String | none | false | Optional fallback endpoint for Elasticsearch data reception. When the primary endpoint is unavailable, data will be routed to this fallback endpoint. |
The pod's IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod's status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod's IP at runtime.
## Usage
The Elasticsearch source allows you to receive log data from Elasticsearch clusters. This source is specifically designed to handle log data ingestion and supports the standard Elasticsearch communication protocols & from Logstash.
# Fluent Forward
Source: https://docs.sawmills.ai/docs/source-fluent-forward
Configure the Fluent Forward source in Sawmills to receive log data from Fluent Bit or Fluentd agents on a configurable address and port.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Required | Default Value | Description |
| :- | :- | :- | :- | :- |
| Name | String | true | none | Unique identifier within Sawmills |
| Address | String | true | `${env:MY_POD_IP}` | Address FluentBit source should bind to |
| Port | Int | true | 8006 | Port number for binding |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# Grafana Loki Source
Source: https://docs.sawmills.ai/docs/source-grafana-loki
Configure the Grafana Loki source in Sawmills to receive log data over HTTP and gRPC with timestamp handling and multi-tenant header support.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Enable HTTP | Boolean | true | true | Enable the Loki source to accept telemetry data over HTTP. |
| HTTP Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) where HTTP data is received. |
| Port (HTTP) | Int | 3500 | true | The port for receiving OTLP data over gRPC. |
| Enable gRPC | Boolean | true | true | Enable the Loki source to accept telemetry data over gRPC. |
| gRPC Hostname | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) where gRPC data is received. |
| Port (gRPC) | Int | 4317 | true | The port for receiving OTLP data over gRPC. |
### Advanced Options
| Field | Type | Default | Required | Description | |
| :- | :- | :- | :- | :- | - |
| Use Incoming Timestamp | Boolean | false | true | Use the source-provided log timestamp if reliable. When off, logs are assigned new timestamps at ingestion to maintain consistency in mixed environments. | |
| Forward X-Scope-OrgID Header | Boolean | true | true | When enabled, the incoming X-Scope-OrgID header is preserved. This is often required for multi-tenant authentication to work properly. | |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# New Relic Source
Source: https://docs.sawmills.ai/docs/source-new-relic
Receive New Relic APM time-slice metrics, Telemetry SDK metrics, and Infrastructure agent samples through one Sawmills source.
## Supported Data Types
📈 **Metrics**
## Overview
The New Relic source supports three different senders. Each sender uses a different protocol and collector endpoint.
| Sender | Data | Collector endpoint label | Common configuration |
| :- | :- | :- | :- |
| APM language agent | APM time-slice metrics | **APM agent** | Agent-specific host setting, such as `NEW_RELIC_HOST` for Java |
| Telemetry SDK | Dimensional Metric API data | **SDK** | Metric API base URL, such as `NEW_RELIC_METRICS_HOST` for Java |
| Infrastructure agent | Host, process, and Kubernetes samples | **Infrastructure agent** | `NRIA_COLLECTOR_URL`, `NRIA_IDENTITY_URL`, and `NRIA_COMMAND_CHANNEL_URL` |
> **Note:** The APM agent and Infrastructure agent are different products. Do
> not use the **APM agent** endpoint for the Infrastructure agent.
Infrastructure-agent support is enabled by default on Sawmills Collector `v1.1097.0` or later. There is no source option to enable or disable this support.
As with all Sawmills pipelines, metric data from this source flows through the configured processors to all pipeline destinations. The New Relic destination reconstructs Infrastructure samples in their native format before it sends them to New Relic.
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | yes | Unique source name in the pipeline. |
| Metrics & Traces Address | String | `${env:MY_POD_IP}` | yes | Collector address that receives New Relic traffic. |
| Port | Integer | `14275` | yes | Collector port that receives all three New Relic protocols. |
### Default protocol paths
| Protocol | Path | Behavior |
| :- | :- | :- |
| Metric API | `/metric/v1` | Converts dimensional metrics to OpenTelemetry metrics. |
| APM agent | `/agent_listener/invoke_raw_method` | Handles `preconnect`, `connect`, and `metric_data` methods. |
| Infrastructure metrics | `/infra/v2/metrics/events/bulk` | Converts native Infrastructure samples to OpenTelemetry metrics for processing. |
| Legacy Infrastructure metrics | `/metrics/events/bulk` | Supports the legacy Infrastructure metrics path. |
| Infrastructure identity | `/identity/v1/connect` | Proxies the identity request to the New Relic Identity API. |
| Infrastructure commands | `/agent_commands/v1/commands` | Proxies the command request to the New Relic Command API. |
| Infrastructure inventory | `/inventory/deltas` | Proxies inventory data to the New Relic Infrastructure API. |
| Infrastructure security | `/security/v1/connect` | Proxies security traffic when the agent sends it. |
You can change Infrastructure paths and upstream service URLs in **Advanced options**. A custom upstream URL overrides automatic regional routing for that service.
## Configure a sender
* For an APM language agent and Telemetry SDK example, see [New Relic Java app](/docs/guide-send-data-from-newrelic-java-app).
* For the New Relic Infrastructure agent, see [New Relic Infrastructure agent](/docs/guide-send-data-from-newrelic-infrastructure-agent).
Open the collector details and find **New Relic ingestion endpoints**. Copy the endpoint for the sender type that you use. Do not copy the endpoint for a different sender type.
## Infrastructure authentication
The Infrastructure agent sends its New Relic license key in the `X-License-Key` header. Sawmills does not put the raw key in OpenTelemetry attributes.
The New Relic destination must use the same license key as the Infrastructure agent. This keeps identity, inventory, and metric data in the same New Relic account. The collector rejects Infrastructure export if the source key and destination key do not match.
> **Warning:** Do not use a New Relic User API key. Use the same ingest license
> key in the Infrastructure agent and the New Relic destination.
## Region routing
Sawmills selects the default New Relic service hosts from the license key region.
| Region | Identity | Commands | Infrastructure data |
| :- | :- | :- | :- |
| US | `identity-api.newrelic.com` | `infrastructure-command-api.newrelic.com` | `infra-api.newrelic.com` |
| EU | `identity-api.eu.newrelic.com` | `infrastructure-command-api.eu.newrelic.com` | `infra-api.eu.newrelic.com` |
The source key and destination key must belong to the same account and region. If you configure a custom upstream URL, configure all related services for the same region.
## Supported Infrastructure versions
The minimum supported `newrelic-infrastructure` Helm chart version is `4.0.1`. Sawmills validated this configuration:
| Component | Validated version |
| :- | :- |
| `nri-bundle` chart | `7.0.4` |
| `newrelic-infrastructure` chart and `nri-kubernetes` image | `4.0.1` |
| `infrastructure-bundle` image | `3.3.22` |
| New Relic Infrastructure agent | `1.73.1` |
Later versions are supported when they keep the same protocol paths and native sample format. Test a later version in staging before you use it in production.
## Metric API format
Supported Metric API types:
* `gauge`
* `count`, with `interval.ms`
* `summary`
## APM agent protocol
Supported methods:
* `preconnect`
* `connect`
* `metric_data`
APM metrics are converted to Summary metrics. The conversion keeps count, sum, minimum, maximum, exclusive time, sum of squares, and transaction scope.
## Related documentation
* [New Relic Infrastructure agent](/docs/guide-send-data-from-newrelic-infrastructure-agent)
* [New Relic Java app](/docs/guide-send-data-from-newrelic-java-app)
* [New Relic Destination](/docs/destination-new-relic)
# OTLP Source
Source: https://docs.sawmills.ai/docs/source-otlp
Configure the OTLP source in Sawmills to receive logs, metrics, and traces over both HTTP and gRPC using the OpenTelemetry protocol standard.
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Enable HTTP | Boolean | true | true | Enable the OTLP source to accept telemetry data over HTTP. |
| HTTP Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) where HTTP data is received. Defaults to your pod’s IP. |
| Port (HTTP) | Int | 4318 | true | The port for receiving OTLP data over HTTP. |
| Enable gRPC | Boolean | true | true | Enable the OTLP source to accept telemetry data over gRPC. |
| gRPC Hostname | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) where gRPC data is received. Defaults to your pod’s IP. |
| Port (gRPC) | Int | 4317 | true | The port for receiving OTLP data over gRPC. |
### Advanced Options
| Field | Type | Default | Required | Description | |
| :- | :- | :- | :- | :- | - |
| Forward X-Scope-OrgID Header | Boolean | true | true | When enabled, the incoming X-Scope-OrgID header is preserved. This is often required for multi-tenant authentication to work properly. | |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# Prometheus Source
Source: https://docs.sawmills.ai/docs/source-prometheus
Configure the Prometheus Remote-Write source in Sawmills to receive metrics data with optional X-Scope-OrgID header forwarding for multi-tenancy.
## Supported Data Types
📈 **Metrics**
## Configuration
| Field | Type | Required | Default Value | Description |
| :- | :- | :- | :- | :- |
| Name | String | true | none | Unique identifier within Sawmills |
| Address | String | true | `${env:MY_POD_IP}` | Address to which the Prometheus Remote-Write source should bind. |
| Port | String | true | 9090 | Port to which the Prometheus Remote-Write source should bind. |
### Advanced Options
| Field | Type | Default | Required | Description | |
| :- | :- | :- | :- | :- | - |
| Forward X-Scope-OrgID Header | Boolean | true | true | When enabled, the incoming X-Scope-OrgID header is preserved. This is often required for multi-tenant authentication to work properly. | |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# Splunk HEC Source
Source: https://docs.sawmills.ai/docs/source-splunk-hec
Configure the Splunk HEC source in Sawmills to ingest logs, metrics, and traces via the Splunk HTTP Event Collector protocol with TLS support.
## Supported Data Types
📘 **Logs** | 📈 **Metrics** | 🚦 **Traces**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) to receive Splunk data. |
| Port | Int | 8088 | true | The port on which Splunk data is received. |
: The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# Syslog Source
Source: https://docs.sawmills.ai/docs/source-syslog
Configure the Syslog source in Sawmills to receive log data via TCP-based syslog using RFC 5424 or RFC 3164 formats with configurable encoding.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) to receive TCP Logs data. |
| Port | Int | 514 | true | The port on which TCP Logs data is received.. |
| Encoding | String | UTF-8 | true | The character encoding to use for parsing syslog messages. If no encoding is selected, no encoding validation occurs and the file is treated as a stream of raw bytes. |
| Protocol | String | RFC 5424 | true | Syslog protocol version that will be used (RFC 5424 or RFC 3164) |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
Currently we only support syslog over TCP
# TCP Logs Source
Source: https://docs.sawmills.ai/docs/source-tcplog
Configure the TCP Logs source in Sawmills to receive raw log data over a TCP connection on a configurable address, port, and encoding.
## Supported Data Types
📘 **Logs**
## Configuration
| Field | Type | Default | Required | Description |
| :- | :- | :- | :- | :- |
| Name | String | none | true | Unique identifier within Sawmills. |
| Address | String | `${env:MY_POD_IP}` | true | The address (IP or hostname) to receive TCP Logs data. |
| Port | Int | 54525 | true | The port on which TCP Logs data is received.. |
The pod’s IP address is retrieved by defining an environment variable `MY_POD_IP` from the pod’s status field `status.podIP`. You can access this value using `${env:MY_POD_IP}` in your configuration to dynamically reference the pod’s IP at runtime.
# Obtaining Tokens
Source: https://docs.sawmills.ai/docs/tokens
Exchange a long-lived Sawmills API key for a short-lived JWT access token and use it to authenticate REST API requests to Sawmills platform services.
Access tokens are short-lived credentials that provide authenticated access to Sawmills services. This guide explains how to exchange your API key for an access token and use it for authentication.
## Overview
Sawmills uses a token exchange system where you can exchange a long-lived API key for a short-lived access token. This provides better security by limiting the exposure time of credentials.
**Token Exchange Flow:**
1. Use your API key to request an access token
2. Use the access token to authenticate with other Sawmills services
3. Refresh the token as needed (tokens expire after 30 minutes by default)
## Exchanging API Key for Token
### API Endpoint
```http theme={null}
POST /v1/token
```
### Authentication
Use your API key in the Authorization header with the `ApiKey` scheme:
```http theme={null}
Authorization: ApiKey sk_test.abc123def4.xyz789uvw0123456789abcdef
```
### Request Body
The request body should be empty:
```json theme={null}
{}
```
### Response
```json theme={null}
{
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
"token_type": "Bearer",
"expires_in": 1800
}
```
**Response Fields:**
* `access_token`: JWT token for authenticating with other services
* `token_type`: Always "Bearer" for Sawmills tokens
* `expires_in`: Token expiration time in seconds (default: 1800 = 30 minutes)
### Example: Exchanging API Key for Token with cURL
```bash theme={null}
curl -X POST "https://api.sawmills.ai/v1/token" \
-H "Authorization: ApiKey sk_test.abc123def4.xyz789uvw0123456789abcdef" \
-H "Content-Type: application/json" \
-d '{}'
```
### Example: Exchanging API Key for Token with JavaScript
```javascript theme={null}
const response = await fetch('https://api.sawmills.ai/v1/token', {
method: 'POST',
headers: {
'Authorization': 'ApiKey sk_test.abc123def4.xyz789uvw0123456789abcdef',
'Content-Type': 'application/json'
},
body: JSON.stringify({})
});
const data = await response.json();
console.log('Access token:', data.access_token);
console.log('Expires in:', data.expires_in, 'seconds');
```
## Using Access Tokens
### Authentication Header
Use the access token in the Authorization header with the `Bearer` scheme:
```http theme={null}
Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...
```
### Example: Using Token with cURL
```bash theme={null}
curl -X GET "https://api.sawmills.ai/v1/user-api-keys" \
-H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
```
### Example: Using Token with JavaScript
```javascript theme={null}
const response = await fetch('https://api.sawmills.ai/v1/user-api-keys', {
method: 'GET',
headers: {
'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...'
}
});
const data = await response.json();
console.log('API Keys:', data.api_keys);
```
## Token Management
### Token Expiration
* **Default TTL**: 30 minutes (1800 seconds)
* **Maximum TTL**: 24 hours (configurable by organization)
* **Refresh Required**: Tokens cannot be refreshed; you must exchange a new token
### Automatic Token Refresh
Implement automatic token refresh in your applications:
```javascript theme={null}
class SawmillsClient {
constructor(apiKey) {
this.apiKey = apiKey;
this.accessToken = null;
this.tokenExpiry = null;
}
async getAccessToken() {
// Check if current token is still valid
if (this.accessToken && this.tokenExpiry && Date.now() < this.tokenExpiry) {
return this.accessToken;
}
// Exchange API key for new token
const response = await fetch('https://api.sawmills.ai/v1/token', {
method: 'POST',
headers: {
'Authorization': `ApiKey ${this.apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({})
});
if (!response.ok) {
throw new Error('Failed to obtain access token');
}
const data = await response.json();
this.accessToken = data.access_token;
this.tokenExpiry = Date.now() + (data.expires_in * 1000);
return this.accessToken;
}
async makeRequest(url, options = {}) {
const token = await this.getAccessToken();
return fetch(url, {
...options,
headers: {
...options.headers,
'Authorization': `Bearer ${token}`
}
});
}
}
// Usage
const client = new SawmillsClient('sk_test.abc123def4.xyz789uvw0123456789abcdef');
const response = await client.makeRequest('https://api.sawmills.ai/v1/user-api-keys');
```
## Error Handling
### Common Error Responses
**401 Unauthorized - Invalid API Key**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "invalid api key credentials"
}
```
**401 Unauthorized - Revoked API Key**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "api key revoked"
}
```
**401 Unauthorized - Invalid Format**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "api key invalid"
}
```
**401 Unauthorized - Missing Authorization**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "authorization header required"
}
```
**401 Unauthorized - Invalid Scheme**
```json theme={null}
{
"code": "UNAUTHENTICATED",
"message": "authorization header must use ApiKey scheme"
}
```
**500 Internal Server Error - Token Issuance Failed**
```json theme={null}
{
"code": "INTERNAL",
"message": "failed to mint access token"
}
```
### Handling Token Errors
```javascript theme={null}
async function handleTokenExchange(apiKey) {
try {
const response = await fetch('https://api.sawmills.ai/v1/token', {
method: 'POST',
headers: {
'Authorization': `ApiKey ${apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({})
});
if (!response.ok) {
const error = await response.json();
switch (error.code) {
case 'UNAUTHENTICATED':
if (error.message.includes('revoked')) {
throw new Error('API key has been revoked. Please create a new one.');
} else if (error.message.includes('invalid')) {
throw new Error('Invalid API key. Please check your credentials.');
} else {
throw new Error('Authentication failed. Please check your API key.');
}
case 'INTERNAL':
throw new Error('Server error. Please try again later.');
default:
throw new Error(`Unexpected error: ${error.message}`);
}
}
return await response.json();
} catch (error) {
console.error('Token exchange failed:', error.message);
throw error;
}
}
```
## Security Considerations
### Token Storage
* Store access tokens in memory only (not persistent storage)
* Never log or expose access tokens
* Implement secure token refresh mechanisms
### API Key Security
* Keep your API key secure and never expose it in client-side code
* Use environment variables or secure key management systems
* Rotate API keys regularly
### Network Security
* Always use HTTPS for token exchange and API calls
* Validate SSL certificates
* Consider using certificate pinning for additional security
## Troubleshooting
### Common Issues
**"Authorization header required"**
* Ensure you're including the Authorization header in your request
* Check that the header is properly formatted
**"Authorization header must use ApiKey scheme"**
* Use `ApiKey` (not `Bearer`) when exchanging API keys for tokens
* Format: `Authorization: ApiKey YOUR_API_KEY`
**"Invalid API key credentials"**
* Verify your API key is correct and active
* Check that the API key hasn't been revoked
* Ensure the API key is correctly formatted
**"Failed to mint access token"**
* This is a server-side error
* Check Sawmills service status
* Contact support if the issue persists
### Debugging Tips
1. **Verify API Key**: Ensure your API key is correctly formatted
2. **Check Token Expiry**: Verify the token hasn't expired before using it
3. **Test with cURL**: Use cURL to test token exchange independently
4. **Review Logs**: Check application logs for detailed error information
## Best Practices
### Application Design
* Implement automatic token refresh
* Handle token expiration gracefully
* Use connection pooling for better performance
* Implement retry logic with exponential backoff
### Error Handling
* Always check response status codes
* Implement proper error handling for all failure cases
* Log errors for debugging (without exposing sensitive data)
* Provide meaningful error messages to users
### Performance
* Cache tokens in memory to avoid unnecessary exchanges
* Use appropriate timeouts for API calls
* Implement circuit breakers for resilience
* Monitor token exchange success rates
## Next Steps
Now that you understand how to obtain and use tokens, you can:
1. [Create API Keys](/docs/api-keys) if you haven't already
2. Integrate token-based authentication into your applications
3. Set up monitoring and alerting for your API usage
# Invite Users
Source: https://docs.sawmills.ai/docs/user-invite
Invite users to your Sawmills organization based on the configured invitation method, assign Admin or Member roles, and manage existing team members.
1. If your organization has chosen the [Automatic Invitations](/docs/configure-invitation-method) method, you will not need to manually enter an email address. users from your organization will be able to log in to the system automatically.
2. If the `No Automatic Enrollment` is selected, follow these steps to invite users
* **Access the Organization Profile**:
* Navigate to the `Organization Profile` section under the user icon located on the top right side of the platform.
![]()
* **Click on the Invite Button**:
* Within the Organization Profile, locate and click the `Invite` button (under the Members tab).
![]()
* You will be prompted to enter the user's email address.
* Choose the appropriate role for the user:
* **Admin**: This role provides the user with administrative privileges.
* **Member**: This role provides the user with standard access rights.
![]()
1. **Send the Invitation**:
* Once you've entered the necessary information, click `Send Invitation`.
* The user will receive an invitation email with instructions on how to complete their registration (if applicable).
1. **Remove user**
* To remove a user, navigate to the `Members Section` in the `Organization Profile` section. Click on the 3 dots all the way to the right of the user name. Click on the 3 dots and choose the **Remove member** option
# User Management
Source: https://docs.sawmills.ai/docs/user-management
Manage users in your Sawmills organization with Admin and Member roles, three invitation methods, and controls for editing or removing team members.
## User Management
### Overview
User Management allows administrators to manage user accounts, permissions, and access levels. It ensures that the right people have the right access to the right resources at the right time, enhancing both security and productivity.
### Features
#### 1. User Roles and Permissions
User roles define what a user can or cannot do within the platform. Each role comes with a set of permissions that grant or restrict access to specific features or data.
* **Admin**: Has full access to all features, including user management.
* **Member**: Can access and interact with resources according to their permissions. For now the main difference is that administrators are the only ones that can invite more users to the organization.
#### 2. User Invitation and Enrollment
Our platform provides three methods for inviting and managing new users:
* **No Automatic Enrollment**: Administrators manually invite users by sending invitations through email. Users must accept the invitation to create an account.
* **Automatic Invitations**: New users are automatically invited based on predefined criteria, such as their email domain or membership in an organization.
* **Automatic Suggestions**: The system suggests potential users based on their interactions with the platform. Administrators can review and approve these suggestions to send invitations.
### Getting Started
To begin managing users, follow these steps:
1. **Define Invitation Method**
* See [Invitation Methods](/docs/configure-invitation-method) to define the preferred way of inviting users.
2. **Invite Users**:
* Go to the **Organization** section and click on **Members**.
* Click on "Add Member" to invite new users. Fill in the necessary details, including their email address and role.
* Users will receive an invitation via email and must follow the provided instructions to join the platform.
3. **Manage Existing Users**:
* View, edit, or remove users from the Members list. You can also adjust roles and permissions as needed.
# Vulnerability Disclosure
Source: https://docs.sawmills.ai/docs/vulnerability-disclosure
How to report security vulnerabilities affecting Sawmills and Mills.
## Report a security issue
If you discover a suspected vulnerability or security incident affecting Sawmills or Mills (Agentmills), email [security@sawmills.ai](mailto:security@sawmills.ai).
This reporting process covers our products, services, infrastructure, and the Sawmills Slack app, including its Mills agent, integrations, and handling of Slack data.
## What to include
To help us investigate, please include:
* A description of the issue and where you found it.
* Steps to reproduce it and its potential impact.
* The date and time you observed it.
* Relevant URLs and redacted screenshots or logs.
* Contact information for follow-up, if you wish.
Do not include passwords, API keys, access tokens, or other people's personal or confidential data in your initial report. We can coordinate an appropriate way to share additional evidence if needed.
## What to expect
Our security team will review your report, assess the issue, and provide updates as the investigation progresses.
Please coordinate disclosure with us and allow a reasonable opportunity to investigate and resolve the issue before sharing details publicly.
## Responsible research
Limit investigation to the minimum necessary to demonstrate the issue. Avoid accessing or modifying other customers' data, disrupting services, or attempting social engineering. If you encounter sensitive data, stop and report what you observed without collecting further data.
## Bug bounty
We do not currently offer a bug bounty program or monetary rewards. Security reports are welcome at [security@sawmills.ai](mailto:security@sawmills.ai).
# Enable Webhook Integration
Source: https://docs.sawmills.ai/docs/webhook-integration
Configure the Sawmills Webhook integration to receive Insight notifications and automated action updates at a custom HTTP endpoint for your team.
## About Webhook Integration
The Webhook Integration allows you to keep up to date with new Insights for your data pipelines.
Configuring the integration will make sure you'll know about recent insights and their impact on your system.
You will also get a updates about Insight actions taken by yourself and your team, such as following the Insight recommendation or archiving the Insight.
## Step-by-Step: Configure Webhook Integration
Follow these steps to configure your Webhook Integration:
### 1. Access Integrations
Open the user menu at the top right and select **Integrations**.
![]()
### 3. Open Webhook Configuration
After completing the authentication, locate and open the **Configure** Webhook drawer.
### 4. Enter the Webhook URL
Enter the Webhook URL for which you want to get notifications.
### 5. Test and Save
Click the test button to make sure the integration work and save your changes.