Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.sawmills.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Configuring a Load Balancer layer

The Sawmills collector supports an optional load balancing layer that sits between your telemetry sources and the backend collector processing layer. This architecture provides enhanced resilience, scalability, and efficient resource utilization by decoupling data ingestion from data processing.

​
Architecture Overview

The load balancing layer introduces a two-tier architecture:
  1. Load Balancer Layer: Receives telemetry data from all sources (Datadog Agent, Fluent, OTLP, etc.), stores it in a queue (in-memory or storage-based), and distributes it to backend collectors.
  2. Backend Collector Layer: Processes telemetry data, applies processors (filtering, masking, aggregation, etc.), and forwards processed data to destinations (Datadog, Splunk, S3, etc.). 
Telemetry Sources -> Load Balancer (Queue) -> Backend Collectors -> Destinations

​
Benefits

​
1. Improved Resilience and Fault Tolerance

  • Queue Buffering: The load balancer queue acts as a buffer, preventing data loss during backend collector restarts, deployments, or temporary failures.
  • Decoupled Operations: Sources continue sending data even when backend collectors are being updated or scaled, ensuring zero downtime deployments.

​
2. Enhanced Scalability with KEDA Integration

When combined with the KEDA scaler, the load balancing layer enables intelligent, metric-driven autoscaling:
  • Queue-Based Scaling: KEDA monitors the queue size and utilization metrics from the load balancer
  • Proactive Scaling: Backend collectors scale up before the queue becomes full, preventing backpressure
  • Cost Optimization: Scale down when queue utilization is low, reducing resource consumption during low-traffic periods
  • Dynamic Response: Automatically adjusts to traffic patterns without manual intervention

​
3. Better Resource Utilization

  • Load Distribution: Evenly distributes telemetry data across multiple backend collector instances
  • Independent Scaling: Load balancer and backend collectors can be scaled independently based on different metrics
  • Resource Efficiency: Backend collectors can focus on processing without handling direct client connections

​
4. Operational Flexibility

  • Zero-Downtime Updates: Update backend collectors without interrupting data ingestion
  • Traffic Spikes Handling: Queue absorbs sudden traffic spikes, giving backend collectors time to scale up
  • Monitoring and Observability: Clear separation of concerns makes it easier to monitor and troubleshoot each layer

​
How It Works with KEDA

The KEDA scaler monitors the load balancer’s queue metrics using Prometheus queries. When queue utilization exceeds the target threshold, KEDA scales up the backend collector replicas. When utilization drops below a lower threshold, it scales down to optimize costs. The scaler queries metrics like:
  • otelcol_exporter_queue_size: Current number of items in the queue
  • otelcol_exporter_queue_capacity: Maximum queue capacity
Based on these metrics, KEDA makes scaling decisions to maintain optimal queue utilization, ensuring the backend collectors can process data efficiently without queue overflow.

​
Configuration

​
Basic Setup

To enable the load balancing layer, add the following configuration to your values.yaml: 
managedChartsValues:
  sawmills-collector:
    loadBalancer:
      enabled: true

​
Setup with KEDA Autoscaling

For optimal performance, combine the load balancer with KEDA autoscaling. This configuration enables queue-based scaling of the backend collector layer: 
managedChartsValues:
  sawmills-collector:
    # Enable load balancing layer
    loadBalancer:
      enabled: true

    # Configure KEDA-based autoscaling
    keda:
      enabled: true
      minReplicas: 1
      maxReplicas: 10
      pollingInterval: 30
      cooldownPeriod: 300
      scaling:
        external:
          enabled: true
          metricType: AverageValue
          metadata:
            query: sum(otelcol_exporter_queue_size{exporter=~"loadbalancing/collector-loadbalancer.*"})
            targetValue: "10"

    # Enable KEDA scaler component
    kedaScaler:
      enabled: true

​
Configuration Parameters

​
Load Balancer

  • loadBalancer.enabled: Enable or disable the load balancing layer (default: false)

​
KEDA Configuration

  • keda.enabled: Enable KEDA autoscaling (default: false)
  • keda.minReplicas: Minimum number of backend collector replicas (default: 1)
  • keda.maxReplicas: Maximum number of backend collector replicas (default: 10)
  • keda.pollingInterval: Interval in seconds for KEDA to check metrics (default: 30)
  • keda.cooldownPeriod: Cooldown period in seconds before scaling down (default: 300)

​
External Scaler Metadata

  • scalerAddress: Address of the KEDA OTLP scaler service. The chart default derives this from the generated scaler service name and release namespace.
  • query: PromQL query that monitors queue utilization metrics
  • targetValue: Target queue utilization percentage to maintain
  • metricType: Type of metric. Use AverageValue for backend collector scaling from load balancer queue metrics.
This ensures the backend collector layer maintains optimal capacity to process queued telemetry data without overflow.

​
Service Names and Routing

If you set resourceBaseName, the chart uses that value as the Kubernetes base name. If you leave it empty, the chart uses the Helm release name. With the default base name sawmills-collector, load balancer mode creates these resources:
ResourceNameUse
Source-facing servicesawmills-collectorSend telemetry sources here when the load balancer is enabled
Load balancer workloadsawmills-collector-lbReceives telemetry and forwards to backend collectors
Backend collector servicesawmills-collector-backendService for backend collector pods
Backend collector headless servicesawmills-collector-backend-headlessUsed by the load balancer to route to backend collector pods
Load balancer headless servicesawmills-collector-headlessUsed for load balancer pod discovery when enabled
KEDA scaler servicesawmills-collector-keda-otel-scalerUsed by KEDA and collector self-telemetry
When load balancer mode is enabled, configure Datadog Agent, OTLP sources, Fluent, Vector, and other telemetry sources to send data to the source-facing service. Do not route source traffic directly to the backend headless service.

​
Prerequisites

Before setting up the load balancer with KEDA:
  1. Kubernetes Cluster: A running Kubernetes cluster
  2. KEDA Installed: KEDA must be installed in your cluster (see KEDA External Scaler for installation instructions)
  3. Helm 3.x: For deploying the Sawmills collector

​
Next Steps

  • Learn more about KEDA configuration in the KEDA External Scaler documentation
  • Configure telemetry sources to send data to the load balancer endpoint
  • Monitor queue metrics and scaling behavior through your observability platform
  • Adjust scaling thresholds based on your traffic patterns and requirements