Skip to main content

​
Configuring a Load Balancer layer

The Sawmills collector supports an optional load balancing layer that sits between your telemetry sources and the backend collector processing layer. This architecture provides enhanced resilience, scalability, and efficient resource utilization by decoupling data ingestion from data processing.

​
Architecture Overview

The load balancing layer introduces a two-tier architecture:
  1. Load Balancer Layer: Receives telemetry data from all sources (Datadog Agent, Fluent, OTLP, etc.), stores it in a queue (in-memory or storage-based), and distributes it to backend collectors.
  2. Backend Collector Layer: Processes telemetry data, applies processors (filtering, masking, aggregation, etc.), and forwards processed data to destinations (Datadog, Splunk, S3, etc.). 
Telemetry Sources → Load Balancer (Queue) → Backend Collectors → Destinations
     (Datadog,          (Buffering &          (Processing &        (Datadog,
      Fluent, etc.)      Distribution)         Transformation)       Splunk, etc.)

​
Benefits

​
1. Improved Resilience and Fault Tolerance

  • Queue Buffering: The load balancer queue acts as a buffer, preventing data loss during backend collector restarts, deployments, or temporary failures.
  • Decoupled Operations: Sources continue sending data even when backend collectors are being updated or scaled, ensuring zero downtime deployments.

​
2. Enhanced Scalability with KEDA Integration

When combined with the KEDA scaler, the load balancing layer enables intelligent, metric-driven autoscaling:
  • Queue-Based Scaling: KEDA monitors the queue size and utilization metrics from the load balancer
  • Proactive Scaling: Backend collectors scale up before the queue becomes full, preventing backpressure
  • Cost Optimization: Scale down when queue utilization is low, reducing resource consumption during low-traffic periods
  • Dynamic Response: Automatically adjusts to traffic patterns without manual intervention

​
3. Better Resource Utilization

  • Load Distribution: Evenly distributes telemetry data across multiple backend collector instances
  • Independent Scaling: Load balancer and backend collectors can be scaled independently based on different metrics
  • Resource Efficiency: Backend collectors can focus on processing without handling direct client connections

​
4. Operational Flexibility

  • Zero-Downtime Updates: Update backend collectors without interrupting data ingestion
  • Traffic Spikes Handling: Queue absorbs sudden traffic spikes, giving backend collectors time to scale up
  • Monitoring and Observability: Clear separation of concerns makes it easier to monitor and troubleshoot each layer

​
How It Works with KEDA

The KEDA scaler monitors the load balancer’s queue metrics using Prometheus queries. When queue utilization exceeds the target threshold, KEDA scales up the backend collector replicas. When utilization drops below a lower threshold, it scales down to optimize costs. The scaler queries metrics like:
  • otelcol_exporter_queue_size: Current number of items in the queue
  • otelcol_exporter_queue_capacity: Maximum queue capacity
Based on these metrics, KEDA makes scaling decisions to maintain optimal queue utilization, ensuring the backend collectors can process data efficiently without queue overflow.

​
Configuration

​
Basic Setup

To enable the load balancing layer, add the following configuration to your values.yaml: 
managedCharts:
  sawmills-collector:
    loadBalancer:
      enabled: true

​
Setup with KEDA Autoscaling

For optimal performance, combine the load balancer with KEDA autoscaling. This configuration enables queue-based scaling of the backend collector layer: 
managedCharts:
  sawmills-collector:
    # Enable load balancing layer
    loadBalancer:
      enabled: true

    # Configure KEDA-based autoscaling
    keda:
      enabled: true
      minReplicas: 1
      maxReplicas: 10
      pollingInterval: 30
      cooldownPeriod: 300
      scaling:
        external:
          enabled: true
          metricType: Value
          metadata:
            scalerAddress: sawmills-collector-keda-otel-scaler.elad.svc.cluster.local:4418
            query: max((otelcol_exporter_queue_size{exporter=~".*loadbalancing/collector-loadbalancer.*"}
              / otelcol_exporter_queue_capacity{exporter=~".*loadbalancing/collector-loadbalancer.*"})
              * 100) > 20 or max((otelcol_exporter_queue_size{exporter=~".*loadbalancing/collector-loadbalancer.*"}
              / otelcol_exporter_queue_capacity{exporter=~".*loadbalancing/collector-loadbalancer.*"})
              * 100) < 5 or vector(10)
            targetValue: "10"

    # Enable KEDA scaler component
    kedaScaler:
      enabled: true

​
Configuration Parameters

​
Load Balancer

  • loadBalancer.enabled: Enable or disable the load balancing layer (default: false)

​
KEDA Configuration

  • keda.enabled: Enable KEDA autoscaling (default: false)
  • keda.minReplicas: Minimum number of backend collector replicas (default: 1)
  • keda.maxReplicas: Maximum number of backend collector replicas (default: 10)
  • keda.pollingInterval: Interval in seconds for KEDA to check metrics (default: 30)
  • keda.cooldownPeriod: Cooldown period in seconds before scaling down (default: 300)

​
External Scaler Metadata

  • scalerAddress: Address of the KEDA OTLP scaler service (format: service.namespace.svc.cluster.local:port)
  • query: PromQL query that monitors queue utilization metrics
  • targetValue: Target queue utilization percentage to maintain
  • metricType: Type of metric (Value for absolute values)
This ensures the backend collector layer maintains optimal capacity to process queued telemetry data without overflow.

​
Prerequisites

Before setting up the load balancer with KEDA:
  1. Kubernetes Cluster: A running Kubernetes cluster
  2. KEDA Installed: KEDA must be installed in your cluster (see KEDA External Scaler for installation instructions)
  3. Helm 3.x: For deploying the Sawmills collector

​
Next Steps

  • Learn more about KEDA configuration in the KEDA External Scaler documentation
  • Configure telemetry sources to send data to the load balancer endpoint
  • Monitor queue metrics and scaling behavior through your observability platform
  • Adjust scaling thresholds based on your traffic patterns and requirements