Secure60 Balancer

Overview

The Secure60 Balancer is a high-performance TCP load balancer specifically designed for handling long-lived TCP connections, such as those used in syslog ingestion. It solves critical challenges in high-volume log ingestion by providing intelligent connection management and load distribution across multiple Secure60 Collectors.

Key Features

• Intelligent TCP connection management
• Optional TCP connection rotation (disabled by default)
• Automatic health checking of backend collectors
• Least-request load balancing algorithm
• DNS-based service discovery
• Real-time connection monitoring and statistics

The Problem: Long-Lived TCP Connections

In high-volume log ingestion scenarios, particularly with syslog, several challenges arise:

1. Connection Imbalance: Traditional load balancers often distribute connections evenly, but this doesn’t account for varying traffic volumes per connection
2. Failure Handling: Long-lived TCP connections can mask backend failures
3. Load Distribution: Without proper health checking, some collectors may become overloaded while others remain underutilized
4. Health Monitoring: Traditional health checks may not detect issues with existing connections

How the Secure60 Balancer Solves These Challenges

1. Health Checking

The balancer continuously monitors backend health through:

• Active TCP health checks every 10 seconds
• Automatic removal of unhealthy backends
• Re-addition of recovered backends
• Circuit breaking to prevent cascading failures

2. Load Balancing Algorithm

Uses the Least-Request algorithm to:

• Distribute new connections to the least-loaded collector
• Consider both connection count and request volume
• Automatically adapt to changing traffic patterns

3. Optional Connection Rotation

The balancer can optionally rotate TCP connections after a configurable duration. This feature is disabled by default. When enabled, it:

• Forces clients to reconnect after the specified duration
• Helps distribute load more evenly across collectors
• Provides regular health checking of all connections

Deployment Guide

Docker Deployment

docker run --rm --name secure60/s60-balancer \
  --network s60-balancer_test-network \
  -e S60_FRONTEND_PORT=514 \
  -e S60_BACKEND_PORT=514 \
  -e S60_BACKENDS=collector1,collector2,collector3 \
  -p 514:514 -p 9901:9901 \
  s60-balancer

Docker Compose Deployment

services:
  s60-balancer:
    image: secure60/s60-balancer
    container_name: s60-balancer
    networks:
      - test-network
    environment:
      - S60_FRONTEND_PORT=514
      - S60_BACKEND_PORT=514
      - S60_BACKENDS=collector1,collector2,collector3
    ports:
      - "514:514"
      - "9901:9901"

Configuration Options

Environment Variables

Health Check Configuration

The balancer performs health checks with the following defaults:

• Interval: 10 seconds
• Timeout: 5 seconds
• Unhealthy threshold: 3 failures
• Healthy threshold: 2 successes

Monitoring and Statistics

The balancer provides real-time statistics through its admin interface (port 9901):

# View all statistics
curl localhost:9901/stats

# View active connections per backend
curl localhost:9901/stats | grep "cluster.secure60_service.*upstream_cx_active"

Best Practices

1. Connection Rotation

   • Keep connection rotation disabled by default (S60_MAX_CONNECTION_DURATION=0)
   • Only enable rotation if you have verified client reconnection behavior
   • Test thoroughly in a non-production environment first
   • Consider using a longer duration (e.g., 3600s) if rotation is needed

2. Backend Configuration

   • Deploy at least 3 collectors for high availability
   • Ensure collectors have sufficient capacity for peak loads
   • Monitor collector health through the admin interface

3. Network Configuration

   • Use a dedicated network for balancer-collector communication
   • Ensure sufficient bandwidth for peak traffic
   • Consider network latency when setting health check intervals

4. Monitoring

   • Regularly check connection statistics
   • Monitor for failed health checks
   • Watch for connection imbalance across collectors

Troubleshooting

Common Issues

1. Data Loss

   • Verify connection rotation is disabled (S60_MAX_CONNECTION_DURATION=0)
   • Check client reconnection behavior if rotation is enabled
   • Monitor for connection drops in logs

2. Load Imbalance

   • Verify Least-Request algorithm is working
   • Check for DNS resolution issues
   • Monitor individual collector loads

3. Health Check Failures

   • Verify network connectivity
   • Check collector availability
   • Review health check configuration

Debug Mode

Enable debug mode for detailed logging:

docker run --rm --name secure60/s60-balancer \
  -e S60_DEBUG_MODE=true \
  # ... other configuration ...
  s60-balancer

Support

For assistance with the Secure60 Balancer, contact our integrations team at integrations@secure60.io