Logging Configuration

How RhodeCode Logging Works

RhodeCode uses a centralized metrics and logging stack:

Metrics Collection:

  • RhodeCode apps send metrics via StatsD to statsd-exporter

  • Prometheus scrapes metrics from statsd-exporter, node-exporter, and traefik

  • Grafana visualizes metrics via dashboards (accessible at /_grafana)

Log Aggregation:

  • Docker Loki Driver captures all container logs and sends to Loki server

  • Loki aggregates and stores logs (part of optional metrics stack)

  • Promtail scrapes host system logs from /var/log

  • Grafana provides log browsing interface

Local Log Files:

Docker maintains local log file copies for the docker logs command. By default, these grow indefinitely.

Configuring Log Retention

Global Configuration

File: docker-compose-base.yaml

Uncomment the log rotation settings:

x-logging: &custom-logging
  driver: loki
  options:
    loki-url: "http://${RC_LOKI_AUTH}127.0.0.1:3100/loki/api/v1/push"
    loki-retries: "5"
    loki-timeout: "1s"
    loki-max-backoff: "800ms"
    max-size: "100m"  # Uncomment
    max-file: "10"    # Uncomment (~1GB total per service)

Restart the stack to apply changes

Per-Service Configuration

File: .custom/docker-compose-apps.override.yaml

Override specific services for custom retention:

services:
  vcsserver:
    logging:
      driver: loki
      options:
        loki-url: "http://${RC_LOKI_AUTH}127.0.0.1:3100/loki/api/v1/push"
        loki-retries: "5"
        loki-timeout: "1s"
        loki-max-backoff: "800ms"
        max-size: "200m"
        max-file: "10"  # ~2GB for heavy indexing

File: .custom/docker-compose-services.override.yaml

services:
  redis:
    logging:
      driver: loki
      options:
        loki-url: "http://${RC_LOKI_AUTH}127.0.0.1:3100/loki/api/v1/push"
        loki-retries: "5"
        loki-timeout: "1s"
        loki-max-backoff: "800ms"
        max-size: "50m"
        max-file: "5"   # ~250MB for minimal logging

Apply changes:

Verifying Configuration

# Check container logging config
docker inspect -f '{{.HostConfig.LogConfig}}' rc_cluster_apps-vcsserver-1

# Expected output includes:
# {loki map[... max-file:10 max-size:100m ...]}

# Check disk usage
LOKI_ID=$(docker plugin inspect loki -f '{{.Id}}')
sudo du -sh /var/lib/docker/plugins/${LOKI_ID}/rootfs/var/log/docker/

Alternative: System-Level Log Rotation

Instead of Docker-level log rotation, you can use system logrotate to manage Docker container logs.

When to use this approach:

  • You prefer system-level log management

  • You want to apply rotation to all containers regardless of logging driver

  • You need more control over compression and log cleanup

Configuration:

Create /etc/logrotate.d/docker:

/var/lib/docker/containers/*/*.log {
    rotate 10
    size 100M
    copytruncate
    compress
    missingok
    notifempty
}

This rotates logs when they reach 100MB, keeping 10 rotated copies (compressed). Both approaches can coexist.