Scaling to handle more traffic

Performance Tuning Overview

RhodeCode 5.x provides multiple scaling dimensions to optimize performance for your workload. The key areas to tune are:

  • Gunicorn workers and threads - Controls concurrent request handling per replica

  • Docker replicas - Horizontal scaling of services

For production deployments, especially after migrating from 4.x, comprehensive tuning of these settings is recommended to achieve optimal performance.

Scaling rhodecode app HTTP Traffic

Adjust the .custom/docker-compose-apps.override.yaml file to scale services horizontally:

services:
  rhodecode:
    deploy:
      replicas: 4

  vcsserver:
    deploy:
      replicas: 4

Scaling SSH Traffic

SSH workers can scale independently from HTTP traffic. For high SSH workloads, increase replicas for the sshd service.

Edit .custom/docker-compose-apps.override.yaml:

services:
  sshd:
    deploy:
      replicas: 4

Note

For production deployments with heavy SSH usage (e.g., many developers using git/hg over SSH), 4-7 replicas are recommended depending on load.

Scaling SVN Traffic

SVN operations can be resource-intensive. For deployments with many SVN repositories, scale the SVN service independently.

Edit .custom/docker-compose-apps.override.yaml:

services:
  svn:
    deploy:
      replicas: 4

Note

SVN operations can be particularly demanding. For large SVN deployments, 4-5 replicas are recommended depending on workload.

Scaling Celery Workers

Celery workers handle background tasks and can scale independently from HTTP traffic.

Edit .custom/docker-compose-apps.override.yaml:

services:
  celery:
    deploy:
      replicas: 4

  celery-beat:
    deploy:
      # Must be 1 to avoid duplicate scheduled tasks
      replicas: 1

Note

  • Celery runs with an autoscaler defined in docker-compose-apps.yaml as --autoscale=${RC_CELERY_MAX_WORKERS:-10},${RC_CELERY_MIN_WORKERS:-2}

  • With the shipped defaults this means 2 processes minimum, scaling up to 10 per replica. Set RC_CELERY_MAX_WORKERS / RC_CELERY_MIN_WORKERS to change it

  • Total capacity = replicas × RC_CELERY_MAX_WORKERS

  • For production: 4 worker replicas provide good balance

  • Important: celery-beat must always be 1 replica to avoid duplicate scheduled tasks

  • Every additional Celery replica or autoscale process can open its own database connections. Re-check the database connection budget before scaling Celery up

Scale without downtime:

./rcstack stack rhodecode up --detach --no-recreate --scale rhodecode=4 --scale vcsserver=4

Gunicorn Configuration

Edit config/_shared/gunicorn_conf_rc.py for rhodecode config and config/_shared/gunicorn_conf_vcs.py for vcsserver config:

# Enable port reuse for better load balancing
reuse_port = True

# Use threaded worker class
worker_class = "gthread"

# Workers handle concurrent requests
# VCS server typically needs around 50% more workers
workers = 4


# Threads per worker for concurrent request handling
threads = 5

# Maximum concurrent connections per worker
worker_connections = 30

Note

Recommended Configuration:

  • For production: 4 replicas with 4 workers and 5 threads for rhodecode

  • VCS server typically needs more workers (6 workers recommended)

  • Replicas of vcsserver and rhodecode should typically be equal

  • Total capacity = replicas × workers × threads

  • Each Gunicorn worker of the rhodecode service has its own main database connection pool and counts towards the database connection budget. VCSServer workers do not use the main RhodeCode database and are not part of that budget

  • Set worker_class = "gthread" and reuse_port = True for both rhodecode and vcsserver

  • Monitor CPU and memory usage to fine-tune these values

Additional Performance Parameters for Gunicorn

For production deployments, consider tuning these additional gunicorn parameters in both config/_shared/gunicorn_conf_rc.py and config/_shared/gunicorn_conf_vcs.py:

# Worker restart settings (prevents memory leaks)
max_requests = 2000
max_requests_jitter = int(max_requests * 0.2)  # Adds randomness to prevent simultaneous restarts

# Connection backlog (pending connections queue)
backlog = 64

# Worker timeout (default: 30s, RhodeCode default: 6hrs for long operations)
timeout = 21600

# Graceful restart timeout
graceful_timeout = 21600

# Keep-alive for client connections
keepalive = 2

Note

Key Parameters:

  • max_requests - Worker restarts after N requests (0 = disabled, default in gunicorn is 0)

  • max_requests_jitter - Randomizes restart timing using randint(0, jitter_value)

  • backlog - Max pending connections (default: 2048). Generally 64-2048 range

  • timeout - Worker silent timeout in seconds (default: 30s, 0 = disabled)

  • graceful_timeout - Time for workers to finish requests during restart (default: 30s)

  • keepalive - Keep-alive timeout for client connections (default: 2s, range: 1-5s)

Database Connection Budget

Every scaling dimension above multiplies the number of processes that can talk to the database. The database server enforces a global connection limit (max_connections in MySQL and PostgreSQL). When it is reached, new connections are rejected, for example with MySQL (1040, 'Too many connections'), and requests or background tasks fail. Re-check the budget below whenever you change replicas, Gunicorn workers or the Celery autoscale maximum.

Each process has its own connection pool. RhodeCode uses one SQLAlchemy engine with a connection pool per operating system process. Threads or greenlets inside one process share that pool, but pools are never shared between processes or replicas. Processes that use the main RhodeCode database are:

  • every Gunicorn worker of each rhodecode replica;

  • every Celery prefork process of each celery replica, up to RC_CELERY_MAX_WORKERS per replica;

  • the celery-beat process;

  • SSH sessions handled by the sshd replicas, each of which runs as a separate process for the duration of the connection.

VCSServer (vcsserver service) does not use the main RhodeCode database, so its workers and replicas are not part of this budget.

Pool ceiling versus real demand. The configured pool limits how many connections one process may open:

pool_ceiling_per_process = sqlalchemy.db1.pool_size + sqlalchemy.db1.max_overflow

In the shipped rhodecode.ini both keys are commented out, so the SQLAlchemy defaults apply (pool_size = 5, max_overflow = 10, ceiling 15). A process only checks out connections for work that runs concurrently inside it, so its real demand is bounded by both values:

effective_process_demand <= min(pool_ceiling_per_process, concurrent_db_users_in_process)

where concurrent_db_users_in_process is:

  • Gunicorn gthread worker: the number of threads;

  • Gunicorn gevent worker: the number of greenlets that use the database at the same time (bounded by worker_connections);

  • Celery prefork process: it runs one task at a time and usually holds one database connection;

  • celery-beat process and each SSH session process: usually one connection.

The expected peak for the whole deployment is then:

expected_peak =
    sum(process_count_for_runtime × effective_process_demand_for_runtime)

required_max_connections >=
    expected_peak
    + connections_of_other_database_clients
    + operational_reserve

where process_count_for_runtime is, for example, replicas × workers for the rhodecode service, replicas × RC_CELERY_MAX_WORKERS for Celery and the highest expected number of concurrent SSH sessions for sshd. connections_of_other_database_clients covers every other application, backup job, monitoring agent or ETL process that uses the same database server. operational_reserve keeps room for administrators, migrations and monitoring when the system is under load.

effective_process_demand describes typical concurrency, not a guarantee: code paths that open an additional session can briefly use more than one connection per process. Validate the estimate against measured high-water values (Max_used_connections and the connection counts per user from the queries below) taken during the busiest period, including scheduled night-time jobs.

Multiplying every process by the full pool_ceiling_per_process gives a deliberately pessimistic theoretical ceiling. It is useful to see the worst case allowed by the pool settings, but it overstates the demand of single-threaded processes such as Celery workers and SSH sessions, and should not be the only number used for sizing.

Keep in mind when reading these numbers:

  • pool_size is not the number of connections that are always held open. A pool opens connections only when concurrent work checks them out and keeps up to pool_size idle connections for reuse afterwards.

  • sqlalchemy.db1.pool_recycle only replaces connections older than the given number of seconds. It does not limit how many connections can be open at the same time.

  • Celery autoscaling adds processes under load, so bursts of background work, such as many scheduled tasks starting at the same time, can raise the number of connections quickly.

  • Do not raise the database max_connections without checking that the database server has the memory and CPU to serve that many connections. Lowering process counts or pool limits for a runtime is often the safer change.

Scheduled tasks and night-time load

If connection errors happen outside business hours, compare their timestamps with the scheduled work rather than with interactive traffic:

  • RhodeCode scheduled tasks (repository maintenance, remote pull/push synchronisation, session and archive cleanup, last-update synchronisation, update checks and any custom scheduled entries). Many entries scheduled for the same minute run concurrently and may make Celery autoscale to its maximum.

  • Jobs outside RhodeCode that use the same database server: backups, maintenance, monitoring checks, reporting or ETL jobs and other applications.

Check whether connections return to their usual level after the jobs finish. If they do, spread the schedules or reduce Celery concurrency. If the baseline keeps growing after every night, collect the diagnostics below and contact RhodeCode support.

Measuring connection usage on MySQL

The following queries return only aggregated numbers and do not expose SQL text, client hosts or database names. Run them while the problem is happening, before restarting services:

SHOW VARIABLES LIKE 'max_connections';

SHOW GLOBAL STATUS
WHERE Variable_name IN (
    'Threads_connected',
    'Threads_running',
    'Max_used_connections',
    'Connection_errors_max_connections',
    'Aborted_clients',
    'Aborted_connects'
);

SELECT
    USER,
    COMMAND,
    COUNT(*) AS connection_count,
    MAX(TIME) AS longest_seconds
FROM information_schema.PROCESSLIST
GROUP BY USER, COMMAND
ORDER BY connection_count DESC;

Database usernames may be anonymised before sharing the result, as long as RhodeCode connections can still be told apart from other clients. A large number of Sleep connections owned by the RhodeCode user points at pool capacity, many long running Query connections point at slow or blocked queries, and connections of other users point at other clients of the same server.

To see when RhodeCode scheduled tasks run, without exposing task arguments, repository names or URLs:

SELECT
    task_dot_notation,
    schedule_type,
    schedule_definition_json,
    COUNT(*) AS enabled_entries,
    MAX(schedule_last_run) AS latest_run
FROM schedule_entries
WHERE schedule_enabled = 1
GROUP BY task_dot_notation, schedule_type, schedule_definition_json
ORDER BY task_dot_notation, schedule_type;