SSH Health Monitoring

The default healthcheck for the sshd container only verifies that SSH accepts a connection and answers Permission denied. That probe stays green while SSH wrapper and git transport processes accumulate inside the container, which is the degradation mode that leads to host-wide memory exhaustion: the container keeps accepting new connections and keeps reporting itself healthy right up to the point where the host runs out of RAM.

rc-ssh-healthcheck inspects what the container is actually running instead. It counts rc-ssh-wrapper-v2 and git transport processes, measures the age of the oldest one, and sums their resident memory. When a configured threshold is exceeded, it exits non-zero, which lets Docker mark the container unhealthy and lets a supervisor such as autoheal recycle it before the host is affected.

The command reads /proc directly. It has no dependency on ps or on any other package being present in the image, and it keeps working while the application itself is degraded.

Reporting current usage

Run the command with --report to see the current numbers and every matched process. In this mode it always exits 0, so it is safe to run during an incident:

docker exec rc_cluster_apps-sshd-1 rc-ssh-healthcheck --report

wrappers=3 git=2 max_age=412s total_rss=284MB
  pid=8121 kind=wrapper age=412s rss=96.4MB /home/rhodecode/venv/bin/python .../rc-ssh-wrapper-v2 ...
  pid=8140 kind=git age=406s rss=61.2MB git -c pack.threads=2 upload-pack /var/opt/...

Use --json for machine readable output, for example when feeding a monitoring system.

Thresholds

Thresholds are read from the environment. Every threshold is disabled by default. A correct value depends on repository sizes and on how much SSH concurrency the installation expects, and a probe that recycles the container on a wrong guess is worse than no probe at all. Size the thresholds from the numbers that --report shows on your own instance under normal load.

Variable

Meaning

RC_SSH_HEALTH_MAX_WRAPPERS

Maximum number of rc-ssh-wrapper processes.

RC_SSH_HEALTH_MAX_GIT_PROCESSES

Maximum number of git-upload-pack / git-receive-pack / git-pack-objects processes.

RC_SSH_HEALTH_MAX_PROCESS_AGE_SECONDS

Maximum age of the oldest SSH wrapper or git process.

RC_SSH_HEALTH_MAX_RSS_MB

Maximum combined resident memory of all of the above.

A value that is empty, zero, negative or non-numeric disables that threshold. A threshold fails only when the measured value is strictly greater than the limit.

As a starting point, take the peak values observed over a normal week and roughly double them. For the age threshold, use at least twice ssh.git.operation_timeout_seconds (default 3600): with that timeout in place, a wrapper older than two hours is stuck rather than busy.

RC_SSH_HEALTH_MAX_WRAPPERS=40
RC_SSH_HEALTH_MAX_GIT_PROCESSES=60
RC_SSH_HEALTH_MAX_PROCESS_AGE_SECONDS=7200
RC_SSH_HEALTH_MAX_RSS_MB=4096

Enable one threshold at a time and watch the result before adding the next one. The RSS threshold is usually the one that correlates best with host-level impact.

Wiring it into the healthcheck

Keep the existing connection probe and add the process probe after it, so that both “SSH is not answering” and “SSH is drowning in stuck processes” are reported as unhealthy:

sshd:
  healthcheck:
    test: >-
      /usr/bin/ssh -p$${RC_SSH_PORT} -o StrictHostKeyChecking=no
      -o PasswordAuthentication=No rhodecode@localhost true 2>&1
      | grep -c 'Permission denied' > /dev/null
      && { command -v rc-ssh-healthcheck > /dev/null 2>&1 || exit 0; }
      && rc-ssh-healthcheck

The command -v guard keeps the healthcheck working against images that predate the probe, so pinning an older RC_VERSION under a newer compose file cannot turn a healthy container unhealthy.

If the probe itself fails, for example because /proc cannot be read, it reports healthy and writes the reason to stderr. This is deliberate: a broken probe must not take a working container down. Pass --strict if you want the opposite behaviour.

Incident runbook

Follow these steps when SSH-related memory growth is suspected. Collect the diagnostics before restarting anything — a restart destroys the evidence, and the state before the restart is the only thing that identifies the cause.

  1. Identify the affected replicas. When sshd is scaled out, processes can accumulate in every replica, so check each one:

    docker ps --filter name=sshd --format '{{.Names}}'
for c in $(docker ps --filter name=sshd --format '{{.Names}}'); do
    echo "== $c"; docker exec "$c" rc-ssh-healthcheck --report
done

  2. Capture the process state and container metrics:

    docker exec <sshd-container> rc-ssh-healthcheck --report --json > ssh-health-pre.json
docker stats --no-stream > docker-stats-pre.txt
docker logs --since 2h <sshd-container> > sshd-logs-pre.txt

  3. Generate a diagnostic package, and note that the pre-restart one is the important artifact. Generate a second one after the restart for comparison.

  4. Identify the source of the traffic. The --report output includes the --user and --key-id arguments of each wrapper, which is usually enough to attribute the load to a single automation account.

  5. Restart only the affected sshd replica:

    docker restart <sshd-container>

  6. Reduce the pressure while the cause is being investigated. Moving a high-volume automation client from SSH to HTTPS is usually the quickest mitigation, and the effect is visible immediately in the wrapper count.

Optional container limits can protect the host, at the cost of interrupting SSH operations when they are hit. Set them only with that tradeoff understood, and size mem_limit above the largest legitimate clone the instance serves:

sshd:
  mem_limit: 4g
  pids_limit: 512