System Administration¶

Apache as subdirectory¶

Use the following example to configure Apache as a sub-directory.

<Location /<someprefix> > # Change <someprefix> into your chosen prefix
  ProxyPass http://127.0.0.1:5000/<someprefix>
  ProxyPassReverse http://127.0.0.1:5000/<someprefix>
  SetEnvIf X-Url-Scheme https HTTPS=1
</Location>

In addition to the regular Apache setup you will need to add the following lines into the production.ini file:

1. In the the [app:main] section of your production.ini file add the following line.

   filter-with = proxy-prefix
   

2. At the end of the production.ini file add the following section.

   [filter:proxy-prefix]
   use = egg:PasteDeploy
   prefix = /<someprefix> # Change <someprefix> into your chosen prefix
   

Apache virtual host reverse proxy example¶

Here is a sample configuration file for using Apache as a reverse proxy.

<VirtualHost *:80>
        ServerName hg.myserver.com
        ServerAlias hg.myserver.com

        <Proxy *>
          Order allow,deny
          Allow from all
        </Proxy>

        #important !
        #Directive to properly generate url (clone url) for pylons
        ProxyPreserveHost On

        #rhodecode instance
        ProxyPass / http://127.0.0.1:5000/
        ProxyPassReverse / http://127.0.0.1:5000/

        #to enable https use line below
        #SetEnvIf X-Url-Scheme https HTTPS=1

</VirtualHost>

Apache WSGI configuration¶

RhodeCode Enterprise can also be set up with Apache under mod_wsgi. To configure this use the following steps.

1. Install mod_wsgi using the following command: aptitude install libapache2-mod-wsgi.

   Note

   If using a Debian-based distro, install he package libapache2-mod-wsgi

2. Enable mod_wsgi using the following command: a2enmod wsgi

3. Create a wsgi dispatch script, using the following examples.

   WSGIDaemonProcess pylons \
   threads=4 \
   # check the python virtual env location
   python-path=/home/web/rhodecode/pyenv/lib/python2.6/site-packages
   # Check the install location
   WSGIScriptAlias / /home/web/rhodecode/dispatch.wsgi
   WSGIPassAuthorization On
   # user=www-data group=www-data # Enable if running Apache as root
   

The following is an example wsgi dispatch script.

import os
os.environ["HGENCODING"] = "UTF-8"
os.environ['PYTHON_EGG_CACHE'] = '/home/web/rhodecode/.egg-cache'

# Set the current dir
os.chdir('/home/web/rhodecode/')

import site
site.addsitedir("/home/web/rhodecode/pyenv/lib/python2.6/site-packages")

from paste.deploy import loadapp
from paste.script.util.logging_config import fileConfig

fileConfig('/home/web/rhodecode/production.ini')
application = loadapp('config:/home/web/rhodecode/production.ini')

Nginx tuning¶

Set the following properties in your /etc/nginx/nginx.conf so it does not timeout during large pushes.

proxy_redirect              off;
proxy_set_header            Host $host;

## needed for container auth
# proxy_set_header            REMOTE_USER $remote_user;
# proxy_set_header            X-Forwarded-User $remote_user;

proxy_set_header            X-Url-Scheme $scheme;
proxy_set_header            X-Host $http_host;
proxy_set_header            X-Real-IP $remote_addr;
proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header            Proxy-host $proxy_host;
proxy_buffering             off;
proxy_connect_timeout       7200;
proxy_send_timeout          7200;
proxy_read_timeout          7200;
proxy_buffers               8 32k;
client_max_body_size        1024m; # Set this to a larger number if you experience timeouts
client_body_buffer_size     128k;
large_client_header_buffers 8 64k;

Increase Database Performance¶

To increase database performance switch to database-based user sessions. File-based sessions are only suitable for smaller setups. The most common issue being file limit errors which occur if there are lots of session files. Therefore, in a large scale deployment, to give better performance, scalability, and maintainability we recommend switching from file-based sessions to database-based user sessions.

To switch to database-based user sessions uncomment the following section in your data/production.ini file.

# db session
beaker.session.type = ext:database
# adjust this property to include your database credentials
beaker.session.sa.url = postgresql://postgres:<pass>@localhost/rhodecode
beaker.session.table_name = db_session

Increase Cache Size¶

When managing hundreds of repositories from the main RhodeCode Enterprise interface the system can become slow when the cache expires. Increasing the cache expiration option improves the response times of the main user interface. To increase your cache size, change the following default value in the data/production.ini file. The value is specified in seconds.

beaker.cache.sql_cache_long.expire=3600  # one day (86400) or a one week (604800)

Note

The RhodeCode Enterprise cache automatically expires for changed repositories.

Move tmp to HGFS¶

HGFS refers to Host Guest File Sharing, a feature of VMWare’s virtualization software. RhodeCode Enterprise components heavily use the /tmp folder, so moving your /tmp folder into to a RAM-based HGFS can lead to a noticeable performance boost.

Change Default Encoding¶

RhodeCode Enterprise uses utf8 encoding by default. You can change the default encoding in the production.ini file. To change the default encoding used by RhodeCode Enterprise, set a new value for the default_encoding.

## default encoding used to convert from and to unicode
## can be also a comma seperated list of encoding in case of mixed encodings
default_encoding = utf8

Note

Changing the default encoding will affect many parts of your RhodeCode Enterprise installation, including committers names, file names, and the encoding of commit messages. RhodeCode Enterprise can detect if the chardet library is installed. If the chardet library is detected, RhodeCode Enterprise will fallback to using it if there are encode/decode errors.

Scale Horizontally¶

Horizontal scaling means adding more machines into your pool of resources. Horizontally scaling RhodeCode Enterprise gives a huge performance increase, especially under large traffic scenarios with a high number of requests. This is very beneficial when RhodeCode Enterprise is serving many users simultaneously, or if continuous integration servers are automatically pulling and pushing code.

To horizontally scale RhodeCode Enterprise you should use the following steps:

1. In the data/production.ini file, set instance_id = *. This enables RhodeCode Enterprise to use multiple nodes. You must set this in the [app:main] section of the production.ini file.
2. Define the number of worker threads using the formula (2 * NUM_CPU) - 1. For example 4 CPU cores would lead to 2*4-1 = 7 workers.
3. For Posix environments we recommend using Gunicorn as it has a built-in support for multiple workers and load balancing. To enable Gunicorn use the following steps.

• Comment out the following section from the [server:main] area in the default production.ini file.

  # WAITRESS
  use = egg:waitress#main
  # number of worker threads
  threads = 5
  # MAX BODY SIZE 100GB
  max_request_body_size = 107374182400
  # use poll instead of select, fixes fd limits, may not work on old
  # windows systems.
  asyncore_use_poll = True
  

• Insert the following replacement code in the [server:main] section.

  # GUNICORN
  # run with gunicorn --log-config <inifile.ini> --paste <inifile.ini>
  use = egg:gunicorn#main
  # is set to more than one worker, recommended value is (2*NUMBER_OF_CPUS+1)
  workers = 5
  # process name
  proc_name = rhodecode
  # type of worker class, one of sync, eventlet, gevent, tornado
  # recommended for bigger setup is using of of other than sync one
  worker_class = sync
  # max number of requests that worker will handle before beeing gracefully
  # restarted, could prevent memory leaks
  max_requests = 1000
  # amount of time a worker can spend with handling a request before it
  # gets killed and restarted
  timeout = 36000
  

• Insert the following replacement code in the [app:main] section.

  # In the [app:main] section
  [app:main]
  # You must set `instance_id = *`
  instance_id = *
  

• Use the following command to run the RhodeCode Enterprise server with Gunicorn: gunicorn --log-config <inifile.ini> --paste <inifile.ini>

4. On Windows you can use multiple paster commands. To start multiple RhodeCode Enterprise servers and then load balance them with an Nginx or Apache web server, use the following steps:

1. Add the following paster variable to the production.ini file

[server:main]
host = 0.0.0.0
port = %(port_var)s

2. Run multiple RhodeCode Enterprise servers using the following command, but specifying different ports for each running version, for example

paster serve example.ini port_var=5000
paster serve example.ini port_var=5001

Note

You should configure your load balancing accordingly. We recommend writing load balancing rules that will separate regular user traffic from automated process traffic like continuous servers or build bots.

If you scale across different machines, each RhodeCode Enterprise instance needs to store its data on a shared disk, preferably together with your repositories. This data directory contains template caches, a whoosh index, and is used for task locking to ensure safety across multiple instances. To do this, set the following properties in the production.ini file to set the shared location across all RhodeCode Enterprise instances.

cache_dir = /file/path                   # set to shared directory location
index_dir = /file/path                   # set to shared directory location
beaker.cache.data_dir = /file/path       # set to shared directory location
beaker.cache.lock_dir = /file/path       # set to shared directory location

Note

If Celery is used on each instance then you should run separate Celery instances, but the message broker should be the same for all of them. This excludes one RabbitMQ shared server.

Reset Admin Account Privileges¶

If you accidentally remove your admin privileges from the admin account you can restore them using ishell. Use the following example to reset your account.

# Where filename.ini is your production .ini
paster ishell filename.ini
# where username is your admin username
you = User.get_by_username('username')
you.admin = True
Session().add(you);Session().commit()

Set to read global .hgrc file¶

By default, RhodeCode Enterprise does not read global hgrc files in /etc/mercurial/hgrc or /etc/mercurial/hgrc.d because it can lead to issues. This is set in the rhodecode_ui table for which there is no UI. If you need to edit this you can manually change the settings using SQL statements with ishell. Use the following example to make changes to this table.

paster ishell production.ini
In [4]: new_option = RhodeCodeUi()
In [5]: new_option.ui_section='web'
In [6]: new_option.ui_key='allow_push'
In [7]: new_option.ui_value='*'
In [8]: Session().add(new_option);Session().commit()

Removing User Problems¶

If you try to remove a user that has any of the following properties you may run into an error:

• open pull requests
• repository owner
• requested as code reviewer within a pull request

To remove a user with any of these properties you must first ensure that all their open pull requests or code review requests are closed. Once these are closed, reassign their repository groups or user groups to a different user. You can then remove that user without errors. To do this use the following steps:

1. Install ipython inside the RhodeCode Enterprise virtualenv

2. With ipython installed you can use ishell to assign a new owner to a repository. Once the repository group has a new owner you can then remove that user from RhodeCode Enterprise. To reassign repository group or user group ownership, use the following example:

   # start ishell
   paster ishell production.ini
   # Display all user groups owned by USERNAME
   In [5]: target_groups = UserGroup.query().filter(UserGroup.user_id==USERNAME).all()
   # Change all user groups to NEWUSER ownership
   In [6]: for ug in target_groups:
       ....:     ug.user_id = NEWUSER
       ....:     Session().add(ug)
       ....:     session().commit()
   

Manually Reset Password¶

If you need to manually reset a user password, use the following steps.

1. Navigate to your RhodeCode Enterprise install location, and install ipython.
2. Run the interactive RhodeCode Enterprise prompt
3. Set a new password

Use the following code example to carry out these steps.

cd ~/rhodecode
# installs ipython
system/bin/pip install ipython
# starts the ishell interactive prompt
system/bin/paster ishell ~/rhodecode/data/production.ini
from rhodecode.lib.utils2 import generate_api_key
from rhodecode.lib.auth import get_crypt_password
# Enter the user name whose password you wish to change
my_user = 'USERNAME'
u = User.get_by_username(my_user)
# If this fails then the user does not exist
u.api_key = generate_api_key(my_user)
# Set the new password
u.password = get_crypt_password('PASSWORD')
Session().add(u)
Session().commit()
exit