Full-text Search¶

Full-text search

RhodeCode provides a full text search capabilities to search inside file content, commit message, and file paths. Indexing is not enabled by default and to use full text search building an index is a pre-requisite.

RhodeCode Enterprise provides support for ElasticSearch 8 as a backend for advanced and scalable search.

Previously RhodeCode was also supporting alternative Whoosh backend to index repositories and provide full-text search, but as of version 5.6.0 Whoosh is deprecated. ElasticSearch 8 is only available for EE users.

Auth Token generation¶

RhodeCode indexer runs on top of RhodeCode Enterprise API and requires an Authentication Token before continuing. To run the indexer you need to have an Authentication Token with admin rights to all of repositories that indexer should process.

To get your API Token, on the RhodeCode Enterprise interface go to Click on the icon with your user in top right corner your-username ‣ My Account ‣ Auth tokens

Put a description for the Authentication Token
Select expiration date if desired
Select api calls role for the token
Click Add
Click on the obfuscated generated token, and copy it.

Indexing¶

To index repositories stored in RhodeCode, you have the option to set the indexer up in a number of ways, for example:

Call the indexer via a cron job. We recommend running this once at night. In case you need everything indexed immediately it’s possible to index few times during the day. Indexer has a special locking mechanism that won’t allow two instances of indexer running at once. It’s safe to run it even every 1hr.
Hook the indexer up with your CI server to reindex after each push.
Set the indexer to infinitely loop and reindex as soon as it has run its previous cycle. This allows to get an instance indexing of content that would be available seconds after changes happen.

The indexer works by indexing new commits added since the last run, and comparing file changes to index only new or modified files across each invocation.

Note

If you wish to build a brand new index from scratch each time, use the force option in the configuration file, or run it with –force flag.

To set up indexing, use the following steps:

Configure the .rhoderc File
Run the Indexer
Schedule the Indexer
Advanced Indexing

Configure the `.rhoderc` File¶

Note

Optionally it’s possible to use indexer without the .rhoderc. Simply instead of executing with –instance-name=rc-idx execute providing the host and token directly: –api-host=https://your-host.example.com –api-key=<auth-token-goes-here>

Note

In some cases the domain could be only available via the custom DNS, you can always refer the instance by it’s docker name and port (http://rhodecode:10020) instead of hostname, for example:

./rcstack cli cmd rhodecode-index --api-host=http://rhodecode:10020 --api-key=xxx

Indexer uses the /home/user/.rhoderc file for connection details to RhodeCode Enterprise instances. You need to configure the details for each instance you want to index.

./rcstack cli cmd rhodecode-setup-config \
--filename=/vol/confvolume/.rhoderc \
--instance-name=rc-idx api_host=https://your-host.example.com,api_key=<auth-token-goes-here>

Here’s an example generated config you might also mount as a file to the docker image.

# Configure .rhoderc with matching details
# This allows the indexer to connect to the instance
[instance:rc-idx]
api_host = https://your-host.example.com
api_key = <auth token goes here>

Run the Indexer¶

Run the indexer using the following command, and specify the instance you want to index:

# Using default simples indexing of all repositories
$ ./rcstack cli cmd rhodecode-index \
   --no-tty --config=/vol/confvolume/.rhoderc \
   --instance-name=rc-idx

# Using a custom mapping file and invocation without ``.rhoderc``
$ ./rcstack cli cmd rhodecode-index \
   --no-tty \
   --api-host=https://your-host.example.com --api-key=xxxxx \
   --mapping=/vol/confvolume/search_mapping.ini

# Using a custom mapping file with indexing rules, and using elasticsearch 6 backend
$ ./rcstack cli cmd rhodecode-index \
   --no-tty --config=/vol/confvolume/.rhoderc \
   --instance-name=rc-idx \
   --mapping=/vol/confvolume/search_mapping.ini \
   --es-version=6 --engine-location=http://elasticsearch:9200

# For some advanced usage, please check --help flag to see what other CLI options are available
``$ ./rcstack cli cmd rhodecode-index --help``

Note

In case of often indexing using Whoosh backend the index may become fragmented. Most often a result of that is error about too many open files. To fix this indexer needs to be executed with –optimize flag. E.g

$ ./rcstack cli cmd rhodecode-index --instance-name=rc-idx --optimize
This should be executed regularly, once a week is recommended. When using ElasticSearch this step can be skipped.

Schedule the Indexer¶

To schedule the indexer, configure the crontab file to run the indexer inside your RhodeCode Tools virtualenv using the following steps.

Open the crontab file, using crontab -e.
Add the indexer to the crontab, and schedule it to run as regularly as you wish.
Save the file.

$ crontab -e

# The virtualenv can be called using its full path, so for example you can
# put this example into the crontab

# Run the indexer daily at 4am using the default mapping settings, --no-tty is required for non interactive calls
0 4 * * * ./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx

# Run the indexer every Sunday at 3am using default mapping
0 3 * * 0 ./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx

# Run the indexer every 15 minutes using a custom mapping file
*/15 * * * * ./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx --mapping=/vol/confvolume/search_mapping.ini

Advanced Indexing¶

Force Re-Indexing single repository¶

Often it’s required to re-index whole repository because of some repository changes, or to remove some indexed secrets, or files. There’s a special –repo-name= flag for the indexer that limits execution to a single repository. For example to force-reindex single repository such call can be made

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx --force --repo-name=rhodecode-vcsserver

Limiting indexing to small number of repos¶

Often to preserve memory usage and system load we might limit the number of repositories processed on each call. There’s a special –repo-limit= flag for the indexer that limits execution to a N repositories.

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx --repo-limit=10

Removing repositories from index¶

A regular index run only processes repositories that currently exist in RhodeCode Enterprise and match the mapping rules. It does not remove documents that are already stored in the index. Documents of repositories that were deleted, renamed (the old name), or later excluded by search_mapping.ini rules stay in the index until they are removed explicitly using one of the commands below.

It’s important to distinguish between:

Excluding a repository from future indexing - add a = 0 rule to the [__INDEX_RULES__] section. Existing documents are kept.
Removing already indexed documents - use --remove-only for a single repository, or --prune for all repositories that are no longer eligible for indexing.
Deleting a whole ElasticSearch index - the rhodecode_commit_8 and rhodecode_file_8 indexes (rhodecode_commit_9 and rhodecode_file_9 for ElasticSearch 9) are shared by all repositories. Never delete those indexes to clean up a single repository. Deleting them removes the search data of every repository and requires a full re-index.

Remove a single repository¶

To remove all indexed commits and files of one repository run the indexer with --remove-only. The --repo-name flag is required, --remove-only without it is rejected.

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx \
  --es-version=8 --engine-location=http://elasticsearch:9200 \
  --remove-only --repo-name=rhodecode-vcsserver

The documents are removed using the native ElasticSearch Delete By Query API, separately from the commit and file index, followed by a verification that no document of the repository is left. Running it again for an already removed repository succeeds and reports zero removed documents.

Prune repositories no longer eligible for indexing¶

--prune-dry-run and --prune compare all repository names stored in both the commit and the file index with the repositories that a full index run would process:

without explicit [repo-name] sections: all repositories reported by the server, filtered by the [__INDEX_RULES__] rules;
with explicit [repo-name] sections in the mapping file: the sections act as an allow-list, only listed repositories that exist on the server are used, and the [__INDEX_RULES__] rules are still applied to them. A listed repository matching a = 0 rule is not eligible. All other indexed repositories become prune candidates. [conf:repo-name] sections only configure a repository and do not add it to the allow-list.

Repositories present in the index but not eligible are stale: deleted repositories, old names of renamed repositories, and repositories excluded by mapping rules. The --repo-limit setting never makes repositories stale, it only applies to index builds.

Always run the dry-run first, with exactly the same mapping file as the prune:

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx \
  --es-version=8 --engine-location=http://elasticsearch:9200 \
  --mapping=/vol/confvolume/search_mapping.ini --prune-dry-run

The dry-run does not modify the index. It prints one line per stale repository, with the number of its commit and file documents, sorted by name, and a summary:

PRUNE-CANDIDATE group/deleted-repository commits=1532 files=4821
PRUNE-CANDIDATE old-group/renamed-repository commits=12 files=40
PRUNE-SUMMARY indexed=1200 eligible=1198 stale=2 commits=1544 files=4861

After reviewing the candidates, remove them with --prune:

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx \
  --es-version=8 --engine-location=http://elasticsearch:9200 \
  --mapping=/vol/confvolume/search_mapping.ini --prune

PRUNED group/deleted-repository commits=1532 files=4821
PRUNED old-group/renamed-repository commits=12 files=40
PRUNE-SUMMARY repositories=2 commits=1544 files=4861 failures=0

The command exits with a non-zero exit code if any repository could not be removed completely: an ElasticSearch timeout, failures, a version conflict, or documents left after the delete. Version conflicts are not retried automatically. Stop the competing writer and run --prune again, it is safe to repeat.

Prune is an explicit action and is never executed by a regular index run. It cannot be combined with --repo-name, --remove-only, --force, --optimize, --show-matched-repos, --source, --repo-limit or --repeat, and --prune cannot be combined with --prune-dry-run. Prune also cannot be combined with --create-mapping. --remove-only cannot be combined with --create-mapping, --force, --optimize or --show-matched-repos.

Note

Prune requires an Authentication Token of a super-admin. The indexer always calls the super-admin only get_server_info API first, and stops before touching ElasticSearch if that call fails. This guarantees the list of repositories used for the comparison is complete, a token that can see only some repositories would otherwise mark all others as stale.

Note

Prune always works with live API responses. --prune and --prune-dry-run are rejected before any API or ElasticSearch call when an API response cache is configured, either as api_cache / api_cache_only in the .rhoderc instance section or with the --api-cache / --api-cache-only flags. A cached repository list may be outdated and would make existing repositories look deleted.

The inventory of indexed repositories is paginated, so it is not limited by the number of repositories in the index, and missing indexes are never created by --prune, --prune-dry-run or --remove-only.

Concurrency¶

Index builds, --prune and --prune-dry-run share the same lock, so they never run at the same time. When the lock is held by another indexer process the command exits with Lock Held. Don’t run manual ElasticSearch _delete_by_query calls or other index writers in parallel with a scheduled index job. Pause the cron job while cleaning up manually.

Verifying and recovering¶

To verify that a repository is gone, count its documents in both indexes. The repository field is an exact keyword, the result should be 0:

curl -s -H 'Content-Type: application/json' http://elasticsearch:9200/rhodecode_commit_8/_count \
  -d '{"query": {"term": {"repository": "group/deleted-repository"}}}'
curl -s -H 'Content-Type: application/json' http://elasticsearch:9200/rhodecode_file_8/_count \
  -d '{"query": {"term": {"repository": "group/deleted-repository"}}}'

Use a JSON aware tool (for example jq) to read responses, instead of parsing pretty printed JSON with grep and sed. For listing indexed repositories use --prune-dry-run, or a paginated composite aggregation on the repository field, a plain terms aggregation silently stops at its size limit.

If a repository was pruned by mistake, for example because of a wrong mapping file, fix the mapping and run the indexer for it again. The repository is re-indexed from scratch:

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx \
  --es-version=8 --engine-location=http://elasticsearch:9200 --repo-name=group/repository

For a full rollback of the index data, take an ElasticSearch snapshot before running --prune on production data.

Using search_mapping.ini file for advanced index rules¶

By default rhodecode-index runs for all repositories, all files with parsing limits defined by the CLI default arguments. You can change those limits by calling with different flags such as –max-filesize=2048kb or –repo-limit=10

For more advanced execution logic it’s possible to use a configuration file that would define detailed rules which repositories and how should be indexed.

To create the search_mapping.ini file manually, use the below command

./rcstack cli cmd rhodecode-index --config=/vol/confvolume/.rhoderc --instance-name=rc-idx \
--create-mapping=/vol/confvolume/search_mapping.ini

Now the indexer can be executed with –mapping flag

Here’s a detailed example of using search_mapping.ini file.

[__DEFAULT__]
; Create index on commits data, and files data in this order. Available options
; are `commits`, `files`
index_types = commits,files

; Commit fetch limit. In what amount of chunks commits should be fetched
; via api and parsed. This allows server to transfer smaller chunks and be less loaded
commit_fetch_limit = 1000

; Commit process limit. Limit the number of commits indexer should fetch, and
; store inside the full text search index. eg. if repo has 2000 commits, and
; limit is 1000, on the first run it will process commits 0-1000 and on the
; second 1000-2000 commits. Help reduce memory usage, default is 50000
; (set -1 for unlimited)
commit_process_limit = 20000

; Limit of how many repositories each run can process, default is -1 (unlimited)
; in case of 1000s of repositories it's better to execute in chunks to not overload
; the server.
repo_limit = -1

; Default patterns for indexing files and content of files. Binary files
; are skipped by default.

; Add to index those comma separated files; globs syntax
; e.g index_files = *.py, *.c, *.h, *.js
index_files = *,

; Do not add to index those comma separated files, this excludes
; both search by name and content; globs syntax
; e.g index_files = *.key, *.sql, *.xml, *.pem, *.crt
skip_files = ,

; Add to index content of those comma separated files; globs syntax
; e.g index_files = *.h, *.obj
index_files_content = *,

; Do not add to index content of those comma separated files; globs syntax
; Binary files are not indexed by default.
; e.g index_files = *.min.js, *.xml, *.dump, *.log, *.dump
skip_files_content = ,

; Force rebuilding an index from scratch. Each repository will be rebuild from
; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo
force = false

; maximum file size that indexer will use, files above that limit are not going
; to have they content indexed.
; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB
max_filesize = 10MB


[__INDEX_RULES__]
; Ordered match rules for repositories. A list of all repositories will be fetched
; using API and this list will be filtered using those rules.
; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include
; When this ordered list is traversed first match will return the include/exclude marker
; For example:
;    upstream/binary_repo = 0
;    upstream/subrepo/xml_files = 0
;    upstream/* = 1
;    special-repo = 1
;    * = 0
; This will index all repositories under upstream/*, but skip upstream/binary_repo
; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches


; == EXPLICIT REPOSITORY INDEXING ==
; If defined, only repositories listed as [NAME] sections are indexed (an allow-list),
; instead of all repositories fetched using the API. The __INDEX_RULES__ above are still
; applied to the listed repositories, a listed repository matching a `= 0` rule is not
; indexed. [conf:NAME] sections don't add a repository to this list.
; --prune and --prune-dry-run treat every indexed repository that is not listed, or is
; excluded by the rules, as stale. To build index just for repo_name_1 and special-repo use:
;    [repo_name_1]
;    [special-repo]

; == PER REPOSITORY CONFIGURATION ==
; This allows overriding the global configuration per repository.
; example to set specific file limit, and skip certain files for repository special-repo
; the CLI flags doesn't override the conf settings.
;    [conf:special-repo]
;    max_filesize = 5mb
;    skip_files = *.xml, *.sql

In case of 1000s of repositories it can be tricky to write the include/exclude rules at first. There’s a special flag to test the mapping file rules and list repositories that would be indexed. Run the indexer with –show-matched-repos to list only the matching repositories defined in .ini file rules

./rcstack cli cmd rhodecode-index --no-tty --config=/vol/confvolume/.rhoderc --instance-name=rc-idx --show-matched-repos --mapping=/vol/confvolume/search_mapping.ini

Enabling ElasticSearch¶

ElasticSearch is available in EE edition only. It provides scalable and advanced search capabilities and advanced query language. It allows advanced filtering by file paths, extensions, use OR statements, ranges etc. Please check query language examples in the search field for some advanced query language usage.

Open the rhodecode.ini file for the instance you wish to edit. The default location is config/_shared/rhodecode.ini
Find the search configuration section:

###################################
## SEARCH INDEXING CONFIGURATION ##
###################################

search.module = rhodecode.lib.index.whoosh
search.location = %(here)s/data/index

and change it to:

search.module = rc_elasticsearch
search.location = http://elasticsearch:9200
## specify Elastic Search version, 8 for latest
search.es_version = 8

where search.location points to the ElasticSearch server by default running on port 9200.

Index invocation also needs change. Please provide –es-version= and –engine-location= parameters to define ElasticSearch server location and it’s version. For example:

rhodecode-index --instance-name=enterprise-1 --es-version=8 --engine-location=http://elasticsearch:9200

Optimizing for Large Repositories¶

When indexing large repositories (thousands of repositories or repositories with extensive history), you may need to optimize the VCS server configuration to handle the increased load during indexing operations.

Edit your VCS server Gunicorn configuration file config/_shared/gunicorn_conf_vcs.py and adjust the following settings:

# Maximum number of requests a worker will process before restarting
# This helps prevent memory leaks during heavy indexing operations
max_requests = 5000

# Add randomness to max_requests to prevent all workers restarting simultaneously
max_requests_jitter = 500

# Increase the socket backlog for handling more concurrent connections
# during intensive indexing operations
backlog = 128

# Restart VCS server to apply new configuration
./rcstack stack rhodecode restart

ElasticSearch Tuning:

For very large repositories with complex nested structures, you may also need to tune ElasticSearch itself. Add the following environment variable to your ElasticSearch service in the docker-compose file:

elasticsearch:
  environment:
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    - "index.mapping.nested_objects.limit=50000"

ElasticSearch Data Migration (4.x to 5.x)¶

Warning

ElasticSearch Index Incompatibility: When upgrading from RhodeCode 4.x to 5.x, ElasticSearch indices created in version 4.x are not compatible with version 5.x.

Required Action: Delete the old ElasticSearch data volume and restart the stack to create a fresh index:

# Stop the services stack
./rcstack stack services down

# Remove the old ElasticSearch data volume
docker volume rm rc_cluster_services_es_data

# Restart the services stack
./rcstack stack services up -d

# Verify ElasticSearch is running properly
./rcstack status

After completing this step, you will need to rebuild your search index using the indexing commands described above.

Full-text Search¶

Auth Token generation¶

Indexing¶

Configure the .rhoderc File¶

Run the Indexer¶

Schedule the Indexer¶

Advanced Indexing¶

Force Re-Indexing single repository¶

Limiting indexing to small number of repos¶

Removing repositories from index¶

Remove a single repository¶

Prune repositories no longer eligible for indexing¶

Concurrency¶

Verifying and recovering¶

Using search_mapping.ini file for advanced index rules¶

Enabling ElasticSearch¶

Optimizing for Large Repositories¶

ElasticSearch Data Migration (4.x to 5.x)¶

Configure the `.rhoderc` File¶