Question about OpenGrok performance

[a-niec]

Hi,
I would like to share some information about OpenGrok performance on our environment.

Node description:

• CPU: Threadripper 3990X (64c 128t)
• RAM: 256GB
• Storage: XFS file system on HDD Raid ~900 MB/sec read/write
• OS: Ubuntu 20.04
• Source base: 360GB in 2.5M files

OpenGrok parameters:

• opengrok docker image – v 1.7.37
• java heap size 32G (-J=-Xmx32g)
• history disabled
• repo disabled
• lucene memory 256M

Typically indexing source base take ~2h50m.

But I noticed that CPU utilization is really small. I’ve been monitoring indexing in htop and dstat, and it shows 300-800% CPU usage so only small portion of Threadripper power (5%) is used. Clearly there is some bottleneck here.

I have question for you guys – what were the largest CPU load you saw when indexing?
Maybe you have some tips how to unlock full potential?

Best Regards,
Adrian

[vladak]

Hi Adrian ! Is this initial or incremental reindex ? If the former, then 3 hours for initial index of 360 GB of data/code is not so bad at all I'd say.

Also, I wonder how exactly you disabled the history. There is no tunable in the official image that would allow this.

Unless the container is configured with CPU limits, it should use all CPU power available. The indexer in the container is run via opengrok-sync, specifically:

opengrok/docker/sync.yml

Lines 11 to 15 in 8a7aa08

	- command: [opengrok-reindex-project, --printoutput,
	--jar, /opengrok/lib/opengrok.jar, -U, '%URL%', -P, '%PROJECT%', --,
	-r, dirbased, -G, -m, '256', --leadingWildCards, 'on',
	-c, /usr/local/bin/ctags, -U, '%URL%', -H, '%PROJECT%']
	limits: {RLIMIT_NOFILE: 1024}

(noting that in high parallelism cases the RLIMIT_NOFILE resource setting might be to low - maybe it should be dynamic).

This means that there is no thread cap imposed by the tooling. Consequently, the indexer will choose the parallelism level based on the number of available CPUs:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java

Lines 1145 to 1154 in 8a7aa08

	/**
	* Gets the value of {@link Configuration#getIndexingParallelism()} -- or
	* if zero, then as a default gets the number of available processors.
	* @return a natural number >= 1
	*/
	public int getIndexingParallelism() {
	int parallelism = syncReadConfiguration(Configuration::getIndexingParallelism);
	return parallelism < 1 ? Runtime.getRuntime().availableProcessors() :
	parallelism;
	}

When the indexer is run without history processing, only 2nd phase of the indexing will be executed. This phase is heavily parallelized.

That said, back to the Docker image. When projects are enabled, the opengrok-reindex-project tools, which really is just a simple wrapper of the java run of the indexer, will be executed for each project in parallel, like so:

opengrok-latest | 2022-11-04 11:01:30,906    DEBUG opengrok_tools.utils.commandsequence | command = 'opengrok-reindex-project --printoutput --jar /opengrok/lib/opengrok.jar -U http://localhost:8080/ -P Tomcat-10 -- -r dirbased -G -m 256 --leadingWildCards on -c /usr/local/bin/ctags -U http://localhost:8080/ -H Tomcat-10'

And again, each of these will run with as many threads as there are CPUs in the system, assuming the container has not been configured otherwise previously. These threads from distinct opengrok-reindex-project runs (within the same container or multiple containers) will be competing for the CPU. So, even if there is some project that would have disproportionately bigger amount of data to be processed to the other projects, any decrease of CPU usage should not be visible, because it has thread count set to number of CPUs. Of course, assuming the indexer is CPU bound.

One way to check this would be to determine the number of spawned threads and/or ctags processes (there is 1-1 correspondence) by the indexer while it is running. If there are no ctags processed spawned by the indexer process, the indexer has not reached the 2nd phase of the indexing yet.

For example, when I run the indexing on my laptop with 8 "CPUs" (cat /proc/cpuinfo | grep ^processor | wc -l says 8 both in the "global zone" and the container) and internal SSD, using the Docker image, once it gets past the history cache creation to the 2nd phase of the indexing, htop output looks like this when indexing a source root that has just a clone of the Linux kernel:

I confirmed the indexer parallelism level by running pgrep -f -l '^\/usr\/local\/bin\/ctags' -c in the container (reported 8 processes).

Anyhow, while this is definitely way above the 5% CPU utilization, it seems there might be some room for improvement, assuming the I/O does not block the indexer too much. Perhaps raising the thread count above the CPU count could help.

[a-niec]

Nice :)

so this is only change that result in speedup?

- new ScheduledThreadPoolExecutor(1, 
+ new ScheduledThreadPoolExecutor(indexingParallelism, 

If so I'll check this in my environment

[a-niec]

Ok, so I've been testing this change with 1 -> indexingParallelism in self-baked docker image.
It is spawning more OpenGrok-xref-writer threads.

But unfortunately there is no big improvement in time (2:50 -> 2:35).
About your results - did you clear page cache between tests ? eg. sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"
Second indexing on cached files is always faster.

I've tested many threads / lucene mem options.

I've even run indexing (smaller set 30GB) of sources on tmpfs and output to tmpfs - to eliminate any io latency.
It was a a bit better ~70% of normal (4x raid) time, but still - server didn't even heat up ;)

Are those Lucene Merging Thread can somehow impact indexing speed? They are always at 100% maybe they are capping indexers?

[vladak]

There was another change I did as part of PR #4093: 60b407e

In my case it made the CPU to be almost fully utilized during indexing. To measure the run times, I ran the indexing multiple times, so in each case the file-system caches should be heated up, although I did not verify.

[vladak]

The changes are part of the today released 1.7.38, so you can try official Docker images.

[a-niec]

I've re-tested latest image on ramdisk input -> ramdisk output
This images speaks for themself.

Tested on 30GB of sources. Finished in 0:03:25 (previously ~0:06:30). Theoretical (ramdisk) throughput is awesome.
Unbelievable what 2 simple variable changes can do to such complex project.

Starting from v1.7.38 OpenGrok is as fast as CPU and storage let him be ;)