Is Large Page support on Linux only Transparent Huge Pages?

[RogerThiede]

I wanted to benchmark the difference (if any) between Stockfish using transparent huge pages on Linux versus reserved huge pages on Linux when provided 0.5 TB RAM or larger.

README.md:
Support on Linux - Large page support on Linux is obtained by the Linux kernel transparent huge pages functionality. Typically, transparent huge pages are already enabled, and no configuration is needed.

I was able to confirm through /proc/meminfo that AnonHugePages (Transparent Huge Pages) were being utilized by Stockfish when I setoption name Hash value 500000. However, transparent huge pages feature only supports 2 MB pages, and I am curious if larger than 2MB page sizes provide any added benefit when using 0.5 TB or 1 TB RAM. Huge Pages however can be customized to various sizes as opposed to Transparent Huge Pages which are 2MB.

Following instructions from Configuring Huge Pages, I am able to disable transparent huge pages, and reserve many huge pages of 1GB size during boot, but Stockfish never appears to consume them or the OS never seems to allocate them to Stockfish. Of course 1GB huge pages may be unrealistic, but I wanted to benchmark various sizes to discover any sweet spots. Stockfish terminates with Killed after setoption name Hash value 500000.

Is Large Page support on Linux only Transparent Huge Pages, and limited to 2MB huge pages?

[vondele]

yes, I think it is, but by way of an experiment might be not too difficult to change, the code is simple

Stockfish/src/misc.cpp

Lines 447 to 462 in 44b1ba8

	void* aligned_large_pages_alloc(size_t allocSize) {
	#if defined(__linux__)
	constexpr size_t alignment = 2 * 1024 * 1024; // assumed 2MB page size
	#else
	constexpr size_t alignment = 4096; // assumed small page size
	#endif
	// round up to multiples of alignment
	size_t size = ((allocSize + alignment - 1) / alignment) * alignment;
	void *mem = std_aligned_alloc(alignment, size);
	#if defined(MADV_HUGEPAGE)
	madvise(mem, size, MADV_HUGEPAGE);
	#endif
	return mem;
	}