zfs_arc_max init option silently ignored if value is less than default arc_c_min

[eric1894]

System information

Type	Version/Name
Distribution Name	Arch
Distribution Version	Rolling release
Linux Kernel	5.4.24-1-lts
Architecture	x86_64
ZFS Version	0.8.3-1
SPL Version	0.8.3-1

Attempting to set zfs_arc_max (in the module initialization options) to less than 1/32 of total RAM will always silently fail.

This appears to be due to the following check that it's less than arc_c_min:

zfs/module/zfs/arc.c

Line 7572 in 9bb3d57

(zfs_arc_max > arc_c_min)) {

The default value of arc_c_min is 1/32 of total RAM:

zfs/module/zfs/arc.c

Line 7820 in 9bb3d57

arc_c_min = MAX(allmem / 32, 2ULL << SPA_MAXBLOCKSHIFT);

It's not sufficient to also set zfs_arc_min to a lower value in the module initialization options, because the value of zfs_arc_min is handled after zfs_arc_max, by which time zfs_arc_max has already been ignored.

As well as fixing the issue, it would be good to improve the logging in arc_tuning_update() in general, rather than silently ignoring parameters set by the user.

[paulhargreaves]

I've validated on a machine with 256GB of ram. I can't set max_arc below (roughly) 8589934592. e.g. 8443623040 causes it to be ignored and set to 135097968640.

In my case I'd prefer only 5GB of RAM to be consumed since this isn't a fileserver...

[ghost]

I have just been looking in this area. I'll see what I can do.

[paulhargreaves]

(Daft question) - did this make it into any shipping releases? I've just upgraded to 2.0 and I'm still seeing 'WARNING: ignoring tunable zfs_arc_max (using 135095138304 instead)' on boot.

If there is a simple way to check what release a particular change went into, happy to be pointed at it to learn how to check myself in future.

[ghost]

Yes it is in the 2.0.0 release. The operative word here was "silently." The warning message you're seeing is part of what's new. Do you have zfs_arc_min set below zfs_arc_max?

The simple way to see what release a change went into is to find the commit on GitHub and look at the bottom of the box at the top of the page: 36a6e23

[paulhargreaves]

Thanks; that helps.
Having played a lot, I now understand the behaviour better and can solve my original issue which is to set arc_max <8GB. Hurrah.

But, it looks like:

1. zfs_arc_min must be lower than zfs_arc_max, though it's unclear why it can't be the same value. Oracle (not that it matters) docs say they can be the same.
2. WARNING messages aren't as helpful as they could be. E.g. why is it 'using 0'? Why the same message twice? The warning doesn't say why the value is being ignored.
3. zfs_arc_min doesn't seem to be working properly, at least per the docs. If the default is 0, then why does it appear to block a zfs_arc_max of 7GB? If I explicitly set a zfs_arc_min value then I can set arc_max when using low values (lower than 1/8 RAM?)

Should I create a new issue? Details below.

My original modprobe.d/zfs.conf

options zfs zfs_arc_min=8589934592
options zfs zfs_arc_max=8589934592

After rebooting

$ dmesg | grep zfs
[    3.414510] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[    3.414514] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[    3.418949] WARNING: ignoring tunable zfs_arc_max (using 135095138304 instead)

It's unclear why I'm getting the same arc_min warning twice.
Removing zfs_arc_min and rebooting works fine - e.g. dmesg | grep zfs shows nothing, and arc_summary shows arc_max is using the 8GB. I've had arc_min and max set the same for the past year or so.

Now if I try to go lower, e.g. 7GB and reboot, with no arc_min set.

options zfs zfs_arc_max=7516192768

$ dmesg | grep zfs
[    3.596221] WARNING: ignoring tunable zfs_arc_max (using 135095138304 instead)

I'm guessing that the default of arc_min is being calculated higher than arc_max, but the warning doesn't say_why_ arc_max is being ignored. The docs for arc_min say that the default is 0, so it shouldn't be blocking arc_max, but it is.

Re-checking the documentation https://openzfs.github.io/openzfs-docs/man/5/zfs-module-parameters.5.html I see that it's saying I can set arc_min to 32MB and arc_max to 64MB, so:

options zfs zfs_arc_min=33554432
options zfs zfs_arc_max=67108864

$ dmesg | grep zfs
[    3.556248] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[    3.556250] WARNING: ignoring tunable zfs_arc_min (using 0 instead)

Looking at arc_summary, I see:

ARC size (current):                                   148.3 %   94.9 MiB
        Target size (adaptive):                       100.0 %   64.0 MiB
        Min size (hard limit):                         50.0 %   32.0 MiB
        Max size (high water):                            2:1   64.0 MiB

It does look like it's applied, though not sure why it's popping out a WARNING.

options zfs zfs_arc_min=33554432
options zfs zfs_arc_max=4294967296

$ dmesg | grep zfs
[    3.571312] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[    3.571315] WARNING: ignoring tunable zfs_arc_min (using 0 instead)

ARC size (current):                                   100.7 %    4.0 GiB
        Target size (adaptive):                       100.0 %    4.0 GiB
        Min size (hard limit):                          0.8 %   32.0 MiB
        Max size (high water):                          128:1    4.0 GiB

That also worked.

Changing zfs_arc_min to just below arc_max doesn't work? The arc_min size appears to be set, but I don't know if that's just a lucky coincidence.

options zfs zfs_arc_min=4294967295
options zfs zfs_arc_max=4294967296

$ dmesg | grep -i zfs
[    3.465662] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[    3.465665] WARNING: ignoring tunable zfs_arc_min (using 0 instead)

ARC size (current):                                    32.1 %    1.3 GiB
        Target size (adaptive):                       100.0 %    4.0 GiB
        Min size (hard limit):                        100.0 %    4.0 GiB

[ghost]

options zfs zfs_arc_min=8589934592
options zfs zfs_arc_max=8589934592

After rebooting

$ dmesg | grep zfs
[ 3.414510] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[ 3.414514] WARNING: ignoring tunable zfs_arc_min (using 0 instead)
[ 3.418949] WARNING: ignoring tunable zfs_arc_max (using 135095138304 instead)

It's unclear why I'm getting the same arc_min warning twice.

The Linux kernel sets each tunable one at a time. One tunable can affect multiple settings in ZFS internally, so there is one procedure that updates all the ARC tunables whenever one changes.

The first warning comes from the update procedure running for options zfs zfs_arc_min=8589934592, showing you have set zfs_arc_min to an invalid value. The second and third warning come from the update procedure running for options zfs zfs_arc_max=8589934592, showing zfs_arc_min is still invalid, and zfs_arc_max is also invalid now.

The logic at play can be found here:

zfs/module/zfs/arc.c

Lines 7216 to 7240 in dcbf847

	uint64_t allmem = arc_all_memory();
	unsigned long limit;
	/* Valid range: 32M - <arc_c_max> */
	if ((zfs_arc_min) && (zfs_arc_min != arc_c_min) &&
	(zfs_arc_min >= 2ULL << SPA_MAXBLOCKSHIFT) &&
	(zfs_arc_min <= arc_c_max)) {
	arc_c_min = zfs_arc_min;
	arc_c = MAX(arc_c, arc_c_min);
	}
	WARN_IF_TUNING_IGNORED(zfs_arc_min, arc_c_min, verbose);
	/* Valid range: 64M - <all physical memory> */
	if ((zfs_arc_max) && (zfs_arc_max != arc_c_max) &&
	(zfs_arc_max >= 64 << 20) && (zfs_arc_max < allmem) &&
	(zfs_arc_max > arc_c_min)) {
	arc_c_max = zfs_arc_max;
	arc_c = MIN(arc_c, arc_c_max);
	arc_p = (arc_c >> 1);
	if (arc_meta_limit > arc_c_max)
	arc_meta_limit = arc_c_max;
	if (arc_dnode_size_limit > arc_meta_limit)
	arc_dnode_size_limit = arc_meta_limit;
	}
	WARN_IF_TUNING_IGNORED(zfs_arc_max, arc_c_max, verbose);

It's not apparent to me why options zfs zfs_arc_min=8589934592 is being ignored, unless it happens to be what the automatic tuning for arc_c_min is on your system. But the same explanation doesn't work for options zfs zfs_arc_min=33554432, so I don't know.

This area could certainly be made more flexible and user friendly.