Persistent L2ARC

[gamanakis]

This commit makes the L2ARC persistent across reboots. It is
based on issue 3525 in Illumos by Saso Kiselkov, which was later
ported by Yuxuan Shui to zfsonlinux (#2672) with modifications by
Jorgen Lundman, and rebased to current master with fixes and new
features by me.

Co-authored-by: Saso Kiselkov skiselkov@gmail.com
Co-authored-by: Jorgen Lundman lundman@lundman.net
Co-authored-by: George Amanakis gamanakis@gmail.com
Ported-by: Yuxuan Shui yshuiv7@gmail.com
Signed-off-by: George Amanakis gamanakis@gmail.com

Motivation and Context

This feature implements a light-weight persistent L2ARC metadata
structure that allows L2ARC contents to be recovered after a reboot.
This significantly eases the impact a reboot has on read performance
on systems with large caches.

Closes #3744

Description

This is a substantial rework of issue 3525 in Illumos and PR #2672
It includes numerous fixes and new features to be listed here.

How Has This Been Tested?

A number of tests have been added in ZTS.
Also running in a production server (VMs with ZVOLs, Samba, NFS, LXC) with unencrypted ZFS since 11/22/2019.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.

[PrivatePuffin]

You could change the commit message to:
"This commit makes the L2ARC persistent across reboots. It is largely
based on 3525 commit on Illumos"

The rest is already in the Authored and ported by tags.

That way it complies with the stylerule a little more.

edit
I've since come to realise i'm a retard and the style rule is 75 PER LINE, not 75 total. Please ignore this.

[PrivatePuffin]

While i'm not versed in ZoL enough to review some/most of the actual function calls.
I do want to note it's very neatly structured and byond-well commented.

[PrivatePuffin]

Quick work fixing the commit message!
Seems only some space-tabs issues and the sort are left now :)
http://build.zfsonlinux.org/builders/Ubuntu%2017.10%20x86_64%20%28STYLE%29/builds/18872/steps/shell_2/logs/stdio

[gamanakis]

@Ornias1993 I accidentally deleted your comment regarding disabling persistent L2ARC. I guess we could make it an option the user can set, ie a zfs property?

[PrivatePuffin]

@gamanakis No worries that was me, I didnt notice I already submitted it and afterwards scrolled up and noted you already had a quite easy/nice hook to disable persistent L2Arc :)

(not a property yet though, so maybe that can be added indeed )

[ahrens]

Awesome, thanks for picking this up!

based on 3525 commit on Illumos

I don't see that commit in illumos. Could you specify which codebase you started from? Is it #2672 ?

[PrivatePuffin]

@ahrens My fault of being unclear with my advice to change the commit message. @gamanakis also accidentily changed the PR summary.
this was the original (that should be put back, in the PR summary but left out of the commit summary):

This commit makes the L2ARC persistent across reboots. It is largely
based on 3525 commit on Illumos by Saso Kiselkov, which was later
ported by Yuxuan Shui to zfsonlinux, and rebased to current master
with fixes by myself.

With regards to the Illumos commit...
I see the mistake now, it is based on a Ilumos Issue, not commit.
https://www.illumos.org/issues/3525

[gamanakis]

@ahrens Yes, I picked up from #2672

[ahrens]

Thanks for clarifying.

[PrivatePuffin]

@behlendorf Do you have anyone in mind to request a review from?

[behlendorf]

@gamanakis thanks for moving this work forward! I haven't had a chance to take a careful look at this PR to provide any detailed feedback. But when skimming I noticed that it doesn't appear to add any new test cases. One of the next steps for this PR would be add some to the tests/zfs-tests/tests/functional directory to verify everything is working as intended.

[gamanakis]

@behlendorf I will work on that. Meanwhile I did a minor cleanup and implemented this as a zpool property, so that the user can set it. Is it ok if I push (or force-push?) those commits in this PR?

[PrivatePuffin]

@gamanakis Usually those changes can get pushed to the same repo no problem, certainly before formal review :)

[scineram]

Thanks for picking this up.

What is the point of another property?

[PrivatePuffin]

@scineram I think @gamanakis means the earlier discussed option to enable/disable the persistance

[PrivatePuffin]

@gamanakis Just as headsup: lookup if your git email on your machine is the same as the one setup on github. it doesn't recognise your commits as linked to your github account. you might want too ;)

[PrivatePuffin]

@gamanakis You might need to add your property to the test suite too, test errors:
zpool_get_002_pos
Found zpool features not in the zpool_get test config 59/60.

[gamanakis]

I pushed a final functional commit finalizing the zpool property, and adding man pages.
I am working on the tests now.

[gamanakis]

f005091: No functional change. Improve the implementation of #10154 and add comments.

1ee5bac: Make L2BLK_*_COMPRESS use SPA_COMPRESSBITS, like BP_*_COMPRESS. Again no functional changes.

e5ef3fd: Minor style changes consolidating code in tests, zdb.c and arc.c. Also give the 007 test more time to retrieve the amount of log blocks using zdb after issuing the off-lining of the cache device. Report the amount of restored blocks in dbgmsg if the rebuild was successful.

303f413: Fix the placement of DTRACE_PROBE4 in l2arc_evict().

412d6f5: Prevent unnecessary iteration of l2arc_evict() if we evict all buffers.

fbf8134: If we encounter a log block with invalid cksum in zdb.c, print out its pointer.

e044ce2: Report the number of restored log blocks in dbgmsg even when the rebuild was aborted prematurely.

b74fa37: Minor fix in zdb.c when dumping log entries.

[adamdmoss]

Still behaving well FWIW. :)

[behlendorf]

Merged. Thanks to everyone who helped get this done!

[tycho]

Thanks for your hard work on this, folks! This is going to be a great benefit to several machines I maintain with ZOL.

[BrainSlayer]

why is this patch enforcing lz4 compression, no matter which compression was selected? its also uses lz4 if compression is disabled.

[gamanakis]

@BrainSlayer the persistent L2ARC does use LZ4 for the compression of the metadata (log blocks). Otherwise the L2ARC buffers (L2ARC data) use whatever compression the filesystem uses.

[gamanakis]

@BrainSlayer you can see what compression algorithm is used for L2ARC data by inspecting the metadata with:
zdb -lll device | less
This produces for example:

lb[   1]        le[   5]        DVA asize: 105472, vdev: 0, offset: 2484060160
|                               birth: 19
|                               lsize: 131072
|                               psize: 105472
|                               compr: 15
|                               type: 1
|                               protected: 0
|                               prefetch: 0
|                               address: 663780352
|

Which tells you: the 5th buffer (le[5]) referenced by the first log block (lb[1]) is using compression algorithm (compr) 15, which in master is lz4 (found in zio_compress.h) because I set the filesystem compression to lz4. If you run this with the ZSTD PR merged in, and in a filesystem where zstd is enabled you will see 16 (the value for zstd in zio_compress.h) in that PR.

[BrainSlayer]

the question is more why you enforce lz4 and why not using the configured algorithm (or even no compression if disabled)

[gamanakis]

@BrainSlayer L2ARC devices can cache from different filesystems with different compression algorithms. Doing what you suggested makes it very complex and prone to bugs. After all only the metadata on L2ARC used for restoring the buffers use LZ4, the actual cached buffers (which were fetched from ARC) still use the compression used for the filesystem.

[gamanakis]

@BrainSlayer actually your suggestion is impossible with the current implementation. The metadata (log blocks) consist of up to 1022 entries which refer to buffers cached from ARC. Each of those 1022 cached buffers may be using a different compression algorithm, dependent on which filesystem it comes from. Ie the log block can contain entries referencing buffers, some of which are LZ4, some ZSTD, some with compression disabled. So which compression algorithm would the log block use?
This is the reason the compression for log blocks (again these are the metadata on L2ARC only) was fixed to LZ4.

[BrainSlayer]

okay. thanks for explaination. i will consider this in my rework of the zstd patch. so your own patch might be already good enough

[fedstryale]

Is it possible for l2arc to cache data even if it's still in ram and not evicted(duplicated cache)? For now, if the ram is still enough to hold cache, the data cached cannot be retrieved between reboots through persistent l2arc. It's hard to accept the most frequently used data in ram cannot be accelerated by persistent l2arc.

[gamanakis]

@fedstryale yes it is. Try setting l2arc_headroom = 0 in current master. That way the L2ARC will scan the full ARC lists for cacheable buffers, long before they are evicted. This is also documented in the zpoolconcepts.8 man page.