Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After zpool replace ZFS restarts resilvering process and does not finish #9551

Closed
GothardDoesLearn opened this issue Nov 5, 2019 · 6 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@GothardDoesLearn
Copy link

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version Ubuntu 18.04.3 LTS
Linux Kernel 4.15.0-65-generic
Architecture x86_64
ZFS Version v0.7.5-1ubuntu16.6
SPL Version 0.7.5-1ubuntu2

Describe the problem you're observing

One of my HDD drives needed to be replaced, due to not beeing recognized. my guess from the sound of the HDD that the motor inside broke.

i shut down the NAS, disconnected the HDD and replaced it with a new one, booted and "zpool replace" the drive with the new one. i did that a couple of times now for the last years if one HDD seems to have an issue and everything worked fine. but this time, the resilvering process keeps restarting, sometimes it goes above 70% some times it restarts at about 20% of the resilvering process. i also tried with a second bought HDD in case of a broken bought HDD, and replaced a 2nd time, also didnt work. i did some research but i could not figure out the problem at all and the forum hints didnt work for me.

thank you for reading and your time you use to help me, if i can provide addional information that you need just ask and i try to provide it.

Describe how to reproduce the problem

I can't tell when it will restart the resilvering process, but i just have to wait and it does not seem to stop. it keeps restarting.

Include any warning/errors/backtraces from the system logs

pool status looks like this:

zpool status -x
pool: storage3
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Nov  5 12:16:33 2019
	4,74T scanned out of 48,8T at 692M/s, 18h33m to go
	462G resilvered, 9,71% done
config:

	NAME                                            STATE     READ WRITE CKSUM
	storage3                                        DEGRADED     0     0     0
	  raidz2-0                                      DEGRADED     0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WXL1H16744AD    ONLINE       0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX11D76NYYX0    ONLINE       0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPJT6    ONLINE       0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3    ONLINE       0     0     0  (resilvering)
	    replacing-4                                 DEGRADED     0     0     0
	      3192384416026124003                       UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPL6F-part1
	      6836373468087217751                       UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX61DC8960TS-part1
	      ata-WDC_WD60EFRX-68L0BN1_WD-WX11DC88FJ8H  ONLINE       0     0     0  (resilvering)
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86E6498    ONLINE       0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KCAUN    ONLINE       0     0     0
	    ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KCRZU    ONLINE       0     0     0
	    wwn-0x50014ee20d96896d                      ONLINE       0     0     0
	    wwn-0x50014ee262ebc598                      ONLINE       0     0     0

errors: No known data errors
event log

TIME                           CLASS
Nov  4 2019 03:24:09.249072466 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xec5598d715b00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x39ec5598aa1db
        vdev_delta_ts = 0x910f925
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x39ec45028665a
        zio_delta = 0x0
        zio_offset = 0x1dc0d85f000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1576
        time = 0x5dbf8bc9 0xed88b52 
        eid = 0x545

Nov  4 2019 03:24:09.249072466 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xec5598dde8100401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x39ec5598aa1db
        vdev_delta_ts = 0x910f925
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x39ec4502881cd
        zio_delta = 0x0
        zio_offset = 0x1dc0d864000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1577
        time = 0x5dbf8bc9 0xed88b52 
        eid = 0x546

Nov  4 2019 03:24:09.249072466 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xec5598e495b00801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x39ec5598aa1db
        vdev_delta_ts = 0x910f925
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x39ec45024a0cf
        zio_delta = 0x0
        zio_offset = 0x1dc0d853000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1573
        time = 0x5dbf8bc9 0xed88b52 
        eid = 0x547

Nov  4 2019 03:24:09.249072466 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xec5598edba000401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x39ec5598aa1db
        vdev_delta_ts = 0x910f925
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x39ec45028cdfd
        zio_delta = 0x0
        zio_offset = 0x1dc0d868000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1578
        time = 0x5dbf8bc9 0xed88b52 
        eid = 0x548

Nov  4 2019 03:24:09.249072466 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xec55990a96400c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x39ec5598aa1db
        vdev_delta_ts = 0x910f925
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x39ec45027b5f3
        zio_delta = 0x0
        zio_offset = 0x1dc0d85b000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1575
        time = 0x5dbf8bc9 0xed88b52 
        eid = 0x549

Nov  4 2019 03:24:15.252997721 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x11a00c1
        history_time = 0x5dbf8bcf
        time = 0x5dbf8bcf 0xf147059 
        eid = 0x54a

Nov  4 2019 03:24:15.280997372 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5dbf8bcf 0x10bfadfc 
        eid = 0x54b

Nov  4 2019 03:24:15.280997372 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "func=2 mintxg=3 maxtxg=18293169"
        history_internal_name = "scan setup"
        history_txg = 0x11a00c1
        history_time = 0x5dbf8bcf
        time = 0x5dbf8bcf 0x10bfadfc 
        eid = 0x54c

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x29359787bf100001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xfe35432b
        zio_timestamp = 0x3b292511e10f1
        zio_delta = 0xffac3832
        zio_offset = 0x1dc0d857000
        zio_size = 0x1d000
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x54d

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x2935979d7c600001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b292511fa7f9
        zio_delta = 0x0
        zio_offset = 0x1dc0d85f000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1576
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x54e

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597a230b00401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b292511e10f1
        zio_delta = 0x0
        zio_offset = 0x1dc0d857000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1574
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x54f

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597a841200c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b29251204156
        zio_delta = 0x0
        zio_offset = 0x1dc0d868000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1578
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x550

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597ae53d00801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b2925120a510
        zio_delta = 0x0
        zio_offset = 0x1dc0d86c000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1579
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x551

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597b41c100401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b2925120ef79
        zio_delta = 0x0
        zio_offset = 0x1dc0d870000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x157a
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x552

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597b8a8100c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b292511ff305
        zio_delta = 0x0
        zio_offset = 0x1dc0d864000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1577
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x553

Nov  4 2019 09:27:04.463083299 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x293597bd4f500801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3b2935977ec3c
        vdev_delta_ts = 0x8712804
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3b292511f0f2e
        zio_delta = 0x0
        zio_offset = 0x1dc0d85b000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1575
        time = 0x5dbfe0d8 0x1b9a1723 
        eid = 0x554

Nov  4 2019 09:27:15.938939946 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x11a1000
        history_time = 0x5dbfe0e3
        time = 0x5dbfe0e3 0x37f7162a 
        eid = 0x555

Nov  4 2019 09:27:15.962939645 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5dbfe0e3 0x39654afd 
        eid = 0x556

Nov  4 2019 09:27:15.962939645 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "func=2 mintxg=3 maxtxg=18293169"
        history_internal_name = "scan setup"
        history_txg = 0x11a1000
        history_time = 0x5dbfe0e3
        time = 0x5dbfe0e3 0x39654afd 
        eid = 0x557

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e50a5a00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x1808a0
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xfdd5da02
        zio_timestamp = 0x3e1f1ff21bc1f
        zio_delta = 0xfdd5e039
        zio_offset = 0x5883d000
        zio_size = 0x1000
        zio_objset = 0x15
        zio_object = 0x17019
        zio_level = 0x1
        zio_blkid = 0x5
        time = 0x5dc0ac50 0x34595998 
        eid = 0x558

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e5886e00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x1808a0
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xfdd49f9e
        zio_timestamp = 0x3e1f1ff213e4f
        zio_delta = 0xfdd4af49
        zio_offset = 0x584ef000
        zio_size = 0x1000
        zio_objset = 0x15
        zio_object = 0x17019
        zio_level = 0x1
        zio_blkid = 0x4
        time = 0x5dc0ac50 0x34595998 
        eid = 0x559

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e67db800c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x1808a8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0x1006781f3
        zio_timestamp = 0x3e1f1fc8c4eaf
        zio_delta = 0x100688f90
        zio_offset = 0x5701c000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17018
        zio_level = 0x0
        zio_blkid = 0x86e
        time = 0x5dc0ac50 0x34595998 
        eid = 0x55a

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e816ce00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x1808a0
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xfdd53eb2
        zio_timestamp = 0x3e1f1ff230549
        zio_delta = 0xfdd54a18
        zio_offset = 0x5937e000
        zio_size = 0x1000
        zio_objset = 0x15
        zio_object = 0x17019
        zio_level = 0x1
        zio_blkid = 0x9
        time = 0x5dc0ac50 0x34595998 
        eid = 0x55b

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e87d6800801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0x100696e51
        zio_timestamp = 0x3e1f1fc8b9397
        zio_delta = 0x10069eb72
        zio_offset = 0x57009000
        zio_size = 0xf000
        time = 0x5dc0ac50 0x34595998 
        eid = 0x55c

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347e98d6a00401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1f1fc8bb325
        zio_delta = 0x0
        zio_offset = 0x57014000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17018
        zio_level = 0x0
        zio_blkid = 0x86d
        time = 0x5dc0ac50 0x34595998 
        eid = 0x55d

Nov  4 2019 23:55:12.878270872 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1f347ea166300801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0x67453bfc6e7e3f39
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x67453bfc6e7e3f39
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/wwn-0x50014ee20d96896d-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1f347e4c678
        vdev_delta_ts = 0x734b8f1
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1f1fc8b9397
        zio_delta = 0x0
        zio_offset = 0x57009000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17018
        zio_level = 0x0
        zio_blkid = 0x86c
        time = 0x5dc0ac50 0x34595998 
        eid = 0x55e

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7a127800c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0x10003acf6
        zio_timestamp = 0x3e1fa9b2266dd
        zio_delta = 0x106d4ca49
        zio_offset = 0xc9454000
        zio_size = 0x20000
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x55f

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7a7beb00801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0x10180871f
        zio_timestamp = 0x3e1fa9b2e08c3
        zio_delta = 0x208c1bdfb
        zio_offset = 0xc9498000
        zio_size = 0x1b000
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x560

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7b78a400401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b291cf6
        zio_delta = 0x0
        zio_offset = 0xc9470000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd0a
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x561

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7be4f700c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b25868b
        zio_delta = 0x0
        zio_offset = 0xc946c000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd09
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x562

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7c3bcc00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b22a2f7
        zio_delta = 0x0
        zio_offset = 0xc9458000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd06
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x563

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7ca9ed00801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b254e54
        zio_delta = 0x0
        zio_offset = 0xc9468000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd08
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x564

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7d012100401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b33efa9
        zio_delta = 0x0
        zio_offset = 0xc94af000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd15
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x565

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7d5a0c00801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b339a67
        zio_delta = 0x0
        zio_offset = 0xc94ab000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd14
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x566

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7dbade00401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b2e08c3
        zio_delta = 0x0
        zio_offset = 0xc9498000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd11
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x567

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7e084400001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b252716
        zio_delta = 0x0
        zio_offset = 0xc9463000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd07
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x568

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7e5fd000401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b2266dd
        zio_delta = 0x0
        zio_offset = 0xc9454000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd05
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x569

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7ea63c00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b2ebdf1
        zio_delta = 0x0
        zio_offset = 0xc94a7000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd13
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x56a

Nov  4 2019 23:55:53.269768176 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x1fcaf7f6df800001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3e1fcaf798097
        vdev_delta_ts = 0x9b263c0
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3e1fa9b2e73c5
        zio_delta = 0x0
        zio_offset = 0xc94a2000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0x17031
        zio_level = 0x0
        zio_blkid = 0xd12
        time = 0x5dc0ac79 0x101455f0 
        eid = 0x56b

Nov  4 2019 23:56:06.157607798 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x11a3430
        history_time = 0x5dc0ac86
        time = 0x5dc0ac86 0x964e776 
        eid = 0x56c

Nov  4 2019 23:56:06.169607648 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5dc0ac86 0xa1c01e0 
        eid = 0x56d

Nov  4 2019 23:56:06.169607648 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "func=2 mintxg=3 maxtxg=18293169"
        history_internal_name = "scan setup"
        history_txg = 0x11a3430
        history_time = 0x5dc0ac86
        time = 0x5dc0ac86 0xa1c01e0 
        eid = 0x56e

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51c084d00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xfec51ee2
        zio_timestamp = 0x3f668b08f6a28
        zio_delta = 0xff621661
        zio_offset = 0x1dc0d857000
        zio_size = 0x1d000
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x56f

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51ce16600401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b091db32
        zio_delta = 0x0
        zio_offset = 0x1dc0d870000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x157a
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x570

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51d2e3d00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b09177ee
        zio_delta = 0x0
        zio_offset = 0x1dc0d86c000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1579
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x571

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51d93d100c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b090175b
        zio_delta = 0x0
        zio_offset = 0x1dc0d85f000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1576
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x572

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51de8a400c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b08fa4f8
        zio_delta = 0x0
        zio_offset = 0x1dc0d85b000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1575
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x573

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51e379c00c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b08f6a28
        zio_delta = 0x0
        zio_offset = 0x1dc0d857000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1574
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x574

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c51fe59700001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c51ba1b8
        vdev_delta_ts = 0x105778f2
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b091165f
        zio_delta = 0x0
        zio_offset = 0x1dc0d868000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1578
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x575

Nov  5 2019 06:10:11.741100784 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x669c522114a00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x3f669c522125b
        vdev_delta_ts = 0x24fed
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x3f668b0907131
        zio_delta = 0x0
        zio_offset = 0x1dc0d864000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1577
        time = 0x5dc10433 0x2c2c4cf0 
        eid = 0x576

Nov  5 2019 06:10:17.789027742 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x11a43ef
        history_time = 0x5dc10439
        time = 0x5dc10439 0x2f079b9e 
        eid = 0x577

Nov  5 2019 06:10:17.817027403 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5dc10439 0x30b2d94b 
        eid = 0x578

Nov  5 2019 06:10:17.817027403 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "func=2 mintxg=3 maxtxg=18293169"
        history_internal_name = "scan setup"
        history_txg = 0x11a43ef
        history_time = 0x5dc10439
        time = 0x5dc10439 0x30b2d94b 
        eid = 0x579

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667faaeebd00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x20080ca8
        zio_stage = 0x800000
        zio_pipeline = 0x840000
        zio_delay = 0xffc21f67
        zio_timestamp = 0x40a65755d7892
        zio_delta = 0xffd0cd3b
        zio_offset = 0x1dc0d85b000
        zio_size = 0x1d000
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57a

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667fac6b9e00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a65755e06f6
        zio_delta = 0x0
        zio_offset = 0x1dc0d868000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1578
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57b

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667faccf7700401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a6575630671
        zio_delta = 0x0
        zio_offset = 0x1dc0d874000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x157b
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57c

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667fad4cc600801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a65755e455e
        zio_delta = 0x0
        zio_offset = 0x1dc0d86c000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1579
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57d

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667fadc80a00001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a65755dcba2
        zio_delta = 0x0
        zio_offset = 0x1dc0d864000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1577
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57e

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667fae1b5f00401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a657562ab31
        zio_delta = 0x0
        zio_offset = 0x1dc0d870000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x157a
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x57f

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667faeb2e900401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a65755d7892
        zio_delta = 0x0
        zio_offset = 0x1dc0d85b000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1575
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x580

Nov  5 2019 12:16:27.647858095 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0xa667faf326900401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x140a4edc0f6540b8
                vdev = 0xe7e4ae08e62d0021
        (end detector)
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xe7e4ae08e62d0021
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86KC4F3-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x40a667faa4748
        vdev_delta_ts = 0x7d27e5f
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        parent_guid = 0xbdafe1305c511e2d
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x5
        zio_flags = 0x3808a8
        zio_stage = 0x800000
        zio_pipeline = 0xb80000
        zio_delay = 0x0
        zio_timestamp = 0x40a65755db0d6
        zio_delta = 0x0
        zio_offset = 0x1dc0d85f000
        zio_size = 0x4000
        zio_objset = 0x15
        zio_object = 0xb252
        zio_level = 0x0
        zio_blkid = 0x1576
        time = 0x5dc15a0b 0x269d87af 
        eid = 0x581

Nov  5 2019 12:16:33.539786014 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x11a535e
        history_time = 0x5dc15a11
        time = 0x5dc15a11 0x202c7b1e 
        eid = 0x582

Nov  5 2019 12:16:33.587785426 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5dc15a11 0x2308e4d2 
        eid = 0x583

Nov  5 2019 12:16:33.587785426 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "storage3"
        pool_guid = 0x140a4edc0f6540b8
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "Server"
        history_internal_str = "func=2 mintxg=3 maxtxg=18293169"
        history_internal_name = "scan setup"
        history_txg = 0x11a535e
        history_time = 0x5dc15a11
        time = 0x5dc15a11 0x2308e4d2 
        eid = 0x584

History Log

History for 'storage3':
2016-12-04.17:19:07 zpool create -f storage3 raidz2 /dev/sdaa /dev/sdac /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdx /dev/sdan
2016-12-04.17:47:19 zpool import -N storage3
2016-12-04.18:08:41 zpool import -N storage3
2016-12-04.23:57:08 zpool import -N storage3
2016-12-06.11:20:08 zpool import -N storage3
2016-12-11.23:56:50 zpool import -N storage3
2016-12-18.23:56:57 zpool import -N storage3
2016-12-25.23:56:45 zpool import -N storage3
2016-12-26.19:04:07 zpool import -N storage3
2017-01-01.23:56:58 zpool import -N storage3
2017-01-08.23:56:55 zpool import -N storage3
2017-01-15.23:56:50 zpool import -N storage3
2017-01-22.23:57:00 zpool import -N storage3
2017-01-29.23:57:13 zpool import -N storage3
2017-02-05.23:56:46 zpool import -N storage3
2017-02-12.23:56:53 zpool import -N storage3
2017-02-15.10:32:35 zpool import -N storage3
2017-02-17.10:28:46 zpool import -N storage3
2017-02-19.23:56:56 zpool import -N storage3
2017-02-26.23:57:09 zpool import -N storage3
2017-03-01.13:44:51 zpool import -N storage3
2017-03-05.23:57:35 zpool import -N storage3
2017-03-12.23:57:29 zpool import -N storage3
2017-03-19.23:57:14 zpool import -N storage3
2017-03-26.23:57:27 zpool import -N storage3
2017-04-02.23:57:07 zpool import -N storage3
2017-04-09.23:57:30 zpool import -N storage3
2017-04-16.23:57:13 zpool import -N storage3
2017-04-23.23:57:13 zpool import -N storage3
2017-04-30.23:57:28 zpool import -N storage3
2017-05-07.23:57:29 zpool import -N storage3
2017-05-14.23:57:34 zpool import -N storage3
2017-05-21.23:57:10 zpool import -N storage3
2017-05-28.23:57:26 zpool import -N storage3
2017-06-04.23:57:08 zpool import -N storage3
2017-06-11.23:57:32 zpool import -N storage3
2017-06-18.23:57:10 zpool import -N storage3
2017-06-25.23:57:08 zpool import -N storage3
2017-06-29.13:49:15 zpool import -N storage3
2017-07-02.23:57:10 zpool import -N storage3
2017-07-09.23:57:28 zpool import -N storage3
2017-07-16.23:57:24 zpool import -N storage3
2017-07-23.23:57:26 zpool import -N storage3
2017-07-30.23:57:30 zpool import -N storage3
2017-08-06.23:57:32 zpool import -N storage3
2017-08-13.23:57:33 zpool import -N storage3
2017-08-20.23:57:31 zpool import -N storage3
2017-08-27.23:57:25 zpool import -N storage3
2017-09-03.23:57:28 zpool import -N storage3
2017-09-10.23:57:34 zpool import -N storage3
2017-09-13.11:35:56 zpool import storage3
2017-09-18.14:03:39 zpool import storage3
2017-09-18.14:52:52 zpool import -c /etc/zfs/zpool.cache -aN
2017-09-18.14:59:19 zpool import -c /etc/zfs/zpool.cache -aN
2017-09-18.15:05:31 zpool import -c /etc/zfs/zpool.cache -aN
2017-09-24.23:57:11 zpool import -c /etc/zfs/zpool.cache -aN
2017-10-01.23:57:26 zpool import -c /etc/zfs/zpool.cache -aN
2017-10-08.00:24:18 zpool scrub storage3
2017-10-08.23:57:14 zpool import -c /etc/zfs/zpool.cache -aN
2017-10-15.23:59:16 zpool import -c /etc/zfs/zpool.cache -aN
2017-10-22.23:58:06 zpool import -c /etc/zfs/zpool.cache -aN
2017-10-29.23:58:30 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-05.23:58:45 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-12.00:24:17 zpool scrub storage3
2017-11-12.23:59:18 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-19.23:59:16 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-26.23:59:25 zpool import -c /etc/zfs/zpool.cache -aN
2017-12-03.23:59:42 zpool import -c /etc/zfs/zpool.cache -aN
2017-12-10.00:24:18 zpool scrub storage3
2017-12-11.00:00:20 zpool import -c /etc/zfs/zpool.cache -aN
2017-12-18.00:00:22 zpool import -c /etc/zfs/zpool.cache -aN
2017-12-25.00:00:33 zpool import -c /etc/zfs/zpool.cache -aN
2018-01-01.00:00:54 zpool import -c /etc/zfs/zpool.cache -aN
2018-01-08.00:01:09 zpool import -c /etc/zfs/zpool.cache -aN
2018-01-14.00:24:18 zpool scrub storage3
2018-01-15.00:01:40 zpool import -c /etc/zfs/zpool.cache -aN
2018-01-21.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2018-01-29.00:02:02 zpool import -c /etc/zfs/zpool.cache -aN
2018-02-05.00:02:18 zpool import -c /etc/zfs/zpool.cache -aN
2018-02-11.00:24:18 zpool scrub storage3
2018-02-12.00:02:42 zpool import -c /etc/zfs/zpool.cache -aN
2018-02-19.00:02:49 zpool import -c /etc/zfs/zpool.cache -aN
2018-02-26.00:03:06 zpool import -c /etc/zfs/zpool.cache -aN
2018-03-05.00:03:20 zpool import -c /etc/zfs/zpool.cache -aN
2018-03-11.00:24:18 zpool scrub storage3
2018-03-12.00:03:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-03-19.00:03:55 zpool import -c /etc/zfs/zpool.cache -aN
2018-03-26.00:04:08 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-02.00:04:24 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-08.00:24:15 zpool scrub storage3
2018-04-09.00:04:57 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-11.17:57:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-15.23:56:51 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-18.03:30:11 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-18.10:55:20 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-19.18:59:25 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-22.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-04-29.23:56:50 zpool import -c /etc/zfs/zpool.cache -aN
2018-05-06.23:56:52 zpool import -c /etc/zfs/zpool.cache -aN
2018-05-13.00:24:19 zpool scrub storage3
2018-05-13.23:57:08 zpool import -c /etc/zfs/zpool.cache -aN
2018-05-20.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-05-27.23:56:50 zpool import -c /etc/zfs/zpool.cache -aN
2018-06-03.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2018-06-10.00:24:16 zpool scrub storage3
2018-06-10.23:57:13 zpool import -c /etc/zfs/zpool.cache -aN
2018-06-17.23:56:57 zpool import -c /etc/zfs/zpool.cache -aN
2018-06-24.23:56:40 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-01.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-08.00:24:15 zpool scrub storage3
2018-07-08.23:57:08 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-12.16:36:46 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-12.16:46:05 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-12.16:54:58 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-12.17:24:42 zpool export storage3
2018-07-12.17:27:57 zpool import storage3
2018-07-15.23:57:05 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-16.01:55:42 zpool export storage3
2018-07-16.01:56:06 zpool import storage3
2018-07-16.02:18:29 zpool scrub storage3
2018-07-21.10:43:19 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-21.10:45:16 zpool export storage3
2018-07-21.10:45:57 zpool import storage3
2018-07-21.11:43:49 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-22.23:57:03 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-27.19:19:55 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-29.23:58:19 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-30.02:09:57 zpool import -c /etc/zfs/zpool.cache -aN
2018-07-30.02:11:31 zpool export storage3
2018-07-30.02:50:08 zpool import storage3
2018-08-04.09:18:38 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-04.09:21:11 zpool export storage3
2018-08-04.09:22:46 zpool import storage3
2018-08-05.23:57:12 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-12.00:24:14 zpool scrub storage3
2018-08-12.14:46:25 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-12.23:58:26 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-14.14:20:31 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-14.14:21:54 zpool export storage3
2018-08-14.14:25:43 zpool import storage3
2018-08-14.14:55:12 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-20.00:00:28 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-26.23:59:35 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-28.14:57:06 zpool import -c /etc/zfs/zpool.cache -aN
2018-08-28.17:08:20 zpool export storage3
2018-08-28.17:09:18 zpool import storage3
2018-09-02.23:57:41 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-09.00:24:18 zpool scrub storage3
2018-09-09.23:58:30 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-16.23:57:57 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-22.07:35:10 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-23.23:57:01 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-25.15:55:43 zpool export storage3
2018-09-25.15:59:03 zpool import -f storage3
2018-09-25.16:28:13 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-25.16:46:51 zpool import -c /etc/zfs/zpool.cache -aN
2018-09-30.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2018-10-07.23:57:02 zpool import -c /etc/zfs/zpool.cache -aN
2018-10-14.00:24:18 zpool scrub storage3
2018-10-14.23:57:30 zpool import -c /etc/zfs/zpool.cache -aN
2018-10-21.23:56:52 zpool import -c /etc/zfs/zpool.cache -aN
2018-10-28.23:56:51 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-04.23:56:52 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-11.00:24:15 zpool scrub storage3
2018-11-11.23:57:10 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-14.16:27:28 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-18.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-20.18:34:29 zpool import -c /etc/zfs/zpool.cache -aN
2018-11-25.23:56:59 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-02.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-04.16:38:20 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-04.16:48:06 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-09.00:24:17 zpool scrub storage3
2018-12-09.23:58:13 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-11.09:06:48 zpool clear storage3
2018-12-11.09:18:11 zpool upgrade storage3
2018-12-16.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-23.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2018-12-30.23:56:56 zpool import -c /etc/zfs/zpool.cache -aN
2019-01-06.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-01-13.00:24:18 zpool scrub storage3
2019-01-13.23:58:15 zpool import -c /etc/zfs/zpool.cache -aN
2019-01-15.17:43:29 zpool clear storage3
2019-01-20.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2019-01-27.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-02-03.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2019-02-10.00:24:11 zpool scrub storage3
2019-02-10.23:57:49 zpool import -c /etc/zfs/zpool.cache -aN
2019-02-17.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-02-24.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-03-03.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-03-10.00:24:18 zpool scrub storage3
2019-03-10.23:58:15 zpool import -c /etc/zfs/zpool.cache -aN
2019-03-17.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-03-24.23:56:52 zpool import -c /etc/zfs/zpool.cache -aN
2019-03-31.23:57:02 zpool import -c /etc/zfs/zpool.cache -aN
2019-04-07.23:58:17 zpool import -c /etc/zfs/zpool.cache -aN
2019-04-14.00:24:18 zpool scrub storage3
2019-04-14.23:58:15 zpool import -c /etc/zfs/zpool.cache -aN
2019-04-21.23:57:01 zpool import -c /etc/zfs/zpool.cache -aN
2019-04-28.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-05-05.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-05-12.00:24:17 zpool scrub storage3
2019-05-12.23:58:15 zpool import -c /etc/zfs/zpool.cache -aN
2019-05-19.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-05-23.18:53:17 zpool import -c /etc/zfs/zpool.cache -aN
2019-05-26.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-06-02.23:56:51 zpool import -c /etc/zfs/zpool.cache -aN
2019-06-09.00:24:17 zpool scrub storage3
2019-06-09.23:58:06 zpool import -c /etc/zfs/zpool.cache -aN
2019-06-16.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-06-23.23:56:50 zpool import -c /etc/zfs/zpool.cache -aN
2019-06-30.23:56:54 zpool import -c /etc/zfs/zpool.cache -aN
2019-07-07.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-07-14.00:24:17 zpool scrub storage3
2019-07-14.23:57:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-07-21.23:56:53 zpool import -c /etc/zfs/zpool.cache -aN
2019-07-28.23:56:58 zpool import -c /etc/zfs/zpool.cache -aN
2019-08-04.23:56:55 zpool import -c /etc/zfs/zpool.cache -aN
2019-08-11.00:24:18 zpool scrub storage3
2019-08-11.23:58:22 zpool import -c /etc/zfs/zpool.cache -aN
2019-08-18.23:56:56 zpool import -c /etc/zfs/zpool.cache -aN
2019-08-25.23:56:57 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-01.23:56:57 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-08.00:24:18 zpool scrub storage3
2019-09-08.23:58:17 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-16.11:08:41 zpool import storage3
2019-09-16.11:26:20 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-16.11:49:40 zpool import storage3
2019-09-20.12:09:01 zpool import storage3
2019-09-20.12:24:38 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-20.12:28:21 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-20.13:26:44 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-21.14:12:53 zpool export storage3
2019-09-21.14:13:43 zpool import -d/dev/disk/by-id storage3
2019-09-21.14:36:48 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-21.14:43:03 zpool replace storage3 /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPL6F-part1 /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX61DC8960TS
2019-09-22.23:57:25 zpool import -c /etc/zfs/zpool.cache -aN
2019-09-29.23:56:57 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-06.23:56:58 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-07.14:01:31 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-10.16:44:45 zpool clear storage3
2019-10-13.18:52:16 zpool clear storage3
2019-10-17.10:42:58 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-17.10:56:25 zpool clear -F storage3
2019-10-20.19:48:48 zpool clear storage3
2019-10-23.09:19:17 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-23.09:22:12 zpool import -c /etc/zfs/zpool.cache -aN
2019-10-23.09:30:03 zpool replace storage3 3192384416026124003 /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11DC88FJ8H
2019-11-03.21:04:10 zpool clear storage3

@GregorKopka
Copy link
Contributor

GregorKopka commented Nov 6, 2019

As the resilver is currently running: stop the ZED daemon and wait for resilver to complete?

@GothardDoesLearn
Copy link
Author

GothardDoesLearn commented Nov 6, 2019

As the resilver is currently running: stop the ZED daemon and wait for resilver to complete?

Ohh! i didnt mention that but i all ready did this :/ this was one of the first suggestions i read in a forum. that did not work for me.

systemctl status zed
● zfs-zed.service - ZFS Event Daemon (zed)
   Loaded: loaded (/lib/systemd/system/zfs-zed.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-10-31 13:39:35 CET; 5 days ago
     Docs: man:zed(8)
 Main PID: 29933 (code=exited, status=0/SUCCESS)

Okt 23 17:42:01 Server zed[4325]: eid=1066 class=checksum pool_guid=0x140A4EDC0F6540B8
Okt 23 17:42:01 Server zed[4419]: eid=1067 class=checksum pool_guid=0x140A4EDC0F6540B8
Okt 24 13:15:00 Server zed[25260]: eid=1076 class=io pool_guid=0x140A4EDC0F6540B8 vdev_path=/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86K
Okt 27 13:36:56 Server zed[19026]: eid=1137 class=checksum pool_guid=0x140A4EDC0F6540B8
Okt 29 02:29:34 Server zed[29250]: eid=1185 class=io pool_guid=0x140A4EDC0F6540B8 vdev_path=/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86K
Okt 30 04:05:26 Server zed[17787]: eid=1197 class=io pool_guid=0x140A4EDC0F6540B8 vdev_path=/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86K
Okt 30 15:29:26 Server zed[30166]: eid=1212 class=io pool_guid=0x140A4EDC0F6540B8 vdev_path=/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D86K
Okt 31 13:39:35 Server systemd[1]: Stopping ZFS Event Daemon (zed)...
Okt 31 13:39:35 Server zed[29933]: Exiting
Okt 31 13:39:35 Server systemd[1]: Stopped ZFS Event Daemon (zed).

@behlendorf behlendorf added the Type: Defect Incorrect behavior (e.g. crash, hang) label Nov 11, 2019
@GregorKopka
Copy link
Contributor

I have seen similar with zfs 0.8.1-r0-gentoo on 4.19.67_p2-r2-debian-sources-lts when turning a single drive pool into a triple mirror: after the two zpool attach operations the resilver restarted every some few seconds.

Detaching only one of the new mirror sides didn't help to stop resilver from restarting, I needed to detach both new mirror sides and then attach only one, wait for resilver to complete, then attach the next. Getting to a state with two drives getting resilvered at the same time was back to restart-o-mania again.

No ZED running.

@GothardDoesLearn
Copy link
Author

Hello, i just recently looked at the print from demesg:


[455778.812516] print_req_error: I/O error, dev sdaj, sector 352942624 flags 700
[455778.812546] ata5: EH complete
[528687.013701] ata4.00: exception Emask 0x0 SAct 0x80001000 SErr 0x0 action 0x0
[528687.013708] ata4.00: irq_stat 0x40000008
[528687.013714] ata4.00: failed command: READ FPDMA QUEUED
[528687.013724] ata4.00: cmd 60/88:60:30:65:2b/00:00:03:00:00/40 tag 12 ncq dma 69632 in
                         res 41/40:00:90:65:2b/00:00:03:00:00/00 Emask 0x409 (media error) <F>
[528687.013729] ata4.00: status: { DRDY ERR }
[528687.013732] ata4.00: error: { UNC }
[528687.015448] ata4.00: configured for UDMA/133
[528687.015472] sd 4:0:0:0: [sdai] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[528687.015477] sd 4:0:0:0: [sdai] tag#12 Sense Key : Medium Error [current] 
[528687.015481] sd 4:0:0:0: [sdai] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed
[528687.015487] sd 4:0:0:0: [sdai] tag#12 CDB: Read(16) 88 00 00 00 00 00 03 2b 65 30 00 00 00 88 00 00
[528687.015491] print_req_error: I/O error, dev sdai, sector 53175696 flags 700
[528687.015526] ata4: EH complete
[528692.408840] ata4.00: exception Emask 0x0 SAct 0x8008 SErr 0x0 action 0x0
[528692.408847] ata4.00: irq_stat 0x40000008
[528692.408853] ata4.00: failed command: READ FPDMA QUEUED
[528692.408864] ata4.00: cmd 60/90:78:58:72:2c/00:00:03:00:00/40 tag 15 ncq dma 73728 in
                         res 41/40:00:c0:72:2c/00:00:03:00:00/00 Emask 0x409 (media error) <F>
[528692.408868] ata4.00: status: { DRDY ERR }
[528692.408872] ata4.00: error: { UNC }
[528692.410480] ata4.00: configured for UDMA/133
[528692.410507] sd 4:0:0:0: [sdai] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[528692.410512] sd 4:0:0:0: [sdai] tag#15 Sense Key : Medium Error [current] 
[528692.410516] sd 4:0:0:0: [sdai] tag#15 Add. Sense: Unrecovered read error - auto reallocate failed
[528692.410521] sd 4:0:0:0: [sdai] tag#15 CDB: Read(16) 88 00 00 00 00 00 03 2c 72 58 00 00 00 90 00 00
[528692.410526] print_req_error: I/O error, dev sdai, sector 53244608 flags 700
[528692.410548] ata4: EH complete
[561036.156479] ata2.00: exception Emask 0x0 SAct 0x42 SErr 0x0 action 0x0
[561036.156486] ata2.00: irq_stat 0x40000008
[561036.156491] ata2.00: failed command: READ FPDMA QUEUED
[561036.156502] ata2.00: cmd 60/e8:08:98:ca:06/00:00:ee:00:00/40 tag 1 ncq dma 118784 in
                         res 41/40:00:18:cb:06/00:00:ee:00:00/00 Emask 0x409 (media error) <F>
[561036.156506] ata2.00: status: { DRDY ERR }
[561036.156510] ata2.00: error: { UNC }
[561036.157983] ata2.00: configured for UDMA/133
[561036.158006] sd 2:0:0:0: [sdah] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[561036.158012] sd 2:0:0:0: [sdah] tag#1 Sense Key : Medium Error [current] 
[561036.158016] sd 2:0:0:0: [sdah] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed
[561036.158021] sd 2:0:0:0: [sdah] tag#1 CDB: Read(16) 88 00 00 00 00 00 ee 06 ca 98 00 00 00 e8 00 00
[561036.158025] print_req_error: I/O error, dev sdah, sector 3993422616 flags 700
[561036.158054] ata2: EH complete

Im not sure what that means ;( maybe someone can enlighten me?
Thank your for reading this and your try to help me.

Gothard

tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Jan 22, 2020
If a device is participating in an active resilver, then it will have a
non-empty DTL. Operations like vdev_{open,reopen,probe}() can cause the
resilver to be restarted (or deferred to be restarted later), which is
unnecessary if the DTL is still covered by the current scan range. This
is similar to the logic in vdev_dtl_should_excise() where the DTL can
only be excised if it's max txg is in the resilvered range.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Gallagher <john.gallagher@delphix.com>
Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl>
Signed-off-by: John Poduska <jpoduska@datto.com>
Issue openzfs#840
Closes openzfs#9155
Closes openzfs#9378
Closes openzfs#9551
Closes openzfs#9588
tonyhutter pushed a commit that referenced this issue Jan 23, 2020
If a device is participating in an active resilver, then it will have a
non-empty DTL. Operations like vdev_{open,reopen,probe}() can cause the
resilver to be restarted (or deferred to be restarted later), which is
unnecessary if the DTL is still covered by the current scan range. This
is similar to the logic in vdev_dtl_should_excise() where the DTL can
only be excised if it's max txg is in the resilvered range.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Gallagher <john.gallagher@delphix.com>
Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl>
Signed-off-by: John Poduska <jpoduska@datto.com>
Issue #840
Closes #9155
Closes #9378
Closes #9551
Closes #9588
@nickcmaynard
Copy link

nickcmaynard commented May 24, 2020

I have just seen potentially similar behaviour on my RAIDZ2 array. replaceing two devices at once - hideously slow resilver, with picket-fence behaviour. The resilver wasn't restarting (at least, the timestamp in zpool status wasn't updating), but it was obscenely slow, with pauses for several seconds at a time.

As soon as I detach one of the new drives, things go back to normal.

I'll do these replaces in serial now, but it feels like the wrong answer.

@agilob
Copy link

agilob commented Oct 3, 2020

I'm seeing the same problem.

Resilver runs in a loop, just disabled zfs-zed, need to wait ~20hours before resilvering finishes single loop.

Oct  3 2020 03:53:40.724065519 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "slave"
        history_internal_str = "errors=318954"
        history_internal_name = "scan done"
        history_txg = 0x71bf8c
        history_time = 0x5f77e7b4
        time = 0x5f77e7b4 0x2b285cef 
        eid = 0x77558

Oct  3 2020 03:53:40.744065491 sysevent.fs.zfs.resilver_finish
        version = 0x0
        class = "sysevent.fs.zfs.resilver_finish"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5f77e7b4 0x2c5989d3 
        eid = 0x77559

Oct  3 2020 03:53:40.744065491 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "slave"
        history_internal_str = "errors=159477"
        history_internal_name = "starting deferred resilver"
        history_txg = 0x71bf8c
        history_time = 0x5f77e7b4
        time = 0x5f77e7b4 0x2c5989d3 
        eid = 0x7755a

Oct  3 2020 03:53:45.850725010 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5f77e7b9 0x32b50892 
        eid = 0x7755b
Oct  3 2020 03:53:40.724065519 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "slave"
        history_internal_str = "errors=318954"
        history_internal_name = "scan done"
        history_txg = 0x71bf8c
        history_time = 0x5f77e7b4
        time = 0x5f77e7b4 0x2b285cef 
        eid = 0x77558

Oct  3 2020 03:53:40.744065491 sysevent.fs.zfs.resilver_finish
        version = 0x0
        class = "sysevent.fs.zfs.resilver_finish"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5f77e7b4 0x2c5989d3 
        eid = 0x77559

Oct  3 2020 03:53:40.744065491 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "slave"
        history_internal_str = "errors=159477"
        history_internal_name = "starting deferred resilver"
        history_txg = 0x71bf8c
        history_time = 0x5f77e7b4
        time = 0x5f77e7b4 0x2c5989d3 
        eid = 0x7755a

Oct  3 2020 03:53:45.850725010 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "wdblack"
        pool_guid = 0x509d876228a22ecc
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5f77e7b9 0x32b50892 
        eid = 0x7755b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

5 participants