Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autoreplace not working #2449

Closed
cohdavid opened this issue Jul 1, 2014 · 9 comments
Closed

Autoreplace not working #2449

cohdavid opened this issue Jul 1, 2014 · 9 comments
Labels
Component: ZED ZFS Event Daemon Type: Feature Feature request or new feature

Comments

@cohdavid
Copy link

cohdavid commented Jul 1, 2014

I'm currently testing ZFS on Linux v0.6.3 on Debian Wheezy, using a LSI2008 based controller with the disks in JBOD mode. And I have configured the system as shown:

zpool status tank

pool: tank
state: ONLINE
scan: none requested
config:

    NAME                                STATE     READ WRITE CKSUM
    tank                                ONLINE       0     0     0
      raidz1-0                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:2:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:3:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:4:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:5:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:6:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:7:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:8:0   ONLINE       0     0     0
      raidz1-1                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:9:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:10:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:11:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:12:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:13:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:14:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:15:0  ONLINE       0     0     0
      raidz1-2                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:16:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:17:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:18:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:19:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:20:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:21:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:22:0  ONLINE       0     0     0
    logs
      mirror-3                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:24:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:25:0  ONLINE       0     0     0
    spares
      pci-0000:03:00.0-scsi-0:0:23:0    AVAIL

errors: No known data errors

And then I pull out a disk to see what happens in an error condition.

zpool status tank

pool: tank
state: DEGRADED
status: One or more devices could not be used because the label is missing
or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:

    NAME                                STATE     READ WRITE CKSUM
    tank                                DEGRADED     0     0     0
      raidz1-0                          DEGRADED     0     0     0
        pci-0000:03:00.0-scsi-0:0:2:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:3:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:4:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:5:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:6:0   UNAVAIL      0     0     0
        pci-0000:03:00.0-scsi-0:0:7:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:8:0   ONLINE       0     0     0
      raidz1-1                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:9:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:10:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:11:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:12:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:13:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:14:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:15:0  ONLINE       0     0     0
      raidz1-2                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:16:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:17:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:18:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:19:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:20:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:21:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:22:0  ONLINE       0     0     0
    logs
      mirror-3                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:24:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:25:0  ONLINE       0     0     0
    spares
      pci-0000:03:00.0-scsi-0:0:23:0    AVAIL

errors: No known data errors

zpool get autoreplace tank

NAME PROPERTY VALUE SOURCE
tank autoreplace on local

And now I wonder why the system has not started resilvering using the hot spare.......
@dweeezil
Copy link
Contributor

dweeezil commented Jul 1, 2014

See https://github.com/zfsonlinux/zfs/blob/master/cmd/zed/zed.d/io-spare.sh#L19-22 and the rest of the script for details (and also the checksum-spare.sh script). At the moment, the replace operation will only be performed in response to I/O or checksum errors and then, only if you've set the appropriate flags in the zed.rc file.

The zed and its scripts are new for 0.6.3 and are a work-in-progress at the moment.

@ryao
Copy link
Contributor

ryao commented Jul 2, 2014

@cohdavid You need to have the zed running as @dweeezil insinuated. Not all distributions include the rc scripts (better known as "init" scripts") to start zed at this time.

@cohdavid
Copy link
Author

cohdavid commented Jul 3, 2014

I understand what you are writing, so please consider this as a feature request instead, to get the situation where a disk becoming unavailable will be considered an error condition, triggering the use of a spare disk.
PS. I have zed running, and manually triggering the resilvering using a 'zpool replace' command works.

@behlendorf behlendorf added this to the 0.6.5 milestone Jul 14, 2014
@behlendorf behlendorf mentioned this issue Oct 29, 2014
Closed
@behlendorf behlendorf modified the milestone: 0.6.5 Nov 8, 2014
@mailinglists35
Copy link

oh, I can't wait to see this in 0.6.5!
If I modify the io-spare.sh in 0.6.4, then in order to perform a correct upgrade from 0.6.4 to 0.6.5, will I have to restore it to original state before the upgrade, right?

update: I have replaced my io-spare.sh with the version above referenced by @dweeezil and restarted the zed daemon, but it still does not initiate a resilver :(
using 0.6.4 on debian jessie kernel 3.16

@mailinglists35
Copy link

this appears to work on a testcase using latest released version - 0.6.4-1.1-2 on debian jessie - see comment #3478 (comment)

however it fails with io-spare.sh exit code 4 under cciss storage driver.

how can I make zed/io-spare detect errors when my disks are not named /dev/sdX but /dev/cciss/cXdY ?

@behlendorf
Copy link
Contributor

@mailinglists35 to make the io-spare.sh script more robust it should use the ZEVENT_VDEV_SPARE_GUIDS rather than the ZEVENT_VDEV_SPARE_PATHS to do the replacement. This way the path name itself becomes irrelevant and there is no ambiguity because the guids are for all intents and purposes unique.

The io-spare.sh script has all of this information today but can't use it because the zpool replace command needs to be extended to accept a guid for the spare device. For example, the following syntax should be supported.

zpool replace <failed vdev guid> <spare vdev guid>

@mailinglists35
Copy link

the zpool replace command needs to be extended to accept a guid for the spare device

@behlendorf thanks for the light; is this the right issue to follow to see when it is woking? #2012

@behlendorf
Copy link
Contributor

@mailinglists35 that's related but unnecessary to fix this. The script already has the guid, we just need to extend the utilities and I don't think there an issue open for that.

@mailinglists35
Copy link

@behlendorf I'm confused, then where/what should I watch for to know when the utilities can do that?

The problem that actually I need to solve is to have zfs automatically use a hot spare when i/o or checksum happen on an old hp P400 controller having each physical drive behind a corresponding logicaldrive under /dev/cciss/c0dX (the new "hpsa" driver replacing "cciss" which has normal scsi naming /dev/sdX does not support the p400 controller).
I did some workaround in issue #3478 by creating with mknod equivalent entries in /dev/sdXY for each cciss/c0dX and cciss/c0dXpY (because otherwise zpool would not partition the disks when creating the pool) but I'm not sure autoreplace will happen when a drive will fail and the system will not have remote hands.

@mailinglists35 mailinglists35 mentioned this issue Jun 22, 2015
Closed
@don-brady don-brady mentioned this issue Nov 2, 2015
Closed
@kernelOfTruth kernelOfTruth mentioned this issue Jul 23, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: ZED ZFS Event Daemon Type: Feature Feature request or new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@behlendorf @ryao @dweeezil @mailinglists35 @cohdavid