zfs mount -a: double free/segfault when mountpoint of dataset is not empty #9560

steven-omaha · 2019-11-07T13:41:48Z

System information

Type	Version/Name
Distribution Name	Arch Linux
Distribution Version	Rolling release / latest
Linux Kernel	5.3.8
Architecture	x86_64
ZFS Version	0.8.2
SPL Version	0.8.2

Describe the problem you're observing

EDIT: Running zfs mount -a multiple times when the mountpoint is not empty causes a double free / segfault.

Describe how to reproduce the problem

EDIT: See the comment further down. #9560 (comment)

Include any warning/errors/backtraces from the system logs

I'm seeing two different kinds of coredumps.
Type 1:

           PID: 1415 (zfs)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2019-11-07 12:50:21 CET (1h 47min ago)
  Command Line: zfs mount -a
    Executable: /usr/bin/zfs
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (xxxxxxxxx)
       Boot ID: aba23826a7c54c36856aadbbbf311b65
    Machine ID: b7c6e36412f14ccdbb6e8ae854ae3ee2
      Hostname: xxxxxxxxxx
       Storage: /var/lib/systemd/coredump/core.zfs.0.aba23826a7c54c36856aadbbbf311b65.1415.1573127421000000000000.lz4 (inaccessible)
       Message: Process 1415 (zfs) of user 0 dumped core.
                
                Stack trace of thread 1417:
                #0  0x00007f0c495de560 __memchr_avx2 (libc.so.6)
                #1  0x00007f0c494f74fc _IO_getline_info (libc.so.6)
                #2  0x00007f0c495011c5 fgets_unlocked (libc.so.6)
                #3  0x00007f0c49578e0e __getmntent_r (libc.so.6)
                #4  0x00007f0c496f1f45 _sol_getmntent (libuutil.so.1)
                #5  0x00007f0c496f1fd4 getmntany (libuutil.so.1)
                #6  0x00007f0c49689930 libzfs_mnttab_find (libzfs.so.2)
                #7  0x00007f0c49695c7f is_mounted (libzfs.so.2)
                #8  0x0000561d088c6913 n/a (zfs)
                #9  0x0000561d088c7efc n/a (zfs)
                #10 0x00007f0c49695acc n/a (libzfs.so.2)
                #11 0x00007f0c496ca092 n/a (libzfs.so.2)
                #12 0x00007f0c496514cf start_thread (libpthread.so.0)
                #13 0x00007f0c495802d3 __clone (libc.so.6)
                
                Stack trace of thread 1416:
                #0  0x00007f0c49503466 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
                #1  0x00007f0c49504676 _IO_default_uflow (libc.so.6)
                #2  0x00007f0c494f753c _IO_getline_info (libc.so.6)
                #3  0x00007f0c495011c5 fgets_unlocked (libc.so.6)
                #4  0x00007f0c49578e0e __getmntent_r (libc.so.6)
                #5  0x00007f0c496f1f45 _sol_getmntent (libuutil.so.1)
                #6  0x00007f0c496f1fd4 getmntany (libuutil.so.1)
                #7  0x00007f0c49689930 libzfs_mnttab_find (libzfs.so.2)
                #8  0x00007f0c49695c7f is_mounted (libzfs.so.2)
                #9  0x0000561d088c6913 n/a (zfs)
                #10 0x0000561d088c7efc n/a (zfs)
                #11 0x00007f0c49695acc n/a (libzfs.so.2)
                #12 0x00007f0c496ca092 n/a (libzfs.so.2)
                #13 0x00007f0c496514cf start_thread (libpthread.so.0)
                #14 0x00007f0c495802d3 __clone (libc.so.6)
                
                Stack trace of thread 1415:
                #0  0x00007f0c49657c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f0c496ca98e tpool_wait (libzfs.so.2)
                #2  0x00007f0c496972e0 zfs_foreach_mountpoint (libzfs.so.2)
                #3  0x0000561d088cb322 n/a (zfs)
                #4  0x0000561d088be3b7 n/a (zfs)
                #5  0x00007f0c494a8153 __libc_start_main (libc.so.6)
                #6  0x0000561d088be4de n/a (zfs)
                
                Stack trace of thread 1419:
                #0  0x00007f0c49657c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f0c496c9e24 n/a (libzfs.so.2)
                #2  0x00007f0c496514cf start_thread (libpthread.so.0)
                #3  0x00007f0c495802d3 __clone (libc.so.6)
                
                Stack trace of thread 1418:
                #0  0x00007f0c49657c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f0c496c9e24 n/a (libzfs.so.2)
                #2  0x00007f0c496514cf start_thread (libpthread.so.0)
                #3  0x00007f0c495802d3 __clone (libc.so.6)

Type 2:

           PID: 8301 (zfs)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2019-11-07 12:57:30 CET (1h 41min ago)
  Command Line: zfs mount -a
    Executable: /usr/bin/zfs
 Control Group: /user.slice/user-1000.slice/session-3.scope
          Unit: session-3.scope
         Slice: user-1000.slice
       Session: 3
     Owner UID: 1000 (xxxxxxxx)
       Boot ID: aba23826a7c54c36856aadbbbf311b65
    Machine ID: b7c6e36412f14ccdbb6e8ae854ae3ee2
      Hostname: xxxxxxxx
       Storage: /var/lib/systemd/coredump/core.zfs.0.aba23826a7c54c36856aadbbbf311b65.8301.1573127850000000000000.lz4 (inaccessible)
       Message: Process 8301 (zfs) of user 0 dumped core.
                
                Stack trace of thread 8305:
                #0  0x00007f4ffa0705a0 __malloc_arena_thread_freeres (libc.so.6)
                #1  0x00007f4ffa1b54f5 start_thread (libpthread.so.0)
                #2  0x00007f4ffa0e42d3 __clone (libc.so.6)
                
                Stack trace of thread 8301:
                #0  0x00007f4ffa2677e3 n/a (libnvpair.so.1)
                #1  0x00007f4ffa26770a nvlist_free (libnvpair.so.1)
                #2  0x00007f4ffa26770a nvlist_free (libnvpair.so.1)
                #3  0x00007f4ffa1ecebd zfs_close (libzfs.so.2)
                #4  0x00005652dd8f6345 n/a (zfs)
                #5  0x00005652dd8e93b7 n/a (zfs)
                #6  0x00007f4ffa00c153 __libc_start_main (libc.so.6)
                #7  0x00005652dd8e94de n/a (zfs)

The text was updated successfully, but these errors were encountered:

AndCycle · 2019-11-07T13:47:47Z

zfs --version to see if you have correct version of kernel module and userland utility.

steven-omaha · 2019-11-07T13:53:01Z

$ zfs --version
zfs-0.8.2-1
zfs-kmod-0.8.2-1

steven-omaha · 2019-11-07T14:59:53Z

Something is strange. zfs mount -a doesn't want to mount all datasets.

$ zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               30.5G   192G       96K  /zroot
zroot/ROOT          28.9G   192G       96K  none
zroot/ROOT/default  28.9G   192G     28.9G  /
zroot/cache         1.58G   192G     1.58G  /zroot/cache
zroot/home            96K   192G       96K  /home
~ $ zfs mount -a
cannot mount '/home': directory is not empty
cannot mount '/zroot': directory is not empty
double free or corruption (fasttop)
[1]    2304 abort (core dumped)  zfs mount -a
~ [134]$ zfs mount -a
cannot mount '/zroot':
cannot mount '/zroot': mount failed
~ [1]$

Also, I got another different core dump this time.

           PID: 2304 (zfs)
           UID: 1000 (xxxxxxx)
           GID: 1000 (xxxxxxx)
        Signal: 6 (ABRT)
     Timestamp: Thu 2019-11-07 15:53:26 CET (5min ago)
  Command Line: zfs mount -a
    Executable: /usr/bin/zfs
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (xxxxxxx)
       Boot ID: 5d11adef8b744cb9b3bf67dd33002281
    Machine ID: b7c6e36412f14ccdbb6e8ae854ae3ee2
      Hostname: xxxxxxx
       Storage: /var/lib/systemd/coredump/core.zfs.1000.5d11adef8b744cb9b3bf67dd33002281.2304.1573138406000000000000.lz4
       Message: Process 2304 (zfs) of user 1000 dumped core.
                
                Stack trace of thread 2305:
                #0  0x00007f1e887a2f25 raise (libc.so.6)
                #1  0x00007f1e8878c897 abort (libc.so.6)
                #2  0x00007f1e887e6258 __libc_message (libc.so.6)
                #3  0x00007f1e887ed77a malloc_printerr (libc.so.6)
                #4  0x00007f1e887ef295 _int_free (libc.so.6)
                #5  0x00007f1e8896f820 libzfs_mnttab_fini (libzfs.so.2)
                #6  0x00007f1e8896f9e1 libzfs_mnttab_find (libzfs.so.2)
                #7  0x00007f1e8897bc7f is_mounted (libzfs.so.2)
                #8  0x000055d8087ff913 n/a (zfs)
                #9  0x000055d808800efc n/a (zfs)
                #10 0x00007f1e8897bacc n/a (libzfs.so.2)
                #11 0x00007f1e889b0092 n/a (libzfs.so.2)
                #12 0x00007f1e889374cf start_thread (libpthread.so.0)
                #13 0x00007f1e888662d3 __clone (libc.so.6)
                
                Stack trace of thread 2306:
                #0  0x00007f1e8885742c __read (libc.so.6)
                #1  0x00007f1e887e9442 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
                #2  0x00007f1e887ea676 _IO_default_uflow (libc.so.6)
                #3  0x00007f1e887dd53c _IO_getline_info (libc.so.6)
                #4  0x00007f1e887e71c5 fgets_unlocked (libc.so.6)
                #5  0x00007f1e8885ee0e __getmntent_r (libc.so.6)
                #6  0x00007f1e889d7f45 _sol_getmntent (libuutil.so.1)
                #7  0x00007f1e889d7fd4 getmntany (libuutil.so.1)
                #8  0x00007f1e8896f930 libzfs_mnttab_find (libzfs.so.2)
                #9  0x00007f1e8897bc7f is_mounted (libzfs.so.2)
                #10 0x000055d8087ff913 n/a (zfs)
                #11 0x000055d808800efc n/a (zfs)
                #12 0x00007f1e8897bacc n/a (libzfs.so.2)
                #13 0x00007f1e889b0092 n/a (libzfs.so.2)
                #14 0x00007f1e889374cf start_thread (libpthread.so.0)
                #15 0x00007f1e888662d3 __clone (libc.so.6)
                
                Stack trace of thread 2308:
                #0  0x00007f1e8893dc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1e889afe24 n/a (libzfs.so.2)
                #2  0x00007f1e889374cf start_thread (libpthread.so.0)
                #3  0x00007f1e888662d3 __clone (libc.so.6)
                
                Stack trace of thread 2307:
                #0  0x00007f1e8893dc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1e889afe24 n/a (libzfs.so.2)
                #2  0x00007f1e889374cf start_thread (libpthread.so.0)
                #3  0x00007f1e888662d3 __clone (libc.so.6)
                
                Stack trace of thread 2304:
                #0  0x00007f1e8893dc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1e889b098e tpool_wait (libzfs.so.2)
                #2  0x00007f1e8897d2e0 zfs_foreach_mountpoint (libzfs.so.2)
                #3  0x000055d808804322 n/a (zfs)
                #4  0x000055d8087f73b7 n/a (zfs)
                #5  0x00007f1e8878e153 __libc_start_main (libc.so.6)
                #6  0x000055d8087f74de n/a (zfs)

steven-omaha · 2019-11-07T19:06:12Z

I'm now able to reproduce the issue on a different machine (the same system information from the first post still apply). This example is probably not minimal (I did the same error twice), but matches the procedure where the bug appeared the first time.

Boiled down: The bug seems to occur when a dataset cannot be mounted, because the mountpoint is not empty.

mkdir -p /tmp/zfs
truncate -s 1G /tmp/zfs/disk1
truncate -s 1G /tmp/zfs/disk2
zpool create -o ashift=12 tank mirror /tmp/zfs/disk1 /tmp/zfs/disk2
zpool export tank
zpool import -R /mnt -d /tmp/zfs tank
zfs create -o canmount=off tank/ROOT
zfs create -o canmount=on tank/ROOT/default
zfs set mountpoint=/ tank/ROOT/default
mkdir /mnt/home
dd if=/dev/urandom bs=32M count=4 of=/mnt/home/out.dat
zfs create tank/home
zfs set mountpoint=/home tank/home
mkdir /mnt/tank/cache
dd if=/dev/urandom bs=32M count=1 of=/mnt/tank/cache/out.dat
zfs set mountpoint=none tank/ROOT
zfs create tank/cache
zpool export tank

(I did a reboot here, not sure if this is necessary to reproduce)

zpool import -R /mnt -d /tmp/zfs tank
zfs mount -a
zfs mount -a
zfs mount -a

Output

/mnt # zfs mount -a
cannot mount '/mnt/home': directory is not empty
cannot mount '/mnt/tank/cache': directory is not empty
/mnt [1]# zfs mount -a
cannot mount '/mnt/home': directory is not empty
cannot mount '/mnt/tank/cache': directory is not empty
free(): double free detected in tcache 2
[1]    3701 abort (core dumped)  zfs mount -a
/mnt [134]# zfs mount -a
cannot mount '/mnt/home': directory is not empty
cannot mount '/mnt/tank/cache': directory is not empty
[1]    4142 segmentation fault (core dumped)  zfs mount -a
/mnt [139]# zfs mount -a

Coredump 1:

           PID: 3701 (zfs)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Thu 2019-11-07 19:51:37 CET (11min ago)
  Command Line: zfs mount -a
    Executable: /usr/bin/zfs
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (xxxxxxxx)
       Boot ID: d65d5dfe747745d69d0d9e1890a8feca
    Machine ID: 8f0c7880d3ae4a0dbf4a428281f020ad
      Hostname: xxxxxxxxx
       Storage: /var/lib/systemd/coredump/core.zfs.0.d65d5dfe747745d69d0d9e1890a8feca.3701.1573152697000000000000.lz4
       Message: Process 3701 (zfs) of user 0 dumped core.
                
                Stack trace of thread 3706:
                #0  0x00007f7cfa4c8f25 raise (libc.so.6)
                #1  0x00007f7cfa4b2897 abort (libc.so.6)
                #2  0x00007f7cfa50c258 __libc_message (libc.so.6)
                #3  0x00007f7cfa51377a malloc_printerr (libc.so.6)
                #4  0x00007f7cfa51559d _int_free (libc.so.6)
                #5  0x00007f7cfa695815 libzfs_mnttab_fini (libzfs.so.2)
                #6  0x00007f7cfa6959e1 libzfs_mnttab_find (libzfs.so.2)
                #7  0x00007f7cfa6a1c7f is_mounted (libzfs.so.2)
                #8  0x000055afe1285913 n/a (zfs)
                #9  0x000055afe1286efc n/a (zfs)
                #10 0x00007f7cfa6a1acc n/a (libzfs.so.2)
                #11 0x00007f7cfa6d6092 n/a (libzfs.so.2)
                #12 0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #13 0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3709:
                #0  0x00007f7cfa57d42c __read (libc.so.6)
                #1  0x00007f7cfa50f442 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
                #2  0x00007f7cfa510676 _IO_default_uflow (libc.so.6)
                #3  0x00007f7cfa50353c _IO_getline_info (libc.so.6)
                #4  0x00007f7cfa50d1c5 fgets_unlocked (libc.so.6)
                #5  0x00007f7cfa584e0e __getmntent_r (libc.so.6)
                #6  0x00007f7cfa6fdf45 _sol_getmntent (libuutil.so.1)
                #7  0x00007f7cfa6fdfd4 getmntany (libuutil.so.1)
                #8  0x00007f7cfa695930 libzfs_mnttab_find (libzfs.so.2)
                #9  0x00007f7cfa6a1c7f is_mounted (libzfs.so.2)
                #10 0x000055afe1285913 n/a (zfs)
                #11 0x000055afe1286efc n/a (zfs)
                #12 0x00007f7cfa6a1acc n/a (libzfs.so.2)
                #13 0x00007f7cfa6d6092 n/a (libzfs.so.2)
                #14 0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #15 0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3710:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3701:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d698e tpool_wait (libzfs.so.2)
                #2  0x00007f7cfa6a32e0 zfs_foreach_mountpoint (libzfs.so.2)
                #3  0x000055afe128a322 n/a (zfs)
                #4  0x000055afe127d3b7 n/a (zfs)
                #5  0x00007f7cfa4b4153 __libc_start_main (libc.so.6)
                #6  0x000055afe127d4de n/a (zfs)
                
                Stack trace of thread 3708:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3713:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3705:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3703:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3702:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3704:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3712:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3711:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)
                
                Stack trace of thread 3707:
                #0  0x00007f7cfa663c45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f7cfa6d5e24 n/a (libzfs.so.2)
                #2  0x00007f7cfa65d4cf start_thread (libpthread.so.0)
                #3  0x00007f7cfa58c2d3 __clone (libc.so.6)

Coredump 2:

           PID: 4142 (zfs)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2019-11-07 19:51:42 CET (12min ago)
  Command Line: zfs mount -a
    Executable: /usr/bin/zfs
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (xxxxxxxxxxx)
       Boot ID: d65d5dfe747745d69d0d9e1890a8feca
    Machine ID: 8f0c7880d3ae4a0dbf4a428281f020ad
      Hostname: xxxxxxxx
       Storage: /var/lib/systemd/coredump/core.zfs.0.d65d5dfe747745d69d0d9e1890a8feca.4142.1573152702000000000000.lz4
       Message: Process 4142 (zfs) of user 0 dumped core.
                
                Stack trace of thread 4150:
                #0  0x00007f6fd8847466 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
                #1  0x00007f6fd8848676 _IO_default_uflow (libc.so.6)
                #2  0x00007f6fd883b53c _IO_getline_info (libc.so.6)
                #3  0x00007f6fd88451c5 fgets_unlocked (libc.so.6)
                #4  0x00007f6fd88bce0e __getmntent_r (libc.so.6)
                #5  0x00007f6fd8a35f45 _sol_getmntent (libuutil.so.1)
                #6  0x00007f6fd8a35fd4 getmntany (libuutil.so.1)
                #7  0x00007f6fd89cd930 libzfs_mnttab_find (libzfs.so.2)
                #8  0x00007f6fd89d9c7f is_mounted (libzfs.so.2)
                #9  0x000055b0d0bf2913 n/a (zfs)
                #10 0x000055b0d0bf3efc n/a (zfs)
                #11 0x00007f6fd89d9acc n/a (libzfs.so.2)
                #12 0x00007f6fd8a0e092 n/a (libzfs.so.2)
                #13 0x00007f6fd89954cf start_thread (libpthread.so.0)
                #14 0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4143:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4148:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4145:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4152:
                #0  0x00007f6fd88b542c __read (libc.so.6)
                #1  0x00007f6fd8847442 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
                #2  0x00007f6fd8848676 _IO_default_uflow (libc.so.6)
                #3  0x00007f6fd883b53c _IO_getline_info (libc.so.6)
                #4  0x00007f6fd88451c5 fgets_unlocked (libc.so.6)
                #5  0x00007f6fd88bce0e __getmntent_r (libc.so.6)
                #6  0x00007f6fd8a35f45 _sol_getmntent (libuutil.so.1)
                #7  0x00007f6fd8a35fd4 getmntany (libuutil.so.1)
                #8  0x00007f6fd89cd930 libzfs_mnttab_find (libzfs.so.2)
                #9  0x00007f6fd89d9c7f is_mounted (libzfs.so.2)
                #10 0x000055b0d0bf2913 n/a (zfs)
                #11 0x000055b0d0bf3efc n/a (zfs)
                #12 0x00007f6fd89d9acc n/a (libzfs.so.2)
                #13 0x00007f6fd8a0e092 n/a (libzfs.so.2)
                #14 0x00007f6fd89954cf start_thread (libpthread.so.0)
                #15 0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4151:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4154:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4153:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4147:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4146:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4149:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)
                
                Stack trace of thread 4142:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0e98e tpool_wait (libzfs.so.2)
                #2  0x00007f6fd89db2e0 zfs_foreach_mountpoint (libzfs.so.2)
                #3  0x000055b0d0bf7322 n/a (zfs)
                #4  0x000055b0d0bea3b7 n/a (zfs)
                #5  0x00007f6fd87ec153 __libc_start_main (libc.so.6)
                #6  0x000055b0d0bea4de n/a (zfs)
                
                Stack trace of thread 4144:
                #0  0x00007f6fd899bc45 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f6fd8a0de24 n/a (libzfs.so.2)
                #2  0x00007f6fd89954cf start_thread (libpthread.so.0)
                #3  0x00007f6fd88c42d3 __clone (libc.so.6)

turbofx · 2019-11-10T19:36:48Z

I can confirm I'm also seeing this issue on "CentOS Linux release 7.7.1908 (Core)"
Kernel version: 3.10.0-1062.4.1.el7.x86_64
ZFS release: 0.8.2

Once I clear the files from the directory, I can issue a "zfs mount mountpoint and it works fine.

As an aside, On Solaris, this is a boot halting event. I'd like to argue for this being an option that could be enabled here as well.

ghost · 2019-11-12T16:40:29Z

Same issue on Debian 9 with ZFS 0.8.2.

kusumi · 2019-12-16T16:36:17Z

I'll take a look if I can reproduce this with Fedora (haven't tried yet).

h1z1 · 2019-12-18T02:12:32Z

Hit this yesterday, same scenario - zfs mount -a
zfs 0.8.0-rc1

Unfortunately it was in single user and no syslog running.

kusumi · 2019-12-18T12:42:49Z

Then this isn't a regression from recent code. I don't recall anyone reported this.

steven-omaha · 2019-12-18T15:02:18Z

I come to a different conclusion based on git bisect. Tested on 4.19.89-1-lts in a virtualbox.

git bisect reset
git bisect start
git bisect good zfs-0.8.0
git bisect bad master
git bisect run zsh -c 'make clean; make distclean; ./autogen.sh; ./configure; make -j4; sudo ./mytest.zsh'

Where mytest.zsh is this one: https://gist.github.com/steven-omaha/0c722b678e30030390c803c6ec3853b8

Output of git bisect log: https://gist.github.com/steven-omaha/497a6c83420beed56704fca51b79aa1e

The culprit seems to be a9cd8bf, which kind of makes sense.

steven-omaha · 2019-12-18T15:34:26Z

Paging @don-brady

TerraTech · 2020-02-23T22:50:38Z

I am hitting this exact same bug on a Gentoo system:
linux 4.19.97
zfs 0.8.3 (builtin module + tools)

Is there a patch available, or is reverting a9cd8bf a viable solution?

bill-mcgonigle · 2020-02-24T20:06:23Z

Seeing this on Debian, current buster-backports dkms:

zfs-0.8.3-1~bpo10+1
zfs-kmod-0.8.3-1~bpo10+1
4.19.0-8-amd64

[  938.580748] ZFS: Loaded module v0.8.3-1~bpo10+1, ZFS pool version 5000, ZFS filesystem version 5
[  967.821373]  zd0: p1 p2 p3
[  967.857106]  zd16: p1
[  967.913742]  zd32: p1
[  967.981277]  zd48: p1 p2 < p5 >
[  968.044034]  zd64: p1
[  968.097512]  zd80: p1
[  976.060629] zfs[3534]: segfault at 0 ip 00007faa7f0d6694 sp 00007faa75ff3420 error 4 in libc-2.28.so[7faa7f07c000+148000]
[  976.060638] zfs[3618]: segfault at 0 ip 00007faa7f1b201c sp 00007faa757f2478 error 4
[  976.060665]  in libc-2.28.so[7faa7f07c000+148000]
[  976.060683] Code: 29 f2 41 ff 55 70 48 85 c0 7e 3b 48 8b 93 90 00 00 00 48 01 43 10 48 83 fa ff 74 0a 48 01 d0 48 89 83 90 00 00 00 48 8b 43 08 <0f> b6 00 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f 1f
[  976.060691] Code: 29 c8 c5 f8 77 c3 0f 1f 84 00 00 00 00 00 48 85 d2 0f 84 5a 02 00 00 89 f9 c5 f9 6e c6 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 44 <c5> fd 74 0f c5 fd d7 c1 85 c0 0f 85 c4 01 00 00 48 83 ea 20 0f 86

I'll see the segfault with 'zfs' on the command line until I clean up the existing mountpoint (we had two pools with datasets referencing the same mountpoints).

On @steven-omaha 's stackstraces, there are two threads using fgets_unlocked - is that OK since it's not threadsafe?

Dacesilian · 2020-04-20T16:36:10Z

I'm experiencing probably the same problem, details are there - https://forum.proxmox.com/threads/zfs-mount-on-start-problem-segfault-at-0-error-4-in-libc-2-28-so-subvolumes-not-mounted.68519/ .

Is there anything I can provide to solve this problem? Thank you.

TerraTech · 2020-04-20T22:54:10Z

@Dacesilian Reverting a9cd8bf has been working fine for me as a bandaid until a formal solution is implemented.

This reverts commit a9cd8bf which introduced a segfault when running `zfs mount -a` multiple times when there are mountpoints which are not empty. This segfault is now seen frequently by the CI after the mount code was updated to directly call mount(2). The original reason this logic was added is described in openzfs#8881. Since then the systemd `zfs-share.target` has been updated to run "After" the `zfs-mount.server` which should avoid this issue. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#9560

behlendorf · 2020-05-26T23:11:13Z

I've reverted commit a9cd8bf to resolve the segfault. The added After systemd dependency should prevent this issue on boot.

This reverts commit a9cd8bf which introduced a segfault when running `zfs mount -a` multiple times when there are mountpoints which are not empty. This segfault is now seen frequently by the CI after the mount code was updated to directly call mount(2). The original reason this logic was added is described in openzfs#8881. Since then the systemd `zfs-share.target` has been updated to run "After" the `zfs-mount.server` which should avoid this issue. Reviewed-by: Don Brady <don.brady@delphix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#9560 Closes openzfs#10364 (cherry picked from commit d1b84da)

This reverts commit a9cd8bf which introduced a segfault when running `zfs mount -a` multiple times when there are mountpoints which are not empty. This segfault is now seen frequently by the CI after the mount code was updated to directly call mount(2). The original reason this logic was added is described in openzfs#8881. Since then the systemd `zfs-share.target` has been updated to run "After" the `zfs-mount.server` which should avoid this issue. Reviewed-by: Don Brady <don.brady@delphix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#9560 Closes openzfs#10364

commit 36bf32f Author: TerraTech <TerraTech@users.noreply.github.com> Date: Wed Jan 6 00:50:13 2021 -0800 libzfs_FQ.c: Extend AUTOSNAP handling Motivation: CATALYST zpool/BUILD/BINDER/portage-GC-REPOS and using 'a{d}' * daily is minimum to be picked up by OSCAR backups - * AUTOSNAP := { D | H | Z | m } + * AUTOSNAP := { D | H | M | W | Y | Z | d | m } * H == no Hourly * D == no Daily + * W == no Weekly + * M == no Monthly + * Y == no Yearly * Z == no ZBhold + * d == daily only commit 418dd7c Author: TerraTech <TerraTech@users.noreply.github.com> Date: Tue May 19 16:18:34 2020 -0700 libzfs_FQ.c: refactor(lib): Deprecate 'FQ_OVERRIDE=b', now uses 'FQ_OVERRIDE=B' commit 28e1c80 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sun May 17 03:31:00 2020 -0700 libzfs_FQ.c: bug: Fix 'FQoverride_gtnd()' strtok() usage commit af02cff Author: TerraTech <TerraTech@users.noreply.github.com> Date: Wed May 13 18:30:36 2020 -0700 Fix missing sys/crypto/impl.h when building inline via --enable-builtin CC [M] fs/zfs/icp/illumos-crypto.o fs/zfs/icp/illumos-crypto.c:37:10: fatal error: sys/crypto/impl.h: No such file or directory 37 | #include <sys/crypto/impl.h> | ^~~~~~~~~~~~~~~~~~~ compilation terminated. make[3]: *** [scripts/Makefile.build:266: fs/zfs/icp/illumos-crypto.o] Error 1 make[2]: *** [scripts/Makefile.build:500: fs/zfs/icp] Error 2 make[1]: *** [scripts/Makefile.build:500: fs/zfs] Error 2 make: *** [Makefile:1691: fs] Error 2 commit b159b95 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Tue May 12 12:42:27 2020 -0700 libzfs_FQ.c: refactor(lib): Implement new visibility 'h{...}' (HIDE) grouping commit 03e3a3e Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat May 16 12:13:57 2020 -0700 refactor(lib): Refactor 'FQoverride_gtnd()' and implement envariable 'FQ_GTND_SKIP' commit 427f7d8 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Mon Apr 13 03:29:37 2020 -0700 libzfs_sendrecv.c: feat(lib): Add 'FQgtnd_override()' -- prepping for refactoring commit 7d48a31 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Mon Apr 13 03:36:04 2020 -0700 libzfs_FQ.c: dry(lib): Add 'FQenv_override()' to centralize override env access commit 19a9c60 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat Apr 11 23:19:36 2020 -0700 libzfs_FQ.c: fix(lib): Fix bug where xattrs were only being parsed if 'a{}' was set, ergo missing 'x' parsing commit 5e23cc4 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat Apr 11 23:17:55 2020 -0700 libzfs_FQ.c: feat(lib): Add 'FQdump_xattrs()' stringer commit 088dd5d Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat Apr 11 23:16:14 2020 -0700 libzfs_FQ.h: style(lib): 'FQgothere()' place brackets around function name commit 3a4fcf5 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat Apr 11 17:19:46 2020 -0700 feat(lib): Add 'FQbackup_skip_missing_snapshot()' to handle a long standing backup problem // // Skip if following xattrs: // 1) A // 2) a{D} // // This handles the following backup malfunction that occurs when dataset has been receiving // daily autosnaps, but later - either 'A' or 'a{D}' xattrs has been set. // // ** cannot send zssd@autosnap_2020-04-04_00:00:00_daily recursively: snapshot zssd/bats@autosnap_2020-04-04_00:00:00_daily does not exist // commit 4d0ab5a Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sat Apr 11 17:14:33 2020 -0700 libzfs_FQ.c: refactor(lib): Refactor xattrs retrieval, parsing and 'FQxattrs_has_*()' checking 1) Now scans and parses the 'x' attrs for better accuracy over 'index()' type checks 2) Standardize on passing 'FQxattrs_t *' everywhere, instead of the split brain FQxattrs_t/xattrs_t 3) Decided to pass in '&xattrs' instead of malloc in FQget_xattrs() to reduce the many free() needed commit 28ca767 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Thu Apr 9 12:46:49 2020 -0700 libzfs_FQ.c: fix: If no xattrs, then minutely was being implicitly included. commit 141d79b Author: TerraTech <TerraTech@users.noreply.github.com> Date: Thu Apr 9 11:07:23 2020 -0700 libzfs_FQ.c: refactor(lib): Wire in 'FQsnapshot_filter()' to replace old method commit b51b574 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Thu Apr 9 11:05:50 2020 -0700 libzfs_FQ.c: feat(lib): Add 'FQis_autosnap_minutely()' inclusion test commit 50f279d Author: TerraTech <TerraTech@users.noreply.github.com> Date: Thu Apr 9 11:03:34 2020 -0700 libzfs_FQ.c: refactor: FQxattrs_has_*() interrogators commit e43980b Author: TerraTech <TerraTech@users.noreply.github.com> Date: Wed Apr 8 14:05:26 2020 -0700 libzfs_FQ.c: fix(exclusion): Non-autosnap snapshots were also being excluded commit 97f857f Author: TerraTech <TerraTech@users.noreply.github.com> Date: Tue Apr 7 22:14:43 2020 -0700 libzfs_dataset.c: fix(holds): holdname was being set to dataset name 'nvlist_t *holds' is: key=dataset name val=holdname commit 3dae4af Author: TerraTech <TerraTech@users.noreply.github.com> Date: Sun Apr 5 23:25:41 2020 -0700 Add 'FQexclude_hold()' ability. For now traps 'ZBhold' only. commit 7c49a5e Author: TerraTech <TerraTech@users.noreply.github.com> Date: Fri Apr 3 01:43:07 2020 -0700 libzfs_dataset.c: Wire in snapshot exclusions via 'FQexclude_snapshot()' commit 3bb177e Author: TerraTech <TerraTech@users.noreply.github.com> Date: Fri Apr 3 01:41:36 2020 -0700 libzfs_sendrecv.c: Move out FQ specific infrastructure to libzfs_FQ commit eff14fd Author: TerraTech <TerraTech@users.noreply.github.com> Date: Fri Apr 3 01:32:17 2020 -0700 Add new libzfs_FQ library for backup overrides and snapshot exclusions commit cb9df84 Author: fqbuild <fqbuild@spunky> Date: Wed Mar 11 22:01:25 2020 -0700 libzfs_sendrecv.c: Add FQbackup_override() ability e.g. to override fq:xattrs=B FQ_OVERRIDE=B zfs send ... commit 35b8ee6 Author: Terra <Service@FutureQuest.net> Date: Sun Feb 23 18:24:11 2020 -0500 From 64c5c8750d738ecc502b6a0192feebf33914acbe Mon Sep 17 00:00:00 2001 Subject: [PATCH] Revert "Let zfs mount all tolerate in-progress mounts" This reverts commit a9cd8bf. REF: openzfs#9560 commit 6e597a7 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Fri Mar 15 13:34:33 2019 -0700 From 95ce7929646908673eecfd7f55db6d653feb2fea Mon Sep 17 00:00:00 2001 Subject: [PATCH 2/2] libzfs_sendrecv.c: zfs send dataset exclude commit bfcb121 Author: TerraTech <TerraTech@users.noreply.github.com> Date: Fri Mar 15 13:28:48 2019 -0700 From 3e2dc2c3c5bcc81225449400c01c58d6c5df5027 Mon Sep 17 00:00:00 2001 Subject: [PATCH 1/2] libzfs_sendrecv.c: Squelch "skipping snapshot"

steven-omaha changed the title ~~zfs mount -a: segfault~~ zfs mount -a: double free/segfault when mountpoint of dataset is not empty Nov 7, 2019

behlendorf added the Type: Defect Incorrect behavior (e.g. crash, hang) label Nov 11, 2019

behlendorf added the good first issue Indicates a good issue for first-time contributors label Nov 13, 2019

behlendorf mentioned this issue May 25, 2020

Revert "Let zfs mount all tolerate in-progress mounts" #10364

Merged

12 tasks

behlendorf closed this as completed in d1b84da May 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zfs mount -a: double free/segfault when mountpoint of dataset is not empty #9560

zfs mount -a: double free/segfault when mountpoint of dataset is not empty #9560

steven-omaha commented Nov 7, 2019 •

edited

Loading

AndCycle commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

turbofx commented Nov 10, 2019

ghost commented Nov 12, 2019

kusumi commented Dec 16, 2019

h1z1 commented Dec 18, 2019

kusumi commented Dec 18, 2019

steven-omaha commented Dec 18, 2019

steven-omaha commented Dec 18, 2019

TerraTech commented Feb 23, 2020 •

edited

Loading

bill-mcgonigle commented Feb 24, 2020

Dacesilian commented Apr 20, 2020

TerraTech commented Apr 20, 2020 •

edited

Loading

behlendorf commented May 26, 2020

zfs mount -a: double free/segfault when mountpoint of dataset is not empty #9560

zfs mount -a: double free/segfault when mountpoint of dataset is not empty #9560

Comments

steven-omaha commented Nov 7, 2019 • edited Loading

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

AndCycle commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

steven-omaha commented Nov 7, 2019

turbofx commented Nov 10, 2019

ghost commented Nov 12, 2019

kusumi commented Dec 16, 2019

h1z1 commented Dec 18, 2019

kusumi commented Dec 18, 2019

steven-omaha commented Dec 18, 2019

steven-omaha commented Dec 18, 2019

TerraTech commented Feb 23, 2020 • edited Loading

bill-mcgonigle commented Feb 24, 2020

Dacesilian commented Apr 20, 2020

TerraTech commented Apr 20, 2020 • edited Loading

behlendorf commented May 26, 2020

steven-omaha commented Nov 7, 2019 •

edited

Loading

TerraTech commented Feb 23, 2020 •

edited

Loading

TerraTech commented Apr 20, 2020 •

edited

Loading