ZFS stuck

[edillmann]

Hi,

I'm did pull the following pull-requests over current master
#1610
#1612
#1614
#1496

zfs build fine but after some time (several hours). zfs is stuck (no io), and I get the following kernel traces

http://pastebin.com/jv3qMVv9

ps|grep zfs show's only a zfs snapshot command

kernel is 3.9.2

[edillmann]

@dweeezil i did of course pull dweeezil/zfs@a0dc667

regards,
Eric

[dweeezil]

@edillmann I've been meaning to try running the Illumos 2882/2883/2900 with some of those recent changes but haven't had a chance to yet. I'm leaving for vacation today so I'm going to be off the grid for a week. I've been trying to keep the patch rebased to current master as well as possible and maybe by the time I get back, some of those other things will have been committed.

Taking a quick look at your stacks, I'll take a shot in the dark and ask whether you've got all your devices using the noop scheduler? Presumably this hang is hit-or-miss to reproduce?

[edillmann]

In fact it seems to be the deadline scheduler :-( !!!

[dweeezil]

@edillmann De-cloaking on my vacation for a moment: The one thing I think is a pain in the butt when using partitions (I always use my own GPT partitions) is to set noop scheduler on the underlying disks. I have a feeling it would be very difficult or hack-ish to get ZFS to do that job automatically.

[casualfish]

@edillmann Does this bug also happen without pulling these changes? Since there are multiple commits involved perhaps you can use git bisect to locate the specific commit causing this issue.

[edillmann]

@casualfish yes, in fact this bug first appears under last master (without patches applied)
In fact i do no understand the relation between io-scheduler noop and zfs, but it seem's to but that way.
For now i force the io-scheduler to be noop by appending (elevator=noop) to kernel command line.
I lost this param when I did upgrade the kernel.

[casualfish]

@edillmann The interaction between vdevs and linux kernel block layer is in file vdev_disk.c. You needn't have to manually set elevator=noop since it's the default scheduler when opening a vdev(disk type) https://github.com/zfsonlinux/zfs/blob/master/module/zfs/vdev_disk.c#L312. According to your kernel traces it looks like deadlock has happened. Could you describe your workload and the exact steps to trigger this bug?

[atonkyra]

Out of curiosity, are you running a pre-emptible kernel?

[edillmann]

@atonkyra no for now i'am using "No Forced Preemption (Server)" Preemption model.
Should i use Voluntary Kernel Preemption ?

[atonkyra]

@edillmann I think trying it out might be worth the shot.

[edillmann]

@atonkyra i will give it a try

[atonkyra]

@edillmann Okay, please report back with any results you get :)

[atonkyra]

Any updates @edillmann ?

[edillmann]

Hi, for now (5 days), changing preemption model to Voluntary Kernel Preemption did the trick, but for how long, who knows, let wait severals days more

[edillmann]

For now 19 days uptime, i'm closing this one.
definitively changing preemption model to Voluntary Kernel Preemption did the trick.

[behlendorf]

@edillmann Thanks for the update.