Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix growpart to avoid race in partition table reload #1058

Merged
merged 1 commit into from
Aug 23, 2020

Conversation

bcressey
Copy link
Contributor

Issue number:
#845

Description of changes:
udevd watches files in /dev via inotify, and when a block disk device is closed after writing, it tells the kernel to reload the partition table.

This was racing with the reload code in growpart; if we were unlucky in our timing, the kernel would be busy with udevd's reload and our call would fail.

To fix this, we stop reloading the partition table in growpart, and use inotify to determine when udevd has deleted and recreated the partition links.

Testing done:
Tested launches and reboots of 250 instances in a cluster. All came up successfully and resized the partition to the expected 20 GiB size.

# systemctl status prepare-local
● prepare-local.service - Prepare Local Directory (/local)
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/prepare-local.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2020-08-21 19:05:08 UTC; 22min ago
   Main PID: 2049 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 9185)
     Memory: 0B
     CGroup: /system.slice/prepare-local.service

Aug 21 19:05:07 localhost growpart[1911]: read partition table from nvme1n1
Aug 21 19:05:07 localhost growpart[1911]: wrote partition table to /dev/nvme1n1
Aug 21 19:05:07 localhost growpart[1911]: saw BOTTLEROCKET-DATA link deleted
Aug 21 19:05:07 localhost growpart[1911]: saw BOTTLEROCKET-DATA link created
Aug 21 19:05:07 localhost growpart[2008]: read partition table from nvme1n1
Aug 21 19:05:07 localhost growpart[2008]: wrote partition table to /dev/nvme1n1
Aug 21 19:05:07 localhost growpart[2008]: saw BOTTLEROCKET-DATA link deleted
Aug 21 19:05:07 localhost growpart[2008]: saw BOTTLEROCKET-DATA link created
Aug 21 19:05:08 localhost systemd-growfs[2026]: Successfully resized "/local" to 19.9G bytes.
Aug 21 19:05:08 localhost systemd[1]: Finished Prepare Local Directory (/local).

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@bcressey bcressey linked an issue Aug 21, 2020 that may be closed by this pull request
@bcressey bcressey requested review from tjkirch and samuelkarp August 21, 2020 19:32
@bcressey bcressey added this to the GA milestone Aug 21, 2020
sources/growpart/src/diskpart.rs Outdated Show resolved Hide resolved
sources/growpart/src/diskpart.rs Outdated Show resolved Hide resolved
udevd watches files in /dev via inotify, and when a block disk device
is closed after writing, it tells the kernel to reload the partition
table.

This was racing with the reload code in growpart; if we were unlucky
in our timing, the kernel would be busy with udevd's reload and our
call would fail.

To fix this, we stop reloading the partition table in growpart, and
use inotify to determine when udevd has deleted and recreated the
partition links.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
@bcressey bcressey merged commit f6d9fcd into bottlerocket-os:develop Aug 23, 2020
@bcressey bcressey deleted the fix-growpart-race branch August 23, 2020 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

occasional failure to prepare local storage on boot
3 participants