Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(vhost-user-blk): switch to crosvm backend in functional tests #4247

Merged
merged 4 commits into from
Nov 20, 2023

Conversation

kalyazin
Copy link
Contributor

@kalyazin kalyazin commented Nov 17, 2023

This is to address intermittent failures related to vhost-user-blk.
Unfortunately this change cannot be split into multiple commits due to high level of entanglement of the changes.

Changes

Switching from the Qemu to crosvm backend comes with the challenge that crosvm locks the disk file exclusively when opening it as rw (which makes sense). We have to make copies of the disk files before we give them to crosvm if we don't configure crosvm as readonly.

One of the ripples of the above is an existing race condition in the partuuid_and_disk_path_host fixture over the disk file kicks in. We are removing this fixture altogether because now we use per microVM copies of the disks.

Finally, this change explicitly configures performance tests to switch to the Qemu backend to preserve the status quo. We may switch them to the crosvm backend too at a later stage.

Reason

The Qemu backend uses direct IO when working with the disk file which makes it scale very poorly. That results in intermittent test failures, because our functional integration tests run in parallel, and our ssh connection timeouts expire if the microVM does not boot in time.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following
Developer Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • [ ] If a specific issue led to this PR, this PR closes the issue.
  • The description of changes is clear and encompassing.
  • [ ] Any required documentation changes (code and docs) are included in this PR.
  • [ ] API changes follow the Runbook for Firecracker API changes.
  • [ ] User-facing changes are mentioned in CHANGELOG.md.
  • All added/changed functionality is tested.
  • New TODOs link to an issue.
  • Commits meet contribution quality standards.

  • This functionality cannot be added in rust-vmm.

Copy link

codecov bot commented Nov 17, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (edbdf2c) 81.71% compared to head (7f16549) 81.69%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4247      +/-   ##
==========================================
- Coverage   81.71%   81.69%   -0.03%     
==========================================
  Files         240      240              
  Lines       29239    29239              
==========================================
- Hits        23893    23887       -6     
- Misses       5346     5352       +6     
Flag Coverage Δ
4.14-c7g.metal 77.16% <100.00%> (ø)
4.14-m5d.metal 79.04% <100.00%> (ø)
4.14-m6a.metal 78.17% <100.00%> (ø)
4.14-m6g.metal 77.16% <100.00%> (ø)
4.14-m6i.metal 79.04% <100.00%> (ø)
5.10-c7g.metal 80.04% <100.00%> (ø)
5.10-m5d.metal 81.70% <100.00%> (ø)
5.10-m6a.metal 80.92% <100.00%> (ø)
5.10-m6g.metal 80.04% <100.00%> (ø)
5.10-m6i.metal 81.69% <100.00%> (ø)
6.1-c7g.metal 80.04% <100.00%> (ø)
6.1-m5d.metal 81.70% <100.00%> (-0.02%) ⬇️
6.1-m6a.metal 80.92% <100.00%> (ø)
6.1-m6g.metal 80.04% <100.00%> (ø)
6.1-m6i.metal 81.69% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This removes 256 byte offest when requesting config
from backend. There is no apparent reason why
we should add the offset.

Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
Unfortunately this change cannot be split into
multiple commits due to high level of entanglement
of the changes.

Reason for the change: the Qemu backend uses direct IO
when working with the disk file which makes it
scale very poorly.  That results in intermittent
test failures, because our functional integration tests
run in parallel, and our ssh connection timeouts
expire if the microVM does not boot in time.

Switching from the Qemu to crosvm backend comes with
the challenge that crosvm locks the disk file exclusively
when opening it as rw (which makes sense).  We have
to make copies of the disk files before we give
them to crosvm if we don't configure crosvm as readonly.

One of the ripples of the above is an existing race
condition in the partuuid_and_disk_path_host fixture
over the disk file kicks in.  We are removing this
fixture altogether because now we use per microVM
copies of the disks.

Finally, this change explicitly configures
performance tests to switch to the Qemu backend
to preserve the status quo.  We may switch them
to the crosvm backend too at a later stage.

Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
@kalyazin kalyazin marked this pull request as ready for review November 17, 2023 18:18
@kalyazin kalyazin self-assigned this Nov 17, 2023
@kalyazin kalyazin added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Nov 17, 2023
Copy link
Contributor

@bchalios bchalios left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. just a single comment for code deduplication

@kalyazin kalyazin merged commit 3f172b9 into firecracker-microvm:main Nov 20, 2023
7 checks passed
@kalyazin kalyazin deleted the vub_conn_fix branch November 20, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Awaiting review Indicates that a pull request is ready to be reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants