mgr/volumes: support to reject CephFS clones if cloner threads are not available #52670

neesingh-rh · 2023-07-27T09:51:16Z

Allow the cloning only when (pending_clones + in-progress_clones) < max_concurrent_clones, otherwise return the error EAGAIN. This way we solves the issue of stale cephfs clones.

Fixes: https://tracker.ceph.com/issues/59714
Signed-off-by: Neeraj Pratap Singh neesingh@redhat.com

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

src/pybind/mgr/volumes/fs/operations/versions/subvolume_v1.py

src/pybind/mgr/volumes/fs/operations/volume.py

src/pybind/mgr/volumes/fs/volume.py

src/pybind/mgr/volumes/fs/operations/volume.py

kotreshhr · 2023-08-07T12:36:01Z

@neesingh-rh I think this should be sent on top of #52765 ?

neesingh-rh · 2023-08-07T15:01:56Z

@neesingh-rh I think this should be sent on top of #52765 ?

Yup, I will do as soon as it merges or should I add those commits in this PR?

kotreshhr · 2023-08-08T08:51:02Z

@neesingh-rh I think this should be sent on top of #52765 ?

Yup, I will do as soon as it merges or should I add those commits in this PR?

Let's wait for it to merge. I think it's in @vshankar 's testing branch.

vshankar · 2023-08-08T09:44:49Z

@neesingh-rh I think this should be sent on top of #52765 ?

Yup, I will do as soon as it merges or should I add those commits in this PR?

Let's wait for it to merge. I think it's in @vshankar 's testing branch.

I'll run PR #52765 through fs:volumes sub-suite to faster the merge process.

vshankar · 2023-08-08T11:57:47Z

jenkins retest this please

kotreshhr

@vshankar I think this feature needs to be optional and enabled by default. It restricts other users to create a clone only when the thread is free. What do you think?

src/pybind/mgr/volumes/fs/volume.py

src/pybind/mgr/volumes/fs/operations/volume.py

qa/tasks/cephfs/test_volumes.py

vshankar · 2023-08-16T08:02:40Z

@vshankar I think this feature needs to be optional and enabled by default. It restricts other users to create a clone only when the thread is free. What do you think?

I think this feature needs to be enabled by the operator.

kotreshhr

Looks good to me otherwise. Let's wait for the input whether we should make this configurable.

src/pybind/mgr/volumes/fs/operations/volume.py

src/pybind/mgr/volumes/fs/volume.py

neesingh-rh · 2023-08-16T11:49:40Z

Looks good to me otherwise. Let's wait for the input whether we should make this configurable.

I was working on addition of this functionality ( taking Venky's suggestion #52670 (comment) in mind for now) and have updated the PR with those changes too, Will do the amendments as per the inputs.

src/pybind/mgr/volumes/fs/async_cloner.py

neesingh-rh · 2024-01-17T14:41:43Z

jenkins retest this please

vshankar

@neesingh-rh PTAL at the failure: /a/vshankar-2024-01-22_07:03:31-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7525715

qa/tasks/cephfs/test_volumes.py

* refs/pull/52670/head: doc: add the reject the clone when threads are not available feature in the doc qa: add test cases for the support to reject clones feature mgr/volumes: support to reject CephFS clones if cloner threads are not available Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

…t available CephFS clone creation have a limit of 4 parallel clones by default at a time and rest of the clone create requests are queued. This makes CephFS cloning very slow when there is large amount of clones being created.After this patch clone requests won't be accepeted when the requests exceed the `max_concurrent_clones` config value. Fixes: https://tracker.ceph.com/issues/59714 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

qa/tasks/cephfs/test_volumes.py

anthonyeleven

wording requests

PendingReleaseNotes

doc/cephfs/fs-volumes.rst

Fixes: https://tracker.ceph.com/issues/59714 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

…in the document Fixes: https://tracker.ceph.com/issues/59714 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

* refs/pull/52670/head: doc: add the reject the clone when threads are not available feature in the document qa: add test cases for the support to reject clones feature mgr/volumes: support to reject CephFS clones if cloner threads are not available Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>

vshankar · 2024-02-19T09:29:39Z

jenkins test windows

vshankar · 2024-02-19T16:02:14Z

@neesingh-rh I had to include an additional commit to gixup failures in the fs:volumes. No need to refresh - I'll merge this and the additional commit tomorrow.

vshankar

https://tracker.ceph.com/projects/cephfs/wiki/Main#19th-Feb-2024

neesingh-rh marked this pull request as draft July 27, 2023 09:51

github-actions bot added cephfs Ceph File System pybind tests labels Jul 27, 2023

neesingh-rh force-pushed the wip-59714 branch from 20d3b66 to abe34f4 Compare August 1, 2023 12:18

vshankar reviewed Aug 2, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/operations/versions/subvolume_v1.py Outdated Show resolved Hide resolved

vshankar requested a review from kotreshhr August 2, 2023 02:55

vshankar marked this pull request as ready for review August 2, 2023 02:55

neesingh-rh force-pushed the wip-59714 branch 4 times, most recently from b96184f to b969f0a Compare August 2, 2023 18:48

vshankar reviewed Aug 3, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/operations/volume.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/volume.py Outdated Show resolved Hide resolved

neesingh-rh force-pushed the wip-59714 branch from b969f0a to f9a2281 Compare August 3, 2023 03:37

vshankar reviewed Aug 3, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/operations/volume.py Outdated Show resolved Hide resolved

neesingh-rh force-pushed the wip-59714 branch from f9a2281 to 15b181c Compare August 3, 2023 08:18

neesingh-rh force-pushed the wip-59714 branch 2 times, most recently from 3fa0299 to 43555ca Compare August 11, 2023 06:21

kotreshhr requested changes Aug 16, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/volume.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/operations/volume.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_volumes.py Show resolved Hide resolved

neesingh-rh force-pushed the wip-59714 branch from 43555ca to a8ba2c6 Compare August 16, 2023 09:54

kotreshhr reviewed Aug 16, 2023

View reviewed changes

neesingh-rh force-pushed the wip-59714 branch from a8ba2c6 to 94e36a8 Compare August 16, 2023 11:46

kotreshhr reviewed Aug 17, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/async_cloner.py Outdated Show resolved Hide resolved

vshankar added the wip-vshankar-testing1 label Jan 17, 2024

vshankar requested changes Jan 24, 2024

View reviewed changes

qa/tasks/cephfs/test_volumes.py Show resolved Hide resolved

vshankar removed the wip-vshankar-testing1 label Jan 24, 2024

rishabh-d-dave removed the needs-qa label Feb 2, 2024

neesingh-rh force-pushed the wip-59714 branch from 7995354 to 7c2c206 Compare February 8, 2024 14:19

kotreshhr reviewed Feb 9, 2024

View reviewed changes

qa/tasks/cephfs/test_volumes.py Show resolved Hide resolved

neesingh-rh force-pushed the wip-59714 branch from 7c2c206 to 353db29 Compare February 9, 2024 15:19

anthonyeleven reviewed Feb 9, 2024

View reviewed changes

PendingReleaseNotes Outdated Show resolved Hide resolved

PendingReleaseNotes Outdated Show resolved Hide resolved

doc/cephfs/fs-volumes.rst Outdated Show resolved Hide resolved

neeraj pratap singh added 2 commits February 13, 2024 14:45

qa: add test cases for the support to reject clones feature

dbdc161

Fixes: https://tracker.ceph.com/issues/59714 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc: add the reject the clone when threads are not available feature …

6a44322

…in the document Fixes: https://tracker.ceph.com/issues/59714 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

neesingh-rh force-pushed the wip-59714 branch from 178b6c9 to 6a44322 Compare February 13, 2024 09:48

neesingh-rh requested review from vshankar, anthonyeleven and kotreshhr February 13, 2024 09:59

kotreshhr approved these changes Feb 13, 2024

View reviewed changes

vshankar added the wip-vshankar-testing2 label Feb 16, 2024

vshankar approved these changes Feb 20, 2024

View reviewed changes

vshankar merged commit 435306d into ceph:main Feb 20, 2024
10 of 11 checks passed

vshankar removed the wip-vshankar-testing2 label Feb 20, 2024

vshankar mentioned this pull request Feb 20, 2024

qa: disable snapshot_clone_no_wait for some tests #55660

Merged

14 tasks

This was referenced Feb 21, 2024

quincy: mgr/volumes: support to reject CephFS clones if cloner threads are not available #55690

Open

reef: mgr/volumes: support to reject CephFS clones if cloner threads are not available #55692

Merged

neesingh-rh mentioned this pull request Mar 5, 2024

squid: mgr/volumes: support to reject CephFS clones if cloner threads are not available #55940

Merged

neesingh-rh mentioned this pull request May 29, 2024

qa: setting the config options value back to default at the end of th… #57627

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mgr/volumes: support to reject CephFS clones if cloner threads are not available #52670

mgr/volumes: support to reject CephFS clones if cloner threads are not available #52670

neesingh-rh commented Jul 27, 2023 •

edited

Loading

kotreshhr commented Aug 7, 2023

neesingh-rh commented Aug 7, 2023

kotreshhr commented Aug 8, 2023

vshankar commented Aug 8, 2023

vshankar commented Aug 8, 2023

kotreshhr left a comment

vshankar commented Aug 16, 2023

kotreshhr left a comment

neesingh-rh commented Aug 16, 2023

neesingh-rh commented Jan 17, 2024

vshankar left a comment

anthonyeleven left a comment

vshankar commented Feb 19, 2024

vshankar commented Feb 19, 2024

vshankar left a comment

mgr/volumes: support to reject CephFS clones if cloner threads are not available #52670

mgr/volumes: support to reject CephFS clones if cloner threads are not available #52670

Conversation

neesingh-rh commented Jul 27, 2023 • edited Loading

Contribution Guidelines

Checklist

kotreshhr commented Aug 7, 2023

neesingh-rh commented Aug 7, 2023

kotreshhr commented Aug 8, 2023

vshankar commented Aug 8, 2023

vshankar commented Aug 8, 2023

kotreshhr left a comment

Choose a reason for hiding this comment

vshankar commented Aug 16, 2023

kotreshhr left a comment

Choose a reason for hiding this comment

neesingh-rh commented Aug 16, 2023

neesingh-rh commented Jan 17, 2024

vshankar left a comment

Choose a reason for hiding this comment

anthonyeleven left a comment

Choose a reason for hiding this comment

vshankar commented Feb 19, 2024

vshankar commented Feb 19, 2024

vshankar left a comment

Choose a reason for hiding this comment

neesingh-rh commented Jul 27, 2023 •

edited

Loading