Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cephfs: enable read affinity #4153

Merged
merged 3 commits into from
Nov 22, 2023
Merged

cephfs: enable read affinity #4153

merged 3 commits into from
Nov 22, 2023

Conversation

iPraveenParihar
Copy link
Contributor

Describe what this PR does

  • cephfs: add capability to automatically enable read affinity
    This commit makes use of crush location labels from node labels to supply crush_location and read_from_replica=localize options during mount. Using these options, cephfs
    will be able to redirect reads to the closest OSD, improving performance.

Fixes: #4149

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@iPraveenParihar iPraveenParihar changed the title cepfs: add capability to automatically enable read affinity cephfs: add capability to automatically enable read affinity Sep 28, 2023
@mergify mergify bot added the component/cephfs Issues related to CephFS label Sep 28, 2023
@iPraveenParihar iPraveenParihar marked this pull request as ready for review September 29, 2023 04:30
@iPraveenParihar iPraveenParihar marked this pull request as draft September 29, 2023 04:35
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Oct 29, 2023
@yati1998 yati1998 removed the stale label Nov 3, 2023
@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch 3 times, most recently from 4d028e7 to 6b766e9 Compare November 15, 2023 10:46
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 15, 2023

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@iPraveenParihar iPraveenParihar self-assigned this Nov 15, 2023
@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch 3 times, most recently from 429c46c to 77a33c5 Compare November 16, 2023 13:51
@iPraveenParihar iPraveenParihar marked this pull request as ready for review November 16, 2023 17:02
@iPraveenParihar iPraveenParihar changed the title cephfs: add capability to automatically enable read affinity cephfs: enable read affinity Nov 16, 2023
@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e-helm/k8s-1.27

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e-helm/k8s-1.27

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@iPraveenParihar
Copy link
Contributor Author 

  I1117 06:00:09.279048   24407 cephcmds.go:98] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs]
  E1117 06:00:09.279084   24407 nodeserver.go:322] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 failed to mount volume 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79: an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs] stderr: 2023-11-17T06:00:08.253+0000 7f4b3c7ce580 -1 init, newargv = 0x55d0d853fe60 newargc=17
  ceph-fuse[43995]: starting ceph client
  fuse: unknown option `read_from_replica=localize'
  ceph-fuse[43995]: fuse failed to start
  2023-11-17T06:00:08.265+0000 7f4b3c7ce580 -1 fuse_lowlevel_new failed

@Rakshith-R, looks like read_from_replica=localize option is not supported by fuse mounter?

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Nov 17, 2023 

  I1117 06:00:09.279048   24407 cephcmds.go:98] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs]
  E1117 06:00:09.279084   24407 nodeserver.go:322] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 failed to mount volume 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79: an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs] stderr: 2023-11-17T06:00:08.253+0000 7f4b3c7ce580 -1 init, newargv = 0x55d0d853fe60 newargc=17
  ceph-fuse[43995]: starting ceph client
  fuse: unknown option `read_from_replica=localize'
  ceph-fuse[43995]: fuse failed to start
  2023-11-17T06:00:08.265+0000 7f4b3c7ce580 -1 fuse_lowlevel_new failed

@Rakshith-R, looks like read_from_replica=localize option is not supported by fuse mounter?

Fuse might not support this option, you can to skip and log it in the nodeplugin.

Rakshith-R
Rakshith-R requested changes Nov 22, 2023
View reviewed changes
docs/deploy-cephfs.md Outdated
Comment on lines 234 to 235
Refer [krbd-options](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options)
for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This points to krbd mount options,
These read affinity mount options are not documented for cephfs kernel mounter. I'll ask for it to be documented at https://docs.ceph.com/en/latest/man/8/mount.ceph/#options.

Let's remove these two lines for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Rakshith-R
Copy link
Contributor

Rakshith-R commented Nov 22, 2023 •
edited
Loading

@Mergifyio rebase

@iPraveenParihar @Rakshith-R
cephfs: add read affinity mount option
55f94b9 
This commit makes use of crush location labels from node
labels to supply `crush_location` and `read_from_replica=localize`
options during mount. Using these options, cephfs
will be able to redirect reads to the closest OSD,
improving performance.

Signed-off-by: Praveen M <m.praveen@ibm.com>
@iPraveenParihar @Rakshith-R
helm: add option to enable read affinity for CephFS
7955318 
This commit adds --enable-read-affinity flag to
enable read affinity for CephFS.

Signed-off-by: Praveen M <m.praveen@ibm.com>
@iPraveenParihar @Rakshith-R
doc: add documentation for read affinity
0cc6db9 
This commit adds documentation about read affinity supported
for CephFS subvolumes.

Signed-off-by: Praveen M <m.praveen@ibm.com>
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 22, 2023

rebase

✅ Branch has been successfully rebased

@Rakshith-R Rakshith-R requested a review from a team November 22, 2023 08:46
@nixpanic
Copy link
Member

@Mergifyio rebase

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 22, 2023

rebase

✅ Nothing to do for rebase action

@nixpanic
Copy link
Member

@Mergifyio queue

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 22, 2023 •
edited
Loading

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 00c12b3

@mergify mergify bot added the ok-to-test Label to trigger E2E tests label Nov 22, 2023
@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-cephfs

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.27

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-rbd

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.27

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.27

@ceph-csi-bot ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Nov 22, 2023
@mergify mergify bot merged commit 00c12b3 into ceph:devel Nov 22, 2023
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Madhu-1 Madhu-1 Madhu-1 left review comments

@Rakshith-R Rakshith-R Rakshith-R approved these changes

@nixpanic nixpanic nixpanic approved these changes

Assignees

@iPraveenParihar iPraveenParihar

Labels
component/cephfs Issues related to CephFS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

cephfs: add capability to automatically enable read affinity
6 participants
@iPraveenParihar @Madhu-1 @Rakshith-R @nixpanic @ceph-csi-bot @yati1998