feat(localpv-device): allow local pv device on select devices #1648

kmova · 2020-04-05T06:12:49Z

What this PR does / why we need it:

This PR makes use of the Label Selector capability
added in the BDC. More details: openebs/openebs#2921

The BDC Label Selector feature makes it possible for
administrators to tag a set of block devices and use
them for creating Local PV (device).

To use the Label Selector feature with Local PV (Devices)
the administrator is expected to tag a set of
block devices by assigning them the label:
openebs.io/block-device-tag=< tag-value >.

The < tag-value > can be passed to Local PV storage class
via cas annotations. If the value is present, then Local PV
device provisioner will set the following additional selector
on the BDC:
openebs.io/block-device-tag=< tag-value >

Signed-off-by: kmova kiran.mova@mayadata.io

Special notes for your reviewer:
BDDs will be pushed in another commit or PR.

Executed the following tests:

Existing BDDs execute without any issues, when the block-device-tag is not added to the storageclass.
Created a GKE cluster. Added label on only one device with openebs.io/block-device-tag=mongo.

kiran_mova_mayadata_io@kmova-dev:maya$ kubectl get bd -n openebs --show-labels
NAME                                           NODENAME                                    SIZE           CLAIMSTATE   STATUS   AGE   LABELS
blockdevice-5a2e9602ff2bde719fa25a8b596ea64c   gke-kmova-helm-default-pool-f0c7b5f6-5ls5   402653184000   Unclaimed    Active   45m   kubernetes.io/hostname=gke-kmova-helm-default-pool-f0c7b5f6-5ls5,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-8c90defa852a91fb7a85fd2b10691fe0   gke-kmova-helm-default-pool-f0c7b5f6-4x48   402653184000   Claimed      Active   45m   kubernetes.io/hostname=gke-kmova-helm-default-pool-f0c7b5f6-4x48,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true,openebs.io/block-device-tag=mongo
blockdevice-b542ca2d4a4afe7edb82b0635bcbb813   gke-kmova-helm-default-pool-f0c7b5f6-7q9j   402653184000   Unclaimed    Active   45m   kubernetes.io/hostname=gke-kmova-helm-default-pool-f0c7b5f6-7q9j,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-bade4695f7101ea410dc45f941b4da92   gke-kmova-helm-default-pool-f0c7b5f6-4x48   402653184000   Unclaimed    Active   45m   kubernetes.io/hostname=gke-kmova-helm-default-pool-f0c7b5f6-4x48,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true

Created a storage class with annotation for BlockDeviceTag as follows:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-device-mongo
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "device"
      - name: BlockDeviceTag
        value: "mongo"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---

Created a Mongo statefulset by specifying storageclass as openebs-device-mongo
Only the labeled block device was claimed and PV was created only for one PVC.

kmova-dev:mongodb$ kubectl get bdc -n openebs
NAME                                           BLOCKDEVICENAME                                PHASE     AGE
bdc-pvc-cc31ced3-f686-4889-821c-1366790a2460   blockdevice8c90defa852a91fb7a85fd2b10691fe0   Bound     34m
bdc-pvc-f21fad99-1c5d-4591-b32a-014de39021fb                                                  Pending   34m

kmova-dev:mongodb$ kubectl get pvc
NAME                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
mongo-pvc-mongo-0   Bound     pvc-cc31ced3-f686-4889-821c-1366790a2460   5G         RWO           openebs-device-mongo   35m
mongo-pvc-mongo-1   Pending                                                                        openebs-device-mongo   34m

The BDC with label selector for device tag looks like this:

kmova-dev:mongodb$ kubectl describe bdc bdc-pvc-f21fad99-1c5d-4591-b32a-014de39021fb -n openebs
Name:         bdc-pvc-f21fad99-1c5d-4591-b32a-014de39021fb
Namespace:    openebs
Labels:       <none>
Annotations:  <none>
API Version:  openebs.io/v1alpha1
Kind:         BlockDeviceClaim
Metadata:
  Creation Timestamp:  2020-04-05T05:45:35Z
  Finalizers:
    local.openebs.io/finalizer
  Generation:        2
  Resource Version:  8199
  Self Link:         /apis/openebs.io/v1alpha1/namespaces/openebs/blockdeviceclaims/bdc-pvc-f21fad99-1c5d-4591-b32a-014de39021fb
  UID:               fc99366c-fd95-44dc-9dbb-427d87ad2fbd
Spec:
  Block Device Node Attributes:
    Host Name:  gke-kmova-helm-default-pool-f0c7b5f6-7q9j
  Device Claim Details:
  Device Type:  
  Host Name:    
  Resources:
    Requests:
      Storage:  5G
  Selector:
    Match Labels:
      openebs.io/block-device-tag:  mongo
Status:
  Phase:  Pending
Events:   <none>

When other block devices were labelled, the BDC was bound and pods came to running state:

kmova-dev:mongodb$ kubectl get bd -n openebs -l openebs.io/block-device-tag=mongo
NAME                                           NODENAME                                    SIZE           CLAIMSTATE   STATUS   AGE
blockdevice-8c90defa852a91fb7a85fd2b10691fe0   gke-kmova-helm-default-pool-f0c7b5f6-4x48   402653184000   Claimed      Active   66m
blockdevice-b542ca2d4a4afe7edb82b0635bcbb813   gke-kmova-helm-default-pool-f0c7b5f6-7q9j   402653184000   Claimed      Active   66m
blockdevice-bade4695f7101ea410dc45f941b4da92   gke-kmova-helm-default-pool-f0c7b5f6-4x48   402653184000   Unclaimed    Active   66m
blockdevice-c9872f642d4ad75c26b4e8317fca9c26   gke-kmova-helm-default-pool-f0c7b5f6-5ls5   402653184000   Claimed      Active   66m
blockdevice-d05cb29fd2042111e6286b32ad880d10   gke-kmova-helm-default-pool-f0c7b5f6-7q9j   402653184000   Unclaimed    Active   66m

kmova-dev:mongodb$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          50m
mongo-1   2/2     Running   0          4m7s
mongo-2   2/2     Running   0          3m57s

Ran another test by tagging all the block devices and created a PVC without a tag. As expected no block device was attached.

Checklist:

Fixes #
Labelled this PR & related issue with documentation tag (if applicable)
PR messages has document related information (if applicable)
Labelled this PR & related issue with breaking-changes tag (if applicable)
PR messages has breaking changes related information (if applicable)
Labelled this PR & related issue with requires-upgrade tag (if applicable)
PR messages has upgrade related information (if applicable)
Commit has unit tests (if applicable)
Commit has integration tests (if applicable)

Ref: openebs/openebs#2916 This PR makes use of the Label Selector capability added in the BDC. More details: openebs/openebs#2921 The BDC Label Selector feature makes it possible for administrators to group a set of block devices and use them for creating Local PV (device). To use the Label Selector feature with Local PV (Devices) the administrator is expected to group(or pool) a set of block devices by assigning them the label: openebs.io/block-device-pool=< pool-name >. The `<pool-name>` can be passed to Local PV storage class via cas annotations. If the value is present, then Local PV device provisioner will set the following additional selector on the BDC: `openebs.io/block-device-pool=< pool-name >` Signed-off-by: kmova <kiran.mova@mayadata.io>

pkg/blockdeviceclaim/v1alpha1/build.go

akhilerm

Why does the BDC bdc-pvc-f21fad99-1c5d-4591-b32a-014de39021fb given in the above example has kubernetes.io/hostname also in the spec.selector.MatchLabels ?

vishnuitta · 2020-04-06T05:44:48Z

cmd/provisioner-localpv/app/config.go

+	// volumeBindingMode: WaitForFirstConsumer
+	// reclaimPolicy: Delete
+	//
+	KeyBDPoolName = "BlockDevicePoolName"


can this be called as 'LocalPVPoolName'?
Not sure how BlockDevicePoolName makes sense here

The name is being derived from label openebs.io/block-device-pool, is set on the block devices.

Just thinking out loud, the pool concept can be implemented using this label selector. We need a fixed label key that is shared by NDM and all its consumers. "Pool" may be a over-loaded term.

How about using "openebs.io/block-device-tag"? And make the corresponding naming changes to the above Key, and other method names and variables.

Is this understood by NDM? If yes then label key should have some indication.
I could think of below label key

blockdevice.ndm.openebs.io/tag

If we want to go with pool then blockdevice.ndm.openebs.io/pool-name

If this label is applicable to BlockDevice & BlockDeviceClaim then NDM is not required in the label key. Sorry for above confusions.

So we can have openebs.io/block-device-tag. The one you had suggested initially.

with this stmt, We need a fixed label key that is shared by NDM and all its consumers are we saying - every NDM consumer should be fixed to this label? We don't need this as there is 'Selector' in spec of BD. If it was part of some labels of BD, it might be the case.
By fixing it to fixed string, 'Selector' in 'spec' of BD can be removed and moved to label of 'BD'.
If this is just for local pv, this key has to get into StorageClass, and in the 'Selector' of BD.
Is my understanding correct?

it will be understood by NDM and Local PV (and other storage engines possibly in future):

NDM will have special handling for this label. Like if BD has label, provide it to only those BDCs coming with the label.

Local PV will use it by reading it from Storage Class and passing it to BDC

CSPC operator might label the block devices with a tag, before claiming them.

is this the same tag for every consumer of NDM?

can we get example of BD for above example?

I meant - example mentioned in PR

Signed-off-by: kmova <kiran.mova@mayadata.io>

vishnuitta · 2020-04-06T06:07:06Z

cmd/provisioner-localpv/app/helper_blockdevice.go

+		WithBlockVolumeMode(blkDevOpts.volumeMode)
+
+	// If bdPoolName is configure, set it on the BDC
+	if len(blkDevOpts.bdPoolName) > 0 {


should we make poolName as compulsory? This will avoid for any race between app/pvc deployment and labeling of BDs.
Even though this doesn't happen usually, this can happen with south bound provisioner.

Signed-off-by: kmova <kiran.mova@mayadata.io>

vishnuitta

changes are good

akhilerm

LGTM

The new builder method WithBlockDeviceTag may need to be checked in to openebs/api also.

Ref: openebs-archive/maya#1648 A new label fixed selector (openebs.io/block-device-tag) has been added to BDC to help with narrowing down the block devices that can be claimed via the BDC. This will allow users to tag block devices and indicate which tagged devices can be used by the BDC. Signed-off-by: kmova <kiran.mova@mayadata.io>

kmova added pr/release-note PR should be included in release notes pr/documentation-pending labels Apr 5, 2020

kmova requested review from AmitKumarDas, akhilerm, avishnu and vishnuitta April 5, 2020 06:12

kmova added this to the 1.9 milestone Apr 5, 2020

akhilerm reviewed Apr 5, 2020

View reviewed changes

pkg/blockdeviceclaim/v1alpha1/build.go Outdated Show resolved Hide resolved

AmitKumarDas reviewed Apr 5, 2020

View reviewed changes

pkg/blockdeviceclaim/v1alpha1/build.go Show resolved Hide resolved

akhilerm reviewed Apr 5, 2020

View reviewed changes

vishnuitta reviewed Apr 6, 2020

View reviewed changes

ut(blockdeviceclaim): for method WithBlockDevicePoolName

2f1d263

Signed-off-by: kmova <kiran.mova@mayadata.io>

vishnuitta reviewed Apr 6, 2020

View reviewed changes

feat(localpv-device): rename Block Device Pool to Tag

41a4a8b

Signed-off-by: kmova <kiran.mova@mayadata.io>

kmova added pr/hold-merge Hold on the merge to complete few activities. and removed pr/hold-merge Hold on the merge to complete few activities. labels Apr 6, 2020

docs(localpv): update instruction on running bdd

9da0aa7

Signed-off-by: kmova <kiran.mova@mayadata.io>

vishnuitta approved these changes Apr 7, 2020

View reviewed changes

akhilerm approved these changes Apr 7, 2020

View reviewed changes

This was referenced Apr 8, 2020

feat(bdc): update bdc builder with adding tag openebs-archive/api#36

Merged

BDD for Local PV Block device with tag. #1658

Closed

vishnuitta merged commit a7b0b0f into openebs-archive:master Apr 8, 2020

kmova mentioned this pull request Apr 14, 2020

Admin should be able to specify the block devices that can be used as Local PV openebs/openebs#2916

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(localpv-device): allow local pv device on select devices #1648

feat(localpv-device): allow local pv device on select devices #1648

kmova commented Apr 5, 2020 •

edited

Loading

akhilerm left a comment

vishnuitta Apr 6, 2020

kmova Apr 6, 2020

AmitKumarDas Apr 6, 2020 •

edited

Loading

AmitKumarDas Apr 6, 2020

AmitKumarDas Apr 6, 2020

vishnuitta Apr 6, 2020

kmova Apr 6, 2020

vishnuitta Apr 6, 2020

vishnuitta Apr 6, 2020

vishnuitta Apr 6, 2020

vishnuitta Apr 6, 2020

vishnuitta left a comment

akhilerm left a comment

feat(localpv-device): allow local pv device on select devices #1648

feat(localpv-device): allow local pv device on select devices #1648

Conversation

kmova commented Apr 5, 2020 • edited Loading

akhilerm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmitKumarDas Apr 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

akhilerm left a comment

Choose a reason for hiding this comment

kmova commented Apr 5, 2020 •

edited

Loading

AmitKumarDas Apr 6, 2020 •

edited

Loading