feat(spc,csp): adding support for pool ReadOnly Threshold limit

[mynktl]

Changes:

• Added support to set readOnly threshold for SPC/CSP.

Sample SPC spec

spec:
  name: cstor-pool
  type: sparse
  maxPools: 3
  poolSpec:
    poolType: striped
    roThresholdLimit: 80

Sample CSP spec:

spec:
  group:
  - blockDevice:
    - deviceID: /var/openebs/sparse/2-ndm-sparse.img
      inUseByPool: true
      name: sparse-362d689cfa30cf1ac2cd11ca6dcf9449
  poolSpec:
    cacheFile: /tmp/pool1.cache
    overProvisioning: false
    poolType: striped
    roThresholdLimit: 80
    thickProvisioning: false

Sample output :

kubectl get csp -n openebs
NAME              ALLOCATED   FREE    CAPACITY   STATUS    READONLY   TYPE      AGE
cstor-pool-1mns   117M        9.82G   9.94G      Healthy   true      striped   3h

Events:
  Type     Reason                 Age                  From       Message
  ----     ------                 ----                 ----       -------
  Normal   Synced                 12m                  CStorPool  Received Resource create event
  Normal   Synced                 12m (x2 over 12m)    CStorPool  Received Resource modify event
  Normal   Created                12m                  CStorPool  Resource created successfully
  Normal   Synced                 8m23s                CStorPool  Received Resource create event
  Normal   Imported               8m22s                CStorPool  Resource imported successfully
  Normal   Synced                 29s (x4 over 8m23s)  CStorPool  Received Resource modify event
  Warning  PoolReadOnlyThreshold  21s                  CStorPool  Pool storage limit reached to threshold. Pool expansion is required to make it's replica RW

Note:

1. Since SPC reconciliation is not supported, user can set roThresholdLimit on SPC during creation only.
2. One can manually update the CSP's roThresholdLimit.
3. By default roThresholdLimit is set to 0.
4. If roThresholdLimit is set to 0 or 100:
   - Writes on the pool will be allowed until it runs out of space

Signed-off-by: mayank mayank.patel@mayadata.io

[mynktl]

test case: openebs/openebs#2909

[mittachaitu]

Provided few comments

[mittachaitu]

Once the pool is set as ReadOnly I think we should avoid creating CVRs on that pool.

[mynktl]

@mittachaitu I have addressed your comments. PTAL.
Regarding CVR creation, we should not block it. CVR creation will happen but IOs won't and that will prevent pool running out of space. So we are ok with CVR creation.

[vishnuitta]

did we covered the cases where pool.SetPoolRDMode is done, but, update of cStorPool.Status.ReadOnly as true in etcd is not done due to any reason like restart of pool-mgmt / update to etcd failed?

[mynktl]

@kmova Should we add default value for pool readonly threshold, like 80, so if pool storage gets consumed up-to 80% then replica controller will set readonly mode to pool and no IOs will be allowed(volume creation will be alloed) on the pool unless user expand the pool.?

[vishnuitta]

@sonasingh46 check whether this fn helps you

[vishnuitta]

changes are good