core: Add command to reset mon quorum #61

travisn · 2022-10-10T23:38:50Z

When quorum is lost, restoring quorum to a single mon is currently a complex manual process. Now with this krew command the admin can with less risk reset the mon quorum and restore the cluster again in disaster scenarios.

Resolves #19

subhamkrai

added some initial reviews

README.md

docs/mons.md

kubectl-rook-ceph.sh

README.md

kubectl-rook-ceph.sh

parth-gr · 2022-10-17T11:50:44Z

kubectl-rook-ceph.sh

-  KUBECTL_NS_CLUSTER wait --for=delete pod/"$deployment_pod" --timeout=60s
+  set -e
+  if [ "$deployment_pod" != "" ]; then
+    # scale the deployment to 0


scaling deployment to 0 can be still be out of if statement

If there is no pod, is there any reason to still scale it to 0? Are you just saying it's a precaution to still scale it down?

Yes for a precaution,
A pod can be not present for certain intervals because of some restarting, etc
But the deployment may exist scaled up

kubectl-rook-ceph.sh

parth-gr · 2022-10-17T11:55:19Z

kubectl-rook-ceph.sh

+
+  # Check for the existence of the toolbox
+  info_msg "Start the toolbox if it is not yet running"
+  wait_for_pod_of_deployment_to_be_running rook-ceph-tools


maybe if we just say
wait_for_pod_to_run

rook-ceph-tools is the name of a deployment, not a pod. How about wait_for_deployment_to_run?

subhamkrai · 2022-10-18T06:44:53Z

kubectl-rook-ceph.sh

+
+  info_msg "Mon quorum was successfully restored to mon $good_mon"
+
+  prompt_to_continue_or_cancel "Start up the operator and expand to full mon quorum again?" "yes"


Suggested change

prompt_to_continue_or_cancel "Start up the operator and expand to full mon quorum again?" "yes"

prompt_to_continue_or_cancel "Start up the operator and expand to full mon quorum again?" "yes"

since Mon are successfully restored(as we are logging above) why we are asking for starting the operator up again, we should be doing every time once mons are up?

It's a good question, it just felt like a good thing to pause and say "I did my job to restore quorum to a single mon, are you sure you're ready to start the operator up again?"

if i understand correctly at this point we have a single monitor running right ? If the user cancels this prompt, they'll know that they're taking a risk to run with a single mon, maybe we should add some warning here if at a point they have a cluster running with a single monitor.
And they would need to start and scale up the operator manually right in such situation further.

Yeah we don't want them to risk staying with a single mon. I think the "press any key to continue" will solve this.

subhamkrai · 2022-10-18T06:53:16Z

kubectl-rook-ceph.sh

+    if [ "$INPUT_VAR" = "$proceed_answer" ]; then
+      info_msg "proceeding"
+    else
+      warn_msg "cancelled"
+      exit 1


Suggested change

if [ "$INPUT_VAR" = "$proceed_answer" ]; then

info_msg "proceeding"

else

warn_msg "cancelled"

exit 1

if [ "$INPUT_VAR" = "$proceed_answer" ]; then

info_msg "proceeding"

else

warn_msg "cancelled"

exit 1

I have a question about this maybe I thinking to much, but since checking for strict "$INPUT_VAR" = "$proceed_answer" and if somehow user has a typo or just does enter( which somewhere means default value or n in y/n case) then we are doing exit 1 which will exit the script in the middle of the process or some step. is this okay for the cluster? maybe we are doing the steps in debug mode pod so we are good?

Also, let's say the user didn't give any input in the prompt or wrong input that will lead to exit then should we delete the debug pod deployment in that case?

These are negative case questions and we can fix in follow-up too according to user feedback?

There are currently prompts in two places and I think it's fine if they exit the script:

All the info is gathered, and we prompt if the user wants to proceed. No debug pod has been started yet.

The mon restore is completed and we just have to scale up the operator. They could scale the operator manually if they exit.
The second one could be annoying that you have to scale up the operator. Perhaps a better approach for the 2nd one is to "press any key to continue when you're ready to scale up the operator".

kubectl-rook-ceph.sh

parth-gr · 2022-10-18T11:10:40Z

kubectl-rook-ceph.sh

@@ -423,15 +630,16 @@ function run_start_debug() {
  [[ -z "${REMAINING_ARGS[0]:-""}" ]] && fail_error "Missing mon or osd deployment name"
  deployment_name="${REMAINING_ARGS[0]}"              # get deployment name
  REMAINING_ARGS=("${REMAINING_ARGS[@]:1}")           # remove deploy name from remaining args
+  set +u


Does plus +u is set as it turns off the unset variables as an error
SO to avoid failing

Correct. I wonder if this was only necessary on mac. I was hitting the same error for the main debug start command even though it was working in the CI.

Ohkay got it

docs/mons.md

travisn · 2022-10-18T19:08:46Z

Here is the output of a test run in minikube, with verbose rocksdb output removed. See attached for the full output.
mon-restore-full-output.txt

~/src/go/src/github.com/rook/kubectl-rook-ceph$ ./kubectl-rook-ceph.sh mons restore-quorum ab
mon=ab, endpoint=10.103.13.182:6789
mon=ae, endpoint=10.102.163.160:6789
mon=af, endpoint=10.101.243.83:6789
Info: Check for the running toolbox
Info: Waiting for the pod from deployment "rook-ceph-tools" to be running
deployment.apps/rook-ceph-tools condition met

Warning: Restoring mon quorum to mon ab (10.103.13.182)
Info: The mons to discard are: ae af
Info: The cluster fsid is 7a2d8bf6-670e-41a8-9ae2-8a61c6732241
Info: Are you sure you want to restore the quorum to mon "ab"? If so, enter: yes-really-restore
yes-really-restore
Info: proceeding
deployment.apps/rook-ceph-operator scaled
deployment.apps/rook-ceph-mon-ab scaled
deployment.apps/rook-ceph-mon-ae scaled
deployment.apps/rook-ceph-mon-af scaled
Info: Waiting for operator and mon pods to stop
pod/rook-ceph-operator-b5c96c99b-hvvlk condition met
pod/rook-ceph-mon-ae-58c8b486d6-4gcr5 condition met
pod/rook-ceph-mon-af-7c99ff79c8-6pj87 condition met
setting debug mode for "rook-ceph-mon-ab"
setting debug command to main container
get pod for deployment rook-ceph-mon-ab
deployment.apps/rook-ceph-mon-ab-debug created
ensure the debug deployment rook-ceph-mon-ab is scaled up
deployment.apps/rook-ceph-mon-ab-debug scaled
Info: Waiting for the pod from deployment "rook-ceph-mon-ab-debug" to be running
deployment.apps/rook-ceph-mon-ab-debug condition met
Info: Started debug pod, restoring the mon quorum in the debug pod
Info: Extracting the monmap
parse error setting 'public_bind_addr' to ''

REMOVED: Verbose rocksdb output

debug2022-10-18T19:03:54.603+0000 7f9d3be0e880 -1 wrote monmap to /tmp/monmap
Info: Printing monmap
monmaptool: monmap file /tmp/monmap
epoch 48
fsid 7a2d8bf6-670e-41a8-9ae2-8a61c6732241
last_changed 2022-10-18T19:03:09.461446+0000
created 2022-10-14T15:12:40.925913+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.103.13.182:3300/0,v1:10.103.13.182:6789/0] mon.ab
1: [v2:10.102.163.160:3300/0,v1:10.102.163.160:6789/0] mon.ae
2: [v2:10.101.243.83:3300/0,v1:10.101.243.83:6789/0] mon.af
Info: Removing mon ae
monmaptool: monmap file /tmp/monmap
monmaptool: removing ae
monmaptool: writing epoch 48 to /tmp/monmap (2 monitors)
Info: Removing mon af
monmaptool: monmap file /tmp/monmap
monmaptool: removing af
monmaptool: writing epoch 48 to /tmp/monmap (1 monitors)
Info: Injecting the monmap
parse error setting 'public_bind_addr' to ''

REMOVED: Verbose rocksdb output

Info: Finished updating the monmap!
Info: Printing final monmap
monmaptool: monmap file /tmp/monmap
epoch 48
fsid 7a2d8bf6-670e-41a8-9ae2-8a61c6732241
last_changed 2022-10-18T19:03:09.461446+0000
created 2022-10-14T15:12:40.925913+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.103.13.182:3300/0,v1:10.103.13.182:6789/0] mon.ab
Info: Restoring the mons in the rook-ceph-mon-endpoints configmap to the good mon
configmap/rook-ceph-mon-endpoints patched
Info: Stopping the debug pod for mon ab
setting debug mode for "rook-ceph-mon-ab-debug"
removing debug mode from "rook-ceph-mon-ab-debug"
deployment.apps "rook-ceph-mon-ab-debug" deleted
deployment.apps/rook-ceph-mon-ab scaled
Info: Check that the restored mon is responding
timed out
command terminated with exit code 1
Info: 0: waiting for ceph status to confirm single mon quorum
Info: sleeping 5
timed out
command terminated with exit code 1
Info: 1: waiting for ceph status to confirm single mon quorum
Info: sleeping 5
timed out
command terminated with exit code 1
Info: 2: waiting for ceph status to confirm single mon quorum
Info: sleeping 5
  cluster:
    id:     7a2d8bf6-670e-41a8-9ae2-8a61c6732241
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 1

  services:
    mon: 1 daemons, quorum ab (age 22s)
    mgr: a(active, since 40m)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

Info: finished waiting for ceph status
Info: Purging the bad mons: ae af
Info: purging old mon: ae
deployment.apps "rook-ceph-mon-ae" deleted
service "rook-ceph-mon-ae" deleted
Info: purging old mon: af
deployment.apps "rook-ceph-mon-af" deleted
service "rook-ceph-mon-af" deleted
Info: Mon quorum was successfully restored to mon ab
Info: Only a single mon is currently running
Info: Press Enter to start the operator and expand to full mon quorum again

Info: continuing
deployment.apps/rook-ceph-operator scaled

subhamkrai

LGTM, just a few small nits

subhamkrai · 2022-10-19T08:19:47Z

README.md

@@ -58,6 +58,7 @@ These are args currently supported:
 - `rbd <args>` : Call a 'rbd' CLI command with arbitrary args

 - `mons` : Print mon endpoints
+  - `restore-quorum <mon-name>` : Restore the mon quorum to a single mon since quorum was lost with the other mons


Suggested change

- `restore-quorum <mon-name>` : Restore the mon quorum to a single mon since quorum was lost with the other mons

- `restore-quorum <mon-name>` : Restore the mon quorum to a single mon since quorum was lost with the other mons

is this right? to a single mon

I'll rephrase it for clarity

subhamkrai · 2022-10-19T08:24:43Z

kubectl-rook-ceph.sh

@@ -445,13 +661,20 @@ function run_start_debug() {
  echo "setting debug command to main container"
  deployment_spec=$(update_deployment_spec_command "$deployment_spec")

+  # scale down the daemon pod if it's running
+  set +e
+  echo "get pod for deployment $deployment_name"


Suggested change

echo "get pod for deployment $deployment_name"

info_msg "get pod for deployment $deployment_name"

?

subhamkrai · 2022-10-19T08:25:26Z

kubectl-rook-ceph.sh

+  set -e
+  if [ "$deployment_pod" != "" ]; then
+    # scale the deployment to 0
+    echo "scale down the deployment $deployment_name"


Suggested change

echo "scale down the deployment $deployment_name"

info_msg "scale down the deployment $deployment_name"

subhamkrai · 2022-10-19T08:25:37Z

kubectl-rook-ceph.sh

+    KUBECTL_NS_CLUSTER scale deployments "$deployment_name" --replicas=0
+
+    # wait for the deployment pod to be deleted
+    echo "waiting for the deployment pod \"$deployment_pod\" to be deleted"


Suggested change

echo "waiting for the deployment pod \"$deployment_pod\" to be deleted"

info_msg "waiting for the deployment pod \"$deployment_pod\" to be deleted"

subhamkrai · 2022-10-19T08:25:53Z

kubectl-rook-ceph.sh

@@ -465,6 +688,8 @@ function run_start_debug() {
    spec:
        $deployment_spec
 EOF
+    echo "ensure the debug deployment $deployment_name is scaled up"


Suggested change

echo "ensure the debug deployment $deployment_name is scaled up"

info_msg "ensure the debug deployment $deployment_name is scaled up"

parth-gr · 2022-10-19T12:01:41Z

.github/workflows/ci-for-diff-ns.yaml

@@ -49,6 +49,15 @@ jobs:
          sleep 5
          kubectl rook_ceph -o test-operator -n test-cluster rbd ls replicapool

+          # test the mon restore to restore to mon a, delete mons b and c, then add d and e
+          export ROOK_PLUGIN_SKIP_PROMPTS=true


this variable is used just for the CI?

Correct it's for the CI. Or if someone else wanted to avoid the prompts they could set it, though I wouldn't expect it for normal use.

gauravsitlani

LGTM

When quorum is lost, restoring quorum to a single mon is currently a complex manual process. Now with this krew command the admin can with less risk reset the mon quorum and restore the cluster again in disaster scenarios. Signed-off-by: Travis Nielsen <tnielsen@redhat.com>

travisn · 2022-10-19T18:45:37Z

Also tested on openshift with three nodes and PVC and non-PVC

core: Add command to reset mon quorum Signed-off-by: parth-gr <paarora@redhat.com>

travisn requested a review from gauravsitlani October 10, 2022 23:39

travisn force-pushed the reset-quorum branch from 56fda02 to 5ffe76b Compare October 10, 2022 23:40

subhamkrai requested changes Oct 11, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

docs/mons.md Show resolved Hide resolved

docs/mons.md Outdated Show resolved Hide resolved

kubectl-rook-ceph.sh Outdated Show resolved Hide resolved

travisn mentioned this pull request Oct 11, 2022

Automate restoring of mon quorum for disaster recovery rook/rook#3985

Closed

travisn force-pushed the reset-quorum branch 2 times, most recently from d14be36 to efe3543 Compare October 11, 2022 23:16

subhamkrai requested changes Oct 12, 2022

View reviewed changes

travisn force-pushed the reset-quorum branch 4 times, most recently from 32b6a5c to 35d4e29 Compare October 14, 2022 20:50

travisn commented Oct 14, 2022

View reviewed changes

kubectl-rook-ceph.sh Show resolved Hide resolved

travisn marked this pull request as ready for review October 14, 2022 20:51

travisn force-pushed the reset-quorum branch 6 times, most recently from 4c71c98 to 58d5906 Compare October 15, 2022 06:01

parth-gr reviewed Oct 17, 2022

View reviewed changes

travisn mentioned this pull request Oct 17, 2022

ci: added timeout instead of sleep commands #48

Merged

travisn force-pushed the reset-quorum branch 7 times, most recently from 83da8c4 to 038a02e Compare October 17, 2022 23:18

subhamkrai requested changes Oct 18, 2022

View reviewed changes

parth-gr reviewed Oct 18, 2022

View reviewed changes

gauravsitlani reviewed Oct 18, 2022

View reviewed changes

docs/mons.md Outdated Show resolved Hide resolved

travisn force-pushed the reset-quorum branch from 038a02e to 649d641 Compare October 18, 2022 19:11

travisn requested review from subhamkrai and gauravsitlani October 18, 2022 19:11

travisn force-pushed the reset-quorum branch from 649d641 to 9cae84c Compare October 18, 2022 19:12

subhamkrai approved these changes Oct 19, 2022

View reviewed changes

subhamkrai mentioned this pull request Oct 19, 2022

Add command to update mon configmap for disaster recovery #19

Closed

2 tasks

parth-gr reviewed Oct 19, 2022

View reviewed changes

travisn force-pushed the reset-quorum branch from 9cae84c to bee8e09 Compare October 19, 2022 16:43

gauravsitlani approved these changes Oct 19, 2022

View reviewed changes

travisn force-pushed the reset-quorum branch from bee8e09 to 8d60ac6 Compare October 19, 2022 18:32

travisn merged commit 973a629 into rook:master Oct 19, 2022

travisn deleted the reset-quorum branch October 19, 2022 18:42

parth-gr pushed a commit to parth-gr/kubectl-rook-ceph that referenced this pull request Oct 20, 2022

Merge pull request rook#61 from travisn/reset-quorum

324c8cf

core: Add command to reset mon quorum Signed-off-by: parth-gr <paarora@redhat.com>

travisn mentioned this pull request Oct 21, 2022

Add Krew command for recovering the cephcluster CR after accidental deletion #68

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Add command to reset mon quorum #61

core: Add command to reset mon quorum #61

travisn commented Oct 10, 2022 •

edited

Loading

subhamkrai left a comment

parth-gr Oct 17, 2022

travisn Oct 17, 2022

parth-gr Oct 18, 2022 •

edited

Loading

parth-gr Oct 17, 2022

travisn Oct 17, 2022

parth-gr Oct 18, 2022

subhamkrai Oct 18, 2022

travisn Oct 18, 2022

gauravsitlani Oct 18, 2022 •

edited

Loading

travisn Oct 18, 2022

subhamkrai Oct 18, 2022 •

edited

Loading

travisn Oct 18, 2022

parth-gr Oct 18, 2022

travisn Oct 18, 2022

parth-gr Oct 19, 2022

travisn commented Oct 18, 2022 •

edited

Loading

subhamkrai left a comment

subhamkrai Oct 19, 2022

travisn Oct 19, 2022

subhamkrai Oct 19, 2022

subhamkrai Oct 19, 2022

subhamkrai Oct 19, 2022

subhamkrai Oct 19, 2022

parth-gr Oct 19, 2022

travisn Oct 19, 2022

gauravsitlani left a comment

travisn commented Oct 19, 2022


		info_msg "Mon quorum was successfully restored to mon $good_mon"

		prompt_to_continue_or_cancel "Start up the operator and expand to full mon quorum again?" "yes"

	- `restore-quorum <mon-name>` : Restore the mon quorum to a single mon since quorum was lost with the other mons
	- `restore-quorum <mon-name>` : Restore the mon quorum to a single mon since quorum was lost with the other mons

	echo "get pod for deployment $deployment_name"
	info_msg "get pod for deployment $deployment_name"

	echo "scale down the deployment $deployment_name"
	info_msg "scale down the deployment $deployment_name"

	echo "waiting for the deployment pod \"$deployment_pod\" to be deleted"
	info_msg "waiting for the deployment pod \"$deployment_pod\" to be deleted"

	echo "ensure the debug deployment $deployment_name is scaled up"
	info_msg "ensure the debug deployment $deployment_name is scaled up"

core: Add command to reset mon quorum #61

core: Add command to reset mon quorum #61

Conversation

travisn commented Oct 10, 2022 • edited Loading

subhamkrai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parth-gr Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravsitlani Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhamkrai Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

travisn commented Oct 18, 2022 • edited Loading

subhamkrai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravsitlani left a comment

Choose a reason for hiding this comment

travisn commented Oct 19, 2022

travisn commented Oct 10, 2022 •

edited

Loading

parth-gr Oct 18, 2022 •

edited

Loading

gauravsitlani Oct 18, 2022 •

edited

Loading

subhamkrai Oct 18, 2022 •

edited

Loading

travisn commented Oct 18, 2022 •

edited

Loading