rpk: add support for recovery mode

[twmb]

This ticket tracks the rpk side of recovery mode.

What we will need to change:

• add recovery as an option for rpk redpanda mode. This will set the redpanda.recovery_mode_enabled field in redpanda.yaml to true (see PR https://github.com/redpanda-data/redpanda/pull/14236/files#diff-54ec8d4efd0da0b3ad48dc6dce13b1599b402f1382fdea16408b2d2148f36726R168)
• add rpk cluster partitions enable and rpk cluster partitions disable, both of which support --topic, --partition. Note we want the flags here to be similar to other rpk cluster partitions commands.
• add rpk cluster partitions list (exists) with an option to filter for disabled. Maybe we always want to print whether something is disabled, too?

The proposed admin APIs are:

echo '{"disabled": true} | POST v1/cluster/partitions/foo
echo '{"disabled": true} | POST v1/cluster/partitions/foo/1
echo '{"disabled": false} | POST v1/cluster/partitions/foo
echo '{"disabled": false} | POST v1/cluster/partitions/foo/1
GET v1/cluster/partitions?disabled=true
GET v1/cluster/partitions/foo?disabled=true

These should be in this PR: #14236

[ztlpn]

#14236 contains only code related to the first bullet point.

I'll publish a pr with APIs needed for remaining bullet points shortly, will link it here.

[ztlpn]

#14461 adds APIs for setting/listing disabled partitions. Doesn't include any real effects but it should be enough to unblock implementing rpk support.

[ztlpn]

Note that for consistency with other admin API endpoints I added a namespace argument to the path. So /v1/cluster/partitions/foo becomes /v1/cluster/partitions/kafka/foo etc.

[ztlpn]

just merged #14461, should unblock this issue

[r-vasquez]

Thanks @ztlpn. For the Admin API request, is it fine if I send the request to any node in the cluster, or should the request be directed to the leader only?

[ztlpn]

@r-vasquez GETs can be sent to any node (but for nodes other than controller leader, results can be slightly out of date). POSTs should be sent to the controller leader (other nodes will reply with a redirect).

[r-vasquez]

@ztlpn If I disable a single partition of a topic (POST /v1/cluster/partitions/kafka/<topic>/<partition>) and later on, I try to enable all partitions in the topic (POST /v1/cluster/partitions/kafka/<topic>/), should it enable the partition that I disabled alone?

I'm getting the opposite:

# Disable a single partition [1]
$ curl -X POST localhost:9644/v1/cluster/partitions/kafka/baz/1 -d '{"disabled":true}'

# I see that it's disabled:
$ curl localhost:9644/v1/cluster/partitions/kafka/baz?disabled=true           
[{"ns": "kafka", "topic": "baz", "partition_id": 1, "replicas": [{"node_id": 2, "core": 2},{"node_id": 0, "core": 2},{"node_id": 1, "core": 2}], "disabled": true}]

# Then I try to enable ALL partitions
$ curl -X POST localhost:9644/v1/cluster/partitions/kafka/baz -d '{"disabled":false}'

# But I still see the partition [1] disabled
$ curl localhost:9644/v1/cluster/partitions/kafka/baz\?disabled=true
[{"ns": "kafka", "topic": "baz", "partition_id": 1, "replicas": [{"node_id": 2, "core": 2},{"node_id": 0, "core": 2},{"node_id": 1, "core": 2}], "disabled": true}]%   

Is this expected? If I disable the enable the single partition back, it's fine though.

[ztlpn]

@r-vasquez yes, this is a bug, thanks for catching! I'll fix it shortly

[ztlpn]

fix for the bug above: #15071

[ztlpn]

we also need this small thing: #15104