[CORE-60] Schema Registry: Support `/mode` #17952

BenPope · 2024-04-18T21:19:33Z

Support READONLY /mode

Documentation

Schema registry supports a read-only mode, which is useful when it is undesirable for modifications to be made to either the whole Schema Registry, or particular subjects.

To allow the mode of the Schema Registry to be changed, the node-level property schema_registry.mode_mutability must be set to true in redpanda.yaml (this is now the default):

schema_registry:
  mode_mutability: true

If authentication is enabled on Schema Registry, then the global or subject-level mode can only be changed by superusers.

To get the global mode:

curl http://localhost:$PORT/mode

To set the global mode to read only:

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READONLY"}' http://localhost:$PORT/mode

To revert the global mode to read-write:

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READWRITE"}' http://localhost:$PORT/mode

To query the mode for a particular subject:
If there is no specific mode set for the subject an error shall be returned.

curl http://localhost:$PORT/mode/subject

To query the mode for a particular subject, regardless of whether it has a specific override

curl http://localhost:$PORT/mode/subject?defaultToGlobal=true

To override the global mode for a particular subject:

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READONLY"}' http://localhost:$PORT/mode/<subject>

To clear the override for a particular subject:

curl -X DELETE http://localhost:$PORT/mode/<subject>

New swagger:

    "/mode": {
      "get": {
        "summary": "Get the global mode.",
        "operationId": "get_mode",
        "consumes": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "parameters": [],
        "produces": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          },
          "500": {
            "description": "Internal Server error",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          }
        }
      },
      "put": {
        "summary": "Set the global mode.",
        "operationId": "put_mode",
        "consumes": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "parameters": [
          {
            "name": "mode",
            "in": "body",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          }
        ],
        "produces": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          },
          "422": {
            "description": "Unprocessable Entity",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          },
          "500": {
            "description": "Internal Server error",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          }
        }
      }
    },
    "/mode/{subject}": {
      "get": {
        "summary": "Get the mode for a subject.",
        "operationId": "get_mode_subject",
        "consumes": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "parameters": [
          {
            "name": "subject",
            "in": "path",
            "required": true,
            "type": "string"
          },
          {
            "name": "defaultToGlobal",
            "in": "query",
            "required": false,
            "type": "boolean"
          }
        ],
        "produces": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          },
          "404": {
            "description": "Not Found",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          },
          "500": {
            "description": "Internal Server error",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          }
        }
      },
      "put": {
        "summary": "Set the mode for a subject.",
        "operationId": "put_mode_subject",
        "consumes": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "parameters": [
          {
            "name": "subject",
            "in": "path",
            "required": true,
            "type": "string"
          },
          {
            "name": "mode",
            "in": "body",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          }
        ],
        "produces": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          },
          "404": {
            "description": "Not Found",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          },
          "422": {
            "description": "Unprocessable Entity",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          },
          "500": {
            "description": "Internal Server error",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          }
        }
      },
      "delete": {
        "summary": "Delete the mode for a subject.",
        "operationId": "delete_mode_subject",
        "consumes": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "parameters": [
          {
            "name": "subject",
            "in": "path",
            "required": true,
            "type": "string"
          }
        ],
        "produces": [
          "application/vnd.schemaregistry.v1+json",
          "application/vnd.schemaregistry+json",
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/mode"
            }
          },
          "404": {
            "description": "Not Found",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          },
          "500": {
            "description": "Internal Server error",
            "schema": {
              "$ref": "#/definitions/error_body"
            }
          }
        }
      }
    },

definitions:

      "mode": {
      "type": "object",
      "properties": {
        "mode": {
          "type": "string",
          "enum": [
            "READWRITE",
            "READONLY"
          ]
        },

Notes for the reviewer

A read-only Schema Registry may receive updates directly to its topic, and those changes should become visible in a timely fashion. This will often require a rq.service().writer().read_sync().

Backports Required

There are a couple of commits that may need to be backported.

Release Notes

Features

Schema Registry: Support /mode endpoints for READONLY

src/v/pandaproxy/schema_registry/types.h

src/v/pandaproxy/schema_registry/sharded_store.cc

src/v/pandaproxy/schema_registry/types.h

michael-redpanda

looks nice, a few questions

src/v/pandaproxy/schema_registry/store.h

michael-redpanda · 2024-04-22T13:54:32Z

src/v/pandaproxy/schema_registry/store.h

+    result<void> check_mutable(force f) const {
+        if (!_mutable && !f) {


question: Isn't the force flag for if there are already registered schemas and the mode is going into IMPORT?

Yes, but it's also used when applying records from the topic onto the store.

src/v/pandaproxy/api/api-doc/schema_registry_definitions.def.json

src/v/pandaproxy/api/api-doc/schema_registry.json

oleiman

lgtm. few questions mostly for my own benefit.

src/v/pandaproxy/schema_registry/test/consume_to_store.cc

src/v/pandaproxy/schema_registry/seq_writer.cc

BenPope · 2024-05-08T16:39:17Z

src/v/pandaproxy/schema_registry/service.cc

+            rq.user = maybe_authorize_request(
+              rq.authn_method,
+              _auth_level,
+              rq.service().authenticator(),
+              *rq.req);
        } catch (unauthorized_user_exception& e) {
            audit_authn_failure(rq, e.get_username(), e.what());
            throw;


@pgellert Looks like if this fails AuthZ (c.f. AuthN) this needs something like audit_authz for failure

BenPope · 2024-05-09T14:43:43Z

Changes in force-push

More tests
More compatibility
Properly wired-up is_mutable
Add support for mode READONLY
Drop support for mode IMPORT

BenPope · 2024-05-09T15:58:08Z

src/v/pandaproxy/api/api-doc/schema_registry_definitions.def.json

+      "type": "object",
+      "properties": {
+        "mode": {
+          "type": "string"


Like this, I believe so!

Suggested change

"type": "string"

"type": "string",

"enum": [

"READWRITE",

"READONLY"

]

kbatuigas · 2024-05-09T22:07:48Z

src/v/pandaproxy/api/api-doc/schema_registry.json

+            "type": "string"
+          },
+          {
+            "name": "defaultToGlobal",


From the description for the query param

query the mode for a particular subject, regardless of whether it has a specific override

and the endpoint (without the query param)

If there is no specific mode set for the subject an error shall be returned.

Does defaultToGlobal mean, if there is no specific mode set for the subject, return the global mode instead? I wasn't sure what "specific override" was referring to

From the description for the query param

query the mode for a particular subject, regardless of whether it has a specific override

and the endpoint (without the query param)

If there is no specific mode set for the subject an error shall be returned.

Does defaultToGlobal mean, if there is no specific mode set for the subject, return the global mode instead? I wasn't sure what "specific override" was referring to

Yeah, so the global mode is in force for all subjects, unless a subject has a mode set, in which case that will override whatever is global, for the subject.

To query whether a subject has a mode set, either don't specify defaultToGlobal=true, or specify defaultToGlobal=false, and detect the success or error.

But if you're just querying what mode is in force, specify defaultToGlobal=true

@kbatuigas I added some short descriptions for the parameters.

Signed-off-by: Ben Pope <ben@redpanda.com>

No functional changes Signed-off-by: Ben Pope <ben@redpanda.com>

Signed-off-by: Ben Pope <ben@redpanda.com>

BenPope · 2024-05-13T14:55:47Z

Changes in force-push:

Address review comments
Return 40401 when deleting a mode for a subject, but it isn't set.

pgellert

lgtm

oleiman

lgtm

BenPope · 2024-05-14T07:10:20Z

We don't do this in the swagger specs currently, but I think our doc readers could benefit from additional explanations added to the description for an operation or parameter.

Do you think it would be useful to add docs as code reviewers for Swagger?

Feediver1 · 2024-05-14T15:55:28Z

We don't do this in the swagger specs currently, but I think our doc readers could benefit from additional explanations added to the description for an operation or parameter.

Do you think it would be useful to add docs as code reviewers for Swagger?

Yes--would mean these updates are reviewed before any customers ever see them. Perfect.

michael-redpanda · 2024-05-14T17:36:34Z

We don't do this in the swagger specs currently, but I think our doc readers could benefit from additional explanations added to the description for an operation or parameter.

Do you think it would be useful to add docs as code reviewers for Swagger?

Yes--would mean these updates are reviewed before any customers ever see them. Perfect.

https://redpandadata.atlassian.net/browse/CORE-2952

michael-redpanda · 2024-05-14T19:06:21Z

@Feediver1 done (#18473)

BenPope · 2024-05-22T08:34:29Z

/backport v24.1.x

BenPope · 2024-05-31T13:11:28Z

/backport v23.3.x

vbotbuildovich · 2024-05-31T13:12:22Z

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-17952-v23.3.x-326 remotes/upstream/v23.3.x
git cherry-pick -x c99992877894ada0f7a69878b4c42ab8ec4f4519 5fb72fcf3d840e4290417d62ed0bfb9db7c508f9 452556b74f8d3afd29198308c9c60414a87d1df4 88e1993bb1c6c0677668cbc0134de489b65d1eed ba05b97ce676bf276cd25b0eee16bbc03b2b9360 390fc71805c0b131f5d3154af4b508e730303827 7aa34ffccee787e6d725b621ebe7fdd77befe1f5 71c5fef6fd374ddba669fde6edad32e1318cd607 6ce966f23e086138f6f18368f1ef17799fb374fa 2776733646c541d335085e36485c993a797a0f0b 282854b2223cd8b423318dac62367e5520b41ba9 8b0915f8e22d9768c5f0458c80048ee4f847af3f b55d09ed7a031345182efe5f45272d118cea0fe8 ccdb7d6f060a66417f6cae7c5828d33e60a67f78 6f2f894891c146fc56d7f57bce868b7930db13ad dc8c2e429a1c23db2d87eaa834ff67d6faebb468 dd6c8312fa26a237af40e5d447ea35d2367474fa 19e51c9b6695f9a81fb777d297e511ec3095ced6

Workflow run logs.

BenPope requested a review from a team as a code owner April 18, 2024 21:19

BenPope self-assigned this Apr 18, 2024

BenPope requested review from oleiman and michael-redpanda and removed request for a team and oleiman April 18, 2024 21:19

github-actions bot added the area/redpanda label Apr 18, 2024

BenPope added area/schema-registry Schema Registry service within Redpanda and removed area/redpanda labels Apr 18, 2024

BenPope changed the title ~~Schema Registry: Support /mode~~ [CORE-60] Schema Registry: Support /mode Apr 18, 2024

BenPope mentioned this pull request Apr 18, 2024

Schema Registry features meta-issue #1878

Open

49 tasks

dotnwat reviewed Apr 19, 2024

View reviewed changes

src/v/pandaproxy/schema_registry/types.h Show resolved Hide resolved

src/v/pandaproxy/schema_registry/sharded_store.cc Show resolved Hide resolved

src/v/pandaproxy/schema_registry/types.h Show resolved Hide resolved

michael-redpanda reviewed Apr 22, 2024

View reviewed changes

oleiman previously approved these changes Apr 22, 2024

View reviewed changes

src/v/pandaproxy/schema_registry/test/consume_to_store.cc Show resolved Hide resolved

src/v/pandaproxy/schema_registry/seq_writer.cc Outdated Show resolved Hide resolved

src/v/pandaproxy/schema_registry/seq_writer.cc Outdated Show resolved Hide resolved

dotnwat previously requested changes Apr 23, 2024

View reviewed changes

src/v/pandaproxy/schema_registry/seq_writer.cc Outdated Show resolved Hide resolved

pgellert self-requested a review May 8, 2024 13:36

BenPope commented May 8, 2024

View reviewed changes

BenPope dismissed oleiman’s stale review via 6def7c3 May 9, 2024 14:41

BenPope force-pushed the schema-registry-mode branch from 070f318 to 6def7c3 Compare May 9, 2024 14:41

github-actions bot added the area/redpanda label May 9, 2024

BenPope requested a review from dotnwat May 9, 2024 15:24

BenPope commented May 9, 2024

View reviewed changes

BenPope added the doc-needed label May 9, 2024

BenPope requested review from a team and michael-redpanda May 9, 2024 19:15

kbatuigas reviewed May 9, 2024

View reviewed changes

oleiman self-requested a review May 9, 2024 23:11

pandaproxy/error: Introduce error_codes for mode

c999928

Signed-off-by: Ben Pope <ben@redpanda.com>

BenPope added 6 commits May 13, 2024 15:43

schema_registry/storage: Support writing mode records

b55d09e

Signed-off-by: Ben Pope <ben@redpanda.com>

schema_registry: Support mode endpoints

ccdb7d6

Signed-off-by: Ben Pope <ben@redpanda.com>

schema_registry/dt: Refactor SchemaRegistryBasicAuthTest

6f2f894

No functional changes Signed-off-by: Ben Pope <ben@redpanda.com>

schema_registry: Require superuser to mutate /mode

dc8c2e4

Signed-off-by: Ben Pope <ben@redpanda.com>

schema_registry: Support mode - readonly

dd6c831

Signed-off-by: Ben Pope <ben@redpanda.com>

schema_registry/swagger: Fix typo

19e51c9

Signed-off-by: Ben Pope <ben@redpanda.com>

BenPope dismissed stale reviews from oleiman and kbatuigas via 19e51c9 May 13, 2024 14:54

BenPope force-pushed the schema-registry-mode branch from 0347421 to 19e51c9 Compare May 13, 2024 14:54

BenPope requested review from pgellert and oleiman May 13, 2024 15:59

pgellert approved these changes May 13, 2024

View reviewed changes

oleiman approved these changes May 13, 2024

View reviewed changes

michael-redpanda requested a review from kbatuigas May 14, 2024 14:22

BenPope merged commit c1ec803 into redpanda-data:dev May 14, 2024
18 checks passed

vbotbuildovich mentioned this pull request May 22, 2024

[v24.1.x] [CORE-60] Schema Registry: Support /mode #18623

Merged

BenPope mentioned this pull request May 22, 2024

[v23.3.x] [CORE-2399] Schema Registry: Improve retries and tombstoning #18624

Merged

vbotbuildovich mentioned this pull request May 31, 2024

[v23.3.x] [CORE-60] Schema Registry: Support /mode #18739

Closed

BenPope mentioned this pull request May 31, 2024

[v23.3.x] [CORE-60] Schema Registry: Support /mode #18742

Merged

BenPope mentioned this pull request Jun 14, 2024

Update properties for 24.1.7 redpanda-data/docs#554

Merged

4 tasks

BenPope mentioned this pull request Jul 16, 2024

[CORE-5540] [v24.1.x] Pandaproxy: Avoid large allocations whilst serializing JSON #21450

Merged

BenPope mentioned this pull request Aug 23, 2024

schema registry: typo in swagger file #8221

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-60] Schema Registry: Support `/mode` #17952

[CORE-60] Schema Registry: Support `/mode` #17952

BenPope commented Apr 18, 2024 •

edited

Loading

michael-redpanda left a comment

michael-redpanda Apr 22, 2024

BenPope May 10, 2024

oleiman left a comment

BenPope May 8, 2024

BenPope commented May 9, 2024 •

edited

Loading

BenPope May 9, 2024 •

edited

Loading

kbatuigas May 9, 2024 •

edited

Loading

BenPope May 9, 2024 •

edited

Loading

BenPope May 13, 2024

BenPope commented May 13, 2024

pgellert left a comment

oleiman left a comment

BenPope commented May 14, 2024

Feediver1 commented May 14, 2024

michael-redpanda commented May 14, 2024

michael-redpanda commented May 14, 2024

BenPope commented May 22, 2024

BenPope commented May 31, 2024

vbotbuildovich commented May 31, 2024

		result<void> check_mutable(force f) const {
		if (!_mutable && !f) {

[CORE-60] Schema Registry: Support /mode #17952

[CORE-60] Schema Registry: Support /mode #17952

Conversation

BenPope commented Apr 18, 2024 • edited Loading

Documentation

Notes for the reviewer

Backports Required

Release Notes

Features

michael-redpanda left a comment

Choose a reason for hiding this comment

michael-redpanda Apr 22, 2024

Choose a reason for hiding this comment

BenPope May 10, 2024

Choose a reason for hiding this comment

oleiman left a comment

Choose a reason for hiding this comment

BenPope May 8, 2024

Choose a reason for hiding this comment

BenPope commented May 9, 2024 • edited Loading

BenPope May 9, 2024 • edited Loading

Choose a reason for hiding this comment

kbatuigas May 9, 2024 • edited Loading

Choose a reason for hiding this comment

BenPope May 9, 2024 • edited Loading

Choose a reason for hiding this comment

BenPope May 13, 2024

Choose a reason for hiding this comment

BenPope commented May 13, 2024

pgellert left a comment

Choose a reason for hiding this comment

oleiman left a comment

Choose a reason for hiding this comment

BenPope commented May 14, 2024

Feediver1 commented May 14, 2024

michael-redpanda commented May 14, 2024

michael-redpanda commented May 14, 2024

BenPope commented May 22, 2024

BenPope commented May 31, 2024

vbotbuildovich commented May 31, 2024

[CORE-60] Schema Registry: Support `/mode` #17952

[CORE-60] Schema Registry: Support `/mode` #17952

BenPope commented Apr 18, 2024 •

edited

Loading

BenPope commented May 9, 2024 •

edited

Loading

BenPope May 9, 2024 •

edited

Loading

kbatuigas May 9, 2024 •

edited

Loading

BenPope May 9, 2024 •

edited

Loading