Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CORE-5540] [v24.1.x] Pandaproxy: Avoid large allocations whilst serializing JSON #21450

Merged
merged 15 commits into from
Jul 17, 2024
Merged

[CORE-5540] [v24.1.x] Pandaproxy: Avoid large allocations whilst serializing JSON #21450

merged 15 commits into from
Jul 17, 2024

Conversation

BenPope
Copy link
Member

@BenPope BenPope commented Jul 16, 2024 •
edited
Loading

Backport of PR #20827

Fixes #21297

BenPope added 15 commits July 16, 2024 22:26
@BenPope
json: Template rjson_serialize on Buffer
1c0c445 
Some structures serialize quite large, so in order to avoid
oversize allocs, it's necessary to use a buffer type that is not
based onn contiguous memmory.

Explicitly instantiate all overloads with StringBuffer.

Fix the test due to the way name lookup works.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 7cea48d)
@BenPope
pandaproxy/json: Template rjson_serialize on Buffer
1228340 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit ebc4c13)
@BenPope
schema_registry: Template rjson_serialize on Buffer
3c97452 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 9fcb311)
@BenPope
pandaproxy/json: Cleanup header includes
03345c4 
Namespace changed due to lookup rules for templates.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit a455d23)

Conflicts:
  src/v/pandaproxy/json/requests/produce.h (utils/tristate))
@BenPope
pandaproxy/json: Avoid temporary for calls to write_body
fdd8d6c 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 5d90218)
@BenPope
schema_registry: Avoid temporary for calls to write_body
e4631da 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit d74ec2a)

Conflicts:
  src/v/pandaproxy/schema_registry/handlers.cc (no JSON in this branch)
@BenPope
json: Introduce chunked_buffer
75e6e6c 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit f17567f)
@BenPope
pandaproxy: Use chunked_buffer
12021ab 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 744b0f0)

Conflicts:
  src/v/pandaproxy/json/requests/produce.h (utils/tristate)
@BenPope
Revert "sr: Streaming json lists"
5ecdb7c 
This reverts commit 313d208.

Some edits required to avoid temporaries..

(cherry picked from commit 5f3236c)
@BenPope
json: Introduce chunked_input_stream
76b49f3 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit c676f6e)
@BenPope
pandaproxy/json: Introduce chunked parsing with rjson_parse
3a4ec22 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 240c682)
@BenPope
pandaproxy/rest: Use alternative rjson_parse
7903d38 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 8560e0a)
@BenPope
schema_registry: Use alternative rjson_parse
b098de6 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 6795578)
@BenPope
pandaproxy: Enable streaming requests
bba5270 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit c934757)
@BenPope
pandaproxy/json: Move string version of rjson_parse
84acab6 
It shouldn't be used in the general case.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit ac7c7dd)
@BenPope BenPope added area/schema-registry Schema Registry service within Redpanda kind/backport PRs targeting a stable branch labels Jul 16, 2024
@BenPope BenPope added this to the v24.1.10 milestone Jul 16, 2024
@BenPope BenPope self-assigned this Jul 16, 2024
@BenPope BenPope requested a review from michael-redpanda July 16, 2024 21:59
oleiman
oleiman approved these changes Jul 16, 2024
View reviewed changes
Copy link
Member

@oleiman oleiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. if I'm reading right, 5ecdb7c was the only really hairy one?

@BenPope
Copy link
Member Author

BenPope commented Jul 16, 2024 •
edited
Loading

lgtm. if I'm reading right, 5ecdb7c was the only really hairy one?

Nah, git did it, I didn't have to touch it for the backport, it was clean.

@oleiman
Copy link
Member

oleiman commented Jul 16, 2024 •
edited
Loading

lgtm. if I'm reading right, 5ecdb7c was the only really hairy one?

Nah, git did it, I didn't have to touch it for the backport, it was clean.

Some edits required to avoid temporaries..

not manual edits? ignore me!

@BenPope
Copy link
Member Author

BenPope commented Jul 17, 2024

Blocked on the merge for #21323

@BenPope BenPope merged commit a022cde into redpanda-data:v24.1.x Jul 17, 2024
20 checks passed
@BenPope BenPope mentioned this pull request Jul 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oleiman oleiman oleiman approved these changes

@michael-redpanda michael-redpanda Awaiting requested review from michael-redpanda

Assignees

@BenPope BenPope

Labels
area/redpanda area/schema-registry Schema Registry service within Redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v24.1.10
Development

Successfully merging this pull request may close these issues.
2 participants
@BenPope @oleiman