[CORE-2736] Adds output batch compression for Data Transforms #18514

oleiman · 2024-05-15T21:22:12Z

This PR adds the ability to configure output batch compression on WASM transforms at deploy time or via metadata patch request. Includes rpk experience for the former. The latter case (metadata patch) is available via direct admin API invocation only.

TODO:

rpk bits and tests for deploying w/ compression turned on

Closes CORE-2736

Backports Required

Release Notes

Improvements

Add output batch compression for Data Transforms (configurable per deployed transform)

oleiman · 2024-05-16T05:45:59Z

/ci-repeat 1

oleiman · 2024-05-16T19:50:09Z

/ci-repeat 1
release
skip-units
skip-redpanda-build

vbotbuildovich · 2024-05-16T21:33:14Z

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2fc-4bd5-b204-a1774afd79bf:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_handling_invalid_ids"

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2ff-46d2-8840-d9e063a709e2:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_no_head_of_line_blocking.different_clusters=False.different_connections=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/50134#019009db-8de9-47d5-a29d-ba3e1ac43ce8:

"rptest.tests.topic_creation_test.TopicRecreateTest.test_topic_recreation_while_producing.workload=ACKS_1.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"

src/v/redpanda/admin/api-doc/transform.json

src/go/rpk/pkg/cli/transform/deploy.go

src/go/rpk/pkg/cli/transform/list_test.go

src/go/rpk/pkg/cli/transform/meta.go

Feediver1 · 2024-05-20T17:43:07Z

src/go/rpk/pkg/cli/transform/meta.go

+at a cost. So, while it does occur asynchronously with respect to transform
+execution, compression may introduce latency on the output topic.


Suggested change

at a cost. So, while it does occur asynchronously with respect to transform

execution, compression may introduce latency on the output topic.

at a cost. So, while it occurs asynchronously with respect to transform

execution, compression may introduce latency on the output topic.

oleiman · 2024-05-31T20:08:15Z

force push to sync w/ dev after a couple weeks of inactivity

src/v/redpanda/admin/api-doc/transform.json

oleiman · 2024-06-18T01:06:24Z

/ci-repeat 1

dotnwat · 2024-06-24T02:28:09Z

src/v/model/model.cc

+              .match("producer", compression::producer);
+    } catch (std::runtime_error& e) {


maybe could avoid the try/catch with

string_switch<std::optional>(c)
...
.default_match(std::nullopt);

if (!c.has_value()) {
i.setstate(...)
}

fair point. force of habit from regular switch, but static analysis is no help here obviously.

dotnwat · 2024-06-24T02:28:57Z

src/v/model/transform.h

@@ -170,6 +173,10 @@ struct transform_metadata_patch {
    std::optional<absl::flat_hash_map<ss::sstring, ss::sstring>> env;
    // Desired paused state for the transform
    std::optional<is_transform_paused> paused;
+    // Desired compression mode for the transform
+    std::optional<compression> compression_mode;


what's the difference between compression_mode:none and nullopt here?

in this context, whether or not to update the setting on persisted metadata. comments on these fields could be better.

oleiman · 2024-06-24T15:43:57Z

force push contents:

marginally improve comments on transform_metadata_patch
improve string_switch handling for compression mode

oleiman · 2024-06-24T21:49:52Z

CI Failure is some pandatriage issue

rockwotj · 2024-06-25T02:12:28Z

src/go/rpk/pkg/cli/transform/deploy.go

@@ -233,12 +240,16 @@ func mergeProjectConfigs(lhs project.Config, rhs project.Config) (out project.Co
 	if len(rhs.OutputTopics) > 0 {
 		out.OutputTopics = rhs.OutputTopics
 	}
+	if rhs.Compression != "" {


This is why the flag should default to empty string 😄

Yeah, that's what I was harping on up here 🤷

I have to push again to fix a conflict, so I think I'll change it back. But I (clearly) don't have a strong opinion about it.

Yeah I was agreeing with you 😄

boost::lexical_cast<compression>("bogus) should raise bad_lexical cast. Previous behavior: runtime_error("Fell off the end of a string-switch") Failure to match the source string to one of the compression lexical cases should set the fail bit on the istream instead of allowing the runtime_error to escape. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Also adds bool transform_metadata_patch::empty() Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Respecting transform_metadata::compression_mode Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- tranform_from_json - deploy Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> dt/rpk

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-06-25T17:23:54Z

force push rebase dev to fix merge conflict

oleiman · 2024-06-27T01:25:10Z

CI Failure:

Looks like CI Failure (MetricCheckFailed) in RaftAvailabilityTest. test_leadership_transfer #19953 (unrelated)

oleiman self-assigned this May 15, 2024

github-actions bot added area/rpk area/redpanda area/wasm WASM Data Transforms labels May 15, 2024

oleiman force-pushed the xform/core-2736/compressed-batches branch 2 times, most recently from 4adf296 to 58df36c Compare May 16, 2024 05:45

oleiman commented May 17, 2024

View reviewed changes

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

oleiman force-pushed the xform/core-2736/compressed-batches branch 2 times, most recently from d9f6691 to f13207f Compare May 17, 2024 18:31

oleiman marked this pull request as ready for review May 17, 2024 18:32

oleiman requested review from a team, twmb, r-vasquez, gene-redpanda and Deflaimun as code owners May 17, 2024 18:32

oleiman added the doc-needed label May 17, 2024

oleiman requested review from a team and michael-redpanda and removed request for a team May 17, 2024 22:55

rockwotj reviewed May 19, 2024

View reviewed changes

src/go/rpk/pkg/cli/transform/deploy.go Show resolved Hide resolved

src/go/rpk/pkg/cli/transform/list_test.go Outdated Show resolved Hide resolved

src/go/rpk/pkg/cli/transform/meta.go Outdated Show resolved Hide resolved

Feediver1 reviewed May 20, 2024

View reviewed changes

oleiman marked this pull request as draft May 31, 2024 19:56

oleiman force-pushed the xform/core-2736/compressed-batches branch from f13207f to 1e48f71 Compare May 31, 2024 20:07

Deflaimun reviewed Jun 3, 2024

View reviewed changes

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

Deflaimun reviewed Jun 3, 2024

View reviewed changes

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

Deflaimun reviewed Jun 3, 2024

View reviewed changes

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

michael-redpanda previously approved these changes Jun 18, 2024

View reviewed changes

dotnwat reviewed Jun 24, 2024

View reviewed changes

oleiman dismissed michael-redpanda’s stale review via 4c2460b June 24, 2024 15:42

oleiman force-pushed the xform/core-2736/compressed-batches branch from d6b3bb0 to 4c2460b Compare June 24, 2024 15:42

oleiman requested review from michael-redpanda, rockwotj and dotnwat June 24, 2024 16:20

rockwotj reviewed Jun 25, 2024

View reviewed changes

oleiman added 12 commits June 25, 2024 08:59

model/xform: Add compression mode to metadata & patch

b70309e

Also adds bool transform_metadata_patch::empty() Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin/swagger: Add compression mode to transform endpoints

939faad

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin/xform: Wire up compression mode to deploy/list/patch

45662f7

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

xform/api: Respect compression_mode for patch upsert

5b0bb68

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

xform/proc: Compress record batches before producing

8e8b39f

Respecting transform_metadata::compression_mode Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform/api: Add compression mode to metadata wrappers

472e6d8

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression to project config

74c073f

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression mode to deploy command

e1281d5

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression mode to list command output

a85f394

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/rpk: Wire up compression for transforms

d81e96b

- tranform_from_json - deploy Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> dt/rpk

dt/xform: Test for setting compression mode

04cd18e

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the xform/core-2736/compressed-batches branch from 4c2460b to 04cd18e Compare June 25, 2024 17:22

michael-redpanda approved these changes Jun 26, 2024

View reviewed changes

michael-redpanda requested a review from r-vasquez June 27, 2024 12:04

r-vasquez approved these changes Jun 27, 2024

View reviewed changes

michael-redpanda merged commit 577a280 into redpanda-data:dev Jun 27, 2024
23 of 26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-2736] Adds output batch compression for Data Transforms #18514

[CORE-2736] Adds output batch compression for Data Transforms #18514

oleiman commented May 15, 2024 •

edited

Loading

oleiman commented May 16, 2024

oleiman commented May 16, 2024

vbotbuildovich commented May 16, 2024 •

edited

Loading

Feediver1 May 20, 2024

oleiman commented May 31, 2024

oleiman commented Jun 18, 2024

dotnwat Jun 24, 2024

oleiman Jun 24, 2024

dotnwat Jun 24, 2024

oleiman Jun 24, 2024

oleiman commented Jun 24, 2024

oleiman commented Jun 24, 2024

rockwotj Jun 25, 2024

oleiman Jun 25, 2024

rockwotj Jun 26, 2024

oleiman commented Jun 25, 2024

oleiman commented Jun 27, 2024

		at a cost. So, while it does occur asynchronously with respect to transform
		execution, compression may introduce latency on the output topic.

		.match("producer", compression::producer);
		} catch (std::runtime_error& e) {

[CORE-2736] Adds output batch compression for Data Transforms #18514

[CORE-2736] Adds output batch compression for Data Transforms #18514

Conversation

oleiman commented May 15, 2024 • edited Loading

Backports Required

Release Notes

Improvements

oleiman commented May 16, 2024

oleiman commented May 16, 2024

vbotbuildovich commented May 16, 2024 • edited Loading

Choose a reason for hiding this comment

oleiman commented May 31, 2024

oleiman commented Jun 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jun 24, 2024

oleiman commented Jun 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jun 25, 2024

oleiman commented Jun 27, 2024

oleiman commented May 15, 2024 •

edited

Loading

vbotbuildovich commented May 16, 2024 •

edited

Loading