Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

c/st: self_test_result rpc: move {start,end}_time to end of layout #21431

Conversation

oleiman
Copy link
Member

@oleiman oleiman commented Jul 16, 2024

Effectively turning the original change into an append and obviating the need for a compat version bump.

It's not immediately clear why the struct was laid out this way in the first place. A reasonable guess is some balance of:

  • The original ordering makes good sense for the aggregate in-memory
  • It's an rpc format and not intended for on-disk persistence

Still, the compat version change means that if a self test goes off during an upgrade across the compat boundary, that rpc will fail. Adjusting the layout via envelope::serde_fields means that we can maintain the nice field order semantics in the aggregate while ensuring that the diff looks like a field append from serde's view.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.1.x
  • v23.3.x
  • v23.2.x

Release Notes

  • none

@oleiman oleiman self-assigned this Jul 16, 2024
@oleiman oleiman marked this pull request as ready for review July 16, 2024 17:29
@oleiman
Copy link
Member Author

oleiman commented Jul 16, 2024

My read of

envelope_for_each_field(t, [&](auto& f) {
using FieldType = std::decay_t<decltype(f)>;
if (h._bytes_left_limit == in.bytes_left()) {
return false;
}
if (unlikely(in.bytes_left() < h._bytes_left_limit)) {
throw serde_exception(fmt_with_ctx(
ssx::sformat,
"field spill over in {}, field type {}: envelope_end={}, "
"in.bytes_left()={}",
type_str<Type>(),
type_str<FieldType>(),
h._bytes_left_limit,
in.bytes_left()));
}
f = read_nested<FieldType>(in, bytes_left_limit);

is that the read end sticks the deser'ed value into an instance of the envelope via a reference to the corresponding field (and does this in the order returned by serde_fields), so there's no need for the order of fields in the aggregate to match the serde field order. Which I guess is the whole point.

Copy link
Contributor

@michael-redpanda michael-redpanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd imagine this should be fine...?

@vbotbuildovich
Copy link
Collaborator

new failures in https://buildkite.com/redpanda/redpanda/builds/51593#0190c1ec-fe5c-4294-b623-f0deed21e0cf:

"rptest.tests.cloud_storage_chunk_read_path_test.CloudStorageChunkReadTest.test_read_chunks"
"rptest.tests.scaling_up_test.ScalingUpTest.test_moves_with_local_retention.use_topic_property=False"

@oleiman
Copy link
Member Author

oleiman commented Jul 18, 2024

CI Failures:

@oleiman
Copy link
Member Author

oleiman commented Jul 18, 2024

/ci-repeat 1
skip-redpanda-build

Effectively turning the original change into an append and obviating
the need for a compat version bump.

It's not immediately clear why the struct was laid out this way in
the first place. A reasonable guess is some balance of:

- The original ordering makes good sense for the aggregate in-memory
- It's an rpc format and not intended for on-disk persistence

Still, the compat version change means that if a self test goes off
during an upgrade across the compat boundary, that rpc will fail.
Adjusting the layout via envelope::serde_fields means that we can
maintain the nice field order semantics in the aggregate while
ensuring that the diff looks like a field append from serde's view.
@oleiman oleiman force-pushed the etc/noticket/self-test-result-layout branch from 7b992c1 to 3c38672 Compare July 18, 2024 21:20
@oleiman
Copy link
Member Author

oleiman commented Jul 18, 2024

force push empty

@oleiman
Copy link
Member Author

oleiman commented Jul 19, 2024

Tests seem to have succeeded here... infra failure? https://buildkite.com/redpanda/redpanda/builds/51744#0190c7fd-83ef-4db7-ac59-a5bbb44a8f41

@michael-redpanda
Copy link
Contributor

Tests seem to have succeeded here... infra failure? https://buildkite.com/redpanda/redpanda/builds/51744#0190c7fd-83ef-4db7-ac59-a5bbb44a8f41

Definitely seems that way... the tests all passed it looks like?

@oleiman
Copy link
Member Author

oleiman commented Jul 19, 2024

Tests seem to have succeeded here... infra failure? https://buildkite.com/redpanda/redpanda/builds/51744#0190c7fd-83ef-4db7-ac59-a5bbb44a8f41

Definitely seems that way... the tests all passed it looks like?

Yes, it looks that way to me. Report looks good anyway. Not immediately clear what failed, but it wasn't in ducktape.

@michael-redpanda michael-redpanda merged commit a80e0de into redpanda-data:dev Jul 19, 2024
16 of 19 checks passed
Comment on lines +293 to +312
auto serde_fields() {
return std::tie(
p50,
p90,
p99,
p999,
max,
rps,
bps,
timeouts,
test_id,
name,
info,
test_type,
duration,
warning,
error,
start_time,
end_time);
}
Copy link
Member

@dotnwat dotnwat Jul 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. more broadly, i kinda think we should change serde to require using serde_fields instead of the implicit ordering in the struct. wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah agreed. I don't really get the need for it generally; besides which i believe it maxes out at 20 fields or something like that. Pretty marginal as a convenience feature IMO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be happy to knock that out soon. See what people think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants