iceberg: initial data structures for logical data types #21415

andrwng · 2024-07-15T15:39:22Z

Adds initial logical types[1] that will be used to represent an Iceberg schema. These include "primitives" (basic types like int, float, string, etc) as well as complex types (types that are composed of other types, like list, map, struct). Complex types are represented in Iceberg as composed of "nested fields" which are types that are themselves composed of other types (primitives or complex types).

This PR introduces the recursive definition of these types, and initial code to serialize them to and from JSON (Iceberg manifests include table schemas represented as JSON). The implementation of these types is similar in style to the Iceberg Rust library, though we are using std::variants instead of Rust enums, since C++ enums aren't expressive enough to express complex nested enums with members

For reference, here are implementations in other languages:

[1] https://iceberg.apache.org/spec/#schemas-and-data-types

Backports Required

Release Notes

none

vbotbuildovich · 2024-07-16T08:34:12Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51570#0190ba63-c9d2-43f5-9d32-452b1fe0b226

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51614#0190beee-8ac8-421c-ae3d-3e6f758c5e7c

jcipar

Awesome! This looks really good.

I left a couple minor comments, but otherwise this looks ready to merge.

src/v/iceberg/datatypes.cc

jcipar · 2024-07-16T19:27:45Z

src/v/iceberg/datatypes.cc

+}
+bool operator==(const nested_field& lhs, const nested_field& rhs) {
+    return lhs.id == rhs.id && lhs.required == rhs.required
+           && lhs.name == rhs.name && lhs.type == rhs.type;


Do we need to require the names be equal? What is this used for?

For now I'm just using these to ensure my serialization is correct.

You're right though that schema type equivalence may have a different criteria. I'd argue though that if we want equivalence of just types, it should be a dedicated comparison method, rather than operator==.

src/v/iceberg/datatypes_json.cc

src/v/iceberg/datatypes.cc

Adds initial logical types[1] that will be used to represent an Iceberg schema. These include "primitives" (basic types like int, float, string, etc) as well as complex types (types that are composed of other types, like list, map, struct). Complex types are represented in Iceberg as composed of "nested fields" which are types that are themselves composed of other types (primitives or complex types). This PR introduces the recursive definition of these types, with a basic equality operator and ostream operator (note, this is for JSON serialization, that will come in a following commit). The implementation of these types is similar in style to the Iceberg Rust library, though we are using std::variants instead of Rust enums, since C++ enums aren't expressive enough to express complex nested enums with members.

Adds some helpers for operating on the json::Value class, that will be useful in building JSON parsing for Iceberg metadata types. Long term it may make sense to move these to some general purpose module, but since this is only used in Iceberg for now, and to avoid distractions, the utilities are just added to the Iceberg module.

dotnwat

lgtm

src/v/iceberg/datatypes_json.cc

dotnwat · 2024-07-17T04:56:41Z

src/v/iceberg/datatypes_json.cc

+          .match("binary", binary_type{});
+    }
+    if (!v.IsObject()) {


handle the default match case with an exception?

This already appears to be handled?

redpanda/src/v/strings/tests/string_switch_test.cc

Lines 58 to 70 in 015afa7

BOOST_CHECK_EXCEPTION(

string_switch<int8_t>("ccc").match("a", 0).match("b", 1).

operator int8_t(),

std::runtime_error,

[](const std::runtime_error& e) {

// check that the error string includes the string we were searching

// for as a weak hint to where the error occurred

if (!std::string(e.what()).ends_with("ccc")) {

BOOST_TEST_FAIL(

"Expected error message to end with ccc but was: " << e.what());

};

return true;

});

ahh i thought it just fell through or something. sgtm

emaxerrno · 2024-07-17T05:19:44Z

Very cool

Adds JSON serialization for newly added Iceberg data types. A test is added to demonstrate the roundtrip with a type comprised of multiple nested fields.

vbotbuildovich · 2024-07-17T08:33:35Z

new failures in https://buildkite.com/redpanda/redpanda/builds/51624#0190bf88-4433-4b6d-b663-2861e98c87c9:

"rptest.tests.scaling_up_test.ScalingUpTest.test_fast_node_addition"

andrwng · 2024-07-17T17:29:17Z

CI failure: #20224

github-actions bot added area/build area/redpanda labels Jul 15, 2024

andrwng force-pushed the iceberg-datatypes branch from 77d6336 to 66cb6c2 Compare July 16, 2024 01:57

andrwng requested review from dotnwat and jcipar July 16, 2024 02:23

andrwng marked this pull request as ready for review July 16, 2024 05:59

andrwng force-pushed the iceberg-datatypes branch 4 times, most recently from a1460f6 to 6c1b97f Compare July 16, 2024 19:24

jcipar previously approved these changes Jul 16, 2024

View reviewed changes

andrwng dismissed jcipar’s stale review via 70d5400 July 16, 2024 20:32

andrwng force-pushed the iceberg-datatypes branch from 6c1b97f to 70d5400 Compare July 16, 2024 20:32

andrwng requested a review from jcipar July 16, 2024 20:32

dotnwat reviewed Jul 17, 2024

View reviewed changes

src/v/iceberg/datatypes.cc Show resolved Hide resolved

src/v/iceberg/datatypes.cc Outdated Show resolved Hide resolved

andrwng added 2 commits July 16, 2024 19:45

andrwng force-pushed the iceberg-datatypes branch from 70d5400 to 93b1ffd Compare July 17, 2024 02:45

andrwng requested a review from dotnwat July 17, 2024 02:46

dotnwat previously approved these changes Jul 17, 2024

View reviewed changes

iceberg: json serde for data types

f4f231a

Adds JSON serialization for newly added Iceberg data types. A test is added to demonstrate the roundtrip with a type comprised of multiple nested fields.

andrwng dismissed dotnwat’s stale review via f4f231a July 17, 2024 05:36

andrwng force-pushed the iceberg-datatypes branch from 93b1ffd to f4f231a Compare July 17, 2024 05:36

andrwng requested a review from dotnwat July 17, 2024 05:38

dotnwat approved these changes Jul 17, 2024

View reviewed changes

dotnwat merged commit e539d40 into redpanda-data:dev Jul 17, 2024
16 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iceberg: initial data structures for logical data types #21415

iceberg: initial data structures for logical data types #21415

andrwng commented Jul 15, 2024 •

edited

Loading

vbotbuildovich commented Jul 16, 2024 •

edited

Loading

jcipar left a comment

jcipar Jul 16, 2024

andrwng Jul 16, 2024

dotnwat left a comment

dotnwat Jul 17, 2024

andrwng Jul 17, 2024

dotnwat Jul 17, 2024

emaxerrno commented Jul 17, 2024

vbotbuildovich commented Jul 17, 2024

andrwng commented Jul 17, 2024

	BOOST_CHECK_EXCEPTION(
	string_switch<int8_t>("ccc").match("a", 0).match("b", 1).
	operator int8_t(),
	std::runtime_error,
	[](const std::runtime_error& e) {
	// check that the error string includes the string we were searching
	// for as a weak hint to where the error occurred
	if (!std::string(e.what()).ends_with("ccc")) {
	BOOST_TEST_FAIL(
	"Expected error message to end with ccc but was: " << e.what());
	};
	return true;
	});

iceberg: initial data structures for logical data types #21415

iceberg: initial data structures for logical data types #21415

Conversation

andrwng commented Jul 15, 2024 • edited Loading

Backports Required

Release Notes

vbotbuildovich commented Jul 16, 2024 • edited Loading

jcipar left a comment

Choose a reason for hiding this comment

jcipar Jul 16, 2024

Choose a reason for hiding this comment

andrwng Jul 16, 2024

Choose a reason for hiding this comment

dotnwat left a comment

Choose a reason for hiding this comment

dotnwat Jul 17, 2024

Choose a reason for hiding this comment

andrwng Jul 17, 2024

Choose a reason for hiding this comment

dotnwat Jul 17, 2024

Choose a reason for hiding this comment

emaxerrno commented Jul 17, 2024

vbotbuildovich commented Jul 17, 2024

andrwng commented Jul 17, 2024

andrwng commented Jul 15, 2024 •

edited

Loading

vbotbuildovich commented Jul 16, 2024 •

edited

Loading