Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement mark join #625

Closed
wants to merge 3 commits into from
Closed

feat: implement mark join #625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6ca533f
Merge branch 'main' of github.com:substrait-io/substrait
EpsilonPrime Apr 29, 2024
899abc9
feat: implement mark join
EpsilonPrime Apr 12, 2024
b0f4beb
Updated documentation to be more clear, removed the mark reference as…
EpsilonPrime Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions proto/substrait/algebra.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,6 +445,7 @@ message Rel {
HashJoinRel hash_join = 13;
MergeJoinRel merge_join = 14;
NestedLoopJoinRel nested_loop_join = 18;
MarkJoinRel mark_join = 23;
ConsistentPartitionWindowRel window = 17;
ExchangeRel exchange = 15;
ExpandRel expand = 16;
Expand Down Expand Up @@ -726,6 +727,20 @@ message NestedLoopJoinRel {
substrait.extensions.AdvancedExtension advanced_extension = 10;
}

// A mark join internally scans the left side, constructing a hash table that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does internally here mean?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This claims that the hash table is built from the left side and then the right side is probed. This is the reverse of the HashJoinRel. If this intentional, then we should point this out (e.g. "Note that the table is built from the left side which is the opposite of the approach taken in the HashJoinRel)

// is used to mark the right side as having a join partner on the left side. A
// mark is a nullable boolean field. The mark join operator is used to
// implement semi-joins, anti-joins, and other join types that are not equijoins.
message MarkJoinRel {
RelCommon common = 1;
Rel left = 2;
Rel right = 3;
// optional, defaults to true (a cartesian join)
Expression expression = 4;

substrait.extensions.AdvancedExtension advanced_extension = 10;
}

// The argument of a function
message FunctionArgument {
oneof arg_type {
Expand Down
23 changes: 23 additions & 0 deletions site/docs/relations/physical_relations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,29 @@ The merge equijoin does a join by taking advantage of two sets that are sorted o
| Post Join Predicate | An additional expression that can be used to reduce the output of the join operation post the equality condition. Minimizes the overhead of secondary join conditions that cannot be evaluated using the equijoin keys. | Optional, defaults true. |
| Join Type | One of the join types defined in the Join operator. | Required |



## Mark Join Operator

A mark join internally scans the left side, constructing a hash table that is used to mark the right side as having a join partner on the left side. This mark can end up being True, False, or NULL. The NULL mark is used to indicate that the right side does not have a join partner on the left side. The mark join operator is used to implement semi-joins, anti-joins, and other join types that are not equijoins.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the mark might be null then it sounds like this is a right join. In other words, there is one output row for each row on the right side. Is that true? Can we describe that more clearly?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A regular right/outer join can still emit multiple output rows per input row, a mark join preserves the cardinality of the left hand side exactly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NULL mark is used to indicate that the right side does not have a join partner on the left side.

This is not correct. From the paper:

the marker may not only be TRUE (had a join partner) or FALSE (had no
join partner), but also NULL (had a join partner where the comparison result is NULL, but
none where the comparison is TRUE)


| Signature | Value |
| -------------------- | ------------------------------------------------------------ |
| Inputs | 2 |
| Outputs | 1 |
| Property Maintenance | Distribution is maintained. Orderedness is eliminated. |
| Direct Output Order | Same as the [Join](logical_relations.md#join-operator) operator. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the mark join actually join two tables or does it just add the mark column? In other words, if I apply mark join to a left table l_key, l_payload and a right table r_key, r_payload then is the output l_key, l_payload, r_key, r_payload, mark or is it l_key, l_payload, mark (which would be very strange if the left table is the build side, see previous comment)

From my read of the mark join algorithm in the paper it appears that a mark join does not actually join two tables, but only adds the mark (the algorithm scans R and S. Only rows from R are inserted into the hash table H. The emit at the end is a scan of H which only contains rows from R. This is reinforced by the notation of the emit (emit r, marker of r) Compare this to the full outer hash join algorithm which contains lines like emit r,s and emit r,NULL)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct - the mark join only adds the mark column, the output is l_key, l_payload, mark (where left is the probe side). The cardinality of the left side (probe side) is preserved - i.e. the mark join neither removes nor adds additional rows.


### Mark Join Properties

| Property | Description | Required |
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
| Left Input | A relational input. | Required |
| Right Input | A relational input. | Required |
| Join Expression | A boolean condition that describes whether each record from the left set "match" the record from the right set. Field references correspond to the direct output order of the data. | Required. Can be (but not expected to be) the literal True. |



## Exchange Operator

The exchange operator will redistribute data based on an exchange type definition. Applying this operation will lead to an output that presents the desired distribution.
Expand Down
Loading