Enable larger join tests

[hendrikmakait]

• Closes Enable larger joins for p2p in test_join.py #641

[ncclementi]

@hendrikmakait is this still a draft PR, or do we want to get this one going?

[hendrikmakait]

@ncclementi: I still have to check why the tests don't run on CI; likely the pytest worker gets OOM-killed.

[hendrikmakait]

@ncclementi: As it turns out, one issue with the tests was that the partition size was very small (~8 MiB). I've adjusted the tests now to use larger partitions, let's see what CI says.

[hendrikmakait]

I've also dropped the 10x join since it would mean spilling 10-20x the cluster memory to disk which would slow tests down significantly. We could (and should) add the occasional BIG data integration test run.

[hendrikmakait]

test_join_big shows unnecessarily high memory usage with p2p due (dask/distributed#7496). It is still performing better, so I count that as a win.

test_join_big_small does not show any effect due to early materialization of the smaller dataframe which circumvents a distributed join (#669).

[ncclementi]

test_join_big shows unnecessarily high memory usage with p2p due (dask/distributed#7496). It is still performing better, so I count that as a win.

This makes sense. Hopefully, when things get fixed we should see a nice drop in the memory.

[ncclementi]

This LGTM, the failure is unrelated.
Only comment is that we should probably add a separate issue to track the case for size=10 as part of integration tests, and then mentioned it on this PR just for completion.

[ncclementi]

Out of curiosity, why did you choose the name "predicate"?

[hendrikmakait]

That's me coming from a DB background. I wanted a name that's more descriptive in this context than x2 and since it's the column used in the join predicate (i.e., the expression used to merge the tables/dataframes), that's what I ended up with. This could also be merge_col or something like that if you find that easier to understand.

[ncclementi]

I see. That's good, no need to change it, it was more out of curiosity and I just learn something new :)
I'm merging this in.

[ncclementi]

Thanks @hendrikmakait, let's open a separate issue to track integration tests for the bigger case.