Join fails if number of records after join exceeds Uint32

[MartinBarus]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

When you join two tables, and the output number of records would exceed UINT32 (4,294,967,295) the result is incorrect. It is most likely related to Index type: UInt32 in the polars version info, however, if that is the case, it would be heplful to see a message/warning/exception saying join result will exceed Index type therefore join will produce incorrect result.

Current behaviour results in a silent failure.

Reproducible example

import polars as pl
x = pl.DataFrame({"a":0, "b":range(65536)})
join = x.join(x, on="a")
join

Expected behavior

Result should be non-empty and contain 65536 * 65536 rows. Instead it is empty DataFrame. The same snippet works if you change 65536 for a smaller number (say 10).

In the case where I noticed it, join result was supposed to contain 9088223035 rows, instead it contained 498288443 rows, where np.uint32( np.int64(9088223035)) yields 498288443

Installed versions

---Version info---
Polars: 0.15.17
Index type: UInt32
Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.17
Python: 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0]
---Optional dependencies---
pyarrow: 8.0.0
pandas: 1.4.4
numpy: 1.21.6
fsspec: 2022.11.0
connectorx: <not installed>
xlsx2csv: <not installed>
deltalake: <not installed>
matplotlib: 3.6.1

[ritchie46]

https://github.com/pola-rs/polars#going-big

[MartinBarus]

https://github.com/pola-rs/polars#going-big

Yeah I found it once I knew the join was incorrect, but the join should fail and explicitely mention there was an issue.

As mentioned, current behaviour pretends as if everything is OK, so once you test your script on small data and run on big data, it is quite likely you won't be able to tell what is wrong and why.

[ritchie46]

https://github.com/pola-rs/polars#going-big

Yeah I found it once I knew the join was incorrect, but the join should fail and explicitely mention there was an issue.

As mentioned, current behaviour pretends as if everything is OK, so once you test your script on small data and run on big data, it is quite likely you won't be able to tell what is wrong and why.

Overflow checks would be very expensive. However we could check the output size and print a warning at u32::MAX value.