feat: add support for HashBytes operation on multiple backends

[pdgarden]

Is your feature request related to a problem?

I would need to use md5 function with several backends that I use on different environments (DuckDB, Pyspark, MSSQL), with Pyspark being the most essential one.

Describe the solution you'd like

Add support for operation ibis.expr.operations.generic.HashBytes on the following backends : DuckDB, Pyspark, MSSQL

What version of ibis are you running?

7.2.0

What backend(s) are you using, if any?

DuckDB, Pyspark, MSSQL

Code of Conduct

• I agree to follow this project's Code of Conduct

[lostmygithubaccount]

it will not be as fast as a native implementation (or a pyarrow UDF instead of python), but you can accomplish this now with a UDF:

[ins] In [1]: import ibis

[ins] In [2]: ibis.options.interactive = True

[ins] In [3]: t = ibis.examples.penguins.fetch()

[ins] In [4]: t.limit(3)
Out[4]:
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

[ins] In [5]: from hashlib import md5

[ins] In [6]: @ibis.udf.scalar.python
         ...: def md5_hash(s: str) -> str:
         ...:     return md5(s.encode()).hexdigest()
         ...:

[ins] In [7]: t = t.mutate(md5_hash=md5_hash(ibis._.species)).relocate("md5_hash")

[ins] In [8]: t.limit(3)
Out[8]:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ md5_hash                         ┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string                           │ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├──────────────────────────────────┼─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ aee095557d7c4c311ffb9718b791ad18 │ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ aee095557d7c4c311ffb9718b791ad18 │ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ aee095557d7c4c311ffb9718b791ad18 │ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
└──────────────────────────────────┴─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

[gforsyth]

Hey @pdgarden -- hashing functions are a bit of a mess depending on the backend in use.

Are you looking for a function that returns a string (like a hexdigest) or bytes (like a digest)?

[pdgarden]

Hi,

Thank you very much for your quick reply. I am looking for a function which would ideally return a string like hexdigest.

[gforsyth]

Hey @pdgarden -- there's a new hexdigest method that was just merged in. It will be available in the next release (sometime next week) and you can use that to get the hexdigest of the input columns. Currently it is supported with DuckDB, mssql, and pyspark. More support coming soon.

[pdgarden]

That's great, thank you for your support and reactivity.