feat(api): support all dtypes in MapGet and MapContains #8648

NickCrews · 2024-03-13T20:10:59Z

I bet a lot of these tests will fail in CI. IDK, a lot of these tests feel like overkill. Could just ensure that the Expr can get created, but skip actually executing them on the backend. That requires a lot more boilerplate. But, it is sort of nice to actually have an up-to-date status on what types each backend supports in maps.

ibis/backends/tests/test_map.py

NickCrews · 2024-03-15T20:07:33Z

EDIT: it looks like these might have been failing for an unrelated reason? rebasing on main with teh maybe fix? @cpcloud what should I do about the failing flink tests in CI? If I run pytest -m flink -k all_types ibis/backends/tests/test_map.py locally, I get collected 1980 items / 1908 deselected / 72 selected. Why are they being skipped?

cpcloud · 2024-03-17T11:52:44Z

@NickCrews When making marker global variables in test modules we write them with lowercase.

NickCrews · 2024-03-18T16:59:57Z

@cpcloud fixed those global variable names. Still am curious if you can help me re: flink above so I can get the flink tests failing locally.

cpcloud

LGTM, thanks for working through this.

ibis/backends/flink/compiler.py

The mark for risingwave was misleading. It was sometimes combined with the postgres mark, saying the reason was that only string -> string is supported. But that isn't true. I had to get rid of the `raises=PsycoPg2InternalError` in the risingwave mark, because for some tests the error was actually raised by a lack of the FromYMD Operation. pytest marks don't have support for "this error is caused by one of these two reasons", so I have to go with the lowest common denominator, and not list a reason.

This should be O(1) now not O(n). Also it unifies the API so that MapGet and MapContains work more similarly, eg the same dtype combos will work. I was running into this in https://github.com/ibis-project/ibis/actions/runs/8364426602/job/22899760255?pr=8648

cpcloud

This is failing on Snowflake pretty hard.

…rrays

cpcloud · 2024-03-21T11:29:10Z

Clouds are passing:

…/ibis on  map-any-key is 📦 v8.0.0 via 🐍 v3.12.2 via ❄️   impure (ibis-3.12.2-env)
❯ pytest -m 'bigquery or snowflake' -n auto --dist loadgroup --snapshot-update
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=2614911091
rootdir: /home/cloud/src/ibis
configfile: pyproject.toml
plugins: xdist-3.5.0, pytest_httpserver-1.0.10, cov-4.1.0, benchmark-4.0.0, hypothesis-6.99.9, randomly-3.15.0, mock-3.12.0, repeat-0.9.3, snapshot-0.9.0, clarity-1.0.1
16 workers [3510 items]
......................x........x..........x....x.....x....x...x...x.x.......x....................x.xx...........xx...........................xxxx.x.xxx.xxx.x.xx........x..x...x.x.....x...x.......x...x [  5%]
............x.x.xx.........x.....x................x..x.xx.x...........x...............x........x..x...x..xx................x..x.......x.......x.....x..................xx............................... [ 11%]
..............xx............x...x..x.x..s.......x...................x....x.x...........x.........x..................x.............x..............x..........s........................................... [ 17%]
........................................................xx...x.............x.........................x....................x.x...................................x.................x...x.x..xx.x.xxx.xx.. [ 22%]
xx.xx.xxxx..x...xxxx..x.x..xxx.xxxxx...xxxxx.x...x...xx.x.xx.xx..x.xx...xx.xxx.x.x.x.x..x...x....x........x..................x..x.x......xx.x...x.x...xx..x.........x..........xx.........x.......x...x. [ 28%]
..x.......x..xx...x.xx...............x.........x.............xx.x..x.....x....x..............x.x...........x..........................x..x...x.......x...x.......x...........x..............xx.......... [ 34%]
...x......x...x.......x.xx.......x...xx.x............x..........sxxxxxxxxxxxx.x.................................x.......x.............x......x..........x...x....x.............x...xx............x...... [ 39%]
.........x.......x...x......x..xx..x.....x.....................xssssssssssssssssssssss.....x......x...........xxxx.......x.x...x...xx..x..........x...x.......x....x...xx.x....x......x...x...x..x...x.. [ 45%]
......xxx.........x..........xxx..xxxx.x.x.x..xxxx.x.x..xx.x..x..x.....x..x....x..x.......x.........x................x......x.x...................................x.x...xx........x.......x.......xxx... [ 51%]
.....................x.........x.....x...xx..x............xxx..................x.....................x.....................s..............x..x.x......x....xxxx.xxx...x.x.x.xxx.....xxx..xx.x.x....x.xx. [ 56%]
..x....xx...............x...............................................................................................x..x.xx...........x..x.......xx........x.........xx.x.......x..x..........x..... [ 62%]
.....x......x..............xx....xx.xxxx.x.xx...xxxxxxxxxxx.xxxxx.xx.xxx.xxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxx.x..xx..x.x...x.............................xx..x...... [ 68%]
..x.......x.x...x.....x....x.x.....xxx.x......x...x...........x....x...........x............x..x..............x...x.........x.xx....x....x..................................xx..........x.x..xxx.x...x.x [ 74%]
.x...x.x........x.....x.........x..x........xx........x..x.......xx.x.x.......s.....x...................x.ss..x..x..x......x....xxxxxxxxx..x.x........................................x.x......s....x... [ 79%]
.............x.........x....................................................x.................x.........xx..x.............x........................x...........................s....x................... [ 85%]
..x...x...................................x.........................................................x................................................................................................... [ 91%]
........................................................................................................................................................................................................ [ 96%]
......................................................................x................Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38640), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38640), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38656), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38656), raddr=('34.102.189.10', 443)>
..Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38668), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 38668), raddr=('34.102.189.10', 443)>
....Exception ignored in: <socket.socket fd=29, family=2, type=1, proto=6, laddr=('192.168.50.72', 43164), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=29, family=2, type=1, proto=6, laddr=('192.168.50.72', 43164), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=29, family=2, type=1, proto=6, laddr=('192.168.50.72', 43184), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=29, family=2, type=1, proto=6, laddr=('192.168.50.72', 43184), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43196), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43196), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43198), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43198), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43214), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 43214), raddr=('34.102.189.10', 443)>
........Exception ignored in: <socket.socket fd=25, family=2, type=1, proto=6, laddr=('192.168.50.72', 43250), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=25, family=2, type=1, proto=6, laddr=('192.168.50.72', 43250), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43264), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43264), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=25, family=2, type=1, proto=6, laddr=('192.168.50.72', 43270), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=25, family=2, type=1, proto=6, laddr=('192.168.50.72', 43270), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43282), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43282), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43284), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 43284), raddr=('34.102.189.10', 443)>
...Exception ignored in: <socket.socket fd=20, family=2, type=1, proto=6, laddr=('192.168.50.72', 40696), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=20, family=2, type=1, proto=6, laddr=('192.168.50.72', 40696), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 50310), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/fqm4hmsh6r2n2dcl1y2j5dqr6g6khwls-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 50310), raddr=('34.102.189.10', 443)>
x....                                                                                           [100%]
========================================================================== 2919 passed, 31 skipped, 560 xfailed in 687.95s (0:11:27) ===========================================================================

NickCrews · 2024-03-21T17:02:53Z

Thanks so much for finishing this one off @cpcloud ! This was a marathon, I started getting tired of figuring this out :)

NickCrews force-pushed the map-any-key branch 3 times, most recently from a1b0b83 to 6678570 Compare March 14, 2024 03:09

cpcloud requested changes Mar 14, 2024

View reviewed changes

ibis/backends/tests/test_map.py Outdated Show resolved Hide resolved

ibis/backends/tests/test_map.py Outdated Show resolved Hide resolved

cpcloud added this to the 9.0 milestone Mar 17, 2024

cpcloud force-pushed the map-any-key branch from d07e177 to b1a15fd Compare March 19, 2024 12:49

cpcloud approved these changes Mar 19, 2024

View reviewed changes

cpcloud added expressions Issues or PRs related to the expression API datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) feature Features or general enhancements labels Mar 19, 2024

cpcloud changed the title ~~feat: support all dtypes in MapGet and MapContains~~ feat(api): support all dtypes in MapGet and MapContains Mar 19, 2024

NickCrews mentioned this pull request Mar 19, 2024

bug: flink isn't installed in local dev env #8696

Closed

1 task

NickCrews force-pushed the map-any-key branch 2 times, most recently from 1fe6ae6 to d5ad0fc Compare March 19, 2024 18:03

cpcloud requested changes Mar 21, 2024

View reviewed changes

ibis/backends/flink/compiler.py Outdated Show resolved Hide resolved

cpcloud force-pushed the map-any-key branch from 53ca4b0 to f2b8755 Compare March 21, 2024 09:43

cpcloud approved these changes Mar 21, 2024

View reviewed changes

NickCrews and others added 10 commits March 21, 2024 06:47

feat: support all dtypes in MapGet and MapContains

c39a69a

chore: apply review suggestions

e2ce2ce

chore: lowercase mark names

2e1406a

chore: skip weird map types on flink

a539c82

chore: xpass more types on flink

b02508e

chore: fixup failing flink tests

504eeb8

test(flink): add map contains null test

b8f7296

fix(flink): cast inputs in map contains impl

3e9b9df

cpcloud force-pushed the map-any-key branch from 8d49109 to 3e9b9df Compare March 21, 2024 10:47

cpcloud requested changes Mar 21, 2024

View reviewed changes

cpcloud added 2 commits March 21, 2024 07:04

fix(snowflake): use upstream function for constructing objects from a…

467430f

…rrays

test(snowflake): xfail on non-string map keys

821c790

cpcloud approved these changes Mar 21, 2024

View reviewed changes

cpcloud enabled auto-merge (squash) March 21, 2024 11:30

cpcloud merged commit 401e0a4 into ibis-project:main Mar 21, 2024
80 of 82 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api): support all dtypes in MapGet and MapContains #8648

feat(api): support all dtypes in MapGet and MapContains #8648

NickCrews commented Mar 13, 2024 •

edited

Loading

NickCrews commented Mar 15, 2024 •

edited

Loading

cpcloud commented Mar 17, 2024

NickCrews commented Mar 18, 2024

cpcloud left a comment

cpcloud left a comment

cpcloud commented Mar 21, 2024

NickCrews commented Mar 21, 2024

feat(api): support all dtypes in MapGet and MapContains #8648

feat(api): support all dtypes in MapGet and MapContains #8648

Conversation

NickCrews commented Mar 13, 2024 • edited Loading

NickCrews commented Mar 15, 2024 • edited Loading

cpcloud commented Mar 17, 2024

NickCrews commented Mar 18, 2024

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Mar 21, 2024

NickCrews commented Mar 21, 2024

NickCrews commented Mar 13, 2024 •

edited

Loading

NickCrews commented Mar 15, 2024 •

edited

Loading