Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix input table jpy restriction #5260

Merged
merged 12 commits into from
Mar 21, 2024

Conversation

arman-ddl
Copy link
Contributor

@arman-ddl arman-ddl commented Mar 16, 2024

Trying to create a custom input table resulted in:

Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: Failed to get input table for 'InputTablesCorePlus.TestViaSpecPython'. : deephaven.dherror.DHError: j_table type is not io.deephaven.engine.table.Table : j_table type is not io.deephaven.engine.table.Table
Traceback (most recent call last):
  File "/usr/illumon/dnd/latest/py/packages/deephaven_enterprise/database.py", line 15, in do_with_dh_error
    result = func()
  File "/usr/illumon/dnd/latest/py/packages/deephaven_enterprise/database.py", line 366, in <lambda>
    return do_with_dh_error(lambda: InputTable(j_table=self.j_wdb.inputTable(namespace, table_name)), F"Failed to get input table for '{namespace}.{table_name}'.")
  File "/usr/illumon/dnd/venv/latest/lib/python3.10/site-packages/deephaven/table_factory.py", line 241, in __init__
    super().__init__(j_table)
  File "/usr/illumon/dnd/venv/latest/lib/python3.10/site-packages/deephaven/table.py", line 434, in __init__
    raise DHError("j_table type is not io.deephaven.engine.table.Table")
deephaven.dherror.DHError: j_table type is not io.deephaven.engine.table.Table : j_table type is not io.deephaven.engine.table.Table
NoneType: None

Somehow missed this line in my testing of Core+ custom input tables until a fresh install. Added a basic test for custom input table creation.

Although BaseArrayBackedInputTable is no longer suitable, should we be more specific than Table jpy requirement, e.g. "QueryTable"?

@arman-ddl arman-ddl marked this pull request as draft March 18, 2024 23:55
@jmao-denver jmao-denver marked this pull request as ready for review March 20, 2024 03:56
Copy link
Contributor

@jmao-denver jmao-denver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified the Java object wrapping logic to always wrap with the most specific (deepest) descendent class in the class hierarchy

py/server/tests/test_table_factory.py Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Show resolved Hide resolved
py/server/deephaven/_wrapper.py Show resolved Hide resolved
py/server/deephaven/table_factory.py Show resolved Hide resolved
jmao-denver and others added 3 commits March 20, 2024 09:19
Co-authored-by: Chip Kent <5250374+chipkent@users.noreply.github.com>
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
py/server/deephaven/_wrapper.py Outdated Show resolved Hide resolved
jmao-denver and others added 2 commits March 20, 2024 14:08
Co-authored-by: Chip Kent <5250374+chipkent@users.noreply.github.com>
chipkent
chipkent previously approved these changes Mar 20, 2024
py/server/tests/test_table_factory.py Show resolved Hide resolved
py/server/deephaven/table_factory.py Show resolved Hide resolved
def wrap_j_object(j_obj: jpy.JType) -> Union[JObjectWrapper, jpy.JType]:
""" Wraps the specified Java object as an instance of a custom wrapper class if one is available, otherwise returns
the raw Java object. """
def _wrap_with_subclass(j_obj: jpy.JType, cls: type) -> Optional[JObjectWrapper]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably, this is slower than the existing behavior. The only place I can imagine where that might matter is something like a PartitionedTable.transform with a Python function. Does the Java constituent Table get wrapped automatically as a Python Table wrapper, and if so , is it slow enough to matter?

@jmao-denver you might need to test this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they are not. When requested (e.g. via. constituent_tables()), the constituent tables are explicitly wrapped in Table, not through this auto wrapping facility.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that addresses my concern. I am talking about when you pass a Python function (adapted to Java) to transform.
deephaven.table.PartitionedTable.transform

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some quick/minimal testing, ~2 - 4 % slower, the more constituent tables, the slower, which is kinda expected (more wrapping):

def transform_func(t: Table) -> Table:
    return t.update("f = a + b")

t = empty_table(100_000_000).update(["a = i", "b = i + 1", "c = i % 10_000", "d = `text`", "e = i % 1000000"])
pt = t.partition_by(by=["c"])
print("num of constituent tables: ", len(pt.constituent_tables))
print("constituent table size: ", pt.constituent_tables[0].size)
with make_user_exec_ctx():
    st = time.process_time_ns()
    transformed_pt = pt.transform(transform_func)
    print("transform time: ", (time.process_time_ns() - st)/10**9)
    self.assertIsNotNone(transformed_pt)

Before:

num of constituent tables:  10000
constituent table size:  10000
transform time:  42.321798386

num of constituent tables:  1000
constituent table size:  100000
transform time:  20.706249172

After:

num of constituent tables:  10000
constituent table size:  10000
transform time:  44.08636249

num of constituent tables:  1000
constituent table size:  100000
transform time:  21.192759225

Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This addressed two of my concerns

@jmao-denver jmao-denver merged commit c301ea6 into deephaven:main Mar 21, 2024
14 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Mar 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants