Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ObjectType and Plugins #1787

Merged
merged 67 commits into from
Jan 24, 2022
Merged
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
a24ec60
ObjectType and Plugins
devinrsmith Dec 30, 2021
ecf7e80
Remove old custom type name
devinrsmith Dec 30, 2021
91bd98a
Fix python integration tests
devinrsmith Dec 30, 2021
74aec62
Move logic into deephaven.server.plugin
devinrsmith Dec 31, 2021
ebda9d1
spotless
devinrsmith Dec 31, 2021
968ec24
Move deephaven.server into deephaven2.server so fix broken integratio…
devinrsmith Jan 1, 2022
bb638e6
Review responses
devinrsmith Jan 4, 2022
fe8414f
f
devinrsmith Jan 5, 2022
e4da7bb
f
devinrsmith Jan 5, 2022
8ee1947
better dagger and docs
devinrsmith Jan 6, 2022
354a5a7
f
devinrsmith Jan 6, 2022
edc0617
f
devinrsmith Jan 6, 2022
ca1875d
Merge remote-tracking branch 'upstream/main' into fetch-object
devinrsmith Jan 6, 2022
781cbf2
Stuff
devinrsmith Jan 6, 2022
9de9a46
remove test from dockerfile
devinrsmith Jan 6, 2022
44bec3b
wrong cast
devinrsmith Jan 7, 2022
18b66c1
Revert "remove test from dockerfile"
devinrsmith Jan 7, 2022
a2dc389
Add ObjectServiceTest
devinrsmith Jan 7, 2022
2881aff
Fix plugin discovery for unit tests
devinrsmith Jan 7, 2022
aa89bad
Refactor into Plugin interface which can be walked
devinrsmith Jan 10, 2022
965337b
Fix python, remove png
devinrsmith Jan 10, 2022
e12c2f8
Ryan's suggestions
devinrsmith Jan 10, 2022
f66297a
Remove matplotlib plugin installation
devinrsmith Jan 10, 2022
5e593e6
Merge remote-tracking branch 'upstream/main' into fetch-object
devinrsmith Jan 10, 2022
7dbc3b0
Implement support for the ABC released deephaven-plugin.
devinrsmith Jan 10, 2022
de91510
remove old comment
devinrsmith Jan 11, 2022
49577a0
Use python native array, not java byte array
devinrsmith Jan 11, 2022
3f0e0df
Ensure that python native objects can be exported
devinrsmith Nov 1, 2021
8d3b2e1
Rename
devinrsmith Jan 11, 2022
45970fd
recursive fetch
devinrsmith Jan 11, 2022
2941ea9
Add init method
devinrsmith Jan 12, 2022
5804556
Fix reference id bytes
devinrsmith Jan 12, 2022
b0afabf
Cleanup and consolidate to TypedTicket
devinrsmith Jan 12, 2022
14452e2
auto service
devinrsmith Jan 13, 2022
7265104
f
devinrsmith Jan 13, 2022
ce855b9
Typing and Reference stuff
devinrsmith Jan 13, 2022
88d1686
Merge remote-tracking branch 'upstream/main' into fetch-object
devinrsmith Jan 13, 2022
a7fc020
Remove FetchFigure
devinrsmith Jan 13, 2022
d905b37
Update generated gwt bindings, new code doesn't work in api yet
niloc132 Jan 14, 2022
e342fe1
Updated Exporter interface to allow caller to be more explicit about …
devinrsmith Jan 14, 2022
5c3beb2
Update console js layer
devinrsmith Jan 14, 2022
7576357
fix test
devinrsmith Jan 14, 2022
8f4e8e3
TypedTicket has Ticket
devinrsmith Jan 14, 2022
9032e66
Use empty string instead of optional
devinrsmith Jan 14, 2022
366151c
Fix js vars, and plumb object service for figures
niloc132 Jan 18, 2022
7bbf573
CHERRY-PICK always set an errorId
niloc132 Jan 18, 2022
30e92c2
Figure shouldn't try to re-export, clean up console vars
niloc132 Jan 18, 2022
01c93e3
Verify CTS gets a sane handle
niloc132 Jan 18, 2022
1dd9a59
More test coverage
devinrsmith Jan 18, 2022
17eea96
Added "typed_" prefix. Fixed FieldInfo.
devinrsmith Jan 19, 2022
21a6185
Merge remote-tracking branch 'dsmith/fetch-object' into fetch-object-js
niloc132 Jan 19, 2022
4009b93
Update generated gwt bindings
niloc132 Jan 19, 2022
7681a3f
Update to new typedticket stuff
niloc132 Jan 19, 2022
fe57ee7
spotless
devinrsmith Jan 20, 2022
eb9863c
Copy regenerated _pb2
devinrsmith Jan 20, 2022
1811bb3
Final fixes
devinrsmith Jan 21, 2022
33cf3d2
Merge remote-tracking branch 'upstream/main' into fetch-object
devinrsmith Jan 21, 2022
b298d47
plumb TypedTicket
devinrsmith Jan 21, 2022
cf36e5b
regen and copy python proto
devinrsmith Jan 21, 2022
e492010
Update test
devinrsmith Jan 21, 2022
95ada5c
Update gwt bindings
niloc132 Jan 21, 2022
15296cb
Update JS API to use typedticket for object fetch
niloc132 Jan 21, 2022
d28c82d
Remove Reference.ticket
devinrsmith Jan 21, 2022
a995b11
fix python side for Reference
devinrsmith Jan 21, 2022
d736c85
FetchObject client don't try Table. Better exceptions for FetchObject
devinrsmith Jan 24, 2022
580b272
Add reference to #1872
devinrsmith Jan 24, 2022
ddfcdaa
add because notes for api dagger dependencies
devinrsmith Jan 24, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move logic into deephaven.server.plugin
devinrsmith committed Dec 31, 2021

Unverified

This user has not yet uploaded their public signing key.
commit 74aec625ee603f8c8fe620c50a13fdcc5d08d6b3
20 changes: 0 additions & 20 deletions Integrations/python/deephaven/plugin/__init__.py

This file was deleted.

68 changes: 68 additions & 0 deletions Integrations/python/deephaven/server/plugin/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import jpy

PluginCallback = jpy.get_type('io.deephaven.plugin.PluginCallback')
Exporter = jpy.get_type('io.deephaven.plugin.type.Exporter')

DEEPHAVEN_PLUGIN_ENTRY_KEY = 'deephaven.plugin'
DEEPHAVEN_PLUGIN_REGISTER_NAME = 'register_into'

def get_plugin_entrypoints(name: str):
import sys
if sys.version_info < (3, 8):
# TODO(deephaven-base-images#6): Add importlib-metadata backport install for future server plugin support
# We can remove the exception handling once above gets merged in
try:
from importlib_metadata import entry_points
except ImportError:
return []
else:
from importlib.metadata import entry_points
return entry_points(group=DEEPHAVEN_PLUGIN_ENTRY_KEY, name=name) or []

def all_plugins_register_into(callback: PluginCallback):
callback_adapter = CallbackAdapter(callback)
for entrypoint in get_plugin_entrypoints(DEEPHAVEN_PLUGIN_REGISTER_NAME):
plugin_register_into = entrypoint.load()
plugin_register_into(callback_adapter)

# TODO(deephaven-core#1791): CallbackAdapter implements CallbackABC
class CallbackAdapter:
def __init__(self, callback: PluginCallback):
self._callback = callback

# TODO(deephaven-core#1791): type hint object_type as ObjectTypeABC
def register_object_type(self, object_type):
self._callback.registerObjectType(ObjectTypeAdapter(object_type))

def __str__(self):
return str(self._callback)

# TODO(deephaven-core#1791): ExporterAdapter implements ExporterABC
class ExporterAdapter:
def __init__(self, exporter: Exporter):
self._exporter = exporter

def new_server_side_export(self, object):
# TODO(deephaven-core#1791): define and use ExportABC
raise NotImplementedError

def __str__(self):
return str(self._exporter)

class ObjectTypeAdapter:
# TODO(deephaven-core#1791): type hint user_object_type as ObjectTypeABC
def __init__(self, user_object_type):
self._user_object_type = user_object_type

@property
def name(self):
return self._user_object_type.name

def is_type(self, object):
return self._user_object_type.is_type(object)

def to_bytes(self, exporter: Exporter, object):
return self._user_object_type.to_bytes(ExporterAdapter(exporter), object)

def __str__(self):
return str(self._user_object_type)
2 changes: 1 addition & 1 deletion docker/server/src/main/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ FROM $BASE as install
#RUN set -eux; \
# python3 -m pip install -q --no-cache-dir \
# importlib-metadata \
# https://github.com/deephaven/deephaven-plugin-matplotlib/releases/download/v0.0.1.dev1/deephaven_plugin_matplotlib-0.0.1.dev1-py3-none-any.whl
# https://github.com/deephaven/deephaven-plugin-matplotlib/releases/download/v0.0.1.dev2/deephaven_plugin_matplotlib-0.0.1.dev2-py3-none-any.whl

COPY licenses/ /

Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package io.deephaven.server.plugin;

import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import io.deephaven.plugin.type.ObjectTypeLookup;

import javax.inject.Singleton;

@Module
public interface ObjectTypesModule {

@Provides
@Singleton
static ObjectTypes providesObjectTypes() {
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved
return new ObjectTypes();
}

@Binds
ObjectTypeLookup providesLookup(ObjectTypes types);
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved
}
Original file line number Diff line number Diff line change
@@ -1,21 +1,8 @@
package io.deephaven.server.plugin;

import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import io.deephaven.plugin.type.ObjectTypeLookup;

import javax.inject.Singleton;

@Module
@Module(includes = { ObjectTypesModule.class })
public interface PluginModule {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this serve a purpose on its own? it seems like it just pulls in the plugin.ObjectTypesModule?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a follow up PR for an application plugin; instead of refactoring the structure in the application plugin, I'm providing the follow-up structure for it here.


@Provides
@Singleton
static ObjectTypes providesObjectTypes() {
return new ObjectTypes();
}

@Binds
ObjectTypeLookup providesLookup(ObjectTypes types);
}
124 changes: 12 additions & 112 deletions server/src/main/java/io/deephaven/server/plugin/PluginsLoader.java
Original file line number Diff line number Diff line change
@@ -1,38 +1,19 @@
package io.deephaven.server.plugin;

import io.deephaven.base.log.LogOutput;
import io.deephaven.base.log.LogOutputAppendable;
import io.deephaven.internal.log.LoggerFactory;
import io.deephaven.io.logger.Logger;
import io.deephaven.plugin.Plugin;
import io.deephaven.plugin.PluginCallback;
import io.deephaven.plugin.type.ObjectType;
import io.deephaven.plugin.type.ObjectTypeBase;
import io.deephaven.plugin.type.Exporter;
import io.deephaven.server.console.ConsoleServiceGrpcImpl;
import org.jpy.PyLib.CallableKind;
import org.jpy.PyModule;
import org.jpy.PyObject;

import javax.inject.Inject;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Objects;
import java.util.ServiceLoader;

public final class PluginsLoader implements PluginCallback {
private static final Logger log = LoggerFactory.getLogger(PluginsLoader.class);

private static void allServiceLoaderRegisterInto(PluginCallback callback) {
for (Plugin provider : ServiceLoader.load(Plugin.class)) {
provider.registerInto(callback);
}
}

private static void allPythonRegisterInto(PluginCallback callback) {
try (final PythonPluginModule module = PythonPluginModule.of()) {
module.all_plugins_register_into(new CallbackAdapter(callback));
}
}

public final ObjectTypes types;

@Inject
@@ -43,120 +24,39 @@ public PluginsLoader(ObjectTypes types) {
public void registerAll() {
log.info().append("Registering plugins...").endl();
final Counting serviceLoaderCount = new Counting();
allServiceLoaderRegisterInto(serviceLoaderCount);
io.deephaven.server.plugin.java.Loader.allRegisterInto(serviceLoaderCount);
final Counting pythonModuleCount = new Counting();
if (ConsoleServiceGrpcImpl.isPythonSession()) {
allPythonRegisterInto(pythonModuleCount);
io.deephaven.server.plugin.python.Loader.allRegisterInto(pythonModuleCount);
}
log.info().append("Registered via service loader: ").append(serviceLoaderCount.count).endl();
log.info().append("Registered via service loader: ").append(serviceLoaderCount).endl();
if (ConsoleServiceGrpcImpl.isPythonSession()) {
log.info().append("Registered via python module: ").append(pythonModuleCount.count).endl();
log.info().append("Registered via python modules: ").append(pythonModuleCount).endl();
}
}

@Override
public void registerObjectType(ObjectType objectType) {
log.info().append("Registering plugin object type: ")
log.info().append("Registering object type: ")
.append(objectType.name()).append(" / ")
.append(objectType.toString())
.endl();
types.register(objectType);
}

private class Counting implements PluginCallback {
private class Counting implements PluginCallback, LogOutputAppendable {

private int count = 0;
private int objectTypeCount = 0;

@Override
public void registerObjectType(ObjectType objectType) {
PluginsLoader.this.registerObjectType(objectType);
++count;
}
}

interface PythonCustomType {

static PythonCustomType of(PyObject object) {
return (PythonCustomType) object.createProxy(CallableKind.FUNCTION, PythonCustomType.class);
}

String name();

boolean is_type(PyObject object);

// TODO(deephaven-core#1785): Use more pythonic wrapper for io.deephaven.plugin.type.Exporter
byte[] to_bytes(Exporter exporter, PyObject object);
}

private static final class Adapter extends ObjectTypeBase {

public static Adapter of(PythonCustomType module) {
return new Adapter(module.name(), module);
}

private final String name;
private final PythonCustomType module;

private Adapter(String name, PythonCustomType module) {
this.name = Objects.requireNonNull(name);
this.module = Objects.requireNonNull(module);
}

@Override
public String name() {
return name;
}

@Override
public boolean isType(Object o) {
if (!(o instanceof PyObject)) {
return false;
}
return module.is_type((PyObject) o);
++objectTypeCount;
}

@Override
public void writeToTypeChecked(Exporter exporter, Object object, OutputStream out) throws IOException {
out.write(module.to_bytes(exporter, (PyObject) object));
}

@Override
public String toString() {
return name + ":" + module;
}
}

private static class CallbackAdapter {
private final PluginCallback callback;

public CallbackAdapter(PluginCallback callback) {
this.callback = Objects.requireNonNull(callback);
public LogOutput append(LogOutput logOutput) {
return logOutput.append("objectType=").append(objectTypeCount);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"objectTypeCount=" perhaps? or just "count="?

Copy link
Member Author

@devinrsmith devinrsmith Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logging at the end of the day is going to look like:

Registered via service loader: objectType=M,...
Registered via python modules: objectType=N,...

I'm not a fan of extraneously duplicating "count" in the log lines (in anticipation of more plugin types besides object type), but maybe it makes more sense to have a better name for the plugin type - ie, "ObjectType" to better infer io.deephaven.plugin.type.ObjectType

}

@SuppressWarnings("unused")
public void register_object_type(PyObject module) {
final PythonCustomType pythonCustomType = PythonCustomType.of(module);
final Adapter adapter = Adapter.of(pythonCustomType);
callback.registerObjectType(adapter);
}

// Remove before merge, but still want code reviewers to be able to use existing matplotlib plugin
@Deprecated
public void register_custom_type(PyObject module) {
register_object_type(module);
}
}

interface PythonPluginModule extends AutoCloseable {

static PythonPluginModule of() {
return (PythonPluginModule) PyModule.importModule("deephaven.plugin")
.createProxy(CallableKind.FUNCTION, PythonPluginModule.class);
}

void all_plugins_register_into(CallbackAdapter callback);

@Override
void close();
}
}
14 changes: 14 additions & 0 deletions server/src/main/java/io/deephaven/server/plugin/java/Loader.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package io.deephaven.server.plugin.java;

import io.deephaven.plugin.Plugin;
import io.deephaven.plugin.PluginCallback;

import java.util.ServiceLoader;

public final class Loader {
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved
public static void allRegisterInto(PluginCallback callback) {
for (Plugin provider : ServiceLoader.load(Plugin.class)) {
provider.registerInto(callback);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package io.deephaven.server.plugin.python;

import io.deephaven.plugin.PluginCallback;
import org.jpy.PyObject;

class CallbackAdapter {

private final PluginCallback callback;

public CallbackAdapter(PluginCallback callback) {
this.callback = callback;
}

@SuppressWarnings("unused")
public void registerObjectType(PyObject objectTypeAdapter) {
callback.registerObjectType(new ObjectTypeAdapter(objectTypeAdapter));
}
}
11 changes: 11 additions & 0 deletions server/src/main/java/io/deephaven/server/plugin/python/Loader.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
package io.deephaven.server.plugin.python;

import io.deephaven.plugin.PluginCallback;

public final class Loader {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we couldn't have a single java type which itself is registered as a Plugin (via the service loader mechanism) and is as such automatically included in the java.Loader, provided it is on the classpath?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a possible improvement in the future - will note ticket.

public static void allRegisterInto(PluginCallback callback) {
try (final Module module = Module.of()) {
module.all_plugins_register_into(new CallbackAdapter(callback));
}
}
}
20 changes: 20 additions & 0 deletions server/src/main/java/io/deephaven/server/plugin/python/Module.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
package io.deephaven.server.plugin.python;

import org.jpy.PyLib.CallableKind;
import org.jpy.PyModule;

interface Module extends AutoCloseable {

static Module of() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can any of these wiring classes live in their own gradle project? Most/all of the java/python loader code seems like it should be able to be external, and possibly only be loaded at runtime (for example, if python isn't even in use, skip it)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a possible improvement in the future - will note ticket.

final PyModule module = PyModule.importModule("deephaven.server.plugin");
if (module == null) {
throw new IllegalStateException("Unable to find `deephaven.server.plugin` module");
}
return (Module) module.createProxy(CallableKind.FUNCTION, Module.class);
}

void all_plugins_register_into(CallbackAdapter callback);

@Override
void close();
}
Loading