Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposed gRPC, Java server API for bidirectional stream widgets #4226

Merged
merged 86 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from 84 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
71ce7e1
Update object.proto
mattrunyon Jun 23, 2023
572d28b
Regenerate auth bindings
mattrunyon Jun 23, 2023
44f3144
Wiring for server and plugin messages
mattrunyon Jun 28, 2023
4f9a514
Update sendmessage
mattrunyon Jun 28, 2023
8d3560f
Update gitignore
mattrunyon Jun 28, 2023
0ca5189
Update gwt proto bindings
mattrunyon Jun 28, 2023
d063264
Revert accidental gwt proto changes
mattrunyon Jun 28, 2023
264e052
Cleanup payload gwt enum
mattrunyon Jun 28, 2023
64921a3
Spotless
mattrunyon Jun 28, 2023
2198868
Add missing gwt bindings
mattrunyon Jun 28, 2023
18206eb
More missing gwt bindings
mattrunyon Jun 28, 2023
2c34658
WIP JsWidget Bidir stream
mattrunyon Jun 28, 2023
4c81e83
Bidi communication
mattrunyon Jul 19, 2023
412ea91
Spotless
mattrunyon Jul 19, 2023
4b686cd
Regenerate py/go/cpp clients
mattrunyon Jul 19, 2023
ad65324
Remove unused bidiobjectbase from python server
mattrunyon Jul 19, 2023
2631937
safely close
niloc132 Jul 20, 2023
7d7583e
Draft of proposed api changes
niloc132 Jul 24, 2023
ce31efa
Merge branch 'main' into bidir-communication
niloc132 Jul 24, 2023
3098730
Update generated clients
niloc132 Jul 24, 2023
3d39c9e
Added javadoc
niloc132 Jul 24, 2023
e1bf8d9
Restore older, necessary APIs, renaming/notes for discussion
niloc132 Jul 25, 2023
1c8f76b
Regenerated gwt bindings
niloc132 Jul 25, 2023
0885692
Update js api to fit
niloc132 Jul 25, 2023
748af11
Revert go changes
niloc132 Jul 25, 2023
592927a
Fix broken test
niloc132 Jul 25, 2023
9eab27a
Bugs fixed while building a sample plugin
niloc132 Jul 28, 2023
a17460f
Provide an API to reference tickets in calls to the server
niloc132 Jul 30, 2023
1ee807d
Update field naming
niloc132 Jul 30, 2023
4be526e
regenerate gwt bindings
niloc132 Jul 30, 2023
a0fdc8a
Merge branch 'main' into bidir-communication
niloc132 Jul 30, 2023
c6fbcda
JS API cleanup, docs, ts wiring
niloc132 Aug 1, 2023
e5cd700
Fix js widget to accept new exports from the server
niloc132 Aug 1, 2023
4de04ad
Refactor to remove most of old writeTo methods, and only do bidi
niloc132 Aug 1, 2023
15b055d
Clean up docs/naming
niloc132 Aug 2, 2023
535159c
Correct python adapters
niloc132 Aug 2, 2023
ac12bf6
Ensure user has a session before starting a message stream
niloc132 Aug 2, 2023
c212879
Review feedback
niloc132 Aug 2, 2023
8610d81
Merge branch 'main' into bidir-communication
niloc132 Aug 2, 2023
b288247
Client cleanup
niloc132 Aug 2, 2023
8ea02b7
Move internal python code to deephaven_internal, add simple docs
niloc132 Aug 2, 2023
aeab9fa
Review feedback, fixes from smoke testing
niloc132 Aug 3, 2023
954d5a0
Without js reconnect, don't bother firing events for messages on init
niloc132 Aug 3, 2023
791ee88
Update proto from review feedback
niloc132 Aug 3, 2023
82a85f1
Update generated py/cpp bindings
niloc132 Aug 3, 2023
7f774ad
Regenerate Gwt Bindings
niloc132 Aug 3, 2023
ce92bf0
Update js api to use new bindings
niloc132 Aug 3, 2023
55a3af2
Rearrange inner types, fix imports
niloc132 Aug 3, 2023
461550c
Draft at ObjectType API enhancements, request queuing
niloc132 Aug 3, 2023
98b69fa
Improve error handling, fix bug in queuing
niloc132 Aug 4, 2023
a9dda85
Spotless
niloc132 Aug 4, 2023
2700320
Add PluginClient
jmao-denver Aug 5, 2023
9821ce8
Fix proto tag to be able to send objects to the server
niloc132 Aug 7, 2023
4a9a2a8
Merge branch 'main' into bidir-communication
niloc132 Aug 7, 2023
e5a4c7a
Tried to make the plugin_client a bit more usable
niloc132 Aug 7, 2023
0457867
Correct server plugin impl to round-trip objects
niloc132 Aug 8, 2023
14284f5
Allow python client to use unfetched exported tables
niloc132 Aug 8, 2023
088e7ec
Add clarifying documentation
niloc132 Aug 8, 2023
720612a
Correct queue's lambda type
niloc132 Aug 8, 2023
c95fa8e
typo in js api
niloc132 Aug 8, 2023
731d236
Update python docs
niloc132 Aug 8, 2023
4e24e71
More type hints, easier to understand name
niloc132 Aug 8, 2023
ee0b4fc
Respond to review comments
jmao-denver Aug 8, 2023
4e67d76
Apply devin's suggestion of a specific type for a pending op
niloc132 Aug 8, 2023
9dc279f
More python cleanup
niloc132 Aug 8, 2023
860d39f
Review feedback from Ryan and Devin
niloc132 Aug 8, 2023
81cb7a1
Tweak lambda, successfully send message or fail
niloc132 Aug 9, 2023
9518aa9
tidy docs in object service grpc impl
niloc132 Aug 9, 2023
aed6499
Make a method static
niloc132 Aug 9, 2023
72125bb
Merge branch 'main' into bidir-communication
niloc132 Aug 9, 2023
326d68b
Fix broken refactor
niloc132 Aug 9, 2023
f535e7c
'Fix' python imports
niloc132 Aug 9, 2023
3b57863
Review with Ryan, addit local testing of legacy widgets
niloc132 Aug 9, 2023
85a1541
Follow move of Callback to upper level
niloc132 Aug 9, 2023
99b02dc
Final py cleanup from Chip
niloc132 Aug 9, 2023
e68fa45
Update plugin vers
niloc132 Aug 9, 2023
5f907a3
Spotless, broken import
niloc132 Aug 9, 2023
fc229cb
Update setup.py to reflect new plugin vers
niloc132 Aug 9, 2023
12af1f2
Proposed proto doc updates
niloc132 Aug 9, 2023
3fa8f2f
Generated updates
niloc132 Aug 9, 2023
bac7992
Merge branch 'main' into bidir-communication
niloc132 Aug 10, 2023
a6f01bf
Bump for deephaven-plugin 0.4.0
devinrsmith Aug 9, 2023
97cedc4
Bump py vers for client tests, plugin vers
niloc132 Aug 10, 2023
addbe97
Prevent closing null server plugin stream
niloc132 Aug 10, 2023
9fd19d9
Update JS API to remove last calls to fetchObject
niloc132 Aug 10, 2023
9bab564
Update the rest of the pandas dataframe plugin to the new api
niloc132 Aug 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ workspaces/
devopsLocal/
pyclient/venv
pyclient/dist
pyclient/*.egg-info
.venv/
*.egg-info/

jenkins/failures

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;
import java.util.stream.Collectors;

/**
Expand Down Expand Up @@ -87,7 +86,7 @@ public PythonDeephavenSession(
scope = pythonEvaluator.getScope();
executionContext.getQueryLibrary().importClass(org.jpy.PyObject.class);
try (final SafeCloseable ignored = executionContext.open()) {
module = (PythonScriptSessionModule) PyModule.importModule("deephaven.server.script_session")
module = (PythonScriptSessionModule) PyModule.importModule("deephaven_internal.script_session")
.createProxy(CallableKind.FUNCTION, PythonScriptSessionModule.class);
}
scriptFinder = new ScriptFinder(DEFAULT_SCRIPT_PATH);
Expand Down Expand Up @@ -116,7 +115,7 @@ public PythonDeephavenSession(
evaluator = null;
this.scope = (PythonScope<PyObject>) scope;
try (final SafeCloseable ignored = executionContext.open()) {
module = (PythonScriptSessionModule) PyModule.importModule("deephaven.server.script_session")
module = (PythonScriptSessionModule) PyModule.importModule("deephaven_internal.script_session")
.createProxy(CallableKind.FUNCTION, PythonScriptSessionModule.class);
}
scriptFinder = null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import io.deephaven.auth.ServiceAuthWiring;
import io.deephaven.proto.backplane.grpc.FetchObjectRequest;
import io.deephaven.proto.backplane.grpc.ObjectServiceGrpc;
import io.deephaven.proto.backplane.grpc.StreamRequest;
import io.grpc.ServerServiceDefinition;

/**
Expand All @@ -30,6 +31,12 @@ default ServerServiceDefinition intercept(ObjectServiceGrpc.ObjectServiceImplBas

serviceBuilder.addMethod(ServiceAuthWiring.intercept(
service, "FetchObject", null, this::onMessageReceivedFetchObject));
serviceBuilder.addMethod(ServiceAuthWiring.intercept(
service, "MessageStream", this::onCallStartedMessageStream, this::onMessageReceivedMessageStream));
serviceBuilder.addMethod(ServiceAuthWiring.intercept(
service, "OpenMessageStream", null, this::onMessageReceivedOpenMessageStream));
serviceBuilder.addMethod(ServiceAuthWiring.intercept(
service, "NextMessageStream", null, this::onMessageReceivedNextMessageStream));

return serviceBuilder.build();
}
Expand All @@ -43,14 +50,77 @@ default ServerServiceDefinition intercept(ObjectServiceGrpc.ObjectServiceImplBas
*/
void onMessageReceivedFetchObject(AuthContext authContext, FetchObjectRequest request);

/**
* Authorize a request to open a client-streaming rpc MessageStream.
*
* @param authContext the authentication context of the request
* @throws io.grpc.StatusRuntimeException if the user is not authorized to invoke MessageStream
*/
void onCallStartedMessageStream(AuthContext authContext);

/**
* Authorize a request to MessageStream.
*
* @param authContext the authentication context of the request
* @param request the request to authorize
* @throws io.grpc.StatusRuntimeException if the user is not authorized to invoke MessageStream
*/
void onMessageReceivedMessageStream(AuthContext authContext, StreamRequest request);

/**
* Authorize a request to OpenMessageStream.
*
* @param authContext the authentication context of the request
* @param request the request to authorize
* @throws io.grpc.StatusRuntimeException if the user is not authorized to invoke OpenMessageStream
*/
void onMessageReceivedOpenMessageStream(AuthContext authContext, StreamRequest request);

/**
* Authorize a request to NextMessageStream.
*
* @param authContext the authentication context of the request
* @param request the request to authorize
* @throws io.grpc.StatusRuntimeException if the user is not authorized to invoke NextMessageStream
*/
void onMessageReceivedNextMessageStream(AuthContext authContext, StreamRequest request);

class AllowAll implements ObjectServiceAuthWiring {
public void onMessageReceivedFetchObject(AuthContext authContext, FetchObjectRequest request) {}

public void onCallStartedMessageStream(AuthContext authContext) {}

public void onMessageReceivedMessageStream(AuthContext authContext, StreamRequest request) {}

public void onMessageReceivedOpenMessageStream(AuthContext authContext,
StreamRequest request) {}

public void onMessageReceivedNextMessageStream(AuthContext authContext,
StreamRequest request) {}
}

class DenyAll implements ObjectServiceAuthWiring {
public void onMessageReceivedFetchObject(AuthContext authContext, FetchObjectRequest request) {
ServiceAuthWiring.operationNotAllowed();
}

public void onCallStartedMessageStream(AuthContext authContext) {
ServiceAuthWiring.operationNotAllowed();
}

public void onMessageReceivedMessageStream(AuthContext authContext, StreamRequest request) {
ServiceAuthWiring.operationNotAllowed();
}

public void onMessageReceivedOpenMessageStream(AuthContext authContext,
StreamRequest request) {
ServiceAuthWiring.operationNotAllowed();
}

public void onMessageReceivedNextMessageStream(AuthContext authContext,
StreamRequest request) {
ServiceAuthWiring.operationNotAllowed();
}
}

class TestUseOnly implements ObjectServiceAuthWiring {
Expand All @@ -61,5 +131,31 @@ public void onMessageReceivedFetchObject(AuthContext authContext, FetchObjectReq
delegate.onMessageReceivedFetchObject(authContext, request);
}
}

public void onCallStartedMessageStream(AuthContext authContext) {
if (delegate != null) {
delegate.onCallStartedMessageStream(authContext);
}
}

public void onMessageReceivedMessageStream(AuthContext authContext, StreamRequest request) {
if (delegate != null) {
delegate.onMessageReceivedMessageStream(authContext, request);
}
}

public void onMessageReceivedOpenMessageStream(AuthContext authContext,
StreamRequest request) {
if (delegate != null) {
delegate.onMessageReceivedOpenMessageStream(authContext, request);
}
}

public void onMessageReceivedNextMessageStream(AuthContext authContext,
StreamRequest request) {
if (delegate != null) {
delegate.onMessageReceivedNextMessageStream(authContext, request);
}
}
}
}
111 changes: 111 additions & 0 deletions cpp-client/deephaven/client/proto/deephaven/proto/object.grpc.pb.cc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading