Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EPIC: Allow deephaven server to be started from python #2221

Open
4 of 8 tasks
niloc132 opened this issue Apr 7, 2022 · 1 comment · Fixed by #2196
Open
4 of 8 tasks

EPIC: Allow deephaven server to be started from python #2221

niloc132 opened this issue Apr 7, 2022 · 1 comment · Fixed by #2196
Assignees
@niloc132
Labels
epic Feature Epic (User Story) python python-server-side

Comments

@niloc132
Copy link
Member

niloc132 commented Apr 7, 2022 •
edited
Loading

  • Make the Jetty-based server implementation easier to start with custom options, bindings
  • Move bootstrap.py to test code, to be clear that this shouldn't be used to start the jvm under normal circumstances. It also probably should be pared down to be simpler to read and understand, as we practically never need that degree of customizability.
  • Move existing top-level python projects into py/
  • Create a new java application project with basic wiring to launch jetty for static content and gRPC services, designed to be easily called from python. This will involve cleaning up some currently rough edges in the dagger bindings, so that it is easier to build custom applications like this
  • Create a new wheel project called deephaven_server which is able to start the jvm and the java components of Deephaven, including the engine and the gRPC service. This will have very little python in it, but will mostly be a container to hold the above java application, and will mostly offer the ability to start different parts of the server, as well as manipulate the script session or application
  • Ensure that deephaven has tools for working with the UGP
  • Update all wheel projects to publish working binaries to pypi, including jpy-deephaven (or jpy), deephaven, and deephaven_server.
  • Provide an optional custom UI experience where no console appears in the UI, but newly bound objects appear when the UI hears about them
@niloc132 niloc132 added python epic Feature Epic (User Story) python-server-side labels Apr 7, 2022
@niloc132 niloc132 self-assigned this Apr 7, 2022
@niloc132 niloc132 linked a pull request Apr 8, 2022 that will close this issue
Simplify and relocate deephaven py server library #2196
Merged
niloc132 added a commit that referenced this issue Apr 11, 2022
@niloc132
Simplify and relocate deephaven py server library (#2196)
7380491 
Removes jpy initialization for the new python apis, to be re-introduced
in a new python module that can start the gRPC server as well as the
engine.

Tests and doc generation have their own version of jpyinit, specific to
their circumstances, to allow some flexibility where required, but
generally reducing the overall supported complexity.

Partial #2221
@niloc132 niloc132 mentioned this issue Apr 11, 2022
Merged
niloc132 added a commit that referenced this issue Apr 13, 2022
@niloc132
Relocate python sources into py/ (#2249)
41a27c9 
This also declares wheels as artifacts, making it simpler to declare
and install dependencies.

Gradle build/ and python dist/ are now excluded from the python build -
if we find this is onerous, buildPyWheel could take a CopySpec
argument. With that said, this shouldn't be necessary soon, since
deephaven-jpy will hopefully soon be replaced by upstream jpy, and
deephaven-legacy will be phased out, leaving only projects that use the
new python-wheel plugin.

Legacy deephaven-wheel is renamed now in gradle as well, but not
relocated.

Partial #2221
This was referenced Apr 14, 2022
Merged
Merged
niloc132 added a commit to niloc132/deephaven-core that referenced this issue Apr 14, 2022
@niloc132
Only include server app resources in the app
14db5f7 
This allows the library to be used from a project that wants to redefine
those resources, without requiring that the downstream project shade the
server library or have a specific classpath ordering.

Partial deephaven#2221
This was referenced Apr 14, 2022
Merged
Merged
niloc132 added a commit that referenced this issue Apr 14, 2022
@niloc132
Permit null log streams, as the code is null safe (#2268)
4f6fac7 
This allows a server implementation to not provide these print streams
if it doesn't want to, by explicitly binding to null.

This is clearly safe, since providesStreamToReal explicitly passes
nulls along rather than consume the offered params, if certain system
properties are set.

Partial #2221
niloc132 added a commit that referenced this issue Apr 14, 2022
@niloc132
Only include server app resources in the app (#2269)
7c2fb8a 
This allows the library to be used from a project that wants to redefine
those resources, without requiring that the downstream project shade the
server library or have a specific classpath ordering.

Partial #2221
niloc132 added a commit that referenced this issue Apr 25, 2022
@niloc132
Defensively include Dockerfile (#2270)
5ac0831 
The gradle Sync task can result in unbuildable images without ensuring
that the nested CopySpecs explicitly include their own Dockerfile.

Partial #2221
@niloc132 niloc132 mentioned this issue Apr 25, 2022
Merged
@niloc132 niloc132 reopened this Apr 26, 2022
@niloc132
Copy link
Member Author

Incorrectly closed, this issue isn't complete yet.

niloc132 added a commit that referenced this issue Apr 26, 2022
@niloc132
Refactor ScriptSession injection (#2337)
5425683 
The current system of requiring that the consumer request the entire
map and already know the session type (or read it from the singleton
not-@singleton configuration instance) is pretty onerous for downstream
apps, and somewhat moot when we know that there is only one possible
scriptsession at a time anyway.

Partial #2221
niloc132 added a commit that referenced this issue Apr 27, 2022
@niloc132
JPy should release the GIL while in Java, with a special method to de…
ed73208 
…liberately acquire it (#2314)

There is one exceptional case to deal with, allowing Java to hold and
keep the GIL when it wants it, such as for cleanup work. Now Java can
grab the GIL if it needs it, without a full trip into python and back
again.

Partial #2221
@niloc132 niloc132 mentioned this issue Apr 27, 2022
Merged
@niloc132 niloc132 mentioned this issue May 31, 2022
Open
4 tasks
niloc132 added a commit that referenced this issue Jun 3, 2022
@niloc132
Python library for embedded server (#2353)
3398355 
Introduces a Python library that can start the Deephaven server from
python code. This consists of a wheel that holds not only some python
glue to start the server, but all of the jars required to be on the
deephaven classpath. Presently this depends on deephaven's own jpy
fork, but soon this limitation will be dropped and the upstream jpy
project will work.

Partial #2221
SuperTails pushed a commit to SuperTails/deephaven-core that referenced this issue Jun 29, 2022
@niloc132 @SuperTails
Python library for embedded server (deephaven#2353)
281b193 
Introduces a Python library that can start the Deephaven server from
python code. This consists of a wheel that holds not only some python
glue to start the server, but all of the jars required to be on the
deephaven classpath. Presently this depends on deephaven's own jpy
fork, but soon this limitation will be dropped and the upstream jpy
project will work.

Partial deephaven#2221
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@niloc132 niloc132

Labels
epic Feature Epic (User Story) python python-server-side
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Simplify and relocate deephaven py server library niloc132/deephaven-core
1 participant
@niloc132