feat: Add console scripts to start Deephaven from the command line

[mofojed]

• Added console scripts to the embedded server to add a deephaven console script that can launch deephaven
• Tested by building and installing the deephaven_server whl
• deephaven is the root script, with server being the only positional "command" argument and ran by default
  • Figured I'd add command as a positional argument, in case we want to add other commands (e.g. run to run a specific file)
• Parse arguments for host, port, jvm-args, and extra-classpath
  • If no key provided, a random key is generated
• After starting, waits for the user to press Ctrl+C to exit

Examples:
deephaven server: Starts up Deephaven server on port 8080 with a random key and opens the web UI.
deephaven server --port 9999: Starts up Deephaven on port 9999.
deephaven server --jvm-args="-Ddeephaven.dataDir=/tmp/deephaven": Starts up Deephaven with /tmp/deephaven as the data directory.
deephaven server --port 9999 --jvm-args="-Ddeephaven.dataDir=/tmp/deephaven-data -Ddeephaven.console.disable=true": Starts up Deephaven on port 9999, using /tmp/deephaven as the data directory and the console disabled.
deephaven --help: Prints help about what commands are available
deephaven server --help: Prints help about the server comand

[devinrsmith]

Something we've talked about before is creating the config file if it doesn't exist and writing the PSK into it; this gives it some persistence. I think that's how jupyter does it?

[mofojed]

Jupyter doesn't seem to persist the token. If I run jupyter lab twice in a row, I get new tokens. I think that makes sense, it shouldn't persist unless you specify it.

That being said I'm not against it and Jupyter certainly doesn't have to be precedence.

[devinrsmith]

Okay; do you know if a fresh jupyter lab writes down any configuration files?

[mofojed]

Yes, it writes a couple of files it looks like:

• /home/bender/.jupyter/lab/workspaces/default-37a8.jupyterlab-workspace: Config for the users workspace, such as layout of the panels
• /home/bender/.local/share/jupyter/runtime/jupyter_cookie_secret: I'm not 100% sure what this cookie is for; it's not the token used for auth (that regenerates each time) and it just regenerates this if you delete the file. Searching for details on what that cookie is used for isn't turning up much (I checked the jupyter and jupyterlab orgs on GitHub). There's some mention of it in forums indicating it'll fail if permissions don't allow reading the file, but it will regenerate if the file doesn't exist.

So again, we can write a config, I don't mind writing the token to a file for persistence, but that doesn't seem to be what Jupyter is storing (and I'm actually not sure what the secret cookie they're storing is anyway. Some sort of base64 string, but it doesn't seem to be stored in the browser as a cookie, doesn't seem to have a reference in any of the accessible Jupyter code that I've searched ... I don't know what it is?

[devinrsmith]

Note from community user:

we noticed the deephaven_server python package doesn't allow for specifying extra class pass paths

We might want to consider inheriting the EXTRA_CLASSPATH environment variable (and or START_OPTS / JAVA_OPTS)?

[devinrsmith]

I've got some suggestions that expose more of the configuration properties to EmbeddedServer, PR mofojed#8.

The one other thing that I think will be important is moving the console entry point out of __init__.py. I think users who import deephaven_server should not need to source the click / cli logic.

[chipkent]

This is looking great.

1. I would add an optional argument to server that takes a path to a script file to run before getting into the wait loop. Basically a way to populate DH with a quick command.
2. I would add a new run command to run a script. Basically the same as server without the auth, browser, and wait. Batch mode.

[mofojed]

Both of those options sound great. Mind if it's a future PR?:

1. That's kind of like application mode, would we want to specify the "app" to start with? I believe the only way to do that now is by setting the app.d directory, so we'd want to think how we have this play out/work with application mode.
2. Similar to question 1, do we still have the application mode scripts run or just the file you specified? What config do we want to override in that case?

In any case with the console scripts added, we can add new commands/options fairly easily.

[chipkent]

Let's do it soon.

[niloc132]

Looks like this prevents a user from specifying mTLS/etc as their preferred option, without forcing that they also have PSK (or anon) enabled as well... and it will override their AuthHandlers value and wipe out their own config.

Unsure if we want to get into the business of parsing AuthHandlers or other config vars here... if we want to go after this it might be time to tighten up our config language/control, or just be prepared for breaking changes.

Or put dh_args back or something, and require that it be used instead of system props for dh configs? and have some kind of schema here for "this is actually a ,-delimited list of strings"?

I kind of hate all of the choices... I do have a draft proposal for "we should use TOML", but I'm not sure now's the time to pick that up in earnest?

[niloc132]

Additionally, these shouldn't be mutually exclusive - one day we'll have authorization hooks that are easy to use, and we'll want to allow both (anon might be simple, psk might be admin).

[devinrsmith]

I'm hoping we can discourage users from setting --anonymous or --psk wrt a deephaven server start command (in my helper PR I originally removed them). When they are not provided, the server will retain it's normal defaults or source from the users configuration file. I also suggested that deephaven set-config ... might be a reasonable command that allows us to explicitly present a way to do "proper" configuration from the CLI (ie, writing down into a configuration file w/ proper safeguards).

There was a similar concern I had for jsPlugins configuration, essentially relying on a prefix call:

class PropertyFile {
...
    /**
     * Collect all of the properties in this property file that begin with a given prefix.
     *
     * @return a new Properties object containing the selected properties, with the prefix removed.
     */
    public Properties getProperties(String prefix) {

to allow independent configuration. ie

deephaven.jsPlugins.foo=/path/to/plugin-1
deephaven.jsPlugins.bar=/path/to/plugin-2

as opposed to needing to do

deephaven.jsPlugins=/path/to/plugin-1,/path/to/plugin-2

[devinrsmith]

I don't think we want or need deephaven server command line options to present the full universe of configuration possibilities to users; in my mind, these options may be better thought of as --only-psk=<key>, --only-anonymous, etc. I am leaning that maybe we should remove them from deephaven server altogether, I don't think it would be a big loss. (I could make similar argument for --host and --port, but I think that's a slightly harder sell.)

[mofojed]

I'd be alright with getting rid of the anonymous/psk options now that there's the targetUrls option, as those options conflicted with existing config. Leaving jvm_args seems alright, it adds to the args in your existing config. host/port seem like they'd be nice and would just override any config you have which makes sense IMO.

[devinrsmith]

That's the route I originally took when creating this feature; but it became clear that I was trying to service this one specific use case as opposed to being more library-like and presenting the components themselves.

I could see an argument for moving this logic into the python side Server; I don't necessarily think the java side EmbeddedServer is the correct place. Once you elevate it to Server though, it becomes a public API and something we need to treat as a library feature more generally.

[mofojed]

@devinrsmith Also this gives the URL to the IDE - just wondering, if we want URLs to the iframe endpoints as well, that could be useful for deephaven-ipywidgets (and others).

[devinrsmith]

This API doesn't assume much about target_url; in the case of anonymous, it just echos it back, and in the case of PSK, just appends ?psk=<key>. It doesn't know anything about IDE nor iframe, etc.

By default, target_url = "http://localhost:10000". As part of CLI, you could append /ide/ to be a bit more explicit, or invoke the logic multiple times to get iframe or other links.

It would be nice to have a more proper API around web apps installed as opposed to doing:

for (String appRoot : List.of("/ide/", "/iframe/table/", "/iframe/chart/", "/iframe/widget/")) {

Until then though, I think you'll need to do something similar in CLI if you want to.

[niloc132]

With that said, this cli is a public API - once we put it out there, we can't break it (without being the kind of jerk that would break any other API).

[jmao-denver]

LGTM

[deephaven-internal]

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#185