Skip to content

Commit

Permalink
Merge branch 'master' of github.com:twosigma/beakerx into lukasz/7204…
Browse files Browse the repository at this point in the history
…_do_not_relly_on_bootstrap_css
  • Loading branch information
piorek committed May 17, 2018
2 parents 19fcb50 + 719ccf8 commit 5e45c70
Show file tree
Hide file tree
Showing 85 changed files with 3,368 additions and 700 deletions.
33 changes: 25 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,27 +121,44 @@ The code is organized into subdirectories as follows:
* [beakerx](beakerx) The Python packages. The main beakerx package has:

* a customized KernelSpec to allow BeakerX to configure the JVMs
started to run the kernels,
that run the kernels,

* a server extension for the javadoc, settings, and version
endpoints,

* the beakerx command line program, which has the bkr2ipynb
converter as well as install and uninstall functions,
converter, the py4j server, utilities, install, and uninstall
functions.

* the Python API for the runtime (tables, plots, easyform),
including automatically installing a displayer for pandas tables,
and autotranslation;

* the webpack (compiled JavaScript, TypeScript, CSS, fonts, images);
and
* the nbextension webpack (compiled JavaScript, TypeScript, CSS,
fonts, images); and

* the compiled Java JARs.
* the compiled Java JARs of each of the kernels, and a directory of
shared JARs.

There is a separate python package (beakerx_magics) for the
`%%groovy` magic so it can always be loaded *without* loading the
regular beakerx package (which would turn on display of pandas
tables with our table widget).
`%%groovy` magic so it can be loaded *without* loading the regular
beakerx package (which would turn on display of pandas tables with
our table widget).

BeakerX [configures ipython](beakerx/beakerx/install.py#L140) to
automatically load the magics in the beakerx_magics package,
`%load_ext` is not required.

The [groovy magic](beakerx/beakerx_magics/kernel_magic.py) uses the standard Jupyter API,
jupyter_client.manager.KernelManager to start the kernel.
It then proxies Comm into the inner kernel.

This package also has the py4j support for the `%%python` magic. In
order for the JVM kernels to be able to start Jupyter kernels they
need to be able to call into Python. There is a `beakerx
py4j_server` subcommand for this purpose (for internal use, not for
the user). It calls into the groovy magic with its Comm proxy,
implemented in Python.

* [doc](doc) Documentation consisting of executable tutorial
notebooks. [StartHere.ipynb](StartHere.ipynb) at the top level
Expand Down
4 changes: 2 additions & 2 deletions StartHere.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"[Groovy](doc/groovy/Groovy.ipynb), [Java](doc/java/Java.ipynb), [Scala](doc/scala/Scala.ipynb), [Clojure](doc/clojure/Clojure.ipynb), [SQL](doc/sql/Sql.ipynb), [Kotlin](doc/kotlin/Kotlin.ipynb).\n",
"\n",
"### Magics\n",
"[Timing](doc/groovy/TimingMagicCommands.ipynb), [Classpath and Imports](doc/groovy/ClasspathMagicCommands.ipynb), [Defining New Magics](doc/groovy/LoadMagicCommand.ipynb).\n",
"[Timing](doc/groovy/TimingMagicCommands.ipynb), [Classpath and Imports](doc/groovy/ClasspathMagicCommands.ipynb), [Polyglot Magics](doc/groovy/PolyglotMagic.ipynb), [Defining New Magics](doc/groovy/LoadMagicCommand.ipynb).\n",
"\n",
"### Options Panels\n",
"[Properties, Heap Size, and other JVM Options](doc/groovy/JavaArgs.ipynb), [UI Options](doc/groovy/UIOptions.ipynb).\n",
Expand Down Expand Up @@ -84,7 +84,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
"version": "3.6.5"
}
},
"nbformat": 4,
Expand Down
20 changes: 20 additions & 0 deletions beakerx/beakerx/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
from notebook import notebookapp as app
from .install import install, uninstall
from .bkr2ipynb import main
from beakerx_magics import Py4JServer


def install_subparser(subparser):
install_parser = subparser.add_parser('install', help='installs BeakerX extensions')
Expand All @@ -29,6 +31,7 @@ def install_subparser(subparser):
action='store_true')
return subparser


def uninstall_subparser(subparser):
uninstall_parser = subparser.add_parser('uninstall', help='uninstalls BeakerX extensions')
uninstall_parser.set_defaults(func=uninstall)
Expand All @@ -40,16 +43,31 @@ def uninstall_subparser(subparser):
action='store_true')
return subparser


def bkr2ipynb_subparser(subparser):
bkr2ipynb_parser = subparser.add_parser('bkr2ipynb', help='converts Beaker notebooks to ipynb format')
bkr2ipynb_parser.set_defaults(func=main)
bkr2ipynb_parser.add_argument('notebooks', nargs='+',
help="Beaker notebooks to be converted. Enter *.bkr in case you want to convert all notebooks at once.")
return subparser


def py4j_server_subparser(subparser):
py4j_server_parser = subparser.add_parser('py4j_server')
py4j_server_parser.set_defaults(func=start_py4j_server)
py4j_server_parser.add_argument("--port")
py4j_server_parser.add_argument("--pyport")
py4j_server_parser.add_argument("--kernel")


def start_py4j_server(args):
Py4JServer(args.port, args.pyport, args.kernel)


def run_jupyter(jupyter_commands):
app.launch_new_instance(jupyter_commands)


def init_parser():

parser = argparse.ArgumentParser()
Expand All @@ -60,8 +78,10 @@ def init_parser():
install_subparser(subparsers)
uninstall_subparser(subparsers)
bkr2ipynb_subparser(subparsers)
py4j_server_subparser(subparsers)
return parser


def parse():
parser = init_parser()
args, jupyter_commands = parser.parse_known_args()
Expand Down
1 change: 1 addition & 0 deletions beakerx/beakerx_magics/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,4 @@
from .kotlin_magic import *
from .scala_magic import *
from .sql_magic import *
from .jvm_kernel_magic import *
111 changes: 111 additions & 0 deletions beakerx/beakerx_magics/jvm_kernel_magic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Copyright 2017 TWO SIGMA OPEN SOURCE, LLC #
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from py4j.clientserver import ClientServer, JavaParameters, PythonParameters
from queue import Empty
from jupyter_client.manager import KernelManager
from jupyter_client.kernelspec import NoSuchKernel
import json
import sys


class JVMKernelMagic:

def __init__(self, kernel_name):
self.km = None
self.kc = None
self.comms = []
self.kernel_name = kernel_name
self.start()

def start(self):
self.km = KernelManager()
self.km.kernel_name = self.kernel_name
self.km.start_kernel()
self.kc = self.km.client()
self.kc.start_channels()
self.kc.wait_for_ready()

def stop_kernel(self):
self.kc.stop_channels()
self.km.shutdown_kernel(now=True)

def run_cell(self, code):
if not self.km:
self.start()
self.kc.execute(code, allow_stdin=True)

def get_shell_msg(self):
return self.kc.get_shell_msg()

def get_iopub_msg(self):
try:
msg = self.kc.get_iopub_msg(timeout=1)
return msg
except Empty:
return None

def pass_msg(self, msg_raw):
msg_json = json.loads(msg_raw)
content = msg_json['content']
msg_type = msg_json['header']['msg_type']
msg = self.kc.session.msg(msg_type, content)
self.kc.shell_channel.send(msg)
return None


class PythonEntryPoint(object):

def __init__(self, kernel_name):
self.pm = JVMKernelMagic(kernel_name)

def evaluate(self, code):
print('code for evaluate {}'.format(code))
self.pm.run_cell(code)
return None

def getShellMsg(self):
shellMsg = self.pm.get_shell_msg()
return json.dumps(shellMsg, default=str)

def getIopubMsg(self):
iopubMsg = self.pm.get_iopub_msg()
return json.dumps(iopubMsg, default=str)

def shutdownKernel(self):
self.pm.stop_kernel()
return None

def sendMessage(self, msg_raw):
self.pm.pass_msg(msg_raw)
return None

class Java:
implements = ["com.twosigma.beakerx.kernel.PythonEntryPoint"]


class Py4JServer:
def __init__(self, port, pyport, kernel_name):
try:
pep = PythonEntryPoint(kernel_name)
except NoSuchKernel:
sys.exit(2)
ClientServer(
java_parameters=JavaParameters(port=int(port)),
python_parameters=PythonParameters(port=int(pyport)),
python_server_entry_point=pep)
print('Py4j server is running')


if __name__ == '__main__':
Py4JServer(sys.argv[1], sys.argv[2], sys.argv[3])
3 changes: 2 additions & 1 deletion beakerx/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@
install_requires = [
'notebook >=4.4.0',
'ipywidgets >=7.0.0',
'pandas'
'pandas',
'py4j'

],
python_requires='>=3',
Expand Down
2 changes: 1 addition & 1 deletion doc/groovy/PlotFeatures.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@
"plot << new Points(y: y1)\n",
"plot << new Points(y: y2, size: 10, color: Color.black)\n",
"plot << new Points(y: y3, size: 15, color: Color.orange, outlineColor: Color.black)\n",
"//plot << new Points(y: y4, size: 20, outlineColor: Color.red, fill: false)"
"plot << new Points(y: y4, size: 20, outlineColor: Color.red, fill: false)"
]
},
{
Expand Down
123 changes: 123 additions & 0 deletions doc/groovy/PolyglotMagic.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Polyglot Magics\n",
"\n",
"The BeakerX kernels have magics to run other kernels, including `%%python`. There are magics for each BeakerX langage: `%%groovy`, `%%java`, `%%scala`, `%%sql`, `%%clojure`, and `%%kotlin`. There is also a `%%kernel` magic to access any kernel available to Jupyter. Widgets work through these magics.\n",
"\n",
"## Python Magic\n",
"\n",
"For example, even though this is a Groovy notebook, you can run Python code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%python\n",
"print(\"regular\")\n",
"{\"output\": \"works\"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Widgets Work"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%python\n",
"import ipywidgets as widgets\n",
"w = widgets.IntSlider()\n",
"w"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%python\n",
"w.value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scala Table"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%scala\n",
"new TableDisplay(Seq(Map(\"a\" -> 1, \"b\" -> 2, \"c\" -> 3),\n",
" Map(\"a\" -> 4, \"b\" -> 5, \"c\" -> 6),\n",
" Map(\"a\" -> 7, \"b\" -> 8, \"c\" -> 8)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Kernel Magic\n",
"\n",
"Access any kernel available to Jupyter by name:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%kernel ir\n",
"x <- rnorm(1000)\n",
"plot(x)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%kernel julia-0.6\n",
"(1 + 1im) * (1 - 1im)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Groovy",
"language": "groovy",
"name": "groovy"
},
"language_info": {
"codemirror_mode": "groovy",
"file_extension": ".groovy",
"mimetype": "",
"name": "Groovy",
"nbconverter_exporter": "",
"version": "2.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
2 changes: 1 addition & 1 deletion doc/python/Heatmap.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.4"
"version": "3.6.4"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit 5e45c70

Please sign in to comment.