Kale: Human-in-the-loop Interactive Distributed Computing with Jupyter

This is an ongoing research effort at Lawrence Berkeley National Laboratory, aimed at extending the Jupyter ecosystem with useful add-ons to enable a smooth interactive experience for scientific researchers running on clusters, HPC systems etc.

Features

* Control long running remote tasks

• Remotely register any python function that can be pickled as a task
• Start tasks
• Stop tasks
• Change parameters of a Stopped task
• Restart a Stopped task or Start a different task

* Monitor resource usage

-- Host/Node level resource usage

• CPU
• Memory
• Disk
• Network

-- Task level resource usage

• CPU
• Memory
• Disk
• Network

Components

* Kale Services

• Manager

  • registration/nameserver for workers
  • persistent between tasks

• Worker

  • Wraps task
  • Task registration
  • Task control (start, stop, pause, resume)
  • Resource usage collection

* Kale Widgets

• Resource Board
  • Host/Node level resource plots
  • Task level resource plots, tables, etc