termkit.txt

 TermKit
+++ -
Goal: next gen terminal / command application

Addresses following problems:
1) Monospace character grid with ansi colors is not rich enough to display modern files / media / visualizations / metadata. Cannot effectively handle large output, long/wide tables or direct interaction.
2) Piping raw/untyped binary or text streams between apps is bad for everyone:
   * Humans have to suffer syntax, cannot reflow/manipulate output in real-time
   * Computers have to suffer ambiguities
3) Synchronous input/output makes you wait. SSH keystroke latency is frustrating.
4) String-based command line requires arcane syntax, results in mistakes, repeated attempts at escaping, etc.
5) Unix commands are "useless by default", and when asked, will only tell you raw data, not useful facts. e.g. "rwxr-xr-x" instead of "You can't edit this file."

+++ -

Programs / commands
* Output processor for common cli tools
* Custom implementation of ls and friends, with support for mimicking classic shell behaviour with a 2.0 twist
* SQL shell

Cool input scenarios:
* As you type, the command is visually tokenized and highlighted. tokens can display autocomplete suggestions, icons and indicators inline.
* Instead of quoting and escaping, keys like " and > just trigger the creation of special tokens which are visually different and represent e.g. a quoted string, an argument, a regular expression. to type the special characters literally, just press them twice. the 'command' is just the concatenation of these tokens, interpreted the same way a shell interprets a command.
* Man pages are consulted inline with autocomplete options for arguments and (later) required arguments

Cool output scenarios:
* Listings of files, anywhere, show an icon with distinguished typography for filename vs meta. Quicklook integration on the icon.
* Can complete several tasks at once asynchronously, show real-time progress for all of them together in e.g. a task-list widget.
* Command output is interactive: has elements which can be focused, selected/unselected, opened, right clicked, dragged and dropped

Good desktop citizen:
* Dragging a file on the terminal window makes a folder icon appear on the side for dropping it on the CWD. Can also drag the file into the command line to reference it as an argument.
* Can drag files, snippets, JSON/CSV off the terminal
* Tear-off tabs/windows

+++ - Roadmap

[0.1] UI prototype - DONE
  [X] simulated in safari/webkit.app
  [X] functional input line with tokenization, cursor keys and backspace/delete
  [X] functional autocomplete on tokens
  [X] simulated commands
  [X] simulated output with collapsible sections
 
[0.2] App prototype - DONE
  [X] cocoa app
  [X] webkit in single window view
  [X] design back-end protocol
  [X] node JS back-end, running separately
  [X] connect socket
  [X] run session worker
  [X] passive command implementation
  [X] JS module template, integrated both runtimes.
  [X] wrap unix executable
  [X] interactive output
 
0.3: Command suite
  [X] Redesign message protocol
  [X] Viewstream integration
  [X] 5-pipe command execution
  [X] fix tokenfield
  [X] filesystem autocomplete
  [X] OS X icon loading
  [X] inline image display
  [X] json pretty printer
  [X] json grep
  [X] pipelined commands
  [X] unix command execution
  [X] code syntax highlighting
  [ ] http get/post data piping
  [ ] command decoration
  [ ] interactive execution
  [ ] inline man-pages tips
  [ ] version control
  [ ] termkit:// link handler for executing commands (intra-shell only)
    (idea: Allan Tan)
  [ ] interactive quicklook
    (tip: mikey_p on hackernews)
    qlmanage -p "$@" >& /dev/null
  [ ] wildcard handling
  [ ] regexp hinter

0.4: Network & Modularity
  [ ] SSH tunneling
  [ ] Stand-alone daemon
  [ ] network preview / local edit
  [ ] tabs
  [ ] split off command processor rules / autocomplete handlers into separable blocks
  [ ] server-side hook system
  [ ] add plug-in mechanism with drop-in functionality

0.5: Interactive & Unix Upgrades
  [ ] widget system
  [ ] graphing suite
  [ ] tear off tabs
  [ ] alt view for LS with usable permissions, dates, etc
  [ ] active 'top' monitor
  [ ] gnu parallel integration/clone

0.6: Theming
  [ ] server-side push css/js / integrated web
  [ ] themes / stylesheets

+++ Components

Node JS daemon = 'NodeKit'.
 + Fast enough for server work, concurrency/scaling included
 + New JS language features
 + Cross-platform on unix
 + Process / io / everything integration
 + Self-contained binary, can be included in .app
 - separate from UI / front-end, forced layer of indirection makes it unlike web programming
 - no mac-specific APIs or low-level C access

 => back-end platform, runs locally, can run remotely, or perhaps tunnel its interaction over SSH

WebKit/Cocoa front-end
 + Rich, stylable display + jQuery
 + New JS language features
 + Intimate OS X access, Obj-C, bridgeable with JS

The split:
Front-end = display, formatting, interaction. Always local. Runs in an (enhanced) webview/browser with a websocket to back-end.
Back-end:

1) Local NodeKit: Start node daemon on startup, connect using direct websocket ws://localhost:2222.
2) Remote NodeKit SSH: Daemon is running, use ssh to set up tunnel between local rand port # and remote 2222. connect local websocket to tunnel.
3) Remote NodeKit WSS: Daemon is running, use WSS to connect directly, must authenticate? don't want to replicate OpenSSH, but rudimentary auth could be useful.
4) Basic remote shell: No nodekit daemon. Only literal commands supported. Enough to execute e.g. "apt-get install termkit".

+++ Protocol considerations


The output of a termkit command is split into data and view. The data is the raw information that is piped from one process to the next. 

The view is a stream of dom-like objects and operators on them.

View and data are fundamentally different:
 * Data is a raw binary stream with meta-data annotation, from one process' stdout to another's stdin
 * View is a packetized stream of UI updates and callback events, going directly to the terminal.


+++ Command architecture

The webkit front end is intended to be a view first and foremost. It is an active view that maintains its own contents based on messages to and from the back-end.

problem: if front-end is agnostic, then how to make commands smarter?

 > shell-OS interface is server-side
 > server-side only executes processes/commands, routes streams and provides output.
 > separeate data in/out from ui in/out. datastream vs viewstream
 
 data in/out:
 content-type: unix/pipe
               application/jsonstream
 ...
 
 
 [ Front-End ]   ----->     [  Back-end ]
 [  WebKit   ]   websocket  [  Node.js  ]
                                 |   
                                 |
                                 v *
                            [   Shell   ]
                            [ Worker.js ]
 
 
 Worker sets up process chain:
 
 
  view in - callbacks / command stream
  view out - view stream
  stdin = data in (mime typed)
  stdout = data out (mime typed)
  
                | callbacks         ^ view stream: view updates
  stdin         v                   |
  ---->     [   "command [args...]"   ]  ---->
                                            stdout

  View streams are multiplexed and sent to the front-end.
  Front-end callbacks are routed back to the right process.
  
  Each process has its own view, referenced by ID number.
  Processes cannot break out of their view.
 
 
+++ Data vs UI glue

e.g.
 get http://.../rss.xml
 content-type: application/xml
 
 -> process streams out xml data

   get | grep
   Data Stream:  .txt|html|... > .txt|html|...
   View Stream: file metadata, download progress, grep stats (# hits)
   Output Formatting:  turn application/xml into dynamic XML tree.
  
   get | ungzip | untar
   Data Stream:  .gz  > .tar  >   nothing
   View Stream:  file metadata, download progress, unzip progress (# files)
   Output Formatting: no data output.
    
   The output formatter takes a mime-typed blob and turns into a display.
   - Plain text or HTML
   - Images
   - Binary data (hex or escaped)
   - Url/file reference with icon
   ...
   
   Existing tools are wrapped to provide richer output, either by parsing
   old school output or by interacting directly with the raw library.
    
+++ stream structure

><{
  termkit: '1'
}

>{
  query: 1
  method: session.open
  args:
}
<{
  answer: 1
  success: true
  args: { 
    session: 1
  }
}
>{
  query: 2
  method: shell.environment
  session: 1
}
<{
  answer: 2,
  success: true
  args: { ... }
}
>{
  query: 3
  method: shell.run
  session: 1
  args: {
    command: ...
    id: 0
  }
}
<{
  method: view.allocate,
  session: 1
  args: {
    id: 0
    views: [0, 1]
  }
}
<{
  method: view.add,
  session: 1  
  args: {
    view: 0
    target: null
    objects: [ ... ]
  }
}
<{
  method: view.add,
  session: 1  
  args: {
    view: 0
    target: [ 0 ]
    objects: [ ... ]
  }
}
>{
  method: view.callback,
  session: 1  
  args: {
    view: 0,
    message: { .. }
  }
}
<{
  method: view.update
  args: { ... }
}
<{
  answer: 3,
  success: true,
  args: {
    code: 0,
  }
}


viewstream:

 > shell-specific interaction rules are server-side.
 > rich widget lib for display, extensible
 > widgets are streamed to client like termkit-ML. objects are smartly typed and have callback commands defined for them. callbacks can be stateful or stateless. though stateful is only intended to be used for interactive commands.

 tableview / listcontainer -> generic, scales from simple list to tabled headers w/ simple syntax
 object references for files and other things. are multi-typed and annotated on server-side.

+++ --
references:

textmate html output features
http://blog.macromates.com/2005/html-output-for-commands/

bcat: browser cat tool

terminator: java-based terminal++

protocol:
 fastcgi, json-rpc

command output:
* rich output, consisting of an element tree
* simple container types (list, table) and various viewers to represent common objects and data types

current widget model:
* use a viewcontroller for each widget
* viewcontroller own its own markup
* simple object instantiation, widgets can create child widgets

more ideas:
* autocorrection of mistyped commands
* unified progress indicator during command
* autocompleted/predicted history access
* guard if command is dangerous / doesn't make sense

+++  To read

http://ripple.fortytwo.net/

http://directwebremoting.org/blog/joe/2009/05/27/command_lines.html

http://kkovacs.eu/cool-but-obscure-unix-tools