Twitter Ambrose is a platform for visualization and real-time monitoring of MapReduce data workflows. It presents a global view of all the map-reduce jobs derived from your workflow after planning and optimization. As jobs are submitted for execution on your Hadoop cluster, Ambrose updates its visualization to reflect the latest job status, polled from your process.
Ambrose provides the following in a web UI:
- A chord diagram to visualize job dependencies and current state
- A table view of all the associated jobs, along with their current state
- A highlight view of the currently running jobs
- An overall script progress bar
Ambrose is built using the following front-end technologies:
Ambrose is designed to support any Hadoop workflow runtime, but current support is limited to Apache Pig.
Follow @Ambrose on Twitter to stay in touch!
- Pig - See pig/README.md
- Cascading - future work
- Scalding - future work
- Cascalog - future work
- Hive - future work
Below is a screenshot of the Ambrose UI. Each arc segment on the circle represents a map-reduce job. Dependencies between jobs are represented by chords which connect job arc segments. Grey jobs have not yet run, bright green jobs are running and light green jobs are completed. When the mouse hovers over a job, its arc and input dependencies are highlighted blue. Clicking on the job will select it, updating the contents of the table to the right of the diagram with information about the selected job.
Note that Each job arc is bisected; Chords on one half of the arc connect to predecessor jobs while chords on the other half connect to successor jobs. For example, in the diagram below jobs 1 and 3 have no predecessors while jobs 16, 20, 21, and 22 have no successors (their outputs represent the final result of this workflow).
Note that the chord diagram shown is our first pass at visualizing the workflow, and there's room for improvement. We'd like to support other visualizations as well, like a graph of the workflow DAG. If you develop an improved visualization, be sure to send us a pull request!
To get started with Ambrose, first clone the Ambrose Github repository:
git clone https://github.com/twitter/ambrose.git
cd ambrose
Next, you can try running the Ambrose demo on your local machine. The demo starts a local web server which serves the front-end client resources and sample data. Start the demo with the following command and then browse to http://localhost:8080/workflow.html?localdata=small:
./bin/ambrose-demo
To run Ambrose with an actual Pig script, you'll need to build the Ambrose Pig distribution and untar it:
mvn package
tar zxvf pig/target/ambrose-pig-$VERSION-bin.tar.gz
You can then run the following commands to execute path/to/my/script.pig with an Ambrose app server
embedded within the Pig client:
cd ambrose-pig-$VERSION
./bin/pig-ambrose -f path/to/my/script.pig
Note that this command delegates to the pig script present in your local
installation of Pig, so make sure $PIG_HOME/bin is in your path. Now, browse
to
http://localhost:8080/web/workflow.html
to see the progress of your script using the Ambrose UI. To override the default
port, export AMBROSE_PORT before invoking pig-ambrose:
export AMBROSE_PORT=4567
An initial release will be pushed to Maven shortly.
Bug fixes, features, and documentation improvements are welcome! Please fork the project and send us a pull request on Github. You can submit issues on Github as well.
Here are some high-level goals we'd love to see contributions for:
- Improve the front-end client
- Add other visualization options, like a DAG view
- Create a new back-end for a different runtime environment
- Create a standalone Ambrose server that's not embedded in the workflow client
For transparency and insight into our release cycle, releases will be numbered with the follow format:
<major>.<minor>.<patch>
And constructed with the following guidelines:
- Breaking backwards compatibility bumps the major
- New additions without breaking backwards compatibility bumps the minor
- Bug fixes and misc changes bump the patch
For more information on semantic versioning, please visit http://semver.org/.
- Bill Graham (@billgraham)
- Andy Schlaikjer (@sagemintblue)
- Nicolas Belmonte (@philogb)
Copyright 2012 Twitter, Inc.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0