Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds links to Hamilton UI #868

Merged
merged 1 commit into from
Apr 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
# Hamilton

The general purpose micro-orchestration framework for building [dataflows](https://en.wikipedia.org/wiki/Dataflow) from python functions. Express data, ML, LLM pipelines/workflows, and web requests in a simple declarative manner.
Hamilton also comes with a [UI](https://hamilton.dagworks.io/en/latest/concepts/ui) to visualize, catalog, and monitor your dataflows.

Hamilton is a novel paradigm for specifying a flow of delayed execution in python. It works on python objects of any type and dataflows of any complexity. Core to the design of Hamilton is a clear mapping of function name to artifact, allowing you to quickly grok the relationship between the code you write and the data you produce.

Expand All @@ -32,12 +33,12 @@ Note: Hamilton describes DAGs. If you're looking for something to handle loops o
✅ Model a dataflow -- If you can model your problem as a DAG in python, Hamilton is the cleanest way to build it.<br/>
✅ Unmaintainable spaghetti code -- Hamilton dataflows are unit testable, self-documenting, and provide lineage.<br/>
✅ Long iteration/experimentation cycles -- Hamilton provides a clear, quick, and methodical path to debugging/modifying/extending your code.<br/>
✅ Reusing code across contexts -- Hamilton encourages code that is independent of infrastructure and can run regardless of execution setting.
✅ Reusing code across contexts -- Hamilton encourages code that is independent of infrastructure and can run regardless of execution setting.<br/>
✅ Collaborating on dataflows & tracking execution + artifacts -- Hamilton comes with an optional [UI](https://hamilton.dagworks.io/en/latest/concepts/ui) to visualize, catalog, and monitor your dataflows, which helps team operate smoothly.

## Problems Hamilton Does not Solve
❌ Provisioning infrastructure -- you want a macro-orchestration system (see airflow, kubeflow, sagemaker, etc...).<br/>
❌ Doing your ML for you -- we organize your code, BYOL (bring your own libraries).<br/>
❌ Tracking execution + associated artifacts -- Hamilton is lightweight, but if this is important to you see the [DAGWorks product](https://www.dagworks.io).

See the table below for more specifics/how it compares to other common tooling.

Expand Down
7 changes: 4 additions & 3 deletions docs/concepts/ui.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ Hamilton UI
Hamilton comes with a fully open-source UI that can be run both for local deployment and on a remote server.
The UI consists of the following features:

1. Telemetry for hamilton executions -- both on the history of executions and the data itself
2. A feature/artifact catalog for browsing/connecting executions of nodes -> results
3. A DAG visualizer for exploring the DAG, looking at the code, and determining lineage
1. Telemetry for hamilton executions -- both on the history of executions and the data itself.
2. A feature/artifact catalog for browsing/connecting executions of nodes -> results.
3. A dataflow (i.e. DAG) visualizer for exploring and looking at your code, and determining lineage.
4. A project explorer for viewing curating projects and viewing versions of your Hamilton dataflows.

--------
Overview
Expand Down
4 changes: 4 additions & 0 deletions docs/main.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
</div>

Hamilton is a general-purpose framework to write dataflows using regular Python functions. At the core, each function defines a transformation and its parameters indicates its dependencies. Hamilton automatically connects individual functions into a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) that can be executed, visualized, optimized, and reported on.
Hamilton also comes with a [UI](concepts/ui.rst) to visualize, catalog, and monitor your dataflows.

```{figure} ./_static/abc.png
:scale: 50
Expand All @@ -16,6 +17,7 @@ The ABC of Hamilton

# Why should you use Hamilton?
**Facilitate collaboration**. By focusing on functions, Hamilton avoids sprawling code hierarchy and generates flat dataflows. Well-scoped functions make it easier to add features, complete code reviews, debug pipeline failures, and hand-off projects. Visualizations can be generated directly from your code to better understand and document it.
Integration with the [Hamilton UI](concepts/ui.rst) allows you to track lineage, catalog code & artifacts, and monitor your dataflows.

**Reduce development time**. Hamilton dataflows are reusable across projects and context (e.g., pipeline vs. web service). The benefits of developing robust and well-tested solutions are multiplied by reusability. Off-the-shelf dataflows are available on the [Hamilton Hub](https://hub.dagworks.io/).

Expand Down Expand Up @@ -71,3 +73,5 @@ Multiple companies are doing cool stuff with Hamilton! Come chat with members of
* **Lexis Nexis** - Feature processing and lineage
* **Opendoor** - Manage PySpark pipelines
* **KI** - Feature engineering
* **AB-InBev** - Machine learning Platform on PySpark
* **Rippling** - Feature engineering on PySpark