Implements Hamilton UI -- OS from previously closed source (#840)

* Implements Hamilton UI -- OS from previously closed source

This has the following components:
1. Frontend -- React + typescript app that works together with the
   server
2. Backend -- Django app connected to a postgres DB + blob store.
   Configurable. Defaults to no authentication in local mode, but can be
   connected with PropelAuth.
3. Deployment -- tools for docker-compose deployment
4. Documentation -- minimally documented for now but has enough to get
   started

Note that this does not yet enable tests/github actions, that will be
done in the next few steps. That said, they exist for now.

* Runs/fixes pre-commit

* Adds backend tests to github actions

We're using github actions so we can conditionally skip it.

* Adds frontend github actions

This just tests a build/lint process. We're using github actions for all
new ones so we can do conditional/iterate faster.

* Simplifies github actions for UI

Github actions can filter on changes based on directory path.

---------

Co-authored-by: Stefan Krawczyk <stefan@dagworks.io>
Co-authored-by: Elijah ben Izzy <elijah@dagworks.io>

.github/workflows/hamilton-ui-backend.yml

@@ -0,0 +1,83 @@
+name: Backend Test Workflow
+
+on:
+  push:
+    branches:
+      - main  # or any specific branches you want to include
+    paths:
+      - 'ui/backend/**'
+
+  pull_request:
+    branches:
+      - main
+    paths:
+      - 'ui/backend/**'
+
+
+jobs:
+  test-backend:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        testdir: [test_lifecycle, test_db_methods]  # Specify your test directories
+    services:
+      postgres:
+        image: postgres:14
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_DB: circleci_test
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 100s
+          --health-retries 10
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+      - name: Install dependencies
+        run: |
+          cd ui/backend/server
+          pip install -r requirements-base.txt
+          pip install -r requirements-test.txt
+      - name: Run migrations
+        env:
+          DB_HOST: localhost
+          DB_USER: postgres
+          DB_PASSWORD: "postgres"
+          DB_NAME: ${{ matrix.testdir }}
+          DB_PORT: 5432
+          HAMILTON_ENV: integration_tests
+          DJANGO_SECRET_KEY: test
+          PGPASSWORD: postgres
+          PGHOST: localhost
+          PGUSER: postgres
+          HAMILTON_BLOB_STORE: local
+          HAMILTON_LOCAL_BLOB_DIR: ./blob_data
+        run: |
+          cd ui/backend/server
+          python manage.py sqlcreate
+          echo $(python manage.py sqlcreate) | psql -U postgres
+          python manage.py migrate
+      - name: Run tests
+        env:
+          DB_HOST: localhost
+          DB_USER: postgres
+          DB_PASSWORD: "postgres"
+          DB_NAME: ${{ matrix.testdir }}
+          DB_PORT: 5432
+          HAMILTON_ENV: integration_tests
+          DJANGO_SECRET_KEY: test
+          PGPASSWORD: postgres
+          PGHOST: localhost
+          PGUSER: postgres
+          HAMILTON_BLOB_STORE: local
+          HAMILTON_LOCAL_BLOB_DIR: ./blob_data
+        run: |
+          cd ui/backend/server
+          python -m pytest tests/${{ matrix.testdir }} -vvvvv

.github/workflows/hamilton-ui-frontend.yml

@@ -0,0 +1,30 @@
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'ui/frontend/**'
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'ui/frontend/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ui/frontend
+    strategy:
+      matrix:
+        node-version: [16.x]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Use Node.js ${{ matrix.node-version }}
+      uses: actions/setup-node@v1
+      with:
+        node-version: ${{ matrix.node-version }}
+    - run: npm install --ignore-scripts
+    - run: npm run lint-fix
+    - run: npm run format
+    - run: npm run build

LICENSE

@@ -1,5 +1,11 @@
-Copyright (c) 2019-2022 Stitch Fix
-Copyright (c) 2023-2023 DAGWorks Inc.
+Portions of this software are licensed as follows:
+
+* All content that resides under the "ui/frontend/src/ee" and "ui/backend/server/trackingserver_auth" directory of this
+repository is licensed under the license defined in "ui/frontend/src/ee/LICENSE" "ui/backend/server/trackingserver_auth/LICENSE" respectively.
+* All third party components incorporated into the Hamilton Software are licensed under the original license provided by the owner of the applicable component.
+* Content outside of the above mentioned directories or restrictions above is available under the "BSD-3 Clear Clause" license as defined below.
+
+Copyright (c) 2023-2024 DAGWorks Inc.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, are permitted
@@ -9,7 +15,7 @@ Redistribution and use in source and binary forms, with or without modification,
 disclaimer.
 * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
 disclaimer in the documentation and/or other materials provided with the distribution.
-* Neither the name of Stitch Fix, DAGWorks, nor the names of its contributors may be used to endorse or promote products derived
+* Neither the name of DAGWorks Inc., nor the names of its contributors may be used to endorse or promote products derived
 from this software without specific prior written permission.
 
 NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE GRANTED BY THIS LICENSE. THIS SOFTWARE IS PROVIDED BY

README.md

@@ -432,6 +432,8 @@ If you do not wish to participate, one can opt-out with one of the following met
    HAMILTON_TELEMETRY_ENABLED=false python NAME_OF_MY_DRIVER.py
    ```
 
+For the hamilton UI you jmust use the environment variable method prior to running docker compose.
+
 # Contributors
 
 ## Code Contributors

docs/concepts/index.rst

@@ -17,4 +17,5 @@ concepts that makes Hamilton unique and powerful.
    builder
    function-modifiers-advanced
    parallel-task
+   ui
    best-practices/index

docs/concepts/ui.rst

@@ -0,0 +1,199 @@
+===========
+Hamilton UI
+===========
+
+Hamilton comes with a fully open-source UI that can be run both for local deployment and on a remote server.
+The UI consists of the following features:
+
+1. Telemetry for hamilton executions -- both on the history of executions and the data itself
+2. A feature/artifact catalog for browsing/connecting executions of nodes -> results
+3. A DAG visualizer for exploring the DAG, looking at the code, and determining lineage
+
+--------
+Overview
+--------
+
+Getting Started
+---------------
+
+The Hamilton UI is contained within a set of docker images. You launch with `docker-compose <https://docs.docker.com/compose/>`_, and it will start up the UI, the backend server,
+and a postgres database.
+
+
+Prerequisites
+-------------
+
+To run this, you'll need:
+
+1. Docker installed -- you can follow instructions  `here <https://docs.docker.com/engine/install/>`_.
+2. A Hamilton workflow -- if you don't have this there's an init command that will create a basic one for you.
+3. The `hamilton` repository cloned locally.
+
+----
+
+Starting the UI
+---------------
+
+Start by ensuring that the hamilton repository is cloned locally. Then, navigate to the `ui` directory:
+
+.. code-block:: bash
+
+    git clone https://github.com/dagworks-inc/hamilton
+
+Then, navigate to the ``ui`` directory:
+
+.. code-block:: bash
+
+    cd hamilton/ui
+
+
+And run the server command:
+.. code-block:: bash
+
+    ./deployment/run.sh
+
+
+This will do the following:
+
+1. Pull all docker images from the docker hub
+2. Start a local postgres database
+3. Start the backend server
+4. Start the frontend server
+
+This takes a bit of time! So be patient. The server will be running on port 3000 -- in the logs you'll see something like this:
+
+.. code-block:: bash
+
+    2024-04-18 15:57:07 You can now view frontend in the browser.
+    2024-04-18 15:57:07
+    2024-04-18 15:57:07   Local:            http://localhost:3000
+    2024-04-18 15:57:07   On Your Network:  http://172.19.0.4:3000
+
+
+Then navigate to ``http://localhost:3000`` in your browser, and enter your email (this will be the username used within the app).
+
+Building the Docker Images locally
+__________________________________
+If building the Docker containers from scratch, increase your Docker memory to 10gb or more -- you can do this in the Docker Desktop settings.
+
+To build the images locally, you can run the following command:
+
+.. code-block:: bash
+
+    # from the hamilton/ui directory
+    ./deployment/dev.sh --build
+
+This will build the containers from scratch. If you just want to mount the local code, you can run just
+
+.. code-block:: bash
+
+    ./deployment/dev.sh
+
+----
+
+Running your first dataflows
+----------------------------
+
+Now that you have your server running, you can run a simple dataflow and watch it in the UI!
+You can follow instructions in the UI when you create a new project, or follow the instructions here.
+
+First, install the SDK:
+
+.. code-block:: bash
+
+    pip install sf-hamilton-sdk
+
+Then, navigate to the project page (dashboard/projects), in the running UI, and click the green ``+ New DAG`` button.
+
+.. image:: ../_static/new_project.png
+
+Remember the project ID -- you'll use it for the next steps.
+
+Add the following adapter to your code:
+
+.. code-block:: python
+
+    from hamilton_sdk import adapters
+
+    tracker = adapters.HamiltonTracker(
+       project_id=PROJECT_ID_FROM_ABOVE,
+       username="EMAIL_YOU_PUT_IN_THE_UI",
+       dag_name="my_version_of_the_dag",
+       tags={"environment": "DEV", "team": "MY_TEAM", "version": "X"}
+    )
+
+    dr = (
+      driver.Builder()
+        .with_config(your_config)
+        .with_modules(*your_modules)
+        .with_adapters(tracker)
+        .build()
+    )
+
+Then run your DAG, and follow the links in the logs!
+
+
+Exploring in the UI
+-------------------
+
+Once you get to the UI, you will be navigated to the projects page. After you create one + log,
+you can navigate to `runs/history` for a history of runs. You can select by tags, date, etc...
+
+----
+
+-----------
+Features
+-----------
+
+The UI has the following features:
+
+
+DAG version tracking
+--------------------
+
+Select DAG versions to compare + visualize.
+
+.. image:: ../_static/version_tracking.png
+    :alt: DAG Version Tracking
+
+Feature/asset Catalog
+---------------------
+
+View functions, nodes, and assets across a history of runs.
+
+.. image:: ../_static/catalog.png
+    :alt: Catalog
+
+Browser
+-------
+
+View DAG shapes + code:
+
+
+.. image:: ../_static/code_browser.png
+    :alt: Browser
+
+.. image:: ../_static/dag_view.png
+    :alt: Browser
+
+Run tracking + telemetry
+------------------------
+
+View a history of runs, telemetry on runs/comparison, and data for specific runs:
+
+.. image:: ../_static/run_tracking.png
+    :alt: Run Tracking
+
+.. image:: ../_static/run_telemetry.png
+    :alt: Run Telemetry
+
+.. image:: ../_static/run_data.png
+    :alt: Run Data
+
+----
+
+------------
+Self-Hosting
+------------
+
+Please reach out to us if you want to deploy on your own infrastructure. Self-hosting documentation will be up soon.

setup.cfg

@@ -12,6 +12,8 @@ ignore =
     W503,
     # invalid escape sequence
     W605
+    # multiple statements on one line
+    E704
 
 [isort]
 profile=black

setup.py

@@ -96,6 +96,7 @@ def load_requirements():
         ],
         "diskcache": ["diskcache"],
         "cli": ["typer"],
+        "sdk": ["sf-hamilton-sdk"],
     },
     entry_points={
         "console_scripts": [