- Introduction
- Prerequisites
- Setting Up the Development Environment
- Project Structure
- Running the Application
- Debugging
- Contributing
- Further Reading
Welcome to the local development documentation for Helix.ml! This guide will help you set up your development environment, understand the project structure, run the application locally, and contribute to the Helix.
Before you start, ensure you have the following software installed on your machine:
- docker
- golang
- Node.js and npm
-
Clone the Repository
git clone git@github.com:helixml/helix.git cd helix
If you are an external contributor, consider working out of a forked repository of Helix.
-
Set Up Environment Variables
Create an
.env
file with settings based on the example values and edit it:cp .env.example-prod .env
The default values for settings are optimised for local development.
Here is an overview of the project structure:
helix/
├── Dockerfiles # Dockerfiles for various environments
├── api/ # Main Control Plane API directory
│ ├── cmd/ # Standard golang project structure within here
│ ├── pkg/ #
│ ├── main.go #
├── llamaindex/ # llamaindex
│ └── src/ #
│ └── ... # Other app-specific files
├── unstructured # Python Unstructured for parsing content
├── scripts # Scripts to get stuff done
├── runner # Runner configurations
├── frontend/ # Frontend in React, ts
│ ├── package.json # npm dependencies
│ └── src/ # Source files for the frontend
└── .env # Environment variables file
./stack up
This will bring up the control plane which serves the front-end and various other components on the stack. Refer Helix architecture docs
The control comes up on http://localhost:8080 by default.
Sanity check your environment with
docker ps
This should show you the running containers
$ docker ps
IMAGE PORTS NAMES
ankane/pgvector 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp helix-pgvector-1
helix-frontend 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp helix-frontend-1
helix-gptscript_runner helix-gptscript_runner-1
registry.helix.ml/helix/llamaindex:latest helix-llamaindex-1
webhookrelay/webhookrelayd helix-webhook_relay_github-1
webhookrelay/webhookrelayd helix-webhook_relay_stripe-1
helix-api 0.0.0.0:8080->80/tcp, :::8080->80/tcp helix-api-1
postgres:12.13-alpine 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp helix-postgres-1
quay.io/keycloak/keycloak:23.0 8080/tcp, 8443/tcp helix-keycloak-1
Follow the instructions on the docs to attach a runner
If you're local machine isn't able to host a runner, you have a few options:
- use a VSCode remote SSH session to develop within a machine that does have the resources.
- spin up a remote runner in Luke's bunker (or equivalent) and connect it back to your localhost via an SSH tunnel.
- spin up a remote runner in runpod (or equivalent) and use webhookrelay to connect that machine back to your localhost.
To connect your localhost to a remote runner via an SSH tunnel, follow these steps:
-
In a separate window, SSH into a remote machine and open a connection from the remote back to local:
ssh -p $SSH_PORT -R 8080:localhost:8080 user@remote.com
Where 8080 is the port that your local API is running on.
-
On the remote:
git clone https://github.com/helixml/helix
somewhere -
On the remote create a
.env
file with the following settings:SERVER_PORT=9080 # By default, the runner runs on 8080, so use another port. API_HOST=http://localhost:8080 # You've just forwarded this port back to your local machine API_TOKEN=oh-hallo-insecure-token # This should match the control plane
-
On the remote start the runner:
docker compose -f docker-compose.runner.yaml up -d
-
Now go back to your local machine and browse to
/dashboard
in Helix. You should see the runner. If not, take a look at the runner logs on the remote.
To connect to a runner via WebhookRelay:
-
Create a free account with https://webhookrelay.com/
-
Create a new access token and make a note of it: https://my.webhookrelay.com/tokens
-
Create a new "tunnel": https://my.webhookrelay.com/tunnels. Give it a name of
helix
, based in theeu
, with a destination ofhttp://localhost:8080
. Click create. Make a note of the "Host" that looks something like:https://9hxxxxxxxxx.webrelay.io
-
Add or edit your
docker-compose.dev.yaml
to include the following configuration. Here we're using host network mode so it can connect to the api which is exposed on 8080. You could change the destination in the previous step to point to somewhere else (likeapi:80
for example).webhook_relay_localhost: image: webhookrelay/webhookrelayd network_mode: "host" environment: - KEY=${WEBHOOK_RELAY_KEY:-} - SECRET=${WEBHOOK_RELAY_SECRET:-} command: ["--mode", "tunnel", "-t", "helix"]
-
Start your remote runner with the API_HOST variable set to the webrelay.io url, e.g.:
--api-host https://xxxx.webrelay.io
. -
Watch the runner logs to make sure it's able to connect to your local helix.
If you're testing, developing or working with Apps then you will need to connect a Github OAuth app to be able to read from user's repositories. The current way of doing this is via https://webhookrelay.com/.
- Create a free account with https://webhookrelay.com/
- Create a "bucket", with "forward to an internal location: https://my.webhookrelay.com/new-internal-destination
- Set the
Destination URL
to:http://localhost:8080/api/v1/github/webhook
- This should produce a url that looks something like
https://xxxx.hooks.webhookrelay.com
- Go to the Github oauth app settings and find the HelixML app called Helix apps development. Generate a new secret and grab the client ID.
- Set the following environmental variables in the
.env
file:
GITHUB_INTEGRATION_ENABLED=true
GITHUB_INTEGRATION_CLIENT_ID=6d04xxxxxxxx
GITHUB_INTEGRATION_CLIENT_SECRET=xxxxxx
GITHUB_INTEGRATION_WEBHOOK_URL=https://3nmlcxxxxx.hooks.webhookrelay.com/api/v1/github/webhook
- Restart the API and create an app.
- If you have any problems check the logs in webhook relay bucket.
./stack rebuild <component>
If you're familiar with tmux you will find it useful to do ./stack start
and ./stack stop
instead.
Both the frontend and the api have hot-reloads when in development mode. Rebuilds should only be required when adding libraries.
Bring down the stack
./stack stop
-
View all Docker logs
docker logs <container-name>
-
Branching Strategy
-
Create a new branch for each feature or bugfix:
git checkout -b feature/your-feature-name
-
-
Code Style
- Format all code with standard language formatters.
- Follow the project's coding guidelines for the frontend.
-
Commit Messages
- Write clear and concise commit messages.
-
Pull Requests
- Submit a pull request to the
main
branch for review.
- Submit a pull request to the
Happy coding! If you have any questions or run into issues, feel free to reach out to the maintainers on 👥 Discord.