CLI: Devtool refactors (#3745)

* Initial refactors * Fix tests * Self review 1 * Self review 2 * Maybe fix npm * Fix Go tests * Add seed command * Fix linter * Review * Upgrade testcontainers to fix non-deterministic Postgres tests * Create cloud state dir upfront * nits * Update READMEs * Fix typo
rilldata · Jan 3, 2024 · 941c4ce · 941c4ce · github-actions · Jan 3, 2024
1 parent cb6abf4
commit 941c4ce
Show file tree

Hide file tree

Showing 41 changed files with 1,678 additions and 917 deletions.
diff --git a/.gitignore b/.gitignore
@@ -21,6 +21,7 @@ tsc-tmp/
 /rill
 /cli/pkg/examples/embed/dist
 /dev-project*
+/dev-cloud-state*
 
 # data files
 *.duckdb

diff --git a/.golangci.yml b/.golangci.yml
@@ -156,10 +156,14 @@ linters-settings:
     disable-all: true
   gosec:
     excludes:
+      - G107
       - G108
       - G112
       - G203
+      - G204
       - G306
+      - G401
+      - G501
   revive:
     ignore-generated-header: false
     severity: "warning"
@@ -172,7 +176,6 @@ linters-settings:
       - name: context-keys-type
       - name: dot-imports
       - name: error-return
-      - name: error-strings
       - name: error-naming
       - name: if-return
       - name: increment-decrement
@@ -189,21 +192,26 @@ linters-settings:
       - name: superfluous-else
       - name: unreachable-code
       - name: redefines-builtin-id
+  stylecheck:
+    checks:
+      [
+        "all",
+        "-ST1000",
+        "-ST1003",
+        "-ST1005",
+        "-ST1016",
+        "-ST1020",
+        "-ST1021",
+        "-ST1022",
+      ]
 
 issues:
+  max-issues-per-linter: 0
+  max-same-issues: 0
   exclude-rules:
     - path: _test\.go # disable some linters for test files
       linters:
         - gocyclo
         - gosec
         - dupl
         - varnamelen
-    - linters:
-        - gosec
-      text: "weak cryptographic primitive"
-    - linters:
-        - stylecheck
-      text: "error strings should not be capitalized"
-
-  max-issues-per-linter: 0
-  max-same-issues: 0
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -8,15 +8,17 @@ This file should serve as an entrypoint for learning about and contributing to R
 
 ## Development environment
 
+If you're a Rill team member, you can run `rill devtool start` from the project root to start a full local development environment. If you select the cloud preset, you can fill it with seed data using `rill devtool seed cloud`. See `cli/cmd/devtool/README.md` for more details.
+
+### Development dependencies
+
 This is a full list of development dependencies:
 
 - [Docker](https://www.docker.com)
 - [Node.js 18](https://nodejs.org/en/) (we recommend installing it with [nvm](https://github.com/nvm-sh/nvm))
 - [Go 1.21](https://go.dev) (on macOS, install with `brew install go`)
 - [Buf](https://buf.build) (Protocol Buffers) (on macOS, install with `brew install bufbuild/buf/buf`)
 
-Run `sh scripts/devtool.sh` from project root to run all services locally. Refer `scripts/devtool/README.md` for more details.
-
 ### Editor setup
 
 - Integrate `golangci-lint` ([instructions](https://golangci-lint.run/usage/integrations))
@@ -71,11 +73,13 @@ The project uses NPM for Node.js (specifically, NPM [workspaces](https://docs.np
 Here's a guide to the top-level structure of the repository:
 
 - `.github` contain CI/CD workflows.
-- `admin` contains the backend control plane for a multi-user, hosted version of Rill (in progress, not launched yet).
+- `admin` contains the backend control plane for the managed, multi-user version of Rill.
 - `cli` contains the CLI and a server for the local frontend (used only in production).
 - `docs` contains the user-facing documentation that we deploy to [docs.rilldata.com](https://docs.rilldata.com).
 - `proto` contains protocol buffer definitions for all Rill components, which notably includes our API interfaces.
-- `runtime` is our data plane, responsible for querying and orchestrating data infra. It currently supports DuckDB and Druid.
-- `web-admin` contains the frontend control plane for a multi-user, hosted version of Rill (in progress, not launched yet).
-- `web-common` contains common functionality shared across the local and cloud frontends.
-- `web-local` contains the local Rill Developer application, including the data modeller and current CLI.
+- `runtime` contains the engine (data plane) responsible for orchestrating and serving data.
+- `scripts` contains various scripts and other resources used in development.
+- `web-admin` contains the frontend control plane for the managed, multi-user version of Rill.
+- `web-auth` contains the frontend code for `auth.rilldata.com` (managed with Auth0).
+- `web-common` contains common functionality shared across the local and admin frontend applications.
+- `web-local` contains the local Rill application, notably the data modeller.
diff --git a/admin/README.md b/admin/README.md
@@ -1,76 +1,26 @@
 # admin
 
-This directory contains the control-plane for multi-user, hosted deployments of Rill.
+This directory contains the control-plane for the managed, multi-user version of Rill (currently available on `ui.rilldata.com`).
 
 ## Running in development
 
-1. Create a `.env` file at the root of the repo containing:
-```
-RILL_ADMIN_DATABASE_DRIVER=postgres
-RILL_ADMIN_DATABASE_URL=postgres://postgres:postgres@localhost:5432/postgres
-RILL_ADMIN_HTTP_PORT=8080
-RILL_ADMIN_GRPC_PORT=9090
-RILL_ADMIN_METRICS_EXPORTER="prometheus"
-RILL_ADMIN_TRACES_EXPORTER=""
-RILL_ADMIN_EXTERNAL_URL=http://localhost:8080
-RILL_ADMIN_FRONTEND_URL=http://localhost:3000
-RILL_ADMIN_ALLOWED_ORIGINS=*
-# Hex-encoded comma-separated list of key pairs. To generate, run "go run ./scripts/generate_keypairs/main.go"
-# For details: https://pkg.go.dev/github.com/gorilla/sessions#NewCookieStore
-RILL_ADMIN_SESSION_KEY_PAIRS=7938b8c95ac90b3731c353076daeae8a,90c22a5a6c6b442afdb46855f95eb7d6
-# JWKS details for signing JWTs. The JWKS must contain *private* keys. To generate, run "go run ./scripts/generate_jwks/main.go"
-RILL_ADMIN_SIGNING_JWKS=
-RILL_ADMIN_SIGNING_KEY_ID=
-# Get these from https://auth0.com/ (or ask a team member)
-RILL_ADMIN_AUTH_DOMAIN=gorillio-stage.auth0.com
-RILL_ADMIN_AUTH_CLIENT_ID=
-RILL_ADMIN_AUTH_CLIENT_SECRET=
-# Get these from https://github.com/ (or ask a team member)
-RILL_ADMIN_GITHUB_APP_ID=302634
-RILL_ADMIN_GITHUB_APP_NAME=rill-cloud-dev
-RILL_ADMIN_GITHUB_APP_PRIVATE_KEY=
-RILL_ADMIN_GITHUB_APP_WEBHOOK_SECRET=
-RILL_ADMIN_GITHUB_CLIENT_ID=
-RILL_ADMIN_GITHUB_CLIENT_SECRET=
-# For email client
-RILL_ADMIN_EMAIL_SMTP_HOST=
-RILL_ADMIN_EMAIL_SMTP_PORT=
-RILL_ADMIN_EMAIL_SMTP_USERNAME=
-RILL_ADMIN_EMAIL_SMTP_PASSWORD=
-RILL_ADMIN_EMAIL_SENDER_EMAIL=
-RILL_ADMIN_EMAIL_SENDER_NAME=
-RILL_ADMIN_EMAIL_BCC=
-```
-2. In a separate terminal, run Postgres in the background:
-```bash
-docker-compose -f admin/docker-compose.yml up 
-# Data is persisted. To clear, run: docker-compose -f admin/docker-compose.yml down --volumes
-```
-3. Run the server:
+Run the following command from the repository root to start a full development environment except the admin service:
 ```bash
-go run ./cli admin start
-```
-4. Ping the server:
-```bash
-go run ./cli admin ping --url http://localhost:9090
+rill devtool start cloud --except admin # optional: --reset 
 ```
 
-You can now call the local admin server from the CLI by overriding the admin API URL. For example:
+For as long as the devtool is running, `rill` commands will target your local development environment instead of `rilldata.com` (you can manually switch environments using `rill devtool switch-env`.)
+
+Then separately start the admin service (and start/stop it when you make code changes):
 ```bash
-go run ./cli org create --api-url http://localhost:9090
+go run ./cli admin start
 ```
 
-## Adding endpoints
+### Using Github webhooks in development
 
-We define our APIs using gRPC and use [gRPC-Gateway](https://grpc-ecosystem.github.io/grpc-gateway/) to map the RPCs to a RESTful API. See `proto/README.md` for details.
+The local development environment is not capable of receiving Github webhooks. In most cases, you can just run `rill project reconcile` to manually trigger a reconcile after pushing changes to Github.
 
-To add a new endpoint:
-1. Describe the endpoint in `proto/rill/admin/v1/api.proto`
-2. Re-generate gRPC and OpenAPI interfaces by running `make proto.generate`
-3. Copy the new handler signature from the `AdminServiceServer` interface in `proto/gen/rill/admin/v1/api_grpc_pb.go`
-4. Paste the handler signature and implement it in a relevant file in `admin/server/`
-
-## Using the Github App in development
+Continue reading only if you are making changes to the Github webhooks code and need to these changes specifically.
 
 We use a Github App to listen to pushes on repositories connected to Rill to do automated deployments. The app has access to read `contents` and receives webhooks on `git push`.
 
@@ -87,17 +37,18 @@ npm install --global smee-client
 smee --port 8080 --path /github/webhook --url https://smee.io/IDENTIFIER
 ```
 
-## CLI login/logout
+## Adding endpoints
+
+We define our APIs using gRPC and use [gRPC-Gateway](https://grpc-ecosystem.github.io/grpc-gateway/) to map the RPCs to a RESTful API. See `proto/README.md` for details.
+
+To add a new endpoint:
+1. Describe the endpoint in `proto/rill/admin/v1/api.proto`
+2. Re-generate gRPC and OpenAPI interfaces by running `make proto.generate`
+3. Copy the new handler signature from the `AdminServiceServer` interface in `proto/gen/rill/admin/v1/api_grpc_pb.go`
+4. Paste the handler signature and implement it in a relevant file in `admin/server/`
+
+## Adding a new user preferences field
 
-For trying out CLI login add api-url parameter to point to local admin server like this:
-```
-go run ./cli login --api-url http://localhost:9090/
-```
-For trying out CLI logout add api-url parameter to point to local admin server like this:
-```
-go run ./cli logout --api-url http://localhost:9090/
-```
-## Adding a new field preferences
 To add a new preference field for the user, follow these steps:
 
 1. Include a new column named `preference_<name>` in the `users` table. This can be accomplished by appending an appropriate `ALTER TABLE` query to a newly created `.sql` file located within the `postgres/migrations` folder. 

diff --git a/admin/docker-compose.yml b/admin/docker-compose.yml
diff --git a/admin/server/server.go b/admin/server/server.go
@@ -50,8 +50,8 @@ type Options struct {
 	GRPCPort               int
 	ExternalURL            string
 	FrontendURL            string
-	SessionKeyPairs        [][]byte
 	AllowedOrigins         []string
+	SessionKeyPairs        [][]byte
 	ServePrometheus        bool
 	AuthDomain             string
 	AuthClientID           string

diff --git a/cli/README.md b/cli/README.md
@@ -18,7 +18,7 @@ make cli
 
 In development, the CLI will serve a dummy frontend and not embed any examples. You can run it like this:
 ```bash
-# Optionally run this to embed the UI and examples in the cli (not needed if using `npm run dev`)
+# Optionally run this to embed the UI and examples in the CLI
 make cli.prepare
 
 # To output usage:

diff --git a/cli/cmd/admin/start.go b/cli/cmd/admin/start.go
@@ -37,18 +37,22 @@ import (
 type Config struct {
 	DatabaseDriver           string                 `default:"postgres" split_words:"true"`
 	DatabaseURL              string                 `split_words:"true"`
+	RedisURL                 string                 `default:"" split_words:"true"`
+	ProvisionerSpec          string                 `split_words:"true"`
 	Jobs                     []string               `split_words:"true"`
-	HTTPPort                 int                    `default:"8080" split_words:"true"`
-	GRPCPort                 int                    `default:"9090" split_words:"true"`
-	DebugPort                int                    `split_words:"true"`
 	LogLevel                 zapcore.Level          `default:"info" split_words:"true"`
 	MetricsExporter          observability.Exporter `default:"prometheus" split_words:"true"`
 	TracesExporter           observability.Exporter `default:"" split_words:"true"`
+	HTTPPort                 int                    `default:"8080" split_words:"true"`
+	GRPCPort                 int                    `default:"9090" split_words:"true"`
+	DebugPort                int                    `split_words:"true"`
 	ExternalURL              string                 `default:"http://localhost:8080" split_words:"true"`
 	ExternalGRPCURL          string                 `envconfig:"external_grpc_url"`
 	FrontendURL              string                 `default:"http://localhost:3000" split_words:"true"`
-	SessionKeyPairs          []string               `split_words:"true"`
 	AllowedOrigins           []string               `default:"*" split_words:"true"`
+	SessionKeyPairs          []string               `split_words:"true"`
+	SigningJWKS              string                 `split_words:"true"`
+	SigningKeyID             string                 `split_words:"true"`
 	AuthDomain               string                 `split_words:"true"`
 	AuthClientID             string                 `split_words:"true"`
 	AuthClientSecret         string                 `split_words:"true"`
@@ -58,17 +62,13 @@ type Config struct {
 	GithubAppWebhookSecret   string                 `split_words:"true"`
 	GithubClientID           string                 `split_words:"true"`
 	GithubClientSecret       string                 `split_words:"true"`
-	ProvisionerSpec          string                 `split_words:"true"`
-	SigningJWKS              string                 `split_words:"true"`
-	SigningKeyID             string                 `split_words:"true"`
 	EmailSMTPHost            string                 `split_words:"true"`
 	EmailSMTPPort            int                    `split_words:"true"`
 	EmailSMTPUsername        string                 `split_words:"true"`
 	EmailSMTPPassword        string                 `split_words:"true"`
 	EmailSenderEmail         string                 `split_words:"true"`
 	EmailSenderName          string                 `split_words:"true"`
 	EmailBCC                 string                 `split_words:"true"`
-	RedisURL                 string                 `default:"" split_words:"true"`
 	ActivitySinkType         string                 `default:"" split_words:"true"`
 	ActivitySinkPeriodMs     int                    `default:"1000" split_words:"true"`
 	ActivityMaxBufferSize    int                    `default:"1000" split_words:"true"`
@@ -243,8 +243,8 @@ func StartCmd(ch *cmdutil.Helper) *cobra.Command {
 					GRPCPort:               conf.GRPCPort,
 					ExternalURL:            conf.ExternalURL,
 					FrontendURL:            conf.FrontendURL,
-					SessionKeyPairs:        keyPairs,
 					AllowedOrigins:         conf.AllowedOrigins,
+					SessionKeyPairs:        keyPairs,
 					ServePrometheus:        conf.MetricsExporter == observability.PrometheusExporter,
 					AuthDomain:             conf.AuthDomain,
 					AuthClientID:           conf.AuthClientID,