From 6c00b60b96a95ff2757dc748a0c14db6388ed537 Mon Sep 17 00:00:00 2001 From: coffeegoddd Date: Wed, 12 Jun 2024 13:55:22 -0700 Subject: [PATCH 1/2] /packages/doltlab/content: wip updating for v2.1.6 --- packages/doltlab/content/SUMMARY.md | 1 + .../content/reference/installer/cli.md | 42 ++++++ .../reference/installer/configuration-file.md | 132 +++++++++++++++++- .../content/reference/release-notes/README.md | 2 + .../content/reference/release-notes/v2.1.6.md | 16 +++ 5 files changed, 192 insertions(+), 1 deletion(-) create mode 100644 packages/doltlab/content/reference/release-notes/v2.1.6.md diff --git a/packages/doltlab/content/SUMMARY.md b/packages/doltlab/content/SUMMARY.md index 13eb98fd..e7ff474e 100644 --- a/packages/doltlab/content/SUMMARY.md +++ b/packages/doltlab/content/SUMMARY.md @@ -35,6 +35,7 @@ - [Configuration file reference](reference/installer/configuration-file.md) - [Command line reference](reference/installer/cli.md) - [Release Notes](reference/release-notes/README.md) + - [v2.1.6](reference/release-notes/v2.1.6.md) - [v2.1.5](reference/release-notes/v2.1.5.md) - [v2.1.4](reference/release-notes/v2.1.4.md) - [v2.1.3](reference/release-notes/v2.1.3.md) diff --git a/packages/doltlab/content/reference/installer/cli.md b/packages/doltlab/content/reference/installer/cli.md index 2f5ca287..b8b089a8 100644 --- a/packages/doltlab/content/reference/installer/cli.md +++ b/packages/doltlab/content/reference/installer/cli.md @@ -186,6 +186,24 @@ _String_. The `dolthubadmin` SQL user password of the `doltlabdb` instance. Configuration file equivalent [admin_password](./configuration-file.md#admin_password). +## doltlabdb-backups-volume-host-path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-backups' Docker volume. + +Configuration file equivalent [backups_volume_path](./configuration-file.md#backups_volume_path). + +## doltlabdb-config-volume-host-path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-configs' Docker volume. + +Configuration file equivalent [configs_volume_path](./configuration-file.md#configs_volume_path). + +## doltlabdb-data-volume-host-path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-data' Docker volume. + +Configuration file equivalent [data_volume_path](./configuration-file.md#doltlabdb-data-volume-path). + ## doltlabdb-dolthubapi-password _String_. The `dolthubapi` SQL user password of the `doltlabdb` instance. @@ -210,6 +228,12 @@ _Number_. The port of `doltlabdb`. Configuration file equivalent [port](./configuration-file.md#doltlabdb-port). +## doltlabdb-root-volume-host-path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-root' Docker volume. + +Configuration file equivalent [root_volume_path](./configuration-file.md#root_volume_path). + ## doltlabdb-tls-skip-verify _Boolean_. If true, will disable TLS verification for connection to `doltlabdb`. @@ -234,6 +258,12 @@ _Number_. The port for `doltlabfileserviceapi`. Configuration file equivalent [port](./configuration-file.md#doltlabfileserviceapi-port). +## doltlabfileserviceapi-uploads-volume-host-path + +_String_. The path to an existing directory on the DoltLab host for persisting the 'doltlab-user-uploads' Docker volume. + +Configuration file equivalent [uploads_volume_path](./configuration-file.md#uploads_volume_path). + ## doltlabgraphql-host _String_. The hostname or IP address for `doltlabgraphql`. @@ -252,6 +282,12 @@ _Number_. The port for `doltlabgraphql`. Configuration file equivalent [port](./configuration-file.md#doltlabgrapqhl-port). +## doltlabremoteapi-data-volume-host-path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlab-remote-storage' Docker volume. + +Configuration file equivalent [data_volume_path](./configuration-file.md#doltlabremoteapi-data-volume-path). + ## doltlabremoteapi-file-server-port _Number_. The port for `doltlabremoteapi`'s file server. @@ -476,6 +512,12 @@ Configuration file equivalent [private_key](./configuration-file.md#private_key) _Boolean_. If true will generate a script to install DoltLab's dependencies on Ubuntu. +## use-env + +_Boolean_. If true, sensitive values will not be written to generated assets and environment variables will be expected instead. + +Configuration file equivalent [use_env](./configuration-file.md#use_env). + ## whitelist-all-users _Boolean_. If true, allows all users create accounts on a DoltLab instance, (default true). diff --git a/packages/doltlab/content/reference/installer/configuration-file.md b/packages/doltlab/content/reference/installer/configuration-file.md index ce623d26..16fed70b 100644 --- a/packages/doltlab/content/reference/installer/configuration-file.md +++ b/packages/doltlab/content/reference/installer/configuration-file.md @@ -12,6 +12,7 @@ host: "127.0.0.1" docker_network: "doltlab" metrics_disabled: false whitelist_all_users: true +use_env: false scheme: "http" services: doltlabdb: @@ -20,6 +21,11 @@ services: admin_password: "*****" dolthubapi_password: "*****" tls_skip_verify: true + volume_paths: + data_volume_path: "/local/path/to/store/database/data" + root_volume_path: "/local/path/to/store/database/root" + backups_volume_path: "/local/path/to/store/database/file/backups" + configs_volume_path: "/local/path/to/store/database/configs" doltlabapi: host: "127.0.0.1" port: 9443 @@ -28,9 +34,13 @@ services: host: "127.0.0.1" port: 50051 file_server_port: 100 + volume_paths: + data_volume_path: "/local/path/to/store/remote/data" doltlabfileserviceapi: host: "127.0.0.1" port: 4321 + volume_paths: + uploads_volume_path: "/local/path/to/store/uploads" doltlabgraphql: host: "127.0.0.1" port: 9000 @@ -106,6 +116,7 @@ The following are top-level `installer_config.yaml` options: - [docker_network](#docker_network) - [metrics_disabled](#metrics_disabled) - [whitelist_all_users](#whitelist_all_users) +- [use_env](#use_env) - [services](#services) - [default_user](#default_user) - [smtp](#smtp) @@ -174,6 +185,17 @@ See [prevent unauthorized user account creation](../../guides/basic.md#prevent-u Command line equivalent [white-list-all-users](./cli.md#white-list-all-users). +## use_env + +_Boolean_. If true, sensitive values will not be written to generated assets and environment variables will be expected instead. + +```yaml +# example installer_config.yaml +use_env: true +``` + +Command line equivalent [use-env](./cli.md#use-env). + ## services _Dictionary_. Configuration options for DoltLab's various services. `doltlabdb` passwords are _Required_ in single host deployments, other service definitions are _Required_ for multi-host deployments. @@ -194,6 +216,7 @@ _Dictionary_. Configuration options for `doltlabdb`. - [admin_password](#admin_password) - [dolthubapi_password](#dolthubapi_password) - [tls_skip_verify](#tls_skip_verify) +- [volume_paths](#doltlabdb-volume-paths)

host

@@ -260,6 +283,71 @@ services: Command line equivalent [doltlabdb-tls-skip-verify](./cli.md#doltlabdb-tls-skip-verify). +

volume_paths

+ +_Dictionary_. Local paths used for persisting `doltlabdb` Docker volumes. + +- [data_volume_path](#doltlabdb-data-volume-path) +- [root_volume_path](#root_volume_path) +- [backups_volume_path](#backups_volume_path) +- [configs_volume_path](#configs_volume_path) + +

data_volume_path

+ +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-data' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabdb: + volume_paths: + data_volume_path: "/local/path/for/persisting/data" +``` + +Command line equivalent [doltlabdb-data-volume-host-path](./cli.md#doltlabdb-data-volume-host-path). + +#### root_volume_path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-root' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabdb: + volume_paths: + root_volume_path: "/local/path/for/persisting/doltlabdb/root" +``` + +Command line equivalent [doltlabdb-root-volume-host-path](./cli.md#doltlabdb-root-volume-host-path). + +#### backups_volume_path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-backups' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabdb: + volume_paths: + backups_volume_path: "/local/path/for/persisting/file/backups" +``` + +Command line equivalent [doltlabdb-backups-volume-host-path](./cli.md#doltlabdb-backups-volume-host-path). + +#### configs_volume_path + +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlabdb-dolt-configs' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabdb: + volume_paths: + configs_volume_path: "/local/path/for/persisting/doltlabdb/configs" +``` + +Command line equivalent [doltlabdb-configs-volume-host-path](./cli.md#doltlabdb-configs-volume-host-path). + ### doltlabapi _Dictionary_. Configuration options for `doltlabapi`. @@ -314,6 +402,7 @@ _Dictionary_. Configuration options for `doltlabremoteapi`. - [host](#doltlabremoteapi-host) - [port](#doltlabremoteapi-port) - [file_server_port](#file_server_port) +- [volume_paths](#doltlabremoteapi-volume-paths)

host

@@ -341,7 +430,7 @@ services: Command line equivalent [doltlabremoteapi-port](./cli.md#doltlabremoteapi-port). -

port

+#### file_server_port _Number_. The port for `doltlabremoteapi`'s file server. _Required_ for [configuring multi-host deployments](../../guides/enterprise.md#deploy-doltlab-across-multiple-hosts). @@ -354,12 +443,33 @@ services: Command line equivalent [doltlabremoteapi-file-server-port](./cli.md#doltlabremoteapi-file-server-port). +

volume_paths

+ +_Dictionary_. Local paths used for persisting `doltlabremoteapi` Docker volumes. + +- [data_volume_path](#doltlabremoteapi-data-volume-path) + +

data_volume_path

+ +_String_. The path to an existing directory on the DoltLab host used for persisting the 'doltlab-remote-storage' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabremoteapi: + volume_paths: + data_volume_path: "/path/for/persisting/remote/data" +``` + +Command line equivalent [doltlabremoteapi-data-volume-host-path](./cli.md#doltlabremoteapi-data-volume-host-path). + ### doltlabfileserviceapi _Dictionary_. Configuration options for `doltlabapifileserviceapi`. - [host](#doltlabfileserviceapi-host) - [port](#doltlabfileserviceapi-port) +- [volume_paths](#doltlabfileserviceapi-volume-paths)

host

@@ -387,6 +497,26 @@ services: Command line equivalent [doltlabfileserviceapi-port](./cli.md#doltlabfileserviceapi-port). +

volume_paths

+ +_Dictionary_. Local paths used for persisting `doltlabfileserviceapi` Docker volumes. + +- [uploads_volume_path](#uploads_volume_path) + +

uploads_volume_path

+ +_String_. The path to an existing directory on the DoltLab host for persisting the 'doltlab-user-uploads' Docker volume. + +```yaml +# example installer_config.yaml +services: + doltlabfileserviceapi: + volume_paths: + uploads_volume_path: "/path/for/persisting/user/uploads" +``` + +Command line equivalent [doltlabfileserviceapi-uploads-volume-host-path](./cli.md#doltlabfileserviceapi-uploads-volume-host-path). + ### doltlabgraphql _Dictionary_. Configuration options for `doltlabgraphql`. diff --git a/packages/doltlab/content/reference/release-notes/README.md b/packages/doltlab/content/reference/release-notes/README.md index 096d4311..045fc4b6 100644 --- a/packages/doltlab/content/reference/release-notes/README.md +++ b/packages/doltlab/content/reference/release-notes/README.md @@ -6,6 +6,8 @@ DoltLab is currently closed-source, but the [dolthub/doltlab-issues](https://git The following is a list of release notes for available DoltLab versions >= `v2.0.0`. DoltLab and DoltHub share the same source code. For this reason, we list fixes related to DoltLab issues in the `DoltLab Features/Bug Fixes` section of the release notes, and fixes corresponding to DoltHub issuers in the `DoltLab Features/Bug Fixes` section. +- [DoltLab v2.1.6](./v2.1.6.md) +- [DoltLab v2.1.5](./v2.1.5.md) - [DoltLab v2.1.4](./v2.1.4.md) - [DoltLab v2.1.3](./v2.1.3.md) - [DoltLab v2.1.2](./v2.1.2.md) diff --git a/packages/doltlab/content/reference/release-notes/v2.1.6.md b/packages/doltlab/content/reference/release-notes/v2.1.6.md new file mode 100644 index 00000000..7c185d4a --- /dev/null +++ b/packages/doltlab/content/reference/release-notes/v2.1.6.md @@ -0,0 +1,16 @@ +--- +title: DoltLab v2.1.6 Release Notes +--- + +Download at [https://doltlab-releases.s3.amazonaws.com/linux/amd64/doltlab-v2.1.6.zip](https://doltlab-releases.s3.amazonaws.com/linux/amd64/doltlab-v2.1.6.zip) + +## DoltLab Features/Bug Fixes +* Supports cloud storage for DoltLab on AWS (DoltLab Enterprise), fixes https://github.com/dolthub/doltlab-issues/issues/19, https://github.com/dolthub/doltlab-issues/issues/37. +* `installer` now supports custom volume paths, fixes https://github.com/dolthub/doltlab-issues/issues/92. +* `installer` now supports `use_env` option which will not write secrets to generated files, fixes https://github.com/dolthub/doltlab-issues/issues/89. +* default `installer_config.yaml` contains all options commented out, fixes https://github.com/dolthub/doltlab-issues/issues/88. +* Docker volumes now explicitly named, fixes https://github.com/dolthub/doltlab-issues/issues/90. + + +## DoltHub Features/Bug Fixes +* Misc. UI design changes and bug fixes From 16004e39514b7296fd2df9164e14e06a52039655 Mon Sep 17 00:00:00 2001 From: coffeegoddd Date: Wed, 12 Jun 2024 14:48:15 -0700 Subject: [PATCH 2/2] /packages/doltlab/content/reference/installer: finish doc updates --- .../content/reference/installer/cli.md | 40 ++++++- .../reference/installer/configuration-file.md | 112 ++++++++++++++++++ 2 files changed, 150 insertions(+), 2 deletions(-) diff --git a/packages/doltlab/content/reference/installer/cli.md b/packages/doltlab/content/reference/installer/cli.md index b8b089a8..c0ae26f3 100644 --- a/packages/doltlab/content/reference/installer/cli.md +++ b/packages/doltlab/content/reference/installer/cli.md @@ -156,6 +156,18 @@ _String_. The docker network to run DoltLab in, (default "doltlab"). Configuration file equivalent [docker_network](./configuration-file.md#docker_network). +## doltlabapi-asyncworker-aws-sqs-queue + +_String_. The name of the SQS queue used for processing asynchronous tasks, DoltLab Enterprise only. + +Configuration file equivalent [asyncworker_aws_sqs_queue](./configuration-file.md#asyncworker_aws_sqs_queue). + +## doltlabapi-aws-region + +_String_. The AWS region for 'doltlabapi' cloud storage AWS resources, DoltLab Enterprise only. + +Configuration file equivalent [aws_region](./configuration-file.md#doltlabapi-aws-region). + ## doltlabapi-csv-port _Number_. The port for `doltlabapi`'s CSV service. @@ -180,7 +192,13 @@ _Number_. The port for `doltlabapi`. Configuration file equivalent [port](./configuration-file.md#doltlabapi-port). -## -doltlabdb-admin-password +## doltlabapi-query-job-aws-s3-bucket + +_String_. The name of the S3 bucket used to store the results of SQL query Jobs, DoltLab Enterprise only. + +Configuration file equivalent [query_job_aws_bucket](./configuration-file.md#query_job_aws_bucket). + +## doltlabdb-admin-password _String_. The `dolthubadmin` SQL user password of the `doltlabdb` instance. @@ -310,7 +328,25 @@ Configuration file equivalent [doltlabremoteapi_only](./configuration-file.md#do _Number_. The port for `doltlabremoteapi`. -Configuration file equivalent [port](#doltlabremoteapi-port). +Configuration file equivalent [port](./configuration-file.md#doltlabremoteapi-port). + +## doltlabremoteapi-storage-aws-bucket + +_String_. The AWS S3 bucket used for storing remote data files. DoltLab Enterprise only. + +Configuration file equivalent [aws_bucket](./configuration-file.md#doltlabremoteapi-aws-bucket). + +## doltlabremoteapi-storage-aws-dynamodb-table + +_String_. The AWS DynamoDb table name used for storing the manifest of remote databases. DoltLab Enterprise only. + +Configuration file equivalent [aws_dynamodb_table](./configuration-file.md#aws_dynamodb_table). + +## doltlabremoteapi-storage-aws-region + +_String_. The AWS region where the DynamoDb table is located. DoltLab Enterprise only. + +Configuration file equivalent [aws_region](./configuration-file.md#doltlabremoteapi-aws-region). ## doltlabui-host diff --git a/packages/doltlab/content/reference/installer/configuration-file.md b/packages/doltlab/content/reference/installer/configuration-file.md index 16fed70b..bb56b3df 100644 --- a/packages/doltlab/content/reference/installer/configuration-file.md +++ b/packages/doltlab/content/reference/installer/configuration-file.md @@ -30,10 +30,19 @@ services: host: "127.0.0.1" port: 9443 csv_port: 9444 + cloud_storage: + aws_region: "us-west-2" + user_import_uploads_aws_bucket: "uploads-bucket" + query_job_aws_bucket: "query-job-bucket" + asyncworker_aws_sqs_queue: "async-queue" doltlabremoteapi: host: "127.0.0.1" port: 50051 file_server_port: 100 + cloud_storage: + aws_region: "us-west-2" + aws_bucket: "data-bucket" + aws_dynamodb_table: "manifest-db" volume_paths: data_volume_path: "/local/path/to/store/remote/data" doltlabfileserviceapi: @@ -355,6 +364,7 @@ _Dictionary_. Configuration options for `doltlabapi`. - [host](#doltlabapi-host) - [port](#doltlabapi-port) - [csv_port](#csv_port) +- [cloud_storage](#doltlabapi-cloud-storage)

host

@@ -395,6 +405,57 @@ services: Command line equivalent [doltlabapi-csv-port](./cli.md#doltlabapi-csv-port). +

cloud_storage

+ +_Dictionary_. Configuration options for persisting `doltlabapi` data to cloud resources. + +- [aws_region](#doltlabapi-aws-region) +- [user_import_uploads_aws_bucket](#user_import_uploads_aws_bucket) +- [query_job_aws_bucket](#query_job_aws_bucket) +- [asyncworker_aws_sqs_queue](#asyncworker_aws_sqs_queue) + +
aws_region
+ +_String_. The AWS region for 'doltlabapi' cloud storage AWS resources, DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabapi: + cloud_storage: + aws_region: "us-east-2" +``` + +Command line equivalent [doltlabapi-aws-region](./cli.md#doltlabapi-aws-region). + +##### user_import_uploads_aws_bucket + +_String_. The name of the S3 bucket used to store user uploaded files, DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabapi: + cloud_storage: + user_import_uploads_aws_bucket: "uploads-bucket" +``` + +Command line equivalent [doltlabapi-user-import-uploads-aws-s3-bucket](./cli.md#doltlabapi-user-import-uploads-aws-s3-bucket). + +##### asyncworker_aws_sqs_queue + +_String_. The name of the SQS queue used for processing asynchronous tasks, DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabapi: + cloud_storage: + asyncworker_aws_sqs_queue: "async-queue" +``` + +Command line equivalent [doltlabapi-asyncworker-aws-sqs-queue](./cli.md#doltlabapi-asyncworker-aws-sqs-queue). + ### doltlabremoteapi _Dictionary_. Configuration options for `doltlabremoteapi`. @@ -403,6 +464,7 @@ _Dictionary_. Configuration options for `doltlabremoteapi`. - [port](#doltlabremoteapi-port) - [file_server_port](#file_server_port) - [volume_paths](#doltlabremoteapi-volume-paths) +- [cloud_storage](#doltlabremoteapi-cloud-storage)

host

@@ -463,6 +525,56 @@ services: Command line equivalent [doltlabremoteapi-data-volume-host-path](./cli.md#doltlabremoteapi-data-volume-host-path). +

cloud_storage

+ +_Dictionary_. Configuration options for persisting `doltlabremoteapi` data to cloud resources. + +- [aws_region](#doltlabremoteapi-aws-region) +- [aws_bucket](#doltlabremoteapi-aws-bucket) +- [aws_dynamodb_table](#aws_dynamodb_table) + +
aws_region
+ +_String_. The AWS region where the DynamoDb table is located. DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabremoteapi: + cloud_storage: + aws_region: "us-west-2" +``` + +Command line equivalent [doltlabremoteapi-storage-aws-region](./cli.md#doltlabremoteapi-storage-aws-region). + +
aws_bucket
+ +_String_. The AWS S3 bucket used for storing remote data files. DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabremoteapi: + cloud_storage: + aws_bucket: "remote-bucket" +``` + +Command line equivalent [doltlabremoteapi-storage-aws-bucket](./cli.md#doltlabremoteapi-storage-aws-bucket). + +##### aws_dynamodb_table + +_String_. The AWS DynamoDb table name used for storing the manifest of remote databases. DoltLab Enterprise only. + +```yaml +# example installer_config.yaml +services: + doltlabremoteapi: + cloud_storage: + aws_dynamodb_table: "manifest-db" +``` + +Command line equivalent [doltlabremoteapi-storage-aws-dynamodb-table](./cli.md#doltlabremoteapi-storage-aws-dynamodb-table). + ### doltlabfileserviceapi _Dictionary_. Configuration options for `doltlabapifileserviceapi`.