Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infrastructure documentation improvements #1640

Merged
merged 4 commits into from
Dec 9, 2024
Merged

Infrastructure documentation improvements #1640

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
18eee97
Infrastructure documentation improvements
ekerfelt Dec 9, 2024
d829408
Fixes for feedback
ekerfelt Dec 9, 2024
5ffad36
Replace all Encore references with Encore Cloud
ekerfelt Dec 9, 2024
c773aa6
Merge branch 'main' into infra-docs
ekerfelt Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 84 additions & 27 deletions docs/go/self-host/configure-infra.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ seotitle: Configure Infrastructure
seodesc: Learn how to configure infrastructure resources for your Encore app.
lang: go
---

If you are using infrastructure resources, such as SQL databases, Pub/Sub, or metrics, you will need to configure your Docker image with the necessary configuration.
The `build` command lets you provide this by specifying a path to a config file using the `--config` flag.

Expand All @@ -20,7 +19,7 @@ This supports configuring things like:
- How to call other services over the network ("service discovery"),
most notably their base URLs.
- Observability configuration (where to export metrics, etc.)
- Metadata about the environment the application is running in, to power Encore's [metadata APIs](/docs/go/develop/metadata).
- Metadata about the environment the application is running in, to power Encore's [metadata APIs](/docs/ts/develop/metadata).
- The values for any application-defined secrets.

This configuration is necessary for the application to behave correctly.
Expand Down Expand Up @@ -56,7 +55,7 @@ Here's an example configuration file you can use.
}
},
"redis": {
"encoreredis": {
"my-redis": {
"database_index": 0,
"auth": {
"type": "acl",
Expand Down Expand Up @@ -88,7 +87,7 @@ Here's an example configuration file you can use.
"type": "gcp_pubsub",
"project_id": "my-project",
"topics": {
"encore-topic": {
"my-topic": {
"name": "gcp-topic-name",
"subscriptions": {
"encore-subscription": {
Expand Down Expand Up @@ -181,23 +180,22 @@ configured when the services are started.
```json
{
"service_discovery": {
"user-service": {
"base_url": "https://user.myencoreapp.com",
"myservice": {
"base_url": "https://myservice.myencoreapp.com",
"auth": [
{
"type": "key",
"id": 1,
"key": {
"$env": "USER_SERVICE_API_KEY"
"$env": "MY_SERVICE_API_KEY"
}
}
]
}
}
}
```

- `user-service`: Configuration for a service named `user-service`.
- `myservice`: This is the name of the service as it is declared in your Encore app.
- `base_url`: The base URL for the service.
- `auth`: Authentication methods used for accessing the service. If no authentication methods are specified, the service will use the auth methods defined in the `auth` section.

Expand Down Expand Up @@ -285,7 +283,7 @@ There must be exactly one database configuration for each declared database. You
"ca": "---BEGIN CERTIFICATE---\n..."
},
"databases": {
"main_db": {
"my-database": {
"max_connections": 100,
"min_connections": 10,
"username": "db_user",
Expand All @@ -299,6 +297,7 @@ There must be exactly one database configuration for each declared database. You
}
```

- `my-database`: This is the name of the database as it is declared in your Encore app.
- `host`: SQL server host, optionally including the port.
- `tls_config`: TLS configuration for secure connections. If the server uses TLS with a non-system CA root, or requires a client certificate, specify the appropriate fields as PEM-encoded strings. Otherwise, they can be left empty.
- `databases`: List of databases, each with connection settings.
Expand All @@ -319,6 +318,8 @@ You can set the secret value directly in the configuration file, or use an envir
}
```

- `API_TOKEN`: This is the name of a secret as it is declared in your Encore app.

#### 7.2. Using Environment Reference
As an alternative, you can use an environment variable reference to set the secret value. The env variable should be set in the environment where the application is running. The content
of the environment variable should be a JSON string where each key is the secret name and the value is the secret value.
Expand All @@ -336,7 +337,7 @@ of the environment variable should be a JSON string where each key is the secret
```json
{
"redis": {
"cache": {
"my-redis": {
"host": "redis.myencoreapp.com:6379",
"database_index": 0,
"auth": {
Expand All @@ -352,6 +353,7 @@ of the environment variable should be a JSON string where each key is the secret
}
```

- `my-redis`: This is the name of the redis resource as it is declared in your Encore app.
- `host`: Redis server host, optionally including the port.
- `auth`: Authentication configuration for the Redis server.
- `key_prefix`: Prefix applied to all keys.
Expand All @@ -373,14 +375,14 @@ The configuration for each provider is different. Below are examples for each pr
"type": "gcp_pubsub",
"project_id": "my-gcp-project",
"topics": {
"user-events": {
"name": "user-events-topic",
"my-topic": {
"name": "my-topic",
"project_id": "my-gcp-project",
"subscriptions": {
"user-notification": {
"name": "user-notification-subscription",
"my-subscription": {
"name": "my-subscription",
"push_config": {
"id": "user-push",
"id": "my-push",
"service_account": "service-account@my-gcp-project.iam.gserviceaccount.com"
}
}
Expand All @@ -392,6 +394,13 @@ The configuration for each provider is different. Below are examples for each pr
}
```

- `my-topic`: This is the name of the topic as it is declared in your Encore app.
- `my-subscription`: This is the name of the subscription as it is declared in your Encore app.
- `project_id`: The default GCP project ID. This can be overridden by setting the `project_id` field in the topic or subscription.
- `name`: The name of the topic or subscription.
- `push_config/id`: The id will be appended to `/__encore/pubsub/push/` to form the full push path of your service, e.g. `/__encore/pubsub/push/<id>`. This is the path your service expects to receive push messages on.
- `push_config/service_account`: The service account configured for the push subscription.

#### 9.2. AWS SNS/SQS

```json
Expand All @@ -400,11 +409,11 @@ The configuration for each provider is different. Below are examples for each pr
{
"type": "aws_sns_sqs",
"topics": {
"user-notifications": {
"arn": "arn:aws:sns:us-east-1:123456789012:user-notifications",
"my-topic": {
"arn": "arn:aws:sns:us-east-1:123456789012:my-topic",
"subscriptions": {
"user-queue": {
"arn": "arn:aws:sqs:us-east-1:123456789012:user-queue"
"my-queue": {
"arn": "arn:aws:sqs:us-east-1:123456789012:my-queue"
}
}
}
Expand All @@ -414,6 +423,10 @@ The configuration for each provider is different. Below are examples for each pr
}
```

- `my-topic`: This is the name of the topic as it is declared in your Encore app.
- `my-queue`: This is the name of the queue as it is declared in your Encore app.
- `arn`: The ARN of the SNS topic or SQS queue.

#### 9.3. NSQ Configuration

```json
Expand All @@ -423,11 +436,11 @@ The configuration for each provider is different. Below are examples for each pr
"type": "nsq",
"hosts": "nsq.myencoreapp.com:4150",
"topics": {
"order-events": {
"name": "order-events-topic",
"my-topic": {
"name": "my-topic",
"subscriptions": {
"order-processor": {
"name": "order-processor-subscription"
"my-subscription": {
"name": "my-subscription"
}
}
}
Expand All @@ -437,10 +450,13 @@ The configuration for each provider is different. Below are examples for each pr
}
```

- `my-topic`: This is the name of the topic as it is declared in your Encore app.
- `my-subscription`: This is the name of the subscription as it is declared in your Encore app.

### 10. Object Storage Configuration
Encore currently supports the following object storage providers:
- `gcs` for [Google Cloud Storage](https://cloud.google.com/storage)
- `s3` for [AWS S3](https://aws.amazon.com/s3/)
- `s3` for [AWS S3](https://aws.amazon.com/s3/) or a custom S3-compatible provider

#### 10.1. GCS Configuration

Expand All @@ -451,15 +467,20 @@ Encore currently supports the following object storage providers:
"type": "gcs",
"buckets": {
"my-gcs-bucket": {
"name": "my-gcs-bucket"
"name": "my-gcs-bucket",
"key_prefix": "my-optional-prefix/",
"public_base_url": "https://my-gcs-bucket-cdn.example.com/my-optional-prefix"
}
}
}
]
}
```

- `my-gcs-bucket`: This is the name of the bucket as it is declared in your Encore app.
- `name`: The full name of the GCS bucket.
- `key_prefix`: An optional prefix to apply to all keys in the bucket.
- `public_base_url`: A URL to use for public access to the bucket. This field is required if you configure your bucket to be public. Encore will append the object key to this URL when generating public URLs. The optional prefix will not be appended.

#### 10.2. S3 Configuration

Expand All @@ -471,15 +492,51 @@ Encore currently supports the following object storage providers:
"region": "us-east-1",
"buckets": {
"my-s3-bucket": {
"name": "my-s3-bucket"
"name": "my-s3-bucket",
"key_prefix": "my-optional-prefix/",
"public_base_url": "https://my-gcs-bucket-cdn.example.com/my-optional-prefix"
}
}
}
]
}
```

- `my-s3-bucket`: This is the name of the bucket as it is declared in your Encore app.
- `region`: The AWS region where the bucket is located.
- `name`: The full name of the S3 bucket.
- `key_prefix`: An optional prefix to apply to all keys in the bucket.
- `public_base_url`: A URL to use for public access to the bucket. This field is required if you configure your bucket to be public. Encore will append the object key to this URL when generating public URLs. The optional prefix will not be appended.

#### 10.3. Custom S3 Provider Configuration
You can also configure a custom S3 provider by specifying the endpoint, access key id, and secret access key. Custom S3 providers are useful if you are using a S3-compatible storage provider such as [Cloudflare R2](https://developers.cloudflare.com/r2/).
```json
{
"object_storage": [
{
"type": "s3",
"region": "auto",
"endpoint": "https://...",
"access_key_id": "...",
"secret_access_key": {
"$env": "BUCKET_SECRET_ACCESS_KEY"
},
"buckets": {
"my-custom-bucket": {
"name": "my-custom-bucket",
"key_prefix": "my-optional-prefix/",
"public_base_url": "https://my-gcs-bucket-cdn.example.com/my-optional-prefix"
}
}
}
]
}
```

- `my-custom-bucket`: This is the name of the bucket as it is declared in your Encore app.
- `region`: The region where the bucket is located.
- `name`: The full name of the bucket
- `key_prefix`: An optional prefix to apply to all keys in the bucket.
- `public_base_url`: A URL to use for public access to the bucket. This field is required if you configure your bucket to be public. Encore will append the object key to this URL when generating public URLs. The optional prefix will not be appended.

This guide covers typical infrastructure configurations. Adjust according to your specific requirements to optimize your Encore app's infrastructure setup.
27 changes: 21 additions & 6 deletions docs/menu.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1003,9 +1003,19 @@
file: "platform/infrastructure/own-cloud"
}, {
kind: "basic"
text: "Infrastructure provisioning"
text: "Infrastructure overview"
path: "/platform/infrastructure/infra"
file: "platform/infrastructure/infra"
}, {
kind: "basic"
text: "GCP Infrastructure"
path: "/platform/infrastructure/gcp"
file: "platform/infrastructure/gcp"
}, {
kind: "basic"
text: "AWS Infrastructure"
path: "/platform/infrastructure/aws"
file: "platform/infrastructure/aws"
}, {
kind: "accordion"
text: "Kubernetes deployment"
Expand All @@ -1025,16 +1035,21 @@
path: "/platform/infrastructure/configure-kubectl"
file: "platform/infrastructure/configure-kubectl"
}]
}, {
kind: "basic"
text: "Managing database users"
path: "/platform/infrastructure/manage-db-users"
file: "platform/infrastructure/manage-db-users"
}, {
kind: "basic"
text: "Neon Postgres"
path: "/platform/infrastructure/neon"
file: "platform/infrastructure/neon"
}, {
kind: "basic"
text: "Cloudflare R2"
path: "/platform/infrastructure/cloudflare"
file: "platform/infrastructure/cloudflare"
}, {
kind: "basic"
text: "Managing database users"
path: "/platform/infrastructure/manage-db-users"
file: "platform/infrastructure/manage-db-users"
}]
}, {
kind: "section"
Expand Down
Loading
Loading