-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #71 from fredliporace/docs
Deployment instructions
- Loading branch information
Showing
3 changed files
with
61 additions
and
91 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,35 +1,74 @@ | ||
| # Deploy your own sat-api | ||
| # Example sat-api Deployment | ||
|
|
||
| Work is ongoing to simplify the deployment process, but the following steps should get you started: | ||
| This deployment powers https://sat-api.developmentseed.org/search. We use [kes](https://www.npmjs.com/package/kes) to deploy sat-api as an application to AWS. | ||
|
|
||
| 1. Make sure you have AWS credentials with necessary access to AWS Lambda and AWS APIGateway (an admin level access will do enough). | ||
| For the deployment to work we use a kes template included in the [@sat-utils/api](https://www.npmjs.com/package/@sat-utils/api) package. This package has all the necessary resources needed for a successful deployment of sat-api. | ||
|
|
||
| 2. To simplify deployment to AWS, we make use of [`kes`](http://devseed.com/kes/), a tool for CloudFormation deployment. It can be installed with: | ||
| You can override all the configurations and options in this template by adding override files to the `.kes/config.yml` folder. | ||
|
|
||
| $ npm install -g kes | ||
| ## Install | ||
|
|
||
| 3. You MUST create a bucket on S3 that is used for storing deployment artifacts and metadata csv files. | ||
| $ yarn install | ||
|
|
||
| 4. Update `.kes/config.yml` and enter the name of the bucket. | ||
| ## Deploy with unpublished code | ||
|
|
||
| 5. If direct access to the elasticsearch instance is needed from fixed IP address, copy `.env.example` to `.env` and add your IP address there. | ||
| If you need to use the latest code on the master branch that is not released to npm yet, or if you need to do live development on an instance deployed to AWS (not recommended), you should follow these steps: | ||
|
|
||
| 6. Deploy the system with `kes`. It can take some time for AWS to create the various resources, so be patient. | ||
| - Clone this repo and install requirements ([follow](../README.md#local-installation)) | ||
| - At the repo root run: `yarn linkall` (this will link packages to your local npm). | ||
| - In the deployment repository (e.g. `example` folder) run the link command with the package name you are using: | ||
| - `yarn link @sat-utils/api` | ||
| - In the `example` folder we have included a shortcut: `yarn linkall` | ||
| - Verify packages are linked: `ls -la node_modules/@sat-utils` | ||
| - This show an output similar to: `lrwxr-xr-x 1 user staff 29 Jul 11 14:19 api -> ../../../sat-api/packages/api` | ||
|
|
||
| $ kes cf deploy -r us-east-1 | ||
|
|
||
| Replace `us-east-1` with any desired region. This will deploy the CloudFormation stack, which includes: | ||
| To restore packages from npm just run `yarn`. | ||
|
|
||
| * API Gateway | ||
| * Lambda functions | ||
| * Step Functions | ||
| * CloudWatch Rules | ||
| * Elasticsearch | ||
| * Associated IAM Roles | ||
| ## Deploy an instance | ||
|
|
||
| Additional calls of `kes cf deploy` will update the existing CloudFormation stack (see below on building the deployment distribution). | ||
| Make sure the you add a deployment to `.kes/config.yml` by adding the following to the file: | ||
|
|
||
| 7. Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type. | ||
| ```yaml | ||
| name-of-my-deployment: | ||
| stackName: <name-of-my-stack> | ||
| system_bucket: <a s3 bucket I have access to> | ||
| ``` | ||
| Then run this command: | ||
| $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile <profile-name> --template node_modules/@sat-utils/api/template --deployment <name-of-deployment> --showOutputs | ||
| The command will return the api endpoint that is created by the operation. | ||
| The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console. | ||
| ### Deploy Devseed's Dev stack | ||
| This command only works if you have access to Devseed's AWS account | ||
| $ ./node_modules/.bin/kes cf deploy --region us-east-1 --template node_modules/@sat-utils/api/template --deployment dev --profile <replace-me> --showOutputs | ||
| ### Deployer Role | ||
| For the CI environment, we use a special IAM role that is assumed by an AWS user. This will allow us to give limited access to the user that is used inside the CI build environment. | ||
| To create the deployer role run: | ||
| $ ./node_modules/.bin/kes cf deploy --kes-folder deployer --profile ds --region us-east-1 --showOutputs | ||
| Then create a user on AWS and give it this policy permission. Replase the value of the resource with the output of the previous command: | ||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "sts:AssumeRole", | ||
| "Resource": "<arn:DeployerRole>" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| When running the deployment command make sure to [include the `--role` flag](.circleci/config.yml#L17). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,72 +1,3 @@ | ||
| # Example sat-api Deployment | ||
|
|
||
| This deployment powers https://sat-api.developmentseed.org/search. We use [kes](https://www.npmjs.com/package/kes) to deploy sat-api as an application to AWS. | ||
|
|
||
| For the deployment to work we use a kes template included in the [@sat-utils/api](https://www.npmjs.com/package/@sat-utils/api) package. This package has all the necessary resources needed for a successful deployment of sat-api. | ||
|
|
||
| You can override all the configurations and options in this template by adding override files to the `.kes/config.yml` folder. | ||
|
|
||
| ## Install | ||
|
|
||
| $ yarn install | ||
|
|
||
| ## Deploy with unpublished code | ||
|
|
||
| If you need to use the latest code on the master branch that is not released to npm yet, or if you need to do live development on an instance deployed to AWS (not recommended), you should follow these steps: | ||
|
|
||
| - Clone this repo and install requirements ([follow](../README.md#local-installation)) | ||
| - At the repo root run: `yarn linkall` (this will link packages to your local npm). | ||
| - In the deployment repository (e.g. `example` folder) run the link command with the package name you are using: | ||
| - `yarn link @sat-utils/api` | ||
| - In the `example` folder we have included a shortcut: `yarn linkall` | ||
| - Verify packages are linked: `ls -la node_modules/@sat-utils` | ||
| - This show an output similar to: `lrwxr-xr-x 1 user staff 29 Jul 11 14:19 api -> ../../../sat-api/packages/api` | ||
|
|
||
| To restore packages from npm just run `yarn`. | ||
|
|
||
| ## Deploy an instance | ||
|
|
||
| Make sure the you add a deployment to `.kes/config.yml` by adding the following to the file: | ||
|
|
||
| ```yaml | ||
| name-of-my-deployment: | ||
| stackName: <name-of-my-stack> | ||
| system_bucket: <a s3 bucket I have access to> | ||
| ``` | ||
| Then run this command: | ||
| $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile <profile-name> --template node_modules/@sat-utils/api/template --deployment <name-of-mystack> --showOutputs | ||
| The command will return the api endpoint that is created by the operation. | ||
| ### Deploy Devseed's Dev stack | ||
| This command only works if you have access to Devseed's AWS account | ||
| $ ./node_modules/.bin/kes cf deploy --region us-east-1 --template node_modules/@sat-utils/api/template --deployment dev --profile <replace-me> --showOutputs | ||
| ### Deployer Role | ||
| For the CI environment, we use a special IAM role that is assumed by an AWS user. This will allow us to give limited access to the user that is used inside the CI build environment. | ||
| To create the deployer role run: | ||
| $ ./node_modules/.bin/kes cf deploy --kes-folder deployer --profile ds --region us-east-1 --showOutputs | ||
| Then create a user on AWS and give it this policy permission. Replase the value of the resource with the output of the previous command: | ||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "sts:AssumeRole", | ||
| "Resource": "<arn:DeployerRole>" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| When running the deployment command make sure to [include the `--role` flag](.circleci/config.yml#L17). | ||
| See [documentation](../docs/deploy.md) |