From 15c80a7783f3af1fe0d9743576c9dd4d009588fe Mon Sep 17 00:00:00 2001 From: Frederico Liporace Date: Thu, 26 Jul 2018 14:42:39 -0300 Subject: [PATCH 1/2] Minor editing in deployment instructions; removed reference to previous deployment method --- README.md | 2 +- docs/README.md | 2 +- docs/deploy.md | 35 ----------------------------------- example/README.md | 6 +++++- 4 files changed, 7 insertions(+), 38 deletions(-) delete mode 100644 docs/deploy.md diff --git a/README.md b/README.md index 1d10d83..10149dc 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Development Seed runs an instance of sat-api for the Landsat-8 and Sentinel-2 im ## Documentation -The docuementation includes information on the STAC spec, how to use the API, manage Elasticsearch, as well as how to deploy your own API on AWS. Access the documentation [here](docs) or on [gitbook](https://sat-utils.gitbook.io/sat-api/). +The documentation includes information on the STAC spec, how to use the API, manage Elasticsearch, as well as how to deploy your own API on AWS. Access the documentation [here](docs) or on [gitbook](https://sat-utils.gitbook.io/sat-api/). ## Development diff --git a/docs/README.md b/docs/README.md index e8bee40..43ebba4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,7 @@ https://www.npmjs.com/org/sat-utils * [Home](README.md) * [What is STAC?](stac.md) -* [Deployment](deploy.md) +* [Deployment](../example/README.md) * [API Docs](api.md) * [Manage Elasticsearch](manage.md) * [Changelog](https://github.com/sat-utils/sat-api/blob/sat-api-as-library/CHANGES.md) diff --git a/docs/deploy.md b/docs/deploy.md deleted file mode 100644 index 737ed86..0000000 --- a/docs/deploy.md +++ /dev/null @@ -1,35 +0,0 @@ -# Deploy your own sat-api - -Work is ongoing to simplify the deployment process, but the following steps should get you started: - -1. Make sure you have AWS credentials with necessary access to AWS Lambda and AWS APIGateway (an admin level access will do enough). - -2. To simplify deployment to AWS, we make use of [`kes`](http://devseed.com/kes/), a tool for CloudFormation deployment. It can be installed with: - - $ npm install -g kes - -3. You MUST create a bucket on S3 that is used for storing deployment artifacts and metadata csv files. - -4. Update `.kes/config.yml` and enter the name of the bucket. - -5. If direct access to the elasticsearch instance is needed from fixed IP address, copy `.env.example` to `.env` and add your IP address there. - -6. Deploy the system with `kes`. It can take some time for AWS to create the various resources, so be patient. - - $ kes cf deploy -r us-east-1 - - Replace `us-east-1` with any desired region. This will deploy the CloudFormation stack, which includes: - - * API Gateway - * Lambda functions - * Step Functions - * CloudWatch Rules - * Elasticsearch - * Associated IAM Roles - - Additional calls of `kes cf deploy` will update the existing CloudFormation stack (see below on building the deployment distribution). - -7. Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type. - -The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console. - diff --git a/example/README.md b/example/README.md index 9fe0ebe..be6bef8 100644 --- a/example/README.md +++ b/example/README.md @@ -36,10 +36,14 @@ name-of-my-deployment: Then run this command: - $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile --template node_modules/@sat-utils/api/template --deployment --showOutputs + $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile --template node_modules/@sat-utils/api/template --deployment --showOutputs The command will return the api endpoint that is created by the operation. +Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type. + +The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console. + ### Deploy Devseed's Dev stack This command only works if you have access to Devseed's AWS account From 605a929e4528a6f88d32a95a5318428e231461b4 Mon Sep 17 00:00:00 2001 From: Frederico Liporace Date: Sun, 29 Jul 2018 11:27:53 -0300 Subject: [PATCH 2/2] Requested review --- docs/README.md | 2 +- docs/deploy.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++ example/README.md | 75 +---------------------------------------------- 3 files changed, 76 insertions(+), 75 deletions(-) create mode 100644 docs/deploy.md diff --git a/docs/README.md b/docs/README.md index 43ebba4..e8bee40 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,7 @@ https://www.npmjs.com/org/sat-utils * [Home](README.md) * [What is STAC?](stac.md) -* [Deployment](../example/README.md) +* [Deployment](deploy.md) * [API Docs](api.md) * [Manage Elasticsearch](manage.md) * [Changelog](https://github.com/sat-utils/sat-api/blob/sat-api-as-library/CHANGES.md) diff --git a/docs/deploy.md b/docs/deploy.md new file mode 100644 index 0000000..6cff941 --- /dev/null +++ b/docs/deploy.md @@ -0,0 +1,74 @@ +# Example sat-api Deployment + +This deployment powers https://sat-api.developmentseed.org/search. We use [kes](https://www.npmjs.com/package/kes) to deploy sat-api as an application to AWS. + +For the deployment to work we use a kes template included in the [@sat-utils/api](https://www.npmjs.com/package/@sat-utils/api) package. This package has all the necessary resources needed for a successful deployment of sat-api. + +You can override all the configurations and options in this template by adding override files to the `.kes/config.yml` folder. + +## Install + + $ yarn install + +## Deploy with unpublished code + +If you need to use the latest code on the master branch that is not released to npm yet, or if you need to do live development on an instance deployed to AWS (not recommended), you should follow these steps: + +- Clone this repo and install requirements ([follow](../README.md#local-installation)) +- At the repo root run: `yarn linkall` (this will link packages to your local npm). +- In the deployment repository (e.g. `example` folder) run the link command with the package name you are using: + - `yarn link @sat-utils/api` + - In the `example` folder we have included a shortcut: `yarn linkall` +- Verify packages are linked: `ls -la node_modules/@sat-utils` + - This show an output similar to: `lrwxr-xr-x 1 user staff 29 Jul 11 14:19 api -> ../../../sat-api/packages/api` + +To restore packages from npm just run `yarn`. + +## Deploy an instance + +Make sure the you add a deployment to `.kes/config.yml` by adding the following to the file: + +```yaml +name-of-my-deployment: + stackName: + system_bucket: +``` + +Then run this command: + + $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile --template node_modules/@sat-utils/api/template --deployment --showOutputs + +The command will return the api endpoint that is created by the operation. + +The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console. + +### Deploy Devseed's Dev stack + +This command only works if you have access to Devseed's AWS account + + $ ./node_modules/.bin/kes cf deploy --region us-east-1 --template node_modules/@sat-utils/api/template --deployment dev --profile --showOutputs + +### Deployer Role + +For the CI environment, we use a special IAM role that is assumed by an AWS user. This will allow us to give limited access to the user that is used inside the CI build environment. + +To create the deployer role run: + + $ ./node_modules/.bin/kes cf deploy --kes-folder deployer --profile ds --region us-east-1 --showOutputs + +Then create a user on AWS and give it this policy permission. Replase the value of the resource with the output of the previous command: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Resource": "" + } + ] +} +``` + +When running the deployment command make sure to [include the `--role` flag](.circleci/config.yml#L17). \ No newline at end of file diff --git a/example/README.md b/example/README.md index be6bef8..54c89ed 100644 --- a/example/README.md +++ b/example/README.md @@ -1,76 +1,3 @@ # Example sat-api Deployment -This deployment powers https://sat-api.developmentseed.org/search. We use [kes](https://www.npmjs.com/package/kes) to deploy sat-api as an application to AWS. - -For the deployment to work we use a kes template included in the [@sat-utils/api](https://www.npmjs.com/package/@sat-utils/api) package. This package has all the necessary resources needed for a successful deployment of sat-api. - -You can override all the configurations and options in this template by adding override files to the `.kes/config.yml` folder. - -## Install - - $ yarn install - -## Deploy with unpublished code - -If you need to use the latest code on the master branch that is not released to npm yet, or if you need to do live development on an instance deployed to AWS (not recommended), you should follow these steps: - -- Clone this repo and install requirements ([follow](../README.md#local-installation)) -- At the repo root run: `yarn linkall` (this will link packages to your local npm). -- In the deployment repository (e.g. `example` folder) run the link command with the package name you are using: - - `yarn link @sat-utils/api` - - In the `example` folder we have included a shortcut: `yarn linkall` -- Verify packages are linked: `ls -la node_modules/@sat-utils` - - This show an output similar to: `lrwxr-xr-x 1 user staff 29 Jul 11 14:19 api -> ../../../sat-api/packages/api` - -To restore packages from npm just run `yarn`. - -## Deploy an instance - -Make sure the you add a deployment to `.kes/config.yml` by adding the following to the file: - -```yaml -name-of-my-deployment: - stackName: - system_bucket: -``` - -Then run this command: - - $ ./node_modules/.bin/kes cf deploy --region us-east-1 --profile --template node_modules/@sat-utils/api/template --deployment --showOutputs - -The command will return the api endpoint that is created by the operation. - -Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type. - -The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console. - -### Deploy Devseed's Dev stack - -This command only works if you have access to Devseed's AWS account - - $ ./node_modules/.bin/kes cf deploy --region us-east-1 --template node_modules/@sat-utils/api/template --deployment dev --profile --showOutputs - -### Deployer Role - -For the CI environment, we use a special IAM role that is assumed by an AWS user. This will allow us to give limited access to the user that is used inside the CI build environment. - -To create the deployer role run: - - $ ./node_modules/.bin/kes cf deploy --kes-folder deployer --profile ds --region us-east-1 --showOutputs - -Then create a user on AWS and give it this policy permission. Replase the value of the resource with the output of the previous command: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "sts:AssumeRole", - "Resource": "" - } - ] -} -``` - -When running the deployment command make sure to [include the `--role` flag](.circleci/config.yml#L17). \ No newline at end of file +See [documentation](../docs/deploy.md)