Skip to content

Latest commit

 

History

History

vpc-endpoint

README.md

Creating Amazon OpenSearch Serverless in VPC!

opensearch-serverless-vpc-endpoint-arch

This CDK project launches an Amazon OpenSearch Serverless in VPC.

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

(.venv) $ pip install -r requirements.txt

Before synthesizing the CloudFormation, you should set approperly the cdk context configuration file, cdk.context.json.

For example:

{
  "opensearch_iam_user": {
    "user_name": "opss-user",
    "initial_password": "PassW0rd!"
  },
  "collection_name": "search-movies"
}

At this point you can now synthesize the CloudFormation template for this code.

(.venv) $ export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
(.venv) $ export CDK_DEFAULT_REGION=$(aws configure get region)
(.venv) $ cdk synth --all \
              -c vpc_name='your-existing-vpc-name' \
              -c collection_name='collection-name' \
              -c ec2_key_pair_name="your-ec2-key-pair-name(exclude .pem extension)"

Use cdk deploy command to create the stack shown above.

(.venv) $ cdk deploy --all \
             -c vpc_name='your-existing-vpc-name' \
              -c collection_name='collection-name' \
              -c ec2_key_pair_name="your-ec2-key-pair-name(exclude .pem extension)"

To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command.

A note about Service-Linked Role

Some cluster configurations (e.g VPC access) require the existence of the AWSServiceRoleForAmazonOpenSearchServerless Service-Linked Role.

When performing such operations via the AWS Console, this SLR is created automatically when needed. However, this is not the behavior when using CloudFormation. If an SLR(Service-Linked Role) is needed, but doesn’t exist, you will encounter a failure message simlar to:

Before you can proceed, you must enable a service-linked role to give Amazon OpenSearch Service...

To resolve this, you need to create the SLR. We recommend using the AWS CLI:

aws iam create-service-linked-role --aws-service-name observability.aoss.amazonaws.com

ℹ️ For more information, see here.

Clean Up

Delete the CloudFormation stack by running the below command.

(.venv) $ cdk destroy --force --all \
              -c vpc_name='your-existing-vpc-name' \
              -c collection_name='collection-name' \
              -c ec2_key_pair_name="your-ec2-key-pair-name(exclude .pem extension)"

Useful commands

  • cdk ls list all stacks in the app
  • cdk synth emits the synthesized CloudFormation template
  • cdk deploy deploy this stack to your default AWS account/region
  • cdk diff compare deployed stack with current state
  • cdk docs open CDK documentation

Enjoy!

Run Test

Prerequisites

  • In order to upload and search data, log into the opensearch client machine by ssh.

    $ ssh -i ~/.ssh/your-ssh-key.pem ec2-user@ec2-instance-public-ip

  • Configure aws credentials for the Opensearch IAM User.

    [ec2-user@ip-172-31-0-19 ~]$ aws configure --profile opensearch
AWS Access Key ID [None]: ****************46FI
AWS Secret Access Key [None]: ****************FdsE
Default region name [None]: us-east-1
Default output format [None]:

  • At this point, you can run queries to the opensearch endpoint.
    The followings are examples using run_opensearch_query.py in the opensearch client machine.

    [ec2-user@ip-172-31-0-19 ~]$ aws configure --profile opensearch
get-pip.py
run_opensearch_query.py

Upload and search data

  • Create a single index called movies-index

    python3 run_opensearch_query.py \
  --host 1csodkhc1h7yj93iekof.us-east-1.aoss.amazonaws.com \
  --region us-east-1 \
  --profile opensearch \
  --create-index \
  --index-name movies-index

  • Adding a document to an index

    python3 run_opensearch_query.py \
  --host 1csodkhc1h7yj93iekof.us-east-1.aoss.amazonaws.com \
  --region us-east-1 \
  --profile opensearch \
  --index-name movies-index \
  --put-doc

  • Upload Data using Bulk API

    python3 run_opensearch_query.py \
  --host 1csodkhc1h7yj93iekof.us-east-1.aoss.amazonaws.com \
  --region us-east-1 \
  --profile opensearch \
  --index-name movies-index \
  --bulk-load

  • List all the documents in the index

    python3 run_opensearch_query.py \
  --host 1csodkhc1h7yj93iekof.us-east-1.aoss.amazonaws.com \
  --region us-east-1 \
  --profile opensearch \
  --index-name movies-index \
  --search-all

  • Searching for a document

    python3 run_opensearch_query.py \
  --host 1csodkhc1h7yj93iekof.us-east-1.aoss.amazonaws.com \
  --region us-east-1 \
  --profile opensearch \
  --index-name movies-index \
  --search \
  --keyword miller

References