Skip to content

Using OpenSearch with Security Enabled

Jump to bottom
Ramesh Maddegoda edited this page Mar 3, 2022 · 3 revisions

Changes Required to use OpenSearch (instead of Elasticsearch) with Security Enabled

Introduction

This page explains the code/config changes required to use OpenSearch (instead of Elasticsearch) with security enabled.

The code/config changes are required in following repositories.

  1. https://github.com/NASA-PDS/registry-harvest-service
  2. https://github.com/NASA-PDS/registry-loader
  3. https://github.com/NASA-PDS/registry

Relevant branches and DRAFT pull requests

  1. https://github.com/NASA-PDS/registry-harvest-service branch: ppds-registry-app-250-opensearch-test

  2. https://github.com/NASA-PDS/registry-loader branch: pds-registry-app-250-opensearch-test

  3. https://github.com/NASA-PDS/registry branch: pds-registry-app-250-opensearch-test

Registry Harvest Service

Update the curl command in docker/entrypoint.sh file as follows (with options to pass user name and password)

echo "Waiting for Elasticsearch registry path to be available..."  1>&2
# TODO Warning: Use the default username and password only for testing purposes in local setup
while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' "${ES_URL}"/registry  -u 'admin:admin' --insecure)" != "200" ]]; do
  sleep 1
done

Registry Loader

  1. Update the registry-manager load-data command in the docker/entrypoint.sh to pass -auth /etc/es-auth.cfg.
# Load data into Elasticsearch
echo "Loading harvested data into Elasticsearch ..." 1>&2
registry-manager load-data -dir /tmp/harvest/out/ -es "$ES_URL" -force -auth /etc/es-auth.cfg
  1. Update the docker/test/cfg/harvest-test-config.xml to pass auth="/etc/es-auth.cfg" to registry
<?xml version="1.0" encoding="UTF-8"?>
<harvest nodeName="PDS_ENG">
  <directories>
    <path>/data</path>
  </directories>
  <registry url="https://elasticsearch:9200" index="registry" auth="/etc/es-auth.cfg" />
  <autogenFields/>
</harvest>

Registry

  1. Update following variables in docker/.env.
# Docker image of Elasticsearch/OpenSearch
ES_IMAGE=opensearchproject/opensearch:1.1.0

# Elasticsearch URL (the host name is the Elasticsearch service name specified in the docker compose)
ES_URL=https://elasticsearch:9200
  1. Add a directory to keep certificates as docker/certs and create following files in the directory. This file can be used to generate self signed certificate in docker/certs/ directory.

docker/certs/generate-certs.sh

#!/bin/sh

# ----------------------------------------------------------------------------------------------------------------
#
# This script is used to generate certificates required to enable https in OpenSearch. Please note that the
# CN=elasticsearch should match with the hostname of the OpenSearch/Elasticsearch.
#
# Read more information at: https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/
#
# ----------------------------------------------------------------------------------------------------------------

# Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=CA/ST=CALIFORNIA/L=LA/O=ORG/OU=PDS/CN=elasticsearch" -out root-ca.pem -days 730

# Node cert 1
openssl genrsa -out node1-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem
openssl req -new -key node1-key.pem -subj "/C=CA/ST=CALIFORNIA/L=LA/O=ORG/OU=PDS/CN=elasticsearch" -out node1.csr
openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node1.pem -days 730

# Cleanup
rm node1-key-temp.pem
rm node1.csr

docker/certs/.gitignore

# Ignore everything in this directory
*
# Except following files
!.gitignore
!generate-certs.sh
  1. Update the following properties in docker/default-config/application.properties.
elasticSearch.username=admin
elasticSearch.password=admin
elasticSearch.ssl=true
  1. Add new file docker/default-config/es-auth.cfg.
trust.self-signed = true
# TODO Warning: Use the default username and password only for testing purposes in local setup
user = admin
password = admin
  1. Update the docker/default-config/harvest-job-config.xml to pass auth="/etc/es-auth.cfg" to registry url.
<harvest nodeName="PDS_ENG">
  <directories>
    <path>/data</path>
  </directories>
  <registry url="https://elasticsearch:9200" index="registry" auth="/etc/es-auth.cfg" />
  <autogenFields/>
</harvest>
  1. Update the following properties in docker/default-config/harvest-server.cfg.
# Elasticsearch URL
es.url = https://elasticsearch:9200

# Elasticsearch authentication file
es.authFile = /etc/es-auth.cfg
  1. Add new file docker/default-config/opensearch.yml.
cluster.name: docker-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
#discovery.type: single-node

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=elasticsearch,OU=client,O=client,L=test, C=us

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
  1. Update the following services in the docker/docker-compose.yml.
  # Executes the Registry Loader component, which contains Harvest and Registry Manager
  registry-loader:
    profiles: ["pds-batch-loader"]
    image: ${REG_LOADER_IMAGE}
    environment:
      - ES_URL=${ES_URL}
    volumes:
      - ${HARVEST_JOB_CONFIG_FILE}:/cfg/harvest-config.xml
      - ${HARVEST_DATA_DIR}:/data
      - ./default-config/es-auth.cfg:/etc/es-auth.cfg
    networks:
      - pds

 # Downloads and harvests test data with the Registry loader
  registry-loader-test-init:
    profiles: ["dev-api", "int-registry-batch-loader"]
    image: ${REG_LOADER_IMAGE}
    environment:
      - ES_URL=${ES_URL}
      - RUN_TESTS=true
      - TEST_DATA_URL=${TEST_DATA_URL}
    volumes:
      - ./scripts/registry-loader-waits-for-elasticsearch.sh:/usr/local/bin/registry-loader-waits-for-elasticsearch.sh
      - ./default-config/es-auth.cfg:/etc/es-auth.cfg
    networks:
      - pds
    entrypoint: /usr/local/bin/registry-loader-waits-for-elasticsearch.sh

 # Starts Elasticsearch
  elasticsearch:
    profiles: ["dev-api", "pds-core-registry", "int-registry-batch-loader", "int-registry-service-loader"]
    image: ${ES_IMAGE}
    environment:
      - discovery.type=${ES_DISCOVERY_TYPE}
    # Uncomment the following environment variable to use OpenSearch without security
    # - plugins.security.disabled=true
    volumes:
      - ./certs/node1.pem:/usr/share/opensearch/config/esnode.pem
      - ./certs/node1-key.pem:/usr/share/opensearch/config/esnode-key.pem
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./default-config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
    ports:
      - "9200:9200"
      - "9600:9600" # required for Performance Analyzer
    networks:
      - pds

 # Initializes Elasticsearch by creating registry and data dictionary indices by utilizing the Registry Loader
  elasticsearch-init:
    profiles: ["dev-api", "pds-core-registry", "int-registry-batch-loader", "int-registry-service-loader"]
    image: ${REG_LOADER_IMAGE}
    environment:
      - ES_URL=${ES_URL}
    volumes:
      - ./scripts/elasticsearch-init.sh:/usr/local/bin/elasticsearch-init.sh
      - ./default-config/es-auth.cfg:/etc/es-auth.cfg
    networks:
      - pds
    entrypoint: ["bash", "/usr/local/bin/elasticsearch-init.sh"]
@@ -121,6 +131,7 @@ services:
    volumes:
      - ${HARVEST_SERVER_CONFIG_FILE}:/cfg/harvest-server.cfg
      - ${HARVEST_DATA_DIR}:/data
      - ./default-config/es-auth.cfg:/etc/es-auth.cfg
    networks:
      - pds
  1. Update curl command and registry-manger command in docker/scripts/elasticsearch-init.sh as follows to pass credentials.
echo "Waiting for Elasticsearch to launch..."  1>&2
# TODO Warning: Use the default username and password only for testing purposes in local setup
while ! curl --output /dev/null --silent --head --fail "$ES_URL" -u 'admin:admin' --insecure; do
  sleep 1
done

echo "Creating registry and data dictionary indices..." 1>&2
registry-manager create-registry -es "$ES_URL" -auth /etc/es-auth.cfg
  1. Update curl command in docker/scripts/registry-api-waits-for-elasticsearch.sh as follows to pass credentials.
echo "Waiting for Elasticsearch to launch..."  1>&2
# TODO Warning: Use the default username and password only for testing purposes in local setup
while ! curl --output /dev/null --silent --head --fail "$ES_URL" -u 'admin:admin' --insecure; do
  echo "waiting for elasticsearch" 1>&2
  sleep 1
done
  1. Update curl command in docker/scripts/registry-loader-waits-for-elasticsearch.sh as follows to pass credentials.
echo "Waiting for Elasticsearch to launch..."  1>&2
# TODO Warning: Use the default username and password only for testing purposes in local setup
while ! curl --output /dev/null --silent --head --fail "$ES_URL" -u 'admin:admin' --insecure; do
  echo "waiting for elasticsearch" 1>&2
  sleep 1
done
Clone this wiki locally