In the example deployment, for the ease of deployment and demo purposes, all the services have SSL security disabled and are using the default built-in users with passwords.

With NiFi 1.15+ HTTPS is enforced, this requires users to generate their own certificates. Some default publicly availble certificates are available in this repo as part of the demo but users should ALWAYS generate their own in production environment setups.

The Elasticsearch instances are now setup also with certificates, mainly cause this would most likely always be a requirement as part of a production deployment.

IMPORTANT: Please note that the actual security configuration will depend on the requirements of the user/team/organisation planning to use the services stack. The information provided in this README hence should be only considered as a hint and consulted with the key stakeholders before considering any production use.

./security
├── certificates_elasticsearch.env <---- env vars for ES certificates, same vars are used for both native ES & Opensearch
├── certificates_general.env <---------- root-ca env vars
├── certificates_nifi.env <------------- NiFi cert env vars
├── create_es_native_certs.sh <--------- Use this to create certificates for Elasticsearch Native (NOT FOR OPENSEARCH!)
├── create_es_native_credentials.sh <--- Use this after starting up the ES containers to create the base users for ES (NOT FOR OPENSEARCH!)
├── create_keystore.sh <---------------- Used for opensearch node cert generation
├── create_opensearch_admin_cert.sh  <-- Admin certs for Opensearch Kibana
├── create_opensearch_client_cert.sh <-- Generates certificates for client apps to access ES
├── create_opensearch_internal_passwords.sh <- Optional way of generating passwords for Opensearch Admin & Kibana accounts
├── create_opensearch_node_cert.sh <---- Use this to create certificates for the OpenSearch ES nodes
├── create_opensearch_users.sh <-------- Script to set up users for Opensearch after start-up, needs manual execution.
├── create_root_ca_cert.sh <------------ Script for generating root CA, used for NiFi/OpenSearch/Jupyterhub/OCR service
├── database_users.env <---------------- DB users env vars, for both production and samples DB
├── elasticsearch_users.env <----------- OpenSearch/ES native users, used in 'deploy/services.yml' and 'elasticsearch.yml' files for Kibana/ES and 'metricbeat.yml'
├── es_certificates <------------------- This is where OpenSearch/Elasticsearch certificates will go once generated.
├── es_native_cert_generator.sh <------- This is the script used to generate native ES certificates (NOT for Opensearch), used in create_es_native_credentials.sh
├── es_roles <-------------------------- This folder stores Elasticsearch native/Opensearch account roles and role_mappings.
├── nginx_users.env <------------------- Nginx users
├── nifi_certificates <----------------- Location of NiFi cerficiates post-generation.
├── nifi_init_create_user_auth.sh <----- Script used to start the NiFi container for singler user account creation
├── nifi_create_single_user_auth.sh <--- Script used create single user credentials for NiFi (executed inside the container)
├── nifi_toolkit_security.sh <---------- Script for generating NiFi certificates
├── root-ca-truststore.key <------------ all `root-ca` files  are generated by the `create_root_ca_cert.sh` script
├── root-ca.key <------------------------|
├── root-ca.keystore.jks <---------------|
├── root-ca.p12 <------------------------|
├── root-ca.pem <------------------------|
├── root-ca.srl <------------------------|
└── ssl-extensions-x509.cnf <----------- x509 settings used in OpenSearch admin cert and node cert script(s)

The .env files are used to define local env variables that are used in the services.yml file and for certificate generation. The ones that are used and should be modified depending on the deployment are: - certificates_nifi.env - nifi certificates vars - certificates_elasticsearch.env - ES certificate definitions, an important bit here are the ES_INSTANCE_NAME_1/2/3 vars, which control the location of the certificates in the services.yml file and also the location of the certificates in the es_certificates folder. - database_users.env - production and sample DB users, the user should be changed for a production environment - elasticsearch_users.env - all users used for ES native and OpenSearch deployments are declared here.

IMPORTANT: RUN EVERY TIME YOU UPDATE ANY SECURITY ENV VARIABLES.

Assuming you are in the security folder:

1. run cd ../deploy
2. run source export_env_vars.sh <-- needed to set the env vars if you have modified them in the above files.
3. run cd ../security

Assuming that one needs to generate self-signed certificates for the services, there are provided some useful scripts:

• create_root_ca_cert.sh - creates root CA key and certificate, used for NiFi, MedCAT service, Jupyterhub, ocr-service etc.
• create_opensearch_client_cert.sh - creates the client key and certificate for external apps
• create_keystore.sh - creates the JKS keystore using previously generated (client) certificates, used in create_opensearch_node_cert.sh
• create_opensearch_users.sh - creates system users for OpenSearch instances, to be used after finishing the container startup(s)
• create_opensearch_admin_cert.sh - creates certs for OpenSearch Dashboard (Kibana)
• create_opensearch_node_cert.sh - creates certificates for OpenSearch nodes
• create_es_native_certs.sh - creates certificates for pure Elasticsearch (ES native) nodes only

Using create_root_ca_cert.sh the key files that are generated are:

• key: root-ca.key
• certificate: root-ca.pem
• keystore: root-ca.keystore.jks
• p12 cert: root-ca.p12
• pem cert: root-ca.pem

Configure certificate settings for NiFi in certificates_nifi.env and for the root CA in certificates_general.env.

Assuming you are in the security folder:

1. run cd ../deploy
2. run source export_env_vars.sh <-- needed to set the env vars if you have modified them in the above files.
3. run cd ../security
4. run bash create_root_ca_cert.sh
5. run bash nifi_toolkit_security.sh

You must run them in the above order as the root CA is required by the NiFi toolkit.

Follow the instructions carefully, there are a few sections detailing the differences between Elastic versions.

Each version has it's own scripts for generating the necessary certificates. All security variables used within the .sh scripts for CERTIFICATE GENERATION are set in the following files:

• ./certificates_elasticsearch.env
• ./certificates_general.env
• ./certificates_nifi.env

Please pay attention to the following sections, they describe what is needed to secure each version of ES deployments(Opensearch/Native ES).

Certificate namings are now common across ES versions, the deployment requires the following certificates, available in the security folder:

• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elastic-stack-ca.crt.pem
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elastic-stack-ca.key.pem
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-1/http-elasticsearch-1.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-1/http-elasticsearch-1.p12
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-1/http-elasticsearch-1.key
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-2/http-elasticsearch-2.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-2/http-elasticsearch-2.key
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-2/http-elasticsearch-2.p12
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-3/http-elasticsearch-3.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-3/http-elasticsearch-3.p12
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-3/http-elasticsearch-3.key

The ${ELASTICSEARCH_VERSION} MUST be set in the deploy/elastiscsearch.env before starting any container! it will also mount all the certificates seaminglessly according to the ES version, for native ES the certificate files are in security/es_certicates/elasticsearch, OpenSearch variant in security/es_certificates/opensearch/.

For information on OpenSearch security features and their configuration please refer to the official documentation.

We have to make sure to execute the following commands bash ./create_opensearch_node_cert.sh elasticsearch-1 && bash ./create_opensearch_node_cert.sh elasticsearch-2 && bash ./create_opensearch_node_cert.sh elasticsearch-3 this will generate the certificates for all 3 nodes, make sure to generate the ADMIN authorization certificate by doing bash ./create_opensearch_admin_cert.sh.

The keystore/truststore certificates are also generated when creating the node certificates, these are used in the NiFi workflows.

We also provide as part of our deployment the native Elastisearch version since it is used across many organisations in production environments documentation. Please note that the deployment of native ES version requires different settings to be changed from the current repository state.

To generate the above certificates all that is needed is to run the create_es_native_certs.sh.

There are a few variables related to the certificate names, pleas read the following carefully:

• ES_INSTANCE_NAME_1, this variable is usually set to the same name as ELASTICSEARCH_NODE_1_NAME from /deploy/elasticsearch.env, it is used to determine the certificate paths, and also in the certificate hostname SUBJ lines, there are two other vars with the same name aside from the numbering for each node.
• ES_INSTANCE_ALTERNATIVE_1_NAME, this is used along with ES_INSTANCE_NAME_1 to provide additional hostnames forr the certificate generation, also useful incase the node name is different from the elastic search hostname.
• ES_HOSTNAMES, set all your hostnames here, they should include the names of the nodes and also additional hostnames & DNS-es, please follow the exact indentation as it is in the .env file. If it does not work, then manually do : export ES_HOSTNAMES="- elasticsearch-1
- elasticsearch-2
- elasticsearch-3
"
• ES_CLIENT_SUBJ_ALT_NAMES and ES_NODE_SUBJ_ALT_NAMES, set these with additional domain names as needed, both client and node should have the nodes and the kibana hostname instances added.

• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-1/elasticsearch-1.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-1/elasticsearch-1.key
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-2/elasticsearch-2.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-2/elasticsearch-2.key
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-3/elasticsearch-3.crt
• es_certificates/${ELASTICSEARCH_VERSION}/elasticsearch/elasticsearch-3/elasticsearch-3.key
• es_certificates/ca/ca.crt

These certificates are generates by the steps mentioned in the above Elasticsearch Native section.

OpenDashboard requires:

• admin.pem
• admin-key.pem
• es_kibana_client.pem
• es_kibana_client.key
• like the Kibana section above, all the certificates are used under the same names, of course, they will come from es_certificates/opensearch/ folder.

Once generated, the files can be further referenced in services/kibana/config/kibana_opensearch.yml and/or linked directly in the Docker compose file with services configuration.

The sample users and passwords are specified in the following .env files in security/ directory:

• elasticsearch_users.env - contains passwords for ElasticSearch internal users.
• database_users.env - containes account details for both production and samples DB instances
• nginx_users.env - nginx account

Please see the security/opensearch folder for the roles mappings and internal users for user data. You can also use the create_es_users.sh script for this.

On the first run, after changing the default passwords, one should change the default admin and kibanaserver passwords as specified in the OpenSearch documentation.

To do so, one can:

• run the script generate_opensearch_internal_passwords.sh to generate hashes,
• modify the internal_users.yml file with the generated hashes,
• restart the stack, but with using docker-compose down -v to remove the volume data.

Following, one should modify the default passwords for the other build-in users (logstash, kibanaro, readall, snapshotrestore) and to create custom users (cogstack_pipeline, cogstack_user, nifi), as specified below. The script create_es_users.sh creates and sets up example users and roles in ElasticSearch cluster.

For configuring default users, please see the following env files:

• ./elasticsearch_users.env which is used in the create_es_native_credentials.sh script post ES container startup, it creates all the default users. If you wish to add more users make sure to take a look at the official documentation on how to create roles and accounts.
• This script also creates a SERVICE ACCOUNT TOKEN which can be used for Kibana configuration. Please copy the token manually into the elasticsearch.env ELASTICSEARCH_SERVICE_ACCOUNT_TOKEN variable.

Example new roles that will be created after running create_es_users.sh:

• ingest - used for data ingestion, only cogstack_* and nifi_* indices can be used,
• cogstack_accesss - used for read-only access to the data only from cogstack_* and nifi_* indices.

Example new users will be created after running create_es_users.sh:

• cogstack_pipeline - uses ingest role (deprecated),
• nifi - uses ingest role,
• cogstack_user - uses cogstack_access role.

Similarly, as in case of ELK stack, one should obtain certificates for JupyterHub to secure the access to the exposed endpoint. The generated certificates (by create_root_ca_cert.sh) can be referenced directly in services.yml file in the example deployment or directly in the internal JupyterHub configuration file. The COOKIE secret is a key used to encrypt browser cookies, please use the generate_cookie_secret.sh(./services/jupyter-hub/generate_cookie_secret.sh) script to generate a new key, make sure it is done before starting the container.

One should also configure and set up users, since the default user is admin, and the password is set the first time the account is logged in to (be careful, if there is a mistake delete the jupyter container and its volumes and restart). See example deployment services for more details.

Once the container is started up you can create your users and also assing them to groups.

You can create users before hand by adding newlines in the userlist(services/jupyter-hub/config/userlist) file, users with admin roles will need to have their role specificed on the same line, e.g: user_name admin.

If you want to create shared folder for users to use add them to the teamlist(services/jupyter-hub/config/teamlist) file, the first column is the shared folder name and the rest are just the usernames assigned to it.

For more information on JupyterHub security features and their configuration please refer to the official documentation.

For securing Apache NiFi endpoint with self-signed certificates please refer to the official documentation.

Regarding connecting to services that use self-signed certificates (such as Elasticsearch), it is required that these certificates use JKS keystore format. The certificates can be generated using create_keystore.sh. Usage: bash create_keystore.sh <cert_name> <jks_store> | the password is optional.

Before starting the NIFI container it's important to take note of the following things if we wish to enable HTTPS functionality:

• this step is optional (as you might have done it before from configuring other certificates), run create_root_ca_cert.sh to create the ROOT certificates, these will be used by NiFi/OpenSearch/OCR_service/Tika/MedcatService/Jupyterhub etc.

• the nifi_toolkit_security.sh script is used to download the nifi toolkit and generate new certificates and keys that are used by the container, take note that inside the localhost folder there is another nifi.properties file that is generated, we must look to the following setttings which are generated randomly and copy them to the nifi/conf/nifi.properties file.

• the trust/store keys generated for production will be in the nifi_certificates/localhost folder and the nifi-cert.pem + nifi-key.key files. in the baes nifi_certificates folder.

• as port of the security process the nifi.sensitive.props.key should be set to a random string or a password of minimum 12 characters. Once this is set do NOT modify it as all the other sensitive passwords will be hashed with this string. By default this is set to cogstackNiFipass

Example:

    nifi.security.keystorePasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
    nifi.security.keyPasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
    nifi.security.truststore=./conf/truststore.jks
    nifi.security.truststoreType=jks
    nifi.security.truststorePasswd=lzMGadNB1JXQjgQEnFStLiNkJ6Wbbgw0bFdCTICKtKo

This is entirely optional, if you have configered the security certs as described in security/README.md then you are good to go.
Default username :

username: admin     
password: cogstackNiFi

• the login-identity-providers.xml file in /nifi/conf/ stores the password for the user account, to generate a password one must use the following command within the container : /opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials USERNAME PASSWORD, once done, you would need to copy the file from /opt/nifi/nifi-current/conf/login-identity-providers.xml locally with docker cp and replace the one in the nifi/conf folder and rebuild the container.

• alternative to the above step: go into the /security folder, set the desired nifi username & password in the /security/nifi_users.env file. Make sure to STOP any running NiFi containers docker stop cogstack-nifi and execute the following script: bash /security/nifi_init_create_user_auth.sh, this script will start a NiFi container for the time of the account creation and then remove itself, after it finishes, go back to the /deploy folder and start your NiFi container, all should be working!

URL: https://localhost:8443/nifi/login

Troubleshooting Security : if you encounter errors related to sensitive key properties not being set please clear/delete the docker volumes of the nifi container or delete all volumes of inactive containers docker volume prune.

If for some reason you do not wish to authenticate every time you connect to NiFi, you can enable the client certificates in the nginx.conf line 86-87 and delete the commented lines.

Alternatively, one can secure the access to selected services by using NGINX reverse proxy. This may be essential in case some of the web services that need to be exposed to end-users do not offer SSL encryption. See the official documentation for more details on using NGINX for that.

Nginx only requires the root-CA certificate by default, so use the above generate cert section to create it.

In order to be able to properly access the nifi instance securely, you also need to start the nifi-nginx container as it is configured to provide access from any source to nifi, available at https://localhost:8443/nifi .