Tethr Custom ARM Templates for Elasticsearch cluster (forked from elastic/azure-marketplace)

This forked repo has been highly-customized in order to support the following items--

• Availability Zones - The templates have been designed to allow for the deployment of Elasticsearch nodes to Zones 1, 2, and 3.

• VM Scale Sets - The templates have been designed to allow for the deployment of dedicated Client (Coordinating) nodes and Ingest nodes as VM Scale Sets.

• Ingest nodes - Dedicated Ingest nodes can be deployed as a VM Scale Set.

Deploying the Tethr custom ARM templates

To deploy, use the Powershell script \src\new-estemplate.ps1.

New parameters have been added to support the new features.

• nodeType
• vmId
• zoneId
• scaleSetInstanceCount

See full details in Parameters table below. Each is labeled as Tethr CUSTOM

Elasticsearch Azure Marketplace offering (Original description from forked repo. Includes Tethr custom templates)

WARNING Though much of the functionality of the original MarketPlace templates remain intact, much has changed; therefore, not all of the original README text below is still accurate.

This repository consists of:

• src/mainTemplate.json - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates with the main template acting as the entry point.
• src/createUiDefinition - Not supported for Tethr Custom. UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters.

Building

After pulling the source, call npm install once to pull in all devDependencies.

You may edit build/allowedValues.json, which the build will use to patch the ARM template and Marketplace UI definition.

Run npm run build; this will validate EditorConfig settings, JSON files, patch the allowedValues and create a zip in the dist folder.

For more details around developing the template, take a look at the Development README

Azure Marketplace

The Azure Marketplace Elasticsearch offering offers a simplified UI and installation experience over the full power of the ARM template.

It will always bootstrap a cluster complete with a trial license of Elastic's commercial X-Pack plugins.

Did you know that you can apply for a free Basic X-Pack license? Go check out our subscription options

Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch (on Azure) but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs. Check out the examples.

---

VERY IMPORTANT

By default, this template does not configure SSL/TLS for communication with Elasticsearch through an external load balancer. It is strongly recommended that you secure communication before using in production.

You can secure external access to the cluster with TLS by using gateway as the loadBalancerType and supplying a PFX certificate with the appGatewayCertBlob parameter. This sets the cluster up to use Application Gateway for load balancing and SSL offload.

You can secure external access from the browser to Kibana with TLS by supplying a certificate and private key with kibanaCertBlob and kibanaKeyBlob, respectively.

---

You can view the UI in developer mode by clicking here. If you feel something is cached improperly use this client unoptimized link instead

Reporting bugs

Have a look at this screenshot to see how you can navigate to the deployment error status message. Please create an issue with that message and in which resource it occured on our github issues

ARM template

The output from the Azure Marketplace UI is fed directly to the ARM deployment template. You can use the ARM template on its own without going through the Marketplace. In fact, there are many features in the ARM template that are not exposed within the Marketplace such as configuring

• Azure Storage account to use with Azure Repository plugin for Snapshot/Restore
• Application Gateway to use for TLS and SSL offload
• The number and size of disks to attach to each data node VM

Check out our examples repository for examples of common scenarios and also take a look at the following blog posts for further information

• Spinning up a cluster with Elastic's Azure Marketplace template
• Elasticsearch and Kibana deployments on Azure

Parameters (Note: Tethr custom parameters have been added)

Parameter	Type	Description	Default Value
artifactsBaseUrl	string	The base url of the Elastic ARM template.	Raw content of the current branch
esVersion	string	A valid supported Elasticsearch version. See this list for supported versions	The latest version of Elasticsearch supported by the ARM template version
esClusterName	string	The name of the Elasticsearch cluster. Required	""
loadBalancerType	string	The load balancer to set up to access the cluster. Can be internal, external or gateway. By choosing internal, only an internal load balancer is deployed. Useful when connecting to the cluster happens from inside the Virtual Network By choosing external, both internal and external load balancers will be deployed. Kibana communicates with the cluster through the internal load balancer. By choosing gateway, Application Gateway will be deployed for load balancing, allowing a PFX certificate to be supplied for transport layer security to and from Application Gateway, and providing SSL offload. An internal load balancer will also deployed. Application Gateway and Kibana communicate with the cluster through the internal load balancer. If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like Elastic's Security, in addition to configuring transport layer security.	internal
nodeType	string	Tethr CUSTOM The ES node type to be deployed. master Deploys a single Master node data Deploys a single Data node. client Deploys a Client VM Scale Set. ingest Deploys an Ingest VM Scale Set.	""
vmId	string	Tethr CUSTOM A unique VM identifier. For Master or Data nodes, enter a number, e.g., 1, 2, 3 For Client or Ingest VM Scale Sets, enter ss	""
zoneId	array	Tethr CUSTOM Enter the Availability Zone number to deploy, i.e., 1, 2, 3. NOTE: For Client and Ingest VM Scale Sets, simply enter 1. By default, 1 VM will deploy to each Zone.	""
scaleSetInstanceCount	string	Tethr CUSTOM Number of Scale Set nodes to provision. By default, 3 VMs will deploy, each to a different Availability Zone.	3
azureCloudPlugin	string	Either Yes or No to install the Azure Cloud plugin for snapshot/restore. When set to Yes, both azureCloudeStorageAccountName and azureCloudStorageAccountKey must be specified to configure the plugin correctly.	No
azureCloudStorageAccountName	string	The name of an existing storage account to use for snapshots with Azure Cloud plugin. Must be a valid Azure Storage Account name.	""
azureCloudStorageAccountKey	securestring	The access key of an existing storage account to use for snapshots with Azure Cloud plugin.	""
xpackPlugins	string	Either Yes or No to install a trial license of the commercial X-Pack plugins: Monitoring, Security, Alerting, Graph and Machine Learning (5.5.0+). If also installing Kibana, it will have Reporting and Profiler installed.	Yes
esAdditionalPlugins	string	Additional Elasticsearch plugins to install. Each plugin must be separated by a semicolon. e.g. analysis-icu;mapper-attachments	""
esAdditionalYaml	string	Additional configuration for Elasticsearch yaml configuration file. Each line must be separated by a newline character \n e.g. "action.auto_create_index: .security\nindices.queries.cache.size: 5%". This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.	""
esHeapSize	integer	The size, in megabytes, of memory to allocate on each Elasticsearch node for the JVM heap. If unspecified, 50% of the available memory will be allocated to Elasticsearch heap, up to a maximum of 31744MB (~32GB). Take a look at the Elasticsearch documentation for more information. This is an expert level feature - setting a heap size larger than available memory on the Elasticsearch VM SKU will fail the deployment.	0
kibana	string	Either Yes or No to provision a machine with a public IP address that has Kibana installed on it. If you have opted to also install the X-Pack plugins using xpackPlugins, a trial license of the commercial X-Pack Kibana plugins will be installed.	Yes
vmSizeKibana	string	Azure VM size of the Kibana instance. See this list for supported sizes. Check that the size you choose is available in the region you choose.	Standard_A2
kibanaCertBlob	string	A Base-64 encoded form of the certificate (.crt) to secure HTTPS communication between the browser and Kibana.	""
kibanaKeyBlob	securestring	A Base-64 encoded form of the private key (.key) to secure HTTPS communication between the browser and Kibana.	""
kibanaKeyPassphrase	securestring	The passphrase to decrypt the private key. Optional as the key may not be encrypted. Supported only in 5.3.0+	""
jumpbox	string	Either Yes or No to optionally add a virtual machine with a public IP to the deployment, which you can use to connect and manage virtual machines on the internal network. NOTE: If you are deploying Kibana, the Kibana virtual machine can act as a jumpbox.	No
vmHostNamePrefix	string	The prefix to use for hostnames when naming virtual machines in the cluster. Hostnames are used for resolution of master nodes on the network, so if you are deploying a cluster into an existing virtual network containing an existing Elasticsearch cluster, be sure to set this to a unique prefix, to differentiate the hostnames of this cluster from an existing cluster. Can be up to 5 characters in length, must begin with an alphanumeric character and can contain alphanumeric and hyphen characters.	""
vmSizeDataNodes	string	Azure VM size of the data nodes. See this list for supported sizes. Check that the size you choose is available in the region you choose.	Standard_D1
vmDataDiskCount	int	Number of managed disks to attach to each data node in RAID 0 setup. Must be equal to or greater than 0. If the number of disks selected is more than can be attached to the data node VM (sku) size, the maximum number of disks that can be attached for the data node VM (sku) size will be used. Equivalent to taking min(vmDataDiskCount, max supported disks for data node VM size) When 1 disk is selected, the disk is not RAIDed. When 0 disks are selected, no disks will be attached to each data node. Instead, the temporary disk will be used to store Elasticsearch data. The temporary disk is ephemeral in nature and not persistent. Consult Microsoft Azure documentation on temporary disks to understand the trade-offs in using it for storage.	40 i.e. the max supported disks for data node VM size
vmDataDiskSize	string	The disk size of each attached disk. Choose Large (1023Gb), Medium (512Gb) or Small (128Gb). For Premium Storage, disk sizes equate to P30, P20 and P10 storage disk types, respectively.	Large
vmDataNodeCount	int	Tethr CUSTOM Deprecated. No longer active. The number of data nodes you wish to deploy. Must be greater than 0.	3
storageAccountType	string	The storage account type of the attached disks. Choose either Default or Standard. The Default storage account type will be Premium Storage for VMs that support Premium Storage and Standard Storage for those that do not.	Default
dataNodesAreMasterEligible	string	Either Yes or No to make all data nodes master eligible. This can be useful for small Elasticsearch clusters however, for larger clusters it is recommended to have dedicated master nodes. When Yes no dedicated master nodes will be provisioned.	No
vmSizeMasterNodes	string	Azure VM size of dedicated master nodes. See this list for supported sizes. By default the template deploys 3 dedicated master nodes, unless dataNodesAreMasterEligible is set to Yes. Check that the size you choose is available in the region you choose.	Standard_D1
vmClientNodeCount	int	Tethr CUSTOM Deprecated. No longer active. The number of client nodes to provision. Must be a positive integer. By default, the data nodes are added to the backendpool of the loadbalancer but if you provision client nodes, these will be added to the loadbalancer instead. Client nodes can be useful in offloading the gather process from data nodes and are necessary to scale an Elasticsearch cluster deployed with this template beyond 100 data nodes (the maximum number of VMs that can be added to a load balancer backendpool).	0
vmSizeClientNodes	string	Azure VM size of the client nodes see this list for supported sizes. Check that the size you choose is available in the region you choose.	Standard_D1
adminUsername	string	Admin username used when provisioning virtual machines. Must be a valid Linux username i.e. avoid any of the following usernames for Ubuntu	""
authenticationType	string	The authentication type for the Admin user. Either password or sshPublicKey	password
adminPassword	securestring	When authenticationType is password this sets the OS level user's password	""
sshPublicKey	securestring	When authenticationType is sshPublicKey this sets the OS level sshKey that can be used to login.	""
securityBootstrapPassword	securestring	Security password for 6.x bootstrap.password key that is added to the keystore. If no value is supplied, a 13 character password will be generated using the ARM template uniqueString() function. The bootstrap password is used to seed the built-in users. Used only in 6.0.0+	""
securityAdminPassword	securestring	Security password Admin user. This is the built-in elastic user. must be > 6 characters	""
securityReadPassword	securestring	Security password for the es_read user with user (read-only) role, must be > 6 characters	""
securityKibanaPassword	securestring	Security password Kibana. This is the built-in kibana user. must be > 6 characters	""
securityLogstashPassword	securestring	Security password for 5.2.0+ built-in logstash_system user. Only used in 5.2.0+. must be > 6 characters	""
location	string	The location where to provision all the items in this template. Defaults to the special [resourceGroup().location] value which means it will inherit the location from the resource group. Any other value must be a valid Azure region.	[resourceGroup().location]
vNetNewOrExisting	string	Whether the Virtual Network is new or existing. An existing Virtual Network in another Resource Group in the same Location can be used.	new
vNetName	string	The name of the Virtual Network. The Virtual Network must already exist when using an existing Virtual Network	es-net
vNetExistingResourceGroup	string	The name of the Resource Group in which the Virtual Network resides when using an existing Virtual Network. Required when using an existing Virtual Network	""
vNetNewAddressPrefix	string	The address prefix when creating a new Virtual Network. Required when creating a new Virtual Network	10.0.0.0/24
vNetLoadBalancerIp	string	The internal static IP address to use when configuring the internal load balancer. Must be an available IP address on the provided vNetClusterSubnetName.	10.0.0.4
vNetClusterSubnetName	string	The name of the subnet to which Elasticsearch nodes will be attached. The subnet must already exist when using an existing Virtual Network	es-subnet
vNetNewClusterSubnetAddressPrefix	string	The address space of the subnet. Required when creating a new Virtual Network	10.0.0.0/25
vNetAppGatewaySubnetName	string	Subnet name to use for the Application Gateway. Required when selecting gateway for load balancing. The subnet must already exist when using an existing Virtual Network	es-gateway-subnet
vNetNewAppGatewaySubnetAddressPrefix	string	The address space of the Application Gateway subnet. Required when creating a new Virtual Network and selecting gateway for load balancing.	10.0.0.128/28
appGatewayTier	string	The tier of the Application Gateway, either Standard or WAF. Required when selecting gateway for load balancing.	Standard
appGatewaySku	string	The size of the Application Gateway. Choose Small, Medium or Large. When choosing appGatewayTier WAF, the size must be at least Medium. Required when selecting gateway for load balancing.	Medium
appGatewayCount	int	The number instances of the Application Gateway. Can be a value between 1 and 10. A minimum of 2 is recommended for production. Required when selecting gateway for load balancing.	2
appGatewayCertBlob	string	A Base-64 encoded form of the PFX certificate for the Application Gateway. This certificate is used to secure HTTPS connections to and from the Application Gateway. Required when selecting gateway for load balancing.	""
appGatewayCertPassword	securestring	The password for the PFX certificate for the Application Gateway. Defaults to "". Required when selecting gateway for load balancing.	""
appGatewayWafStatus	string	The firewall status of the Application Gateway, either Enabled or Disabled. Required when selecting gateway for load balancing and using appGatewayTier WAF.	Enabled
appGatewayWafMode	string	The firewall mode of the Application Gateway, either Detection or Prevention. Required when selecting gateway for load balancing and using appGatewayTier WAF.	Detection
userCompany	string	The name of your company.	""
userEmail	string	Your email address	""
userFirstName	string	Your first name	""
userLastName	string	Your last name	""
userJobTitle	string	Your job title. Pick the nearest one that matches from the list of job titles	Other
userCountry	string	The country in which you are based.	""

Command line deploy

You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell

Azure CLI 1.0

1. Log into Azure

azure login

2. Ensure you are in arm mode

azure config mode arm

3. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

azure group create <name> <location>

4. Use our published template directly using --template-uri

azure group deployment create --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json --parameters-file parameters/password.parameters.json -g <name>

or if your are executing commands from a clone of this repo using --template-file

azure group deployment create --template-file src/mainTemplate.json --parameters-file parameters/password.parameters.json -g <name>

where <name> refers to the resource group you just created.

NOTE

The --parameters-file can specify a different location for the items that get provisioned inside of the resource group. Make sure these are the same prior to deploying if you need them to be. Omitting location from the parameters file is another way to make sure the resources get deployed in the same location as the resource group.

Azure PowerShell

1. Log into Azure

Login-AzureRmAccount

2. Select a Subscription Id

Select-AzureRmSubscription -SubscriptionId "<subscriptionId>"

3. Define the parameters object for your deployment

$clusterParameters = @{
    "artifactsBaseUrl"="https://raw.githubusercontent.com/elastic/azure-marketplace/master/src"
    "esVersion" = "6.2.1"
    "esClusterName" = "elasticsearch"
    "loadBalancerType" = "internal"
    "vmDataDiskCount" = 1
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityAdminPassword" = "Password123"
    "securityReadPassword" = "Password123"
    "securityKibanaPassword" = "Password123"
    "securityLogstashPassword" = "Password123"
}

4. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"

5. Use our template directly from GitHub

New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters

Targeting a specific template version

You can target a specific version of the template by modifying the URI of the template and the artifactsBaseUrl parameter of the template.

Targeting a specific template version is recommended for repeatable deployments.

For example, to target the 6.2.2 tag release with PowerShell

$templateVersion = "6.2.2"
$templateBaseUrl = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src"

$clusterParameters = @{
    "artifactsBaseUrl" = $templateBaseUrl
    "esVersion" = "6.2.2"
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityAdminPassword" = "Password123"
    "securityReadPassword" = "Password123"
    "securityKibanaPassword" = "Password123"
    "securityLogstashPassword" = "Password123"
}

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"
New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "$templateBaseUrl/mainTemplate.json" -TemplateParameterObject $clusterParameters

Web based deploy

The above button will take you to the autogenerated web based UI based on the parameters from the ARM template.

License

This project is MIT Licensed and is based on the Elasticsearch azure quick start arm template