ElasticSearch Logs

Configuration

Scale stores its logs in an ElasticSearch cluster by datestamped index. This daily index is created automatically by Logstash as logs are streamed from executing jobs. The default deployment of Scale includes an Elasticsearch instance, but this is not to be used for anything other than demonstration purposes. In a production scenario, Elasticsearch should either be a single instance or multiple instances fronted by a load-balancer that provides fault tolerance and high availability. We also support X-Pack Security with username and password protected Elasticsearch deployments. The following are examples of how the SCALE_ELASTICSEARCH_URLS should be configured under the above mentioned scenarios.

Single instance Elasticsearch:

http://my-elastic-server:9200

Load balancer fronted Elasticsearch cluster (AWS ELB example):

http://my-elastic-cluster-myaccount.us-east-2.elb.amazonaws.com:9200

X-Pack Security protected cluster:

https://username:password@my-elastic-cluster-myaccount.us-east-2.elb.amazonaws.com

Indices

You can perform REST API calls to retrieve log information from ElasticSearch. To retrieve logs in correct order, you should order first by @timestamp ascending and then by scale_order_num ascending. Scale stores its logs in a JSON format with the following fields:

message

Type: String
The log message

@timestamp

Type: String
The ISO-8601 timestamp marking when the message was logged

scale_order_num

Type: Integer
A sequence number used to indicate correct log message order when multiple messages share the same @timestamp value. To retrieve logs in correct order, you should order first by @timestamp ascending and then by scale_order_num ascending.

scale_task

Type: String
The ID of the Scale task that produced this log message

scale_job_exe

Type: Integer
The unique cluster ID of the Scale job execution that produced this log message

scale_job_type

Type: Integer
The unique cluster ID of the Scale job type that produced this log message

scale_node

Type: Integer
The host name of the Scale node that executed the Scale task

stream

Type: String
Indicates which stream produced the log message, either “stdout” or “stderr”