Saved scripted fields

[rashidkpc]

Now that elasticsearch scripting is safe we could abstract out script fields and let people provide computed fields, say from a combination of several fields, or with math done on numeric fields.

• Add scripted fields to IndexPattern Scripted fields in index pattern #1982
• Add CRUD interface for scripted fields Scripted fields interface #2076
• Use scripted fields in the aggregation builder in Visualize Scripted fields in Visualize #2124
• Use scripted fields in Discover Select scripted fields #2146
• Merge scripted fields into _formatted fields in Discover Merge stored/index fields into _formatted #1998

[elvarb]

Would it be possible to script based on "external" data?

The use case I can think of is if you have a "amount" field and "currency", to be able to look up what is needed to convert the field to some specific currency.

[rashidkpc]

It would not be possible to script based on external data unless that data was available to elasticsearch, this would use Elasticsearch's scripting support.

[rashidkpc]

This could also be used to combine time fields, for example if you had startTime and endTime and want to see the duration.

[elvarb]

Nice use case, for the start and end time, wanted this for a long time. Thought about using Logstash's ruby filter to do the calculations there. If we have scripted fields like this, would we be able to sort by those fields or query against them? (for example, sort by duration or show only durations over x)

To do the currency calculation, would it be possible to use a percolator instead of elasticsearch scripting and then use Kibana's scripted fields to multiply one entry field with a percolator field?

That way you would update the percolator daily with new currency data and the scripted fields would automatically translate any currency in a field to the same currency.

[rashidkpc]

Other uses for this include running a terms facet over time field to show say, what hour of the day is most popular, or what day of the week, over several weeks.

[andersmartinsson]

Will it be possible to make a scripted field that calculates the margin from for example a cost and sales price field and use that value in a histogram? We are using kibana for sales figures and so far I have not been able to show the margin just the sales figures.... Being able to do this would give us alot better view of the profitmargin over time.

[ellkay-joshe]

When you say abstract out script fields, do you mean that any aspect of scripting we want to use will have to be explicitly supported in Kibana? One of the things I've wanted for a while was to use a script in Kibana to pull the day out of a timestamp in ES, so would I be selecting a field, then selecting "Day" or something or would I be able to just write a script expression to pull out the value I want?

<edit> oh wow I totally missed the thing you said above that actually covered my exact case.

[jayswan]

See this thread for additional discussion of Kibana and ES scripting:

https://groups.google.com/forum/#!topic/logstash-users/crgmYEkXbGI

Specifically, I'd like to have a Kibana-based method to extract data from arbitrary fields and (ideally) get stats on them, as in a terms aggregation. My use case is this: some common log sources (e.g., Cisco routers & switches) have log formats that aren't consistently formatted enough to be able to parse their details in advance. Frequently, I want to extract a field using a regex or field position, then to the equivalent of a terms aggregation for it.

Today, I find the appropriate query in Kibana, paste it into curl, extract the message field using jq, then use Unix toolchain commands (awk, grep, wc, etc) to manually summarize data.

This request is similar to the search-time field extraction feature in Splunk.

[w33ble]

All tasks complete and merged into master for release in Beta 3.

[tomryanx]

Could I get a pointer to how to implement this: "Other uses for this include running a terms facet over time field to show say, what hour of the day is most popular, or what day of the week, over several weeks."

[w33ble]

@tomryanx Unfortunately, that functionality was provided for the dynamic groovy scripting, which has since been disabled for security reasons. You could still expose those scripts via indexed scripting or file based scripting. If you'd like specific pointers, please post the question over at our discussion boards located at https://discuss.elastic.co/c/kibana

[GrahamHannington]

#561 was closed in favor of this issue. Similar to requirements described by other users, I would like to split charts by day of week (to compare number of transactions at the same time each day), or by month of year. Guess I'll go looking for some groovy scripts.