Add Groovy as a scripting language, add groovy sandboxing #6233

dakrone · 2014-05-19T14:31:14Z

Sandboxes the groovy scripting language with multiple configurable
whitelists:

script.groovy.sandbox.receiver_whitelist: comma-separated list of string
classes for objects that may have methods invoked.
script.groovy.sandbox.package_whitelist: comma-separated list of
packages under which new objects may be constructed.
script.groovy.sandbox.class_whitelist comma-separated list of classes
that are allowed to be constructed.

As well as a method blacklist:

script.groovy.sandbox.method_blacklist: comma-separated list of
methods that are never allowed to be invoked, regardless of target
object.

The sandbox can be entirely disabled by setting:

script.groovy.sandbox.enabled: false

s1monw · 2014-05-19T14:56:23Z

docs/reference/modules/scripting.asciidoc

@@ -6,28 +6,30 @@ expressions. For example, scripts can be used to return "script fields"
 as part of a search request, or can be used to evaluate a custom score
 for a query and so on.

-The scripting module uses by default http://mvel.codehaus.org/[mvel] as


we should really have a section that says that we added this in 1.3 and where to find the docs for the prev. version?

Sure, I added the "coming in 1.3.0" part, but not sure where to link for older documentation since 1.2 hasn't been released yet. Once it's released I'll add a commit to point to the older documentation.

Docs are built for branches, ie 1.x, master, not for released versions. So you should use a deprecated tag for mvel eg deprecated[1.3.0,Replace by groovy as the default scripting language, see old docs here <>]

@dakrone can you address clintons comments?

Added a commit to rearrange and (hopefully) clarify the mvel deprecation.

s1monw · 2014-05-19T15:03:25Z

oh man this looks awesome!

rmuir · 2014-05-20T13:56:10Z

src/main/java/org/elasticsearch/script/groovy/GroovyScriptEngineService.java

+            "java.lang.Double", "[D", "[[D", "[[[D",
+            "java.lang.Long", "[J", "[[J", "[[[J",
+            "java.lang.Short", "[S", "[[S", "[[[S",
+            "java.lang.Char", "[C", "[[C", "[[[C",


shouldn't this be java.lang.Character?

Whoops, good catch, fixed.

dakrone · 2014-06-03T09:14:12Z

Added a number of commits to address the feedback, @s1monw can you take another look?

s1monw · 2014-06-12T10:17:22Z

src/main/java/org/elasticsearch/script/ScriptService.java

@@ -74,7 +74,7 @@ public ScriptService(Settings settings, Environment env, Set<ScriptEngineService
        TimeValue cacheExpire = componentSettings.getAsTime("cache.expire", null);
        logger.debug("using script cache with max_size [{}], expire [{}]", cacheMaxSize, cacheExpire);

-        this.defaultLang = componentSettings.get("default_lang", "mvel");


any chance we can make this a constant?

Yes, I will make everything a constant that can be for this.

s1monw · 2014-06-12T10:59:41Z

src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java

@@ -951,7 +951,7 @@ public void script_Score() {
        SearchResponse response = client().prepareSearch("idx").setTypes("type")
                .setQuery(functionScoreQuery(matchAllQuery()).add(ScoreFunctionBuilders.scriptFunction("doc['" + SINGLE_VALUED_FIELD_NAME + "'].value")))
                .addAggregation(terms("terms")
-                        .script("ceil(_doc.score/3)")
+                        .script("ceil(_doc.score()/3)")


mabye we can use _score here too?

I looked into this, _score doesn't work for either Groovy or Mvel (probably due to this being in an aggregation?). If this is something we want to support I think we should open a separate issue for it.

oh I see makes sense!

s1monw · 2014-06-12T11:00:44Z

did another review.... looks pretty close... left some commetns

s1monw · 2014-06-16T10:28:45Z

src/main/java/org/elasticsearch/script/groovy/UpdateableFloat.java

+/**
+ * Float encapsulation that allows updating the value with public member access
+ */
+public class UpdateableFloat extends Number {


IMO we should make this class package private and final? Can you also put in the comment where this is used and for what reason just be a little more verbose.... we might also just put it as a inner class?

dakrone · 2014-06-16T12:51:39Z

@s1monw updated this PR with all of the changes you recommended :)

s1monw · 2014-06-18T11:17:32Z

this actually LGTM maybe @kimchy want's to take a look?
good job!

kimchy · 2014-06-18T11:50:20Z

@dakrone this looks great!, a few comments:

I would add a 3 value setting to dynamic script enable now, true, false, and sandbox.

For 2.0:

I would remove mvel completely
Default to sandbox in dynamic script setting, and groovy as the lang.

For 1.3:

I would suggest also removing mvel completely, aside from security, its just buggy, if we decide to, then its the same changes as 2.0, including properly documenting how to install the mvel plugin and setting mvel as the lang by default, and properly mark it as a breaking change.
If there is resistance from removing mvel, then keep mvel around and keep it as the default lang, next to groovy being an option. Default dynamic script should still be sandbox.

dakrone · 2014-06-18T14:31:17Z

@kimchy Sounds good, I updated the documentation and completely removed mvel as well. I also added the 3-value setting for dynamic scripting.

s1monw · 2014-06-18T18:59:17Z

so this means that upgrading to 1.3 from any prev version means you need to install the mvel plugin on the upgrade and pass mvel as a param in the request to opt in to this language? I think we really need to invest into an upgrade guide here!

dakrone · 2014-06-19T09:04:21Z

@s1monw yes, the mvel plugin will have to be installed and the script.default_lang: mvel option would have to be added to continue using mvel as the default.

For an upgrade guide, do we want to add the two settings to the release notes? It will also be on the scripting blog post to come out after this is merged as well.

Sandboxes the groovy scripting language with multiple configurable whitelists: `script.groovy.sandbox.receiver_whitelist`: comma-separated list of string classes for objects that may have methods invoked. `script.groovy.sandbox.package_whitelist`: comma-separated list of packages under which new objects may be constructed. `script.groovy.sandbox.class_whitelist` comma-separated list of classes that are allowed to be constructed. As well as a method blacklist: `script.groovy.sandbox.method_blacklist`: comma-separated list of methods that are never allowed to be invoked, regardless of target object. The sandbox can be entirely disabled by setting: `script.groovy.sandbox.enabled: false`

As for elasticsearch 1.3.0, `groovy` is the new default scripting language. Related to: elastic/elasticsearch#6233 Closes #61. (cherry picked from commit 170a2cd)

As for elasticsearch 1.3.0, `groovy` is the new default scripting language. Related to: elastic/elasticsearch#6233 Closes #61.

dakrone added v1.3.0 labels May 19, 2014

s1monw reviewed May 19, 2014
View reviewed changes

rmuir reviewed May 20, 2014
View reviewed changes

javanna mentioned this pull request Jun 3, 2014

Add a sandboxed scripting language #5988

Closed

s1monw reviewed Jun 12, 2014
View reviewed changes

s1monw removed the review label Jun 12, 2014

s1monw reviewed Jun 16, 2014
View reviewed changes

dakrone added the review label Jun 17, 2014

dakrone added the breaking label Jun 18, 2014

s1monw removed the review label Jun 18, 2014

nik9000 mentioned this pull request Jun 19, 2014

Mappings: Add transform to document before index #6566

Closed

dakrone changed the title ~~Add Groovy as a scripting language, change default from mvel -> groovy~~ Add Groovy as a scripting language, add groovy sandboxing Jun 20, 2014

dakrone removed the breaking label Jun 20, 2014

dakrone merged commit c70f6d0 into elastic:master Jun 20, 2014

dakrone mentioned this pull request Jun 20, 2014

Switch to Groovy as the default scripting language #6571

Merged

s1monw added the release highlight label Jul 9, 2014

clintongormley changed the title ~~Add Groovy as a scripting language, add groovy sandboxing~~ Scripting: Add Groovy as a scripting language, add groovy sandboxing Jul 16, 2014

clintongormley added the feature label Jul 16, 2014

dadoonet added a commit to elastic/elasticsearch-river-couchdb that referenced this pull request Jul 18, 2014

Use groovy as default scripting language

170a2cd

As for elasticsearch 1.3.0, `groovy` is the new default scripting language. Related to: elastic/elasticsearch#6233 Closes #61.

dakrone deleted the feature/add-groovy-scripting branch September 9, 2014 13:47

clintongormley added the :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache label Jun 6, 2015

clintongormley changed the title ~~Scripting: Add Groovy as a scripting language, add groovy sandboxing~~ Add Groovy as a scripting language, add groovy sandboxing Jun 6, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Groovy as a scripting language, add groovy sandboxing #6233

Add Groovy as a scripting language, add groovy sandboxing #6233

dakrone commented May 19, 2014

s1monw May 19, 2014

dakrone May 19, 2014

clintongormley May 19, 2014

s1monw May 23, 2014

dakrone May 30, 2014

s1monw commented May 19, 2014

rmuir May 20, 2014

dakrone May 20, 2014

dakrone commented Jun 3, 2014

s1monw Jun 12, 2014

dakrone Jun 16, 2014

s1monw Jun 12, 2014

dakrone Jun 13, 2014

s1monw Jun 16, 2014

s1monw commented Jun 12, 2014

s1monw Jun 16, 2014

dakrone commented Jun 16, 2014

s1monw commented Jun 18, 2014

kimchy commented Jun 18, 2014

dakrone commented Jun 18, 2014

s1monw commented Jun 18, 2014

dakrone commented Jun 19, 2014

Add Groovy as a scripting language, add groovy sandboxing #6233

Add Groovy as a scripting language, add groovy sandboxing #6233

Conversation

dakrone commented May 19, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented May 19, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Jun 3, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Jun 12, 2014

Choose a reason for hiding this comment

dakrone commented Jun 16, 2014

s1monw commented Jun 18, 2014

kimchy commented Jun 18, 2014

dakrone commented Jun 18, 2014

s1monw commented Jun 18, 2014

dakrone commented Jun 19, 2014