Build: Move shadow customizations into common code #32014

nik9000 · 2018-07-12T19:09:27Z

Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:

The non-shadow jar will not be built
The shadow jar will not have a "classifier"
Tests will run against the shadow jar
Javadoc will include all of the shadowed classes
Service files in META-INF/services will be merged

nik9000 · 2018-07-12T19:12:58Z

Rather than declare any sort of "configure if shadowed" section I opted to add the shadow customizations to the place in the build where the thing being customized is already being configured. I hope that this reduced the element of surprise when debugging some issue with a feature that is customized by shadowing.

elasticmachine · 2018-07-12T19:14:07Z

Pinging @elastic/es-core-infra

nik9000 · 2018-07-12T19:16:04Z

I've marked this as a 6.x blocker because I'd like to build on it to shadow the new x-pack:protocol project into the x-pack-transport client.

nik9000 · 2018-07-13T02:30:19Z

Looks like I broke the benchmarks! I'll have a look in a bit.

nik9000 · 2018-07-16T18:58:54Z

@rjernst , could you have a look at this when you get a chance? I believe we should get this or something like it merged so we can cleanly add the shadow plugin to the x-pack transport client.

rjernst

I have a couple suggestions.

rjernst · 2018-07-16T19:36:50Z

build.gradle

+        if (shadowed) {
+          /*
+           * Include the source of shadowed upstream projects so we don't
+           * to publish their javadoc.


to publish -> publish?

"have to publish" is what I was going for. I'll fix.

rjernst · 2018-07-16T19:41:59Z

build.gradle

+          project.javadoc.classpath += upstreamProject.javadoc.classpath
+        } else {
+          // Link to non-shadowed dependant projects
+          project.javadoc.dependsOn "${upstreamProject.path}:javadoc"


Cross project task dependencies like this are highly frowned upon in gradle, as they limit the ability to detect when a project needs to be evaluated (which is a goal they are working towards with the new lazy task api). Normally one would only use dependencies to describe this relationship, and use a different configuration than the default/runtime one. Is that possible here?

In this case I'm just reindenting some code that we've had for a while but I'll look into it.

Ok, sorry I missed that. A followup to look at that is fine then.

rjernst · 2018-07-16T19:45:27Z

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

+             */
+            SourceSet testSourceSet = project.sourceSets.findByName('test')
+            if (testSourceSet != null) {
+                testSourceSet.compileClasspath += project.configurations.shadow


Can you add a comment that this is ok for only test because we would not be using shading for rest integ tests or qa tests (since they have no artifacts)?

I can add a comment about how wouldn't make sense for qa projects because they don't have artifacts. I think I should fail the build if you try to add the shadow plugin and the es plugin build plugin. That isn't likely to work and we may as well catch it and warn anyone that tries.

Moves the customizations to the build to produce nice shadow jars and javadocs into common build code, mostly BuildPlugin with a little into the root build.gradle file. This means that any project that applies the shadow plugin will automatically be set up just like the high level rest client: * The non-shadow jar will not be built * The shadow jar will not have a "classifier" * Tests will run against the shadow jar * Javadoc will include all of the shadowed classes * Service files in `META-INF/services` will be merged

* 6.x: Fix rollup on date fields that don't support epoch_millis (#31890) Revert "Introduce a Hashing Processor (#31087)" (#32179) [test] use randomized runner in packaging tests (#32109) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Fix BwC Tests looking for UUID Pre 6.4 (#32158) (#32169) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) Fix CP for namingConventions when gradle home has spaces (#31914) Convert Version to Java - clusterformation part1 (#32009) Fix Java 11 javadoc compile problem Improve docs for search preferences (#32098) Configurable password hashing algorithm/cost(#31234) (#32092) [DOCS] Update TLS on Docker for 6.3 ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Build: Move shadow customizations into common code (#32014) Painless: Add PainlessClassBuilder (#32141) Fix accidental duplication of bwc test for script behavior Handle missing values in painless (#30975) (#31903) Build: Make additional test deps of check (#32015) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Adjust translog after versionType removed in 7.0 (#32020) Disable C2 from using AVX-512 on JDK 10 (#32138) [Rollup] Add new capabilities endpoint for concrete rollup indices (#32111) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ [ML] Wait for aliases in multi-node tests (#32086) Ensure to release translog snapshot in primary-replica resync (#32045) Docs: Fix missing example script quote (#32010) Add Index UUID to `/_stats` Response (#31871) (#32113) [ML] Move analyzer dependencies out of categorization config (#32123) [ML][DOCS] Add missing 6.3.0 release notes (#32099) Updates the build to gradle 4.9 (#32087) Update monitoring template version to 6040099 (#32088) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012)

@return

* master: Painless: Simplify Naming in Lookup Package (#32177) Handle missing values in painless (#32207) add support for write index resolution when creating/updating documents (#31520) ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864) Remove indication of future multi-homing support (#32187) Rest test - allow for snapshots to take 0 milliseconds Make x-pack-core generate a pom file Rest HL client: Add put watch action (#32026) Build: Remove pom generation for plugin zip files (#32180) Fix comments causing errors with Java 11 Fix rollup on date fields that don't support epoch_millis (#31890) Detect and prevent configuration that triggers a Gradle bug (#31912) [test] port linux package packaging tests (#31943) Revert "Introduce a Hashing Processor (#31087)" (#32178) Remove empty @return from JavaDoc Adjust SSLDriver behavior for JDK11 changes (#32145) [test] use randomized runner in packaging tests (#32109) Add support for field aliases. (#32172) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Fix BwC Tests looking for UUID Pre 6.4 (#32158) Improve docs for search preferences (#32159) use before instead of onOrBefore Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) A replica can be promoted and started in one cluster state update (#32042) Fix Java 11 javadoc compile problem Fix CP for namingConventions when gradle home has spaces (#31914) Fix `range` queries on `_type` field for singe type indices (#31756) [DOCS] Update TLS on Docker for 6.3 (#32114) ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Remove versionType from translog (#31945) Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Painless: Add PainlessClassBuilder (#32141) Build: Make additional test deps of check (#32015) Disable C2 from using AVX-512 on JDK 10 (#32138) Build: Move shadow customizations into common code (#32014) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Remove empty @param from Javadoc Re-disable packaging tests on suse boxes Docs: Fix missing example script quote (#32010) [ML] Wait for aliases in multi-node tests (#32086) [ML] Move analyzer dependencies out of categorization config (#32123) Ensure to release translog snapshot in primary-replica resync (#32045) Handle TokenizerFactory TODOs (#32063) Relax TermVectors API to work with textual fields other than TextFieldType (#31915) Updates the build to gradle 4.9 (#32087) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ Check that client methods match API defined in the REST spec (#31825) Enable testing in FIPS140 JVM (#31666) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012) [Test] Modify assert statement for ssl handshake (#32072)

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in elastic#32014

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in #32014

nik9000 added 8 commits July 11, 2018 16:36

Basic + Maven

039865c

Jar configuration

417ab3f

Tests

4f46cb3

Javadoc

1b06e99

New javadoc

5d80fa4

Cleanup

57396ec

More cleanup

dedc9c0

Fix links to shadow configuration

3cd2807

nik9000 requested a review from rjernst July 12, 2018 19:13

nik9000 added >enhancement blocker :Delivery/Build Build or test infrastructure v7.0.0 v6.4.0 labels Jul 12, 2018

Merge branch 'master' into standard_shadow

bdb1d5b

nik9000 changed the title ~~Build: Move shadow customizations into common code|~~ Build: Move shadow customizations into common code Jul 16, 2018

Fix benchmarks

ce14e7f

Remove oops

117fb41

rjernst reviewed Jul 16, 2018

View reviewed changes

rjernst approved these changes Jul 16, 2018

View reviewed changes

Iter

b088863

nik9000 force-pushed the standard_shadow branch from 573d0c4 to b088863 Compare July 16, 2018 21:43

Merge branch 'master' into standard_shadow

c80f065

nik9000 merged commit 1b97652 into elastic:master Jul 17, 2018

nik9000 added the backport pending label Jul 17, 2018

nik9000 removed the backport pending label Jul 18, 2018

rjernst added a commit to rjernst/elasticsearch that referenced this pull request Nov 29, 2018

Build: Fix jdbc jar pom to not include deps

a3723f8

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in elastic#32014

rjernst mentioned this pull request Nov 29, 2018

Build: Fix jdbc jar pom to not include deps #36036

Merged

rjernst added a commit that referenced this pull request Nov 29, 2018

Build: Fix jdbc jar pom to not include deps (#36036)

95d9cef

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in #32014

rjernst added a commit that referenced this pull request Nov 29, 2018

Build: Fix jdbc jar pom to not include deps (#36036)

c0c4785

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in #32014

rjernst added a commit that referenced this pull request Nov 29, 2018

Build: Fix jdbc jar pom to not include deps (#36036)

11b944f

This commit adds back the exclusion of dependencies from the pom file for the jdbc jar, which was accidentally lost in #32014

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: Move shadow customizations into common code #32014

Build: Move shadow customizations into common code #32014

nik9000 commented Jul 12, 2018

nik9000 commented Jul 12, 2018

elasticmachine commented Jul 12, 2018

nik9000 commented Jul 12, 2018

nik9000 commented Jul 13, 2018

nik9000 commented Jul 16, 2018

rjernst left a comment

rjernst Jul 16, 2018

nik9000 Jul 16, 2018

rjernst Jul 16, 2018

nik9000 Jul 16, 2018

rjernst Jul 16, 2018

rjernst Jul 16, 2018

nik9000 Jul 16, 2018

Build: Move shadow customizations into common code #32014

Build: Move shadow customizations into common code #32014

Conversation

nik9000 commented Jul 12, 2018

nik9000 commented Jul 12, 2018

elasticmachine commented Jul 12, 2018

nik9000 commented Jul 12, 2018

nik9000 commented Jul 13, 2018

nik9000 commented Jul 16, 2018

rjernst left a comment

Choose a reason for hiding this comment

rjernst Jul 16, 2018

Choose a reason for hiding this comment

nik9000 Jul 16, 2018

Choose a reason for hiding this comment

rjernst Jul 16, 2018

Choose a reason for hiding this comment

nik9000 Jul 16, 2018

Choose a reason for hiding this comment

rjernst Jul 16, 2018

Choose a reason for hiding this comment

rjernst Jul 16, 2018

Choose a reason for hiding this comment

nik9000 Jul 16, 2018

Choose a reason for hiding this comment