[Enhancement] Parallel test jobs for CI #2861

pawel-gudel-eliatra · 2023-06-14T12:58:04Z

Description

Currently "build" job in Github Actions runs for around 60-80 minutes on average. Long time for tests to be done can slow down development process. Easiest way to make this situation little better is to split tests into separate task and reconfigure Github Actions to run tests in parallel. This change allows to cut run of CI from 60-80 minutes to around 28-30 minutes.

We've determined which tests run are the longest and moved them into separate tasks (defined in build.gradle file). We do not want to break test on Developers side so running gradle test command would run full test suite (test task depends on new ones). This also allows new test to be added run automatically.

Issues Resolved

#2798

Testing

There's no changes in test themselves.

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-06-14T13:19:02Z

Codecov Report

Merging #2861 (e48b74e) into main (c808692) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #2861      +/-   ##
============================================
+ Coverage     62.23%   62.26%   +0.02%     
+ Complexity     3408     3332      -76     
============================================
  Files           266      266              
  Lines         19673    19651      -22     
  Branches       3331     3330       -1     
============================================
- Hits          12244    12235       -9     
+ Misses         5823     5785      -38     
- Partials       1606     1631      +25

see 20 files with indirect coverage changes

willyborankin

Hi @pawel-gudel-eliatra, thank you for the contribution. Could you please remove opensslTest task since you already defined the new one sslTest. I'm wondering how will it work in IDE?
As I understand IDE starts all list of tasks in case you select any test or I'm wrong?

build.gradle

nibix · 2023-06-14T13:52:52Z

FYI: At the moment, the fact that the tests run in parallel breaks the code coverage information. (See the codecov report #2861 (comment) )

Thus, one still open research task is to find a way on how to regain this information.

build.gradle

.github/workflows/ci.yml

build.gradle

* Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

* Revised excludes for: `sslTest`, `securityIntegrationTest` and `indicesTest` * Removed `opensslTest`, all its test are in `sslTest` Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

1. Default 'test' task will run full test suite. 2. For CI builds (run on Github Actions) we've copied existing configuration to task 'citest' which will be run only from CI. 3. We've moved all dependencies to 'citest' 4. 'opensslTest' is still removed and incorporated into both 'test' (as a part of full suite) and 'citest' (as a separate 'sslTest' task) Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

We've decided to remove dependency between `citest` and new, generated, test tasks. This is not needed for CI runs as they will be always started in separate Github Jobs. This is not significant change - it will only reduce complexity a bit. This also required to change order of tasks in build.gradle file. This commit also fixes issue with copyExtraTestResources task. It's paths were hardcoded which was fine for default test task but did not work for every other, newly created, tasks. We've also added dependencies between all new tasks and copyExtraTestResources Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Right now it allows for more sophisticated configuration. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

peternied

I still need to see resolution with the openssl tests and how the the tasks are used and cleanup of the command line parameters. Is there anything you'd need help with to make forward progress?

.github/workflows/ci.yml

build.gradle

nibix · 2023-06-21T07:01:39Z

@peternied

I still need to see resolution with the openssl tests

As there was lots of back and forth on the openssl tests, let me just verify what the resolution of the openssl tests would be. Is it just removing openssltest in favour of opensslcitest?

peternied · 2023-06-21T16:15:58Z

removing openssltest in favour of opensslcitest

👍 it looks like that block of tests should be run on its own that's all

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Unfortunately running them from one list lenghtens time a lot Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Is it possible to run windows jobs before ubuntu? Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

* Fix for not excluded tests. * Reordered matrix strategy variables. * Restored unified list of Gradle task.. again! Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

peternied

Total duration
🔥 24m 54s 🔥

peternied · 2023-06-28T19:39:19Z

@willyborankin anything else you are looking for out of this PR?

willyborankin

LGTM

opensearch-trigger-bot · 2023-06-28T20:42:15Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-2861-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e4f4817e0d2d9caf80da0d0cb1da73e4913f62e3
# Push it to GitHub
git push --set-upstream origin backport/backport-2861-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2861-to-2.x.

peternied · 2023-06-28T20:44:56Z

@pawel-gudel-eliatra Would you be open to create a backport PR to the 2.x branch of this repository? The instructions for that process are in the post [above] this one.

* Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

* add search model group permission to ml_read_access role (#2855) * add search model group permission to ml_read_access role Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> * IntegrationTest spotless (#2863) Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Format everything (#2866) * Use boucycastle PEM reader instead of reg expression (#2864) Use BouncyCastle PEMReader instead of regular expression to read and parse private key pem files. Signed-off-by: Andrey Pleskach <ples@aiven.io> * Adding field level security test cases for FlatFields (#2876) Signed-off-by: Peter Nied <petern@amazon.com> * Update snappy to 1.1.10.1 and guava to 32.0.1-jre (#2886) * Update snappy to 1.1.10.1 and guava to 32.0.1-jre Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade kafka to 3.5.0 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Force snappy Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add runtime dependency on org.scala-lang.modules:scala-java8-compat_3:1.0.2 to fix issue with KafkaSinkTest Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> * Role permissions order tool and workflow (#2733) * Check Permissions Order tool and workflow Adds a NodeJS tool that can inspect yaml role definitions, check if they are in alphabetical order, correct them if required. Signed-off-by: Peter Nied <peternied@hotmail.com> * Apply fixes to roles.yml files Signed-off-by: Peter Nied <peternied@hotmail.com> * Fixing busted test, adding findArrayInJson for response bodies Signed-off-by: Peter Nied <petern@amazon.com> --------- Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Peter Nied <petern@amazon.com> * Misc changes (#2902) Moved isStatic and isReserved methods to the SecurityDynamicConfiguration class Signed-off-by: Andrey Pleskach <ples@aiven.io> * Update triaging guidelines (#2899) * Update triaging guidelines Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * fix cluster perm classification for msearch template (#2892) * fix cluster perm classification for msearch template Signed-off-by: Derek Ho <dxho@amazon.com> * move test to unit test file Signed-off-by: Derek Ho <dxho@amazon.com> * fully revert integration test file Signed-off-by: Derek Ho <dxho@amazon.com> * Update src/test/java/org/opensearch/security/privileges/PrivilegesEvaluatorUnitTest.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * spotless Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * [Doc] Add architecture document (#2869) * Add initial architecture document Signed-off-by: Peter Nied <petern@amazon.com> * [Enhancement] Parallel test jobs for CI (#2861) * Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> * Bump BouncyCastle from jdk15on to jdk15to18 (#2901) jdk15to18 contains fix for - CVE-2023-33201 - Medium Severity Vulnerability Signed-off-by: Andrey Pleskach <ples@aiven.io> * Spotless Apply Signed-off-by: Ryan Liang <jiallian@amazon.com> --------- Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Co-authored-by: Bhavana Ramaram <rbhavna@amazon.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Andrey Pleskach <ples@aiven.io> Co-authored-by: Peter Nied <petern@amazon.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Derek Ho <derek01778@gmail.com> Co-authored-by: pawel-gudel-eliatra <136344230+pawel-gudel-eliatra@users.noreply.github.com>

* Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> (cherry picked from commit e4f4817) Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

* [Enhancement] Parallel test jobs for CI (#2861) * Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> (cherry picked from commit e4f4817) Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> * Generalize Backwards Compatibility tests so we can test from any version to any version With an issue reported indicating that there are serialization issue between 1.3 and 2.X, making sure that we can reproduce the errors. This new workflow(s) will make sure that we aren't breaking BWC with changes we are adding to 2.X and will give us the flexibility to test certain migration workflows. Fixing an issue where the BWC tests were not actually building or executing causing the clusters to spin up and then immediately spin down. We will need to invest more energy into running multiple kinds of security plugin specific scenarios through the test system. Signed-off-by: Peter Nied <petern@amazon.com> * Remove seperate integration test workflow Signed-off-by: Peter Nied <petern@amazon.com> * Adjust BWC setting to point to correct BWC versions Signed-off-by: Peter Nied <petern@amazon.com> * Fix spotless issues Signed-off-by: Peter Nied <petern@amazon.com> --------- Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> Signed-off-by: Peter Nied <petern@amazon.com> Co-authored-by: Peter Nied <petern@amazon.com>

This change combines the many updates from the following commits: * 5f62e8a dependabot: bump commons-io:commons-io from 2.11.0 to 2.13.0 (opensearch-project#3074) * 2f69a10 bump com.github.wnameless.json:json-base from 2.4.0 to 2.4.1 (opensearch-project#3062) * c0e50da dependabot: bump org.cryptacular:cryptacular from 1.2.4 to 1.2.5 (opensearch-project#3071) * d3488e8 dependabot: bump kafka_version from 3.5.0 to 3.5.1 (opensearch-project#3041) * ab6778d Update ospackage, checker-qual, zcxvbn and error_prone_annotations, camel-xmlsecurity (opensearch-project#3023) * 0e6608d Bump JSON libs (opensearch-project#2926) * df07bea SAML 4.3.0 addition persmission (opensearch-project#2987) * e5348eb Change maven repo location for compatibility check (opensearch-project#2980) * 4a1ec53 Bump jaxb to 2.3.8 (opensearch-project#2977) * 9599155 Bump guava to 32.1.1-jre (opensearch-project#2976) * 06eed60 dependabot: bump org.glassfish.jaxb:jaxb-runtime from 2.3.4 to 4.0.3 (opensearch-project#2970) * 1113244 Bump eventbus to 3.3.1 (opensearch-project#2965) * 99ff7b3 dependabot: bump org.apache.bcel:bcel from 6.6.0 to 6.7.0 (opensearch-project#2969) * 0794c3f dependabot: bump jakarta.xml.bind:jakarta.xml.bind-api (opensearch-project#2968) * 9e6aab3 dependabot: bump com.google.j2objc:j2objc-annotations from 1.3 to 2.8 (opensearch-project#2963) * 8227f64 dependabot: bump com.sun.istack:istack-commons-runtime (opensearch-project#2960) * 8e044a6 dependabot: bump org.apiguardian:apiguardian-api from 1.0.0 to 1.1.2 (opensearch-project#2964) * 49cbf52 Remove commons-collections 3.2.2 (opensearch-project#2924) * 092e8f5 Bump SAML libs (opensearch-project#2927) * 8ab7cb4 Resolve CVE-2023-2976 by forcing use of Guava 32.0.1 (opensearch-project#2937) * 4eef662 Clean up and bump Apache libs (opensearch-project#2925) * 9a72355 Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901) * e4f4817 [Enhancement] Parallel test jobs for CI (opensearch-project#2861) * d871af3 Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886) * c808692 Format everything (opensearch-project#2866) Signed-off-by: Peter Nied <petern@amazon.com>

@nibix

…t" (#2935) As a part of discussions in #2861 there's been determined that CI workflow in Github Actions is excluding a lot of unnecessary task. @nibix created #2913 to address this problem. Most of the unneeded tasks has been already removed in #2861. This PR removes last existing part. @peternied's confirmed removal of dependency in his [comment](#2913 (comment)). This change removes the dependency between "_jacocoTestReport_" and "_test_" tasks. Specifically the dependency that forces to start "_test_" task always when "_jacocoTestReport_" is called. This allows us to always generate coverage report in the end of every task run in "CI" workflow without having to exclude "_test_" task in every single one of them. Unfortunately this will break functionality to create code coverage by starting only "_jacocoTestReport_" task. It looks like this was not used widely and was certainly not used in any of Github Actions Workflows. This will not break creating coverage reports in the end of "_test_" task. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

commit 1e24bbb Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 12:06:53 2023 -0700 Fixed the exception in keyutils Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 4b406c5 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 11:41:14 2023 -0700 Change the null check right after the jwtparserbuilder Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a805843 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 11:11:13 2023 -0700 Change to use copyof in getSecurityRoles in AuthCredentials class Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 308f269 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 11:02:04 2023 -0700 Add the null or empty check for signingkey in keyUtils Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 40eed32 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 09:55:19 2023 -0700 Fix V6 and V7 and lint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 3c76151 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 25 09:29:17 2023 -0700 Fix comment - Craig Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 7f2fc19 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 24 23:01:24 2023 -0700 Fix some comments 08/24 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 4841b25 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 24 10:50:24 2023 -0700 Add the constant for defaut service in create obo endpoint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 477b505 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 20:18:05 2023 -0700 Remove the unrelated line in AccountApiTest l77 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit f42d2f5 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 19:40:33 2023 -0700 Re-edit the error msg for createoboendpoint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a272ccb Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 19:34:56 2023 -0700 Rename the obo endpoint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit c021473 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 19:30:45 2023 -0700 Correct the getClusterName() Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 8b5158d Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 19:11:07 2023 -0700 Use ClusterInfoHolder to pass clusterName Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 336aa57 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 18:18:42 2023 -0700 Change the error msg in jwtvendorTests too Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 682379d Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 17:21:11 2023 -0700 Switch to assertThat in obo authenticator test Signed-off-by: Ryan Liang <jiallian@amazon.com> commit cb3406a Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:59:30 2023 -0700 Add comment in DynamicConfigModelV7 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 15c8530 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:47:13 2023 -0700 Change to assertThrows for obo authenticator tests Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e56bf01 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:39:24 2023 -0700 Rename to OnBehalfOfSettings in ConfigV6 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 62cfb4f Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:29:31 2023 -0700 Add comment for authentication failure in obo authenticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit d0ebe91 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:22:08 2023 -0700 Specify the error msg of missing signing key in jwtvendor Signed-off-by: Ryan Liang <jiallian@amazon.com> commit ca95380 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 16:12:46 2023 -0700 Refactor the jwtvendor expiry and set up upper limit Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 884f7a1 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 13:51:11 2023 -0700 Flip the boolean logic of roleSecurityMode Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e1021c2 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 13:36:05 2023 -0700 Refactor the bwc mode into roleSecurityMode Signed-off-by: Ryan Liang <jiallian@amazon.com> commit c1a825b Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 12:03:17 2023 -0700 Refactor in jwtVendor 1 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 8eac5cd Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 23 09:53:24 2023 -0700 Change the comment in backend registry Signed-off-by: Ryan Liang <jiallian@amazon.com> commit b0ac41a Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 17:58:15 2023 -0700 Revert the unrelated change in SecurityRestFilter Signed-off-by: Ryan Liang <jiallian@amazon.com> commit b64460d Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 17:53:32 2023 -0700 Remove stale function Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 54bca2a Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 17:36:05 2023 -0700 Refactor the obo endpoint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e429d7b Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 17:16:47 2023 -0700 Refactor the KeyUtils OBOAuthenticator and JwtAuthenticator with jwtParserBuilder Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 72dcec1 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 16:22:42 2023 -0700 Some minor refactoring in obo authenticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 6f0e79b Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 15:46:19 2023 -0700 Remove the malformed token warning for backendroles extraction Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 1ba378e Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 15:42:38 2023 -0700 Refactor the EncryptionDecryptionUtil Signed-off-by: Ryan Liang <jiallian@amazon.com> commit b315559 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 12:02:06 2023 -0700 Fix the exception type in JwtVendorTests Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 6f49801 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 11:35:10 2023 -0700 Remove the if condition in oboconfig for integ testing Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 034aa34 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 11:22:45 2023 -0700 Set up creatJwkException in exceptionUtils and apply that in JwtVendor constructor Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 0f0478d Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 10:14:55 2023 -0700 Rename the KeyUtils Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a4e7aff Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 09:46:27 2023 -0700 Refactor the backendroles claim into br Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 2ff746e Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 09:31:50 2023 -0700 Fix lint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 9ce36dc Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 01:11:24 2023 -0700 Refactor the OBO Authenticator part2 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e52c5ce Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 00:55:33 2023 -0700 Refactor the backendrole extraction in oboauthenticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 9c9e060 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 00:41:51 2023 -0700 Refactor the role extraction in oboauthenticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 387027b Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 00:18:31 2023 -0700 Refactor the logic in JwtVendor Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 267255c Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 22 00:01:15 2023 -0700 Add integration test case for obo permission Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 93bc8c6 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 19:18:49 2023 -0700 Remove unused constants variable in OBO authenticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 878a107 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 19:10:27 2023 -0700 Refactor the encryptiondecryptionutilstests Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 1c1bae6 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 14:53:45 2023 -0700 Remove the null check in oboconfig Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 7e3824e Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 14:44:07 2023 -0700 Remove the wording of seconds in obo endpoint and make the expiry into configconstants Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a126512 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 12:48:33 2023 -0700 Use constant util in Obo integration test Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e5a32c6 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 12:13:50 2023 -0700 Rename the obo endpoint path to generateobotoekn Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e09a902 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 21 09:15:53 2023 -0700 Remove the unused instance in configV6 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit af8aaa7 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 18 13:05:04 2023 -0700 Fix the lint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 9103b23 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 18 13:01:56 2023 -0700 Add permission obo/create for accessing create obo endpoint Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 2349213 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 15:54:38 2023 -0700 Change the name into keyUtils with the s Signed-off-by: Ryan Liang <jiallian@amazon.com> commit dae0ac7 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 15:48:12 2023 -0700 Switch to try/catch + assertEquals for JwtVendorTest Signed-off-by: Ryan Liang <jiallian@amazon.com> commit d918d7a Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 15:39:46 2023 -0700 Change the JwtVendorTest with manually built-up assertThrow Signed-off-by: Ryan Liang <jiallian@amazon.com> commit f47026b Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 14:35:13 2023 -0700 Fix the typo in exceptionUtils Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 43b8d5d Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 14:33:22 2023 -0700 Remove stacktrace debug statement in OBOAutehnticator Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a2c6db1 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 13:04:30 2023 -0700 Change some of the methods name into camle case instead of snake case Signed-off-by: Ryan Liang <jiallian@amazon.com> commit ee79b49 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 11:55:11 2023 -0700 Add unit tests for EncryptionDecryptionUtil Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 3ebff2b Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 17 09:43:43 2023 -0700 Work around for not set static cluster service Signed-off-by: Ryan Liang <jiallian@amazon.com> commit a4efad6 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 13:31:57 2023 -0700 Modify the getDynamicOnBehalfOfSettings() to return settings.Empty if there is no changes Signed-off-by: Ryan Liang <jiallian@amazon.com> commit e23d757 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 12:26:39 2023 -0700 Encapsulate the logic for endpoints access checking into a method Signed-off-by: Ryan Liang <jiallian@amazon.com> commit be26148 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 12:09:25 2023 -0700 Remove useless comments Signed-off-by: Ryan Liang <jiallian@amazon.com> commit bef85da Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 11:57:39 2023 -0700 Remove the enforcing of token type for OBO auth Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 1f79431 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 11:45:45 2023 -0700 Change the field name reason in obo endpoint to description Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 1f12e5e Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 16 11:15:56 2023 -0700 Change the backendrole check's claim name into br Signed-off-by: Ryan Liang <jiallian@amazon.com> commit b2c7d75 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 15 23:30:25 2023 -0700 Address some comment 2 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit d79973c Author: Ryan Liang <jiallian@amazon.com> Date: Tue Aug 15 23:24:16 2023 -0700 Address some comment 1 Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 8a96cab Author: Sam <128482925+samuelcostae@users.noreply.github.com> Date: Fri Aug 18 14:43:07 2023 +0100 Feature/extensions bwc setting (opensearch-project#3180) ### Description This Draft PR includes the new setting bwcPluginMode (backward compatible plugin mode for extensions ) ### Issues Resolved opensearch-project#2616 Is this a backport? If so, please add backport PR # and/or commits # ### Testing [Please provide details of testing done: unit testing, integration testing and manual testing] ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [x] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Sam <samuel.costa@eliatra.com> commit 91f4478 Merge: e42e4d3 88b6d23 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 14 23:30:37 2023 -0700 Merge branch 'main' into feature/extensions commit e42e4d3 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Mon Aug 14 10:12:58 2023 -0700 [Feature/Extension] Remove hostmapping from create OBO endpoint. (opensearch-project#3161) Signed-off-by: Ryan Liang <jiallian@amazon.com> commit ade34b4 Merge: 6d8e0e2 05f12d8 Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 11 11:37:21 2023 -0700 Merge branch 'main' into feature/extensions commit 6d8e0e2 Merge: 493b53f 3139c18 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Aug 10 21:11:43 2023 -0700 Merge branch 'main' into feature/extensions commit 493b53f Merge: 30cf5b1 46989b5 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 9 12:00:14 2023 -0700 Merge branch 'main' into feature/extensions commit 30cf5b1 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Wed Aug 9 10:10:24 2023 -0700 [Feature/Extension] Add cluster id check for OBO Authenticator (opensearch-project#3117) --------- Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 058f8ec Merge: d643fb2 6cc90e6 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Aug 7 12:33:57 2023 -0700 Merge branch 'main' into feature/extensions commit d643fb2 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Fri Aug 4 22:57:18 2023 -0700 [Feature/Extension] Restrict OBO token's usage for certain endpoints (opensearch-project#3008) Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 2319059 Merge: d634d60 527495d Author: Ryan Liang <jiallian@amazon.com> Date: Fri Aug 4 08:50:34 2023 -0700 Merge branch 'main' into feature/extensions commit d634d60 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Wed Aug 2 13:09:03 2023 -0700 [Feature/Extension] Add configuration of disable OBO (opensearch-project#3047) Signed-off-by: Ryan Liang <jiallian@amazon.com> commit df3dba3 Merge: 1268dee 5384272 Author: Ryan Liang <jiallian@amazon.com> Date: Wed Aug 2 09:42:04 2023 -0700 Merge branch 'main' into feature/extensions commit 1268dee Merge: a9451dd 8063e1b Author: Ryan Liang <jiallian@amazon.com> Date: Tue Jul 25 11:23:05 2023 -0700 Merge branch 'main' into feature/extensions commit a9451dd Merge: 671c772 59e2657 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Jul 24 13:30:37 2023 -0700 Merge branch 'main' into feature/extensions commit 671c772 Merge: 67515bc f1be2d7 Author: Ryan Liang <jiallian@amazon.com> Date: Tue Jul 18 09:04:25 2023 -0700 Merge branch 'main' into feature/extensions commit 67515bc Merge: 88f32e9 0e6608d Author: Ryan Liang <jiallian@amazon.com> Date: Thu Jul 13 11:18:07 2023 -0700 Merge branch 'main' into feature/extensions commit 88f32e9 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Fri Jul 7 11:42:21 2023 -0700 [Feature/Extension] Add oboauthcbackend registry and set up e2e endpoint testing flow (opensearch-project#2857) * Add OBO Authbackend Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Co-authored-by: Peter Nied <petern@amazon.com> commit 8c3c639 Merge: 21891d7 4eef662 Author: Ryan Liang <jiallian@amazon.com> Date: Mon Jul 3 10:34:27 2023 -0700 Merge branch 'main' into feature/extensions commit 21891d7 Merge: 8ad24ad 7546c05 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Jun 29 13:28:12 2023 -0700 Merge branch 'feature-branch-sync-629' into feature/extensions commit 8ad24ad Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Thu Jun 29 13:23:04 2023 -0700 Revert "Feature branch sync 06/29/2023 (opensearch-project#2918)" (opensearch-project#2920) This reverts commit 748a711. commit 748a711 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Thu Jun 29 12:37:08 2023 -0700 Feature branch sync 06/29/2023 (opensearch-project#2918) * add search model group permission to ml_read_access role (opensearch-project#2855) * add search model group permission to ml_read_access role Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> * IntegrationTest spotless (opensearch-project#2863) Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Format everything (opensearch-project#2866) * Use boucycastle PEM reader instead of reg expression (opensearch-project#2864) Use BouncyCastle PEMReader instead of regular expression to read and parse private key pem files. Signed-off-by: Andrey Pleskach <ples@aiven.io> * Adding field level security test cases for FlatFields (opensearch-project#2876) Signed-off-by: Peter Nied <petern@amazon.com> * Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886) * Update snappy to 1.1.10.1 and guava to 32.0.1-jre Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade kafka to 3.5.0 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Force snappy Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add runtime dependency on org.scala-lang.modules:scala-java8-compat_3:1.0.2 to fix issue with KafkaSinkTest Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> * Role permissions order tool and workflow (opensearch-project#2733) * Check Permissions Order tool and workflow Adds a NodeJS tool that can inspect yaml role definitions, check if they are in alphabetical order, correct them if required. Signed-off-by: Peter Nied <peternied@hotmail.com> * Apply fixes to roles.yml files Signed-off-by: Peter Nied <peternied@hotmail.com> * Fixing busted test, adding findArrayInJson for response bodies Signed-off-by: Peter Nied <petern@amazon.com> --------- Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Peter Nied <petern@amazon.com> * Misc changes (opensearch-project#2902) Moved isStatic and isReserved methods to the SecurityDynamicConfiguration class Signed-off-by: Andrey Pleskach <ples@aiven.io> * Update triaging guidelines (opensearch-project#2899) * Update triaging guidelines Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * fix cluster perm classification for msearch template (opensearch-project#2892) * fix cluster perm classification for msearch template Signed-off-by: Derek Ho <dxho@amazon.com> * move test to unit test file Signed-off-by: Derek Ho <dxho@amazon.com> * fully revert integration test file Signed-off-by: Derek Ho <dxho@amazon.com> * Update src/test/java/org/opensearch/security/privileges/PrivilegesEvaluatorUnitTest.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * spotless Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * [Doc] Add architecture document (opensearch-project#2869) * Add initial architecture document Signed-off-by: Peter Nied <petern@amazon.com> * [Enhancement] Parallel test jobs for CI (opensearch-project#2861) * Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> * Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901) jdk15to18 contains fix for - CVE-2023-33201 - Medium Severity Vulnerability Signed-off-by: Andrey Pleskach <ples@aiven.io> * Spotless Apply Signed-off-by: Ryan Liang <jiallian@amazon.com> --------- Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Co-authored-by: Bhavana Ramaram <rbhavna@amazon.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Andrey Pleskach <ples@aiven.io> Co-authored-by: Peter Nied <petern@amazon.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Derek Ho <derek01778@gmail.com> Co-authored-by: pawel-gudel-eliatra <136344230+pawel-gudel-eliatra@users.noreply.github.com> commit 7546c05 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Jun 29 11:50:58 2023 -0700 Spotless Apply Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 81b7818 Author: Andrey Pleskach <ples@aiven.io> Date: Thu Jun 29 15:54:21 2023 +0200 Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901) jdk15to18 contains fix for - CVE-2023-33201 - Medium Severity Vulnerability Signed-off-by: Andrey Pleskach <ples@aiven.io> commit 95efddd Author: pawel-gudel-eliatra <136344230+pawel-gudel-eliatra@users.noreply.github.com> Date: Wed Jun 28 22:41:46 2023 +0200 [Enhancement] Parallel test jobs for CI (opensearch-project#2861) * Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> commit 766389b Author: Peter Nied <petern@amazon.com> Date: Wed Jun 28 15:28:11 2023 -0500 [Doc] Add architecture document (opensearch-project#2869) * Add initial architecture document Signed-off-by: Peter Nied <petern@amazon.com> commit c1d2127 Author: Derek Ho <derek01778@gmail.com> Date: Wed Jun 28 15:21:04 2023 -0400 fix cluster perm classification for msearch template (opensearch-project#2892) * fix cluster perm classification for msearch template Signed-off-by: Derek Ho <dxho@amazon.com> * move test to unit test file Signed-off-by: Derek Ho <dxho@amazon.com> * fully revert integration test file Signed-off-by: Derek Ho <dxho@amazon.com> * Update src/test/java/org/opensearch/security/privileges/PrivilegesEvaluatorUnitTest.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * spotless Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> commit 37f277e Author: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Mon Jun 26 15:28:13 2023 -0400 Update triaging guidelines (opensearch-project#2899) * Update triaging guidelines Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> commit 926bdda Author: Andrey Pleskach <ples@aiven.io> Date: Mon Jun 26 20:09:39 2023 +0200 Misc changes (opensearch-project#2902) Moved isStatic and isReserved methods to the SecurityDynamicConfiguration class Signed-off-by: Andrey Pleskach <ples@aiven.io> commit 9cd0198 Author: Peter Nied <petern@amazon.com> Date: Mon Jun 26 10:04:39 2023 -0500 Role permissions order tool and workflow (opensearch-project#2733) * Check Permissions Order tool and workflow Adds a NodeJS tool that can inspect yaml role definitions, check if they are in alphabetical order, correct them if required. Signed-off-by: Peter Nied <peternied@hotmail.com> * Apply fixes to roles.yml files Signed-off-by: Peter Nied <peternied@hotmail.com> * Fixing busted test, adding findArrayInJson for response bodies Signed-off-by: Peter Nied <petern@amazon.com> --------- Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Peter Nied <petern@amazon.com> commit 4bb144f Author: Craig Perkins <cwperx@amazon.com> Date: Wed Jun 21 09:31:47 2023 -0400 Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886) * Update snappy to 1.1.10.1 and guava to 32.0.1-jre Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade kafka to 3.5.0 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Force snappy Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add runtime dependency on org.scala-lang.modules:scala-java8-compat_3:1.0.2 to fix issue with KafkaSinkTest Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> commit c71d9b3 Author: Peter Nied <petern@amazon.com> Date: Tue Jun 20 13:36:01 2023 -0500 Adding field level security test cases for FlatFields (opensearch-project#2876) Signed-off-by: Peter Nied <petern@amazon.com> commit e3b4f8f Author: Andrey Pleskach <ples@aiven.io> Date: Mon Jun 19 16:18:28 2023 +0200 Use boucycastle PEM reader instead of reg expression (opensearch-project#2864) Use BouncyCastle PEMReader instead of regular expression to read and parse private key pem files. Signed-off-by: Andrey Pleskach <ples@aiven.io> commit ef6224c Author: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Thu Jun 15 23:32:10 2023 -0400 Format everything (opensearch-project#2866) commit ef048a2 Author: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Thu Jun 15 11:57:25 2023 -0400 IntegrationTest spotless (opensearch-project#2863) Signed-off-by: Stephen Crawford <steecraw@amazon.com> commit b6bfb11 Author: Bhavana Ramaram <rbhavna@amazon.com> Date: Tue Jun 13 17:00:34 2023 -0700 add search model group permission to ml_read_access role (opensearch-project#2855) * add search model group permission to ml_read_access role Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> commit 26244e9 Merge: 56e77fe 1691ca7 Author: Craig Perkins <cwperx@amazon.com> Date: Tue Jun 13 16:14:33 2023 -0400 Merge branch 'format-feature-extensions-apply' into feature/extensions commit 1691ca7 Merge: efcadd4 2e263b8 Author: Craig Perkins <cwperx@amazon.com> Date: Tue Jun 13 16:05:59 2023 -0400 Merge branch 'main' into format-feature-extensions-apply commit efcadd4 Merge: 1a09a87 ceb5ad2 Author: Craig Perkins <cwperx@amazon.com> Date: Fri Jun 9 10:16:02 2023 -0400 Merge branch 'main' into format-feature-extensions-apply commit 1a09a87 Author: Craig Perkins <cwperx@amazon.com> Date: Fri Jun 9 09:57:42 2023 -0400 Run spotlessApply Signed-off-by: Craig Perkins <cwperx@amazon.com> commit 01917ff Author: Craig Perkins <cwperx@amazon.com> Date: Fri Jun 9 09:55:09 2023 -0400 Remove other spotless section Signed-off-by: Craig Perkins <cwperx@amazon.com> commit c83ad28 Author: Craig Perkins <cwperx@amazon.com> Date: Fri Jun 9 09:53:35 2023 -0400 Add formatting changes in feature/extensions Signed-off-by: Craig Perkins <cwperx@amazon.com> commit 56e77fe Merge: fa0fcc3 33aebb9 Author: Ryan Liang <jiallian@amazon.com> Date: Wed May 24 11:31:54 2023 -0700 Merge branch 'main' into feature/extensions commit fa0fcc3 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Tue May 16 10:55:57 2023 -0700 [Feature/Extension] Rename the term 'extension' into 'on_behalf_of' (opensearch-project#2774) Signed-off-by: Ryan Liang <jiallian@amazon.com> commit f1cee3b Merge: 95f9c77 15860b6 Author: Ryan Liang <jiallian@amazon.com> Date: Tue May 16 09:33:59 2023 -0700 Merge branch 'main' into feature/extensions commit 95f9c77 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Thu May 11 11:50:12 2023 -0700 [Security/Extension] Extension Authentication Backend (opensearch-project#2672) * Extension Authentication-backend Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 8f02d8d Merge: 9515181 9d758f9 Author: Ryan Liang <jiallian@amazon.com> Date: Wed May 10 12:12:20 2023 -0700 Merge branch 'main' into feature/extensions commit 9515181 Author: Ryan Liang <jiallian@amazon.com> Date: Wed May 10 09:51:26 2023 -0700 Fix the conflicts Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 06055c3 Merge: df75a37 f4def32 Author: Ryan Liang <jiallian@amazon.com> Date: Wed May 10 09:34:05 2023 -0700 merge main into security extension feature branch commit df75a37 Author: MaciejMierzwa <dev.maciej.mierzwa@gmail.com> Date: Tue May 2 15:44:04 2023 +0200 Extensions config for JWT signing/encryption key (opensearch-project#2671) * Extensions config for JWT signing/encryption key Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com> commit d4e5f1f Merge: 4da62c3 6997f97 Author: Ryan Liang <jiallian@amazon.com> Date: Thu Apr 27 07:05:39 2023 -0700 Merge branch 'main' into feature/extensions commit 4da62c3 Merge: 73ab1fc 6ace852 Author: Craig Perkins <cwperx@amazon.com> Date: Tue Apr 25 09:38:52 2023 -0400 Merge branch 'main' into feature/extensions commit 73ab1fc Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Thu Apr 6 13:38:19 2023 -0700 [Security/Extension] Role encryption/decryption (opensearch-project#2620) * Encryption/Decryption of `roles` Signed-off-by: Ryan Liang <jiallian@amazon.com> commit 1681823 Author: Ryan Liang <109499885+RyanL1997@users.noreply.github.com> Date: Fri Mar 31 06:58:56 2023 -0700 [Security/Extension] JWT Vendor for extensions (opensearch-project#2567) * JWT Vendor for extensions Signed-off-by: Ryan Liang <jiallian@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com>

pawel-gudel-eliatra requested review from cliu123, cwperks, DarshitChanpura, davidlago, peternied, RyanL1997, stephen-crawford, reta and willyborankin as code owners June 14, 2023 12:58

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 1a6c826 to bf86452 Compare June 14, 2023 13:16

willyborankin requested changes Jun 14, 2023

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

build.gradle Outdated Show resolved Hide resolved

build.gradle Outdated Show resolved Hide resolved

peternied reviewed Jun 14, 2023

View reviewed changes

build.gradle Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

stephen-crawford reviewed Jun 15, 2023

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

build.gradle Show resolved Hide resolved

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 61078e7 to 9002220 Compare June 16, 2023 12:53

pawel-gudel-eliatra added 4 commits June 16, 2023 15:03

Applied :spotlessApply changes

25f4645

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Minor fixes after PR

0e1c958

* Revised excludes for: `sslTest`, `securityIntegrationTest` and `indicesTest` * Removed `opensslTest`, all its test are in `sslTest` Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 9002220 to 38e7a37 Compare June 16, 2023 13:05

pawel-gudel-eliatra added 4 commits June 16, 2023 17:44

Restored opensslTest

4db5e47

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Add more options to splitTestConfig

156675d

Right now it allows for more sophisticated configuration. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Hopefully fix for Code Coverage issues.

5673984

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 163950a to 5673984 Compare June 20, 2023 20:01

Restored split of SSL tests.

33f8318

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

peternied requested changes Jun 20, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

build.gradle Outdated Show resolved Hide resolved

build.gradle Show resolved Hide resolved

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 412a3f5 to ae391e6 Compare June 23, 2023 21:51

Shorten number of excludes; Shorten ci.yml

77cf616

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

pawel-gudel-eliatra force-pushed the split-ci-tests branch from 49b0ddd to 77cf616 Compare June 23, 2023 22:44

pawel-gudel-eliatra added 3 commits June 27, 2023 23:45

Restored split build tasks

d5085bc

Unfortunately running them from one list lenghtens time a lot Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Added short explanation

57b483e

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

Reorder matrix strategy.

1dc9f57

Is it possible to run windows jobs before ubuntu? Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

pawel-gudel-eliatra force-pushed the split-ci-tests branch from feb5587 to 1dc9f57 Compare June 27, 2023 22:06

nibix mentioned this pull request Jun 28, 2023

Clean up -x task exclusions in ci.yml #2913

Closed

Minor optimizations and fixes.

ebae0eb

* Fix for not excluded tests. * Reordered matrix strategy variables. * Restored unified list of Gradle task.. again! Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

pawel-gudel-eliatra force-pushed the split-ci-tests branch from d93ed57 to ebae0eb Compare June 28, 2023 09:22

Removed unnecessary args from intergrationTest GHA job

e48b74e

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

peternied approved these changes Jun 28, 2023

View reviewed changes

willyborankin approved these changes Jun 28, 2023

View reviewed changes

peternied merged commit e4f4817 into opensearch-project:main Jun 28, 2023

peternied added the backport 2.x backport to 2.x branch label Jun 28, 2023

This was referenced Jun 30, 2023

Improve serialization speeds #2802

Merged

Create concept of persistent ThreadContext headers that are unstashable opensearch-project/OpenSearch#8291

Merged

pawel-gudel-eliatra mentioned this pull request Jul 5, 2023

[Enchancement] Removed dependency between "test" and "jacocoTestReport" #2935

Merged

3 tasks

pawel-gudel-eliatra mentioned this pull request Jul 5, 2023

[Backport] Parallel test jobs for CI (#2861) #2936

Merged

DarshitChanpura mentioned this pull request Jul 12, 2023

[BUG] [CI] Tests under src/integrationTest folder not being executed in CI checks #2998

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Parallel test jobs for CI #2861

[Enhancement] Parallel test jobs for CI #2861

pawel-gudel-eliatra commented Jun 14, 2023

codecov bot commented Jun 14, 2023 •

edited

Loading

willyborankin left a comment

nibix commented Jun 14, 2023

peternied left a comment

nibix commented Jun 21, 2023

peternied commented Jun 21, 2023

peternied left a comment

peternied commented Jun 28, 2023

willyborankin left a comment

opensearch-trigger-bot bot commented Jun 28, 2023

peternied commented Jun 28, 2023

[Enhancement] Parallel test jobs for CI #2861

[Enhancement] Parallel test jobs for CI #2861

Conversation

pawel-gudel-eliatra commented Jun 14, 2023

Description

Issues Resolved

Testing

Check List

codecov bot commented Jun 14, 2023 • edited Loading

Codecov Report

willyborankin left a comment

Choose a reason for hiding this comment

nibix commented Jun 14, 2023

peternied left a comment

Choose a reason for hiding this comment

nibix commented Jun 21, 2023

peternied commented Jun 21, 2023

peternied left a comment

Choose a reason for hiding this comment

peternied commented Jun 28, 2023

willyborankin left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jun 28, 2023

peternied commented Jun 28, 2023

codecov bot commented Jun 14, 2023 •

edited

Loading