Feature branch sync 06/29/2023 (opensearch-project#2918)

* add search model group permission to ml_read_access role (opensearch-project#2855) * add search model group permission to ml_read_access role Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> * IntegrationTest spotless (opensearch-project#2863) Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Format everything (opensearch-project#2866) * Use boucycastle PEM reader instead of reg expression (opensearch-project#2864) Use BouncyCastle PEMReader instead of regular expression to read and parse private key pem files. Signed-off-by: Andrey Pleskach <ples@aiven.io> * Adding field level security test cases for FlatFields (opensearch-project#2876) Signed-off-by: Peter Nied <petern@amazon.com> * Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886) * Update snappy to 1.1.10.1 and guava to 32.0.1-jre Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade kafka to 3.5.0 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Force snappy Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add runtime dependency on org.scala-lang.modules:scala-java8-compat_3:1.0.2 to fix issue with KafkaSinkTest Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> * Role permissions order tool and workflow (opensearch-project#2733) * Check Permissions Order tool and workflow Adds a NodeJS tool that can inspect yaml role definitions, check if they are in alphabetical order, correct them if required. Signed-off-by: Peter Nied <peternied@hotmail.com> * Apply fixes to roles.yml files Signed-off-by: Peter Nied <peternied@hotmail.com> * Fixing busted test, adding findArrayInJson for response bodies Signed-off-by: Peter Nied <petern@amazon.com> --------- Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Peter Nied <petern@amazon.com> * Misc changes (opensearch-project#2902) Moved isStatic and isReserved methods to the SecurityDynamicConfiguration class Signed-off-by: Andrey Pleskach <ples@aiven.io> * Update triaging guidelines (opensearch-project#2899) * Update triaging guidelines Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * fix cluster perm classification for msearch template (opensearch-project#2892) * fix cluster perm classification for msearch template Signed-off-by: Derek Ho <dxho@amazon.com> * move test to unit test file Signed-off-by: Derek Ho <dxho@amazon.com> * fully revert integration test file Signed-off-by: Derek Ho <dxho@amazon.com> * Update src/test/java/org/opensearch/security/privileges/PrivilegesEvaluatorUnitTest.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * spotless Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * [Doc] Add architecture document (opensearch-project#2869) * Add initial architecture document Signed-off-by: Peter Nied <petern@amazon.com> * [Enhancement] Parallel test jobs for CI (opensearch-project#2861) * Split multiple tests into separate gradle tasks. * Tasks are configured in "splitTestConfig" map in build.gradle file. Map allows to use all patterns from TestFilter like: includeTestsMatching, excludeTestsMatching, includeTest etc. * Tasks are automatically generated from "splitTestConfig" map. * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to output task names to console. First one outputs them as a JSON and second - in gradlew "-x <TASK>" format to use in CLI. * Patterns included in tasks are automatically excluded from main "test" task but at the same time generated tasks are dependencies for "test". Running "gradlew test" will run whole suite at once. * CI pipeline has been configured to accomodate all changes. * New 'master' task to generate list of jobs to run in parallel. * Updated matrix strategy to include task name to start. Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> * Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901) jdk15to18 contains fix for - CVE-2023-33201 - Medium Severity Vulnerability Signed-off-by: Andrey Pleskach <ples@aiven.io> * Spotless Apply Signed-off-by: Ryan Liang <jiallian@amazon.com> --------- Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com> Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Co-authored-by: Bhavana Ramaram <rbhavna@amazon.com> Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Andrey Pleskach <ples@aiven.io> Co-authored-by: Peter Nied <petern@amazon.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Derek Ho <derek01778@gmail.com> Co-authored-by: pawel-gudel-eliatra <136344230+pawel-gudel-eliatra@users.noreply.github.com>
peternied · Jun 29, 2023 · 748a711 · 748a711
1 parent 26244e9
commit 748a711
Show file tree

Hide file tree

Showing 282 changed files with 23,462 additions and 17,558 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -6,13 +6,34 @@ env:
   GRADLE_OPTS: -Dhttp.keepAlive=false
 
 jobs:
-  build:
-    name: build
+  generate-test-list:
+    runs-on: ubuntu-latest
+    outputs:
+      separateTestsNames: ${{ steps.set-matrix.outputs.separateTestsNames }}
+    steps:
+    - name: Set up JDK for build and test
+      uses: actions/setup-java@v2
+      with:
+        distribution: temurin # Temurin is a distribution of adoptium
+        java-version: 17
+
+    - name: Checkout security
+      uses: actions/checkout@v2
+
+    - name: Generate list of tasks
+      id: set-matrix
+      run: |
+        echo "separateTestsNames=$(./gradlew listTasksAsJSON -q --console=plain | tail -n 1)" >> $GITHUB_OUTPUT
+
+  test:
+    name: test
+    needs: generate-test-list
     strategy:
       fail-fast: false
       matrix:
+        gradle_task: ${{ fromJson(needs.generate-test-list.outputs.separateTestsNames) }}
+        platform: [windows-latest, ubuntu-latest]
         jdk: [11, 17]
-        platform: ["ubuntu-latest", "windows-latest"]
     runs-on: ${{ matrix.platform }}
 
     steps:
@@ -29,12 +50,8 @@ jobs:
       uses: gradle/gradle-build-action@v2
       with:
         arguments: |
-          build test -Dbuild.snapshot=false
-          -x integrationTest
-          -x spotlessCheck
-          -x checkstyleMain
-          -x checkstyleTest
-          -x spotbugsMain
+          ${{ matrix.gradle_task }} -Dbuild.snapshot=false
+          -x test
 
     - name: Coverage
       uses: codecov/codecov-action@v1
@@ -59,7 +76,7 @@ jobs:
       fail-fast: false
       matrix:
         jdk: [17]
-        platform: ["ubuntu-latest", "windows-latest"]
+        platform: [ubuntu-latest, windows-latest]
     runs-on: ${{ matrix.platform }}
 
     steps:
@@ -78,18 +95,13 @@ jobs:
       with:
         arguments: |
           integrationTest -Dbuild.snapshot=false
-          -x spotlessCheck
-          -x checkstyleMain
-          -x checkstyleTest
-          -x spotbugsMain
 
   backward-compatibility:
-
     strategy:
       fail-fast: false
       matrix:
         jdk: [11, 17]
-        platform: ["ubuntu-latest", "windows-latest"]
+        platform: [ubuntu-latest, windows-latest]
     runs-on: ${{ matrix.platform }}
 
     steps:

diff --git a/.github/workflows/code-hygiene.yml b/.github/workflows/code-hygiene.yml
@@ -57,3 +57,26 @@ jobs:
       - uses: gradle/gradle-build-action@v2
         with:
           arguments: spotbugsMain
+
+  check-permissions-order:
+    runs-on: ubuntu-latest
+    name: Check permissions orders
+    steps:
+    - uses: actions/checkout@v2
+    - run: npm install yaml
+
+    - name: Check permissions order
+      run: |
+        exclude_pattern="(^|/)roles_invalidxcontent.yml($|/)
+                          (^|/)invalid_config/config.yml($|/)"
+        # Set pattern to exclude certain files
+        set -e
+        exit_code=0
+        for file in $(find . -name '*.yml' | grep -Ev "$exclude_pattern"); do
+          if ! node check-permissions-order.js "$file" --slient; then
+            exit_code=1
+            echo "Error: $file requires changes. Run the following command to fix:"
+            echo "node check-permissions-order.js $file --fix"
+          fi
+        done
+        exit $exit_code
diff --git a/.gitignore b/.gitignore
@@ -43,3 +43,7 @@ out/
 build/
 gradle-build/
 .gradle/
+
+# nodejs
+node_modules/
+package-lock.json
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -0,0 +1,131 @@
+# OpenSearch Security Plugin Architecture
+
+OpenSearch’s core systems do not include security features, these features are added by installing the Security Plugin. The Security Plugin extends OpenSearch to provide authentication, authorization, end to end Encryption, audit logging, and management interfaces.
+
+## Components
+
+The Security Plugin is packaged into a standard plugin zip file used by OpenSearch which can be installed by using the plugin tool. The security configuration is accessible on disk for modification before the node has been turned on.  After node startup, the admin tools or API endpoints can be used for dynamic changes.
+
+```mermaid
+graph TD
+    subgraph OpenSearch Node
+        subgraph File System
+            cfg[Security Configuration files]
+            adm[Admin Tools]
+        end
+        subgraph Indices
+            idx(Index 1..n)
+            secIdx[Security Index]
+        end
+        subgraph Plugins
+           pgns(Plugins 1..n)
+           sec[Security Plugin]
+        end
+
+        sec -- bootstrap security config --> cfg
+        sec -- refresh security config from cluster --> secIdx
+        adm -- backup/restore security config --> sec
+    end
+```
+
+### Security Plugin
+
+The runtime of the Security Plugin uses extension points to insert itself into the path actions. Several security management actions are registered in OpenSearch so they can be changed through REST API actions.
+
+### Security Configuration
+
+The security configuration is stored in an system index that is replicated to all nodes. When a change has been made to the configuration, the Security Plugin is reloaded to cleanly initialize its components with the new settings.
+
+#### Configuration Files
+
+When starting up with no security index detected in the cluster, the Security Plugin will attempt to load configuration files from disk into a new security index. The configuration files can be manually modified or sourced from a backup of a security index created using the admin tools.
+
+### Admin Tools
+
+For OpenSearch nodes to join a cluster, they need to have the same security configuration. Complete security configurations will include SSL settings and certificate files. The admin tools allow users to manage these settings and other features.
+
+## Flows
+
+### Authentication / Authorization
+
+The Security Plugin supports multiple authentication backends including an internal identity provider which works with HTTP basic authentication as well as support [external providers](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/) such as OpenId Connect (OIDC) and SAML.
+
+Authorization is governed by roles declared in the security configuration. Roles control resource access by referencing the transport action name and/or index names in combination with OpenSearch action names.
+
+Users are assigned roles via the role mappings. These mappings include backend role assignments from authentication providers as well as internal roles defined in the Security Plugin.
+
+```mermaid
+sequenceDiagram
+    title Basic Authorization flow
+    autonumber
+    participant C as Client
+    participant O as OpenSearch
+    participant SP as Security Plugin
+    participant RH as Request Handler
+    participant AL as Audit Log
+
+    C->>O: Request
+    O->>SP: Request Received
+    activate SP
+    SP->>SP: Authenticate user via internal/external auth providers
+    SP->>SP: Resolve Authorization for user
+    SP-->>O: Allow/Deny request
+    SP->>AL: Update Audit Log asynchronously
+    deactivate SP
+    O->>RH: Request continues to request handler
+    RH-->>O: Result
+    O->>C: Response
+```
+
+#### Multiple Authorization Provider flow
+
+Based on the order within the Security Plugin's configuration authentication providers are iterated through to discover which provider can authenticate the user.
+
+```mermaid
+sequenceDiagram
+    title Multiple Authorization Provider flow
+    autonumber
+    participant C as Client
+    participant SP as Security Plugin
+    participant IAP as Internal Auth Provider
+    participant EAP as External Auth Provider*
+    participant SC as Security Configuration
+
+    C->>SP: Incoming request
+    SP->>IAP: Attempt to authenticate internally
+    IAP-->>SP: Internal user result
+    loop for each External Auth Provider
+        SP->>EAP: Attempt to authenticate
+        EAP-->>SP: External user result
+    end
+    SP->>SC: Check Authorization rules
+    SC->>SC: Match user roles & permissions
+    SC-->>SP: Authorization result
+    SP-->>C: Response
+```
+
+#### Rest vs Transport flow
+
+OpenSearch treats external REST requests differently than internal transport requests. While REST requests allow for client-to-node communication and make use of API routes, transport requests are more structured and are used to communicate between nodes.
+
+```mermaid
+sequenceDiagram
+    title Rest vs Transport Flow
+    autonumber
+    participant C as Client
+    participant O as OpenSearch
+    participant SP as Security Plugin (Rest Filter & Security Interceptor)
+    participant AH as Action Handler
+
+    C->>O: Request
+    O->>SP: REST Request Received
+    SP->>SP: If using client cert, Authenticate
+    SP-->>O: Continue request
+    O->>SP: Transport Request Received
+    SP->>SP: Authenticate user via internal/external auth providers
+    SP->>SP: Resolve Authorization for user
+    SP-->>O: Allow/Deny request
+    O->>AH: Send transport request to action handler
+    AH-->>O: Result
+    O->>C: Response
+```
diff --git a/TRIAGING.md b/TRIAGING.md
@@ -25,9 +25,8 @@ Meetings are lightly structured as follows:
 1. Announcements: If there are any announcements to be made they will happen at the start of the meeting.
 2. Review of new issues: The meetings always start with reviewing all untriaged [issues](https://github.com/search?q=label%3Auntriaged+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=created&o=desc) for the security and security-dashboards repositories.
 3. Untriaged items: Review any [issues](https://github.com/search?q=-label%3Atriaged+is%3Aopen++is%3Aissue+repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues) that might have had the 'untriaged' label removed but require additional triage discussion.
-4. Open discussion: Next, we open the floor in case anyone wants to highlight an issue.
-5. Backlog discussion: Then, we review issues from the [backlogs](https://github.com/search?q=label%3A%22sprint+backlog%22+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=created&o=desc) of the security and security-dashboards repositories.
-6. Least recent discussed issue: Finally, to close out the meeting we will [review the oldest](https://github.com/search?q=+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=updated&o=asc) issues from both repositories, security and security-dashboards, to help identify issues that have languished.
+4. Pull request discussion: Then, we review the status of outstanding [pull requests](https://github.com/search?q=+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=pullrequests&ref=advsearch) from the security and security-dashboards repositories.
+5. Open discussion: Finally, we open the floor in case anyone wants to highlight an issue.
 
 There is no specific ordering within each category.