[Question] How do administrators control what extensions can do?

[peternied]

Problem Statement

Administrators need to be able to quickly understand what an extension might be able to do with an OpenSearch Cluster. Administrators might be familiar with the OpenSearch concepts, or this might be there first interaction. The way that extensions are allowed to do certain actions against the cluster should be clear for both skill sets.

Application Scopes for Extensions

Extensions get access through Application scopes that control broad behavior within the OpenSearch cluster from the extension configuration file[1]. For an example of other application permissions via scope, check out the documentation from Slacks application integration [2].

Allowing scopes

extension.yml

extensionName: myExtension
hostAddress: 127.0.0.1
...
scope: [
  "deprecation.action_user_context:read", # This extension gets user context on incoming action requests, this is a feature set for deprecation
  "extension_point.action:allowed", # This extension can use the ActionPlugin extension point
  "extension_point.cluster:allowed", # This extension can use the ClusterPlugin extension point
  "index:read", # This extension is allowed to read from indexes
  "request.impersonate_user:allowed", # This extension is provided user tokens that allow for impersonation of that user on requests
  "request.service_account:allowed" # This extension is provided with a service account to connect with the cluster
]

Scopes conventions

Scopes format is an single optional namespace + the name of the scope + : + the type of action. When presented, they are always sorted alphabetically to ensure consistent readability.

Scopes are provided on discovery

When the extensions is initialized, in the discovery call, the list of scopes are send to the extension so it can acknowledge or reject initialization.

sequenceDiagram
    actor Admin
    box white OpenSearch Process
       participant Node
       participant ExtensionManager
    end
    box white Separate Host
       participant Extension
    end

    Admin->>Node: Start OpenSearch
    Node->>ExtensionManager: Initialize
    ExtensionManager->>ExtensionManager: Read configuration
    ExtensionManager->>Extension: Initialize via REST API<BR>Including Scopes:[]
    Extension-->>ExtensionManager: Acknowledge/Reject request
    ExtensionManager->>Node: Fully initialized
    Admin->>Node: GET _cluster/health
    Node-->>Admin: `{ "status" : "green", ... }`

Loading

[P2] Dynamically Adjust Scopes

Scope can be adjusted dynamically without restarting a cluster. REST APIs for CRUD access to /extensions/{extensionId} allow updates to ExtensionSettings. When there are changes extensions will have an transport action SETTINGS_UPDATE that will include a copy of the ExtensionSettings including the scopes.

? Might be able to re-use REQUEST_EXTENSION_UPDATE_SETTINGS for this

Implicit scope enforcement for actions

The identity service is aware of the context of the all activity in the cluster. By using checkPermissions(...) api will support extension scopes.

identityService.getSubject().checkPermission(ApplicationScope.REQUEST_SERVICE_ACCOUNT);

By performing checks inside of NodeClient all local or transport actions will be denied by default. Only if the action has an annotation(s) and the subject's application scope matches one of those annotations will these actions be allowed.

This will allow rolling out these scopes over time as more access is unlocked.

sequenceDiagram
    participant Client
    participant OpenSearch
    participant SecurityPlugin
    participant NodeClient

    Client->>+OpenSearch: Sends request
    OpenSearch->>+SecurityPlugin: Sends request to
    SecurityPlugin->>SecurityPlugin: PrivilegeEvaluator to check request allowed
    SecurityPlugin->>OpenSearch: Accept/Reject request
    OpenSearch->>NodeClient: Sends request to
    NodeClient->>NodeClient: ApplicationScope to check request allowed
    NodeClient->>OpenSearch: Accept/Reject request
    OpenSearch-->>-Client: Returns response

Loading

Fine grain controls

Extensions high level application scopes are handled independently of any other security system. This allows for the Security Plugin to continue to manage fine grain levels of access, or alternative security plugin implementations.

Scopes are not a fine grain access system. Extension are limited in the number of ways they can be called by the OpenSearch cluster via extension points. The number of ways extensions can make requests into the cluster are also limited to impersonate user, which are triggered by actions in an OpenSearch cluster, and service account. The service account associated with an extension is where cluster administrators can use all access control systems that are available for users. When using the Security Plugin this includes role mappings, advanced document features such as document level and field level security.

• [1] https://github.com/opensearch-project/opensearch-sdk-java/blob/main/CREATE_YOUR_FIRST_EXTENSION.md#creating-an-extension-from-scratch
• [2] https://api.slack.com/legacy/oauth-scopes
• Allow extensions to register roles #2595
• [Design] What is the user model for defining what plugin/extension can do? OpenSearch#5934

[peternied]

@scrawfor99 Love to get some initial thoughts

[cwperks]

@peternied I'd like to add my 2 cents on this too.

I'll try to summarize succinctly in 3 parts:

1. For installation purposes I think we should be broad. Signal to the cluster admin what kind of extension they are installing.

For AD that means:

• ActionExtension - it will extend OpenSearch by adding new endpoints
• ScriptExtension
• JobSchedulerExtension

We can document precisely what it means for an extension to be any one of these types and when UX is developed for installation this could be a prompt on installation.

Using the ./bin/opensearch-plugin install tool you get prompted to accept the plugin's policy on installation. This prompt would be analogous and upfront explain to the cluster admin how the extension would like to extend OpenSearch.

➜  opensearch-3.0.0-SNAPSHOT ./bin/opensearch-plugin install file:/Users/cwperx/Desktop/distributions/opensearch-security-3.0.0.0-SNAPSHOT.zip
-> Installing file:/Users/cwperx/Desktop/distributions/opensearch-security-3.0.0.0-SNAPSHOT.zip
-> Downloading file:/Users/cwperx/Desktop/distributions/opensearch-security-3.0.0.0-SNAPSHOT.zip
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission /proc/sys/net/core/somaxconn#plus read
* java.lang.RuntimePermission accessClassInPackage.com.sun.jndi.*
* java.lang.RuntimePermission accessClassInPackage.sun.misc
* java.lang.RuntimePermission accessClassInPackage.sun.nio.ch
* java.lang.RuntimePermission accessClassInPackage.sun.security.x509
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission accessUserInformation
* java.lang.RuntimePermission createClassLoader
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission shutdownHooks
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission getNetworkInformation
* java.net.NetPermission getProxySelector
* java.net.SocketPermission * connect,accept,resolve
* java.security.SecurityPermission getProperty.ssl.KeyManagerFactory.algorithm
* java.security.SecurityPermission insertProvider.BC
* java.security.SecurityPermission org.apache.xml.security.register
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setProperty.ocsp.enable
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission org.apache.xml.security.ignoreLineBreaks write
* javax.security.auth.AuthPermission doAs
* javax.security.auth.AuthPermission modifyPrivateCredentials
* javax.security.auth.kerberos.ServicePermission * accept
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]

2. There should be a policy that governs how requests coming from an extension can interact with opensearch - it looks like a role, smells like a role and can be implemented like a role, but this is a policy that governs requests coming from an extension and should not be mappable to users.

request_from_ad_logs_readonly:
    extension_policy: true # Should not be mapped to a user
    reserved: true
    cluster_permissions:
        - 'cluster_monitor'
    index_permissions:
        - index_patterns:
            - 'logs-*'
        allowed_actions:
            - 'indices_monitor'
            - 'indices:admin/aliases/get'
            - 'indices:admin/mappings/get'

3. The service account created for an extension (assuming the extensions needs one - not all do) can have a token issued for it and that token can have a single role associated with it. For most cases, it should be narrowly targeting the system index reserved by an extension so it can interact with it.

ephemeral_role:
    reserved: true
    index_permissions:
        - index_patterns:
            - '.opendistro-anomaly-results*'
            - '.opendistro-anomaly-detector*'
            - '.opendistro-anomaly-checkpoints'
            - '.opendistro-anomaly-detection-state'
        allowed_actions:
            - 'indices_all'

[peternied]

@cwperks I would like to separate fine-grain access control from this discussion, it's important and terribly complex. Supporting these scenarios is a better fit for service accounts and the tools associated with them.

• [Question] Service Account Specifications #2597

I think I might be missing a concern of yours could you call it out specifically or what is the user scenario that is missed or incomplete? Extensions honoring security posture are new we have considerable flexibility in what we implement.

[cwperks]

@peternied Yes, let me try to explain more in detail. I really would like to dig into this further so hear me out. On a high-level, I think if we define these policies (the policy that governs how requests coming from a service on behalf of a user can interact with opensearch) like roles that there will be a great deal of code re-use and it would also deliver a high bar of security and flexibility right off the bat. In the comment above I described a convention for naming a policy per extension/service, I now think that's too inflexible so I have another proposal but before I introduce that let's clearly state the problem.

Problem

The security plugin will now be issuing tokens on behalf of a user to give to an external service (I will be using extension/service interchangeably. Service refers to something more generic than an extension. An extension is a service.) When a service makes a request on behalf of the user utilizing this token it must be authorized by the security plugin before the request is executed.

How is this request authorized?

For some background assume the payload of an (on-behalf-of) access token looks like:

# Payload
{
  "iss": "<cluster_name>",
  "iat":1676908684,
  "exp":1676908744,
  "sub":"<principal_identifier_token>",
  "r":"<encrypted_mapped_roles>", # r for roles
  "br": "<encrypted_backend_roles>", # br for backend_roles
  "aud": "{extensionUniqueId}" // The identifier of the service this token is issued for
}

When authorizing this request, privilege evaluation will be done at 2 gates:

1. Service gate - In this gate privilege evaluation is performed to ensure that the request on-behalf-of a user from the specific service is permitted
2. User gate - In this gate, the user's privileges are evaluated to ensure the user can perform the action

Proposal: Map service + user to role(s) [policy] that specify what a request on behalf of the user from the service can do

For this proposal, I want you to imagine the all_access role renamed. For this proposal think of the all_access role as act_as_original_user

# roles.yml

act_as_original_user:
  extension_policy: true
  reserved: true
  description: "This policy permits any request from a service on-behalf-of a user. In effect, the service is allowed to act as if it were the user."
  cluster_permissions:
    - "*"
  index_permissions:
    - index_patterns:
        - "*"
      allowed_actions:
        - "*"
  extension_permissions: # Yes, this permits calls to other extensions too
    - "*"

This policy could be defined in the roles section of the security plugin and I believe we can also use roles_mappings to map this to a user in the following way.

# roles_mapping.yml

act_as_original_user:
  reserved: false
  description: "Maps <service_id> extension to act_as_original_user"
  services_mappings:
   <service_id>:
      users: 
        - "*"

Read this as map all users of the <service_id> extension can be mapped to the act_as_original_user role (policy) that will be utilized when evaluating privileges at the service gate.

Zooming in on the service gate

The service gate is the first gate the request must get through to be considered authorized. At the service gate the request must be resolved to a set of roles (policies) that can be checked to see if the request is permitted.

Imagine a function:

Set<String> mappedPolicies = mapPolicies(user, serviceId)

This function goes through the roles_mapping as defined above and resolves to act_as_original_user

If this set is empty we will block the request - some policy needs to be resolved to.

If we can resolve to a set of policies then let's re-use the PrivilegesEvaluator using this set of roles and if granted, proceed to the User gate.

The term roles is overloaded

The term roles is overloaded in the security plugin. To be clear, when I speak of roles there are 3 concepts to consider:

1. role - I think of these as internal roles - these are the roles defined in roles.yml
2. backend_roles - Backend roles are typically extracted from another IdP, but not necessarily. They can be assigned in OSD as a grab bag list of strings associated with a user.
3. mappedRoles - The mapped roles are the result of the roles resolution process which in effect maps backend roles to internal roles. Roles can also be mapped by the caller's IP Address/Hostname.

The hostname mapping I take a bit of an issue with because I think its more apt to conceptually think about it as a policy. An IP Address gets mapped to a role so a set of privileges can be evaluated, but IMO its better to think about the roles as a policy that governs how requests from that IP Address can interact with the cluster.

Comments on Service Account / Service Account Token

A service account token is used for the extension to interact with the OpenSearch cluster as itself and not on behalf of a user.

My thoughts is that absent of UX, we should have a cluster admin define if they want a service account + a token through extensions/extensions.yml. Since this file is controlled by the cluster admin, I think we should assume that any setup in this file has the explicit permission of the cluster admin so we should take it as is.

In the special case of an extension requesting to reserve indices we may wish to automatically provide a token scoped to those reserved indices.

Take for example ad_settings.yml - the settings file for AD extension:

extensionName: anomaly-detection
hostAddress: 127.0.0.1
hostPort: 4532
opensearchAddress: 127.0.0.1
opensearchPort: 9200
reservedIndices: [".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state"]

Inside of extensions/extensions.yml the cluster admin can grant AD extension reserved indices via a setting like:

extensions:
  - name: anomaly-detection
    uniqueId: ad
    hostAddress: '127.0.0.1'
    port: '4532'
    version: '1.0'
    opensearchVersion: '3.0.0'
    minimumCompatibleVersion: '3.0.0'
    allowReservedIndices: true

Since this extension is requesting to reserve indices and the administrator has given permissions, we should automatically create a service account and a service account token scoped to those indices and send it to the extension.

Question: What if the extensions wants reserved indices and other permissions?

Additional service account permissions should live in extensions/extensions.yml which is owned by the cluster admin to ensure the cluster admin is the one that explicitly grants the permissions. An extension developer who desires their extension to have the ability to do more on its own must provide installation instructions to the cluster admin to tell them what to add in extensions.yml.

This is contrived, but imagine that there is an AD extension that wants to act as a daemon on logs-*. The developer of such extension would instruct the cluster admin to add serviceAccountPermissions inside the extensions configuration in extensions.yml.

# ad-settings.yml

extensionName: anomaly-detection
hostAddress: 127.0.0.1
hostPort: 4532
opensearchAddress: 127.0.0.1
opensearchPort: 9200
reservedIndices: [".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state"]

and on the cluster admin side in extensions.yml

# Read this as the extensions is granted permission to read from logs-* as a daemon - on its own

extensions:
  - name: anomaly-detection
    uniqueId: ad
    hostAddress: '127.0.0.1'
    port: '4532'
    version: '1.0'
    opensearchVersion: '3.0.0'
    minimumCompatibleVersion: '3.0.0'
    allowReservedIndices: true
    serviceAccountPermissions:
      index_permissions:
        - index_patterns:
            - 'logs-*'
          allowed_actions:
            - 'read'

In this case we could either create two roles associated with the service account, 1 scoped to the reserved indices and 1 and with the permissions defined in serviceAccountPermissions or we could combine them to create a single role. Maybe 2 roles makes sense here to make it straightforward to implement.

Using these settings its possible to create in-memory (ephemeral) roles with the serviceAccountPermissions and give strong ownership of the reserved indices to the extension by looking up the extension's reserved indices in the registry on a request made using the extension's service account token.

Please let me know your thoughts and I'd really love to dive in further because I'm becoming pretty convinced that this is both a good design and maximizes code re-use.

Edit: Adding section on complications with FLS, DLS and Field Masking

Complications with FLS/DLS/Field Masking

Roles in security plugin also allow for fine-grained access control on data. When a user has multiple roles, any additional restrictions are OR-ed together across the roles to return all documents and fields visible to a user across the roles. With on-behalf-of tokens, there are 2 gates for authorization: The service gate and the user gate. Ideally, it would be desired to allow for these restrictions at the service gate in addition to the user gate and return the intersection of what they are both permitted to see. Take a contrived example below:

# User 1 mapped to employee_role:

employee_role:
  index_permissions:
    - index_patterns:
        - 'employee*'
      allowed_actions:
        - 'read'
      dls: '{"bool": {"should": [{"match": {"state": "NY"}}, {"match": {"state": "MA"}}, {"match": {"state": "CA"}}]}}'

# Extension policy

only_allow_service_to_see_ny_ma_wa:
  index_permissions:
    - index_patterns:
        - 'employee*'
      allowed_actions:
        - 'read'
      dls: '{"bool": {"should": [{"match": {"state": "NY"}}, {"match": {"state": "TX"}}, {"match": {"state": "WA"}}]}}' 

In such a case, if a SearchRequest was performed on employee* index, the user might expect the query only to return documents with a field called state equal to NY and no other documents. A feature like this is currently not supported in the security plugin and its not immediately obvious how to provide support for this feature. The link to where this issue is tracked can be found here: [Provide link to issue]

For the experimental release, data controls will not be permitted on service roles/policies.

[peternied]

@cwperks I like the notion of service gate and user gate. I think that this is an example of something that is specific to extensions that I would call Application Scope, typically dropping the word application. I also think that how service gates are implement need not be the same as user gates - I am going to advocate they are implemented completely separately.

Look at these scopes from Slack [1] like channels.write files.read, reminders.write, ... they cover all the features that can be interacted with a slack instance and are readable. I would rather make a clear contract with administrators that are limited, rather than give them the flexibility / complexity of the Security Plugin permissions.

I've been tweaking the description, and I'll be continuing to make changes, thanks for the feedback.

• [1] https://api.slack.com/legacy/oauth-scopes

[cwperks]

@peternied If we could try to enumerate a list of initial scopes I think that would facilitate the conversation greatly. Would it be possible to have a scope version of the following portion of a roles definition?

index_permissions:
    - index_patterns:
        - 'logs-*'
      allowed_actions:
        - 'read'

I was aiming for code re-use with the proposal above, but I see how there can be some difficulty in devising what the policy definition for a service could be.

There could be a catalog of policies for common-cases similar to the reserved roles that are automatically created today:

act_as_original_user:
  extension_policy: true
  reserved: true
  description: "This policy permits any request from a service on-behalf-of a user. In effect, the service is allowed to act as if it were the user."
  cluster_permissions:
    - "*"
  index_permissions:
    - index_patterns:
        - "*"
      allowed_actions:
        - "*"
  extension_permissions: # Yes, this permits calls to other extensions too
    - "*"

indices_readonly:
  extension_policy: true
  reserved: true
  description: "This policy permits any read requests to OpenSearch indices from a service acting on behalf of a user"
  index_permissions:
    - index_patterns:
        - "*"
      allowed_actions:
        - "read"

[stephen-crawford]

Hi @peternied, sorry for the late review. I think that your proposal about scope use seems like a strong option for handling broad permissioning of extensions. I appreciate the similarity that this would share with existing models such as GitHub and Slack. I also feel that this option would make the permissioning of extensions faster once users learned the new syntax. Assuming the alternative is more fine-grained permissions, it seems obvious that scopes will be easier to implement once users learn the syntax.

My only real hesitancy with this implementation is that it requires users to remember yet another syntax. One of my major gripes with the existing model is that we make use of numerous syntax patterns. This can complicate the setup process for users. Scope definition seems simple in abstract but again would likely be another thing administrators must remember.

Regardless, I do not have any real concerns with the use of scopes and would be on board with implementing this design.

[stephen-crawford]

Hi @cwperks, thank you for your very thoughtful comment. You clearly have put a lot of time into this design. As with Peter's proposal, I do not see anything you described that I would not be on board with. The use of the service and user gates seems slightly outside the area of the scope conversation but I recognize that they would be a powerful tool for managing extensions permissions.

I also appreciate that this would not require the addition of an existing permission and would make use of the the existing code base. My primary concern with this option is assuring the correct patterns etc. are used seems like a tall task. The fine grain permissions seem great on the surface but may not be the most intuitive expression of the permissions. I think that this option would have a steeper learning curve for completely new users than moving to scopes would have for existing users. I also wonder how broad these lists would have to become in order for basic operations to be performed. Would administrators need to define roles with 10+ permissions in order to grant all_access? Please let me know what you think. Again, great work with this.

[davidlago]

In order to close this issue we need to review and agree on a proposal. Leaving open for now.