[Kerberos] Add Kerberos authentication support #32263
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds the framework to support Kerberos authN in elasticsearch. ES is the service protected by Kerberos, each ES service node will have its own keytab. Keytab is the file with Service principal name and encrypted key. This can be then used to validate the authenticator coming in the request. This change only adds support for SPNEGO mechanism and uses JGSS. JVM options -Djava.security.krb5.conf can be used to specify krb5.conf with additional settings if required. For Kerberos Realm, KerberosRealmSettings: Captures settings required for Kerberos Usually keytab (stored in the config), cache settings and krb debug flag KerberosAuthenticationToken: Handles extraction of token from request Extracts the token from request header: "Authorization: Negotiate " If any error condition occurs, throws Exception with Rest status 401 Also adds response header "WWW-Authenticate: Negotiate" KerberosTicketValidator: Used for kerberos ticket validation and gss context establishment. On service side, we need to login first, uses Jaas to complete service login. To avoid more file configurations, we generate the JAAS configuration with required modules in memory. The token extracted from authnToken is passed on to GSSContext which uses service credentials (keytab) to verify the passed token and generates output token. If GSS context is established it returns tuple of client-username and out token (can be empty). If out token is present but context is yet not established then it will return tuple with no username and out token. The out token needs to be returned as response header 401 and "WWW-Authenticate: Negotiate " for ongoing negotiation. This will continue till either it fails or successful authentication on context establishment. Changes in plugin-security policy to add required permissions Few settings like Jaas config and kerberos keytab access requires permissions. For testing, KerberosTestCase is the base class to start/stop kdc server and build test settings. SimpleKdcLdapServer is a wrapper around SimpleKdcServer(ApacheDS), which simplifies in memory testing with KDC and uses in-memory LDAP server as its backend.
As there are some system properties like `java.security.krb5.kdc` , `java.security.krb5.realm` which can specify values that are applicable to whole JVM. This is the reason for having only one instance of Kerberos realm. Each ES node will have a Kerberos keytab with credentials. This keytab must exist for Kerberos authentication to work. `KerberosRealmBootstrapCheck` performs these checks for given configuration.
Till now we had support for 'Basic', 'Bearer' auth schemes and this was sufficient for us to reply `WWW-Authenticate` header with one value either for `Basic` or `Bearer` for unauthorized access. After introducing Kerberos we will be supporting `Negotiate` scheme. As per [RFC7235](https://tools.ietf.org/html/rfc7235#section-4.1), we may respond with the list of challenges. This list is of auth schemes supported by the server. We can also have custom Realms defining their own response header value for 'WWW-Authenticate' header. This commit introduces a `getWWWAuthenticateHeaderValue` in `Realm` to identify the scheme which it wants to use. By default it uses 'Basic' auth scheme. This can be overriden by realms like KerberosRealm to specify 'Negotiate' scheme or OAuth to specify 'Bearer' or custom realms added by security extensions to specify their own scheme. SAML specifications do not specify anything related to the header but unofficially many have used 'SAML' as auth scheme or used 'Bearer' auth scheme for passing SAML tokens. But most of the realms would use the existing schemes like 'Basic', 'Digest', 'Bearer', 'Negotiate' etc. At the startup, `Security#createComponents` will take care of creating `DefaultAuthenticationFailureHandler` with default response header values for 'WWW-Authenticate' as a list of configured and enabled auth schemes.
This commit adds authentication realm for handling Kerberos authentication by spnego mechanism. The class `KerberosRealm` authenticates user for given kerberos ticket after validating the ticket using `KerberosTicketValidator`. It uses native role mapping store to find user details and then creates an authenticated `User`. On successful authentication, it will return populated `User` object with roles. On failure to authenticate, it will terminate authentication process with a failure message. The failure could be due to gss context negotiation failure requiring further negotiation and it might return outToken to be communicated with peer as value for header `WWW-Authenticate` in the form 'Negotiate oYH1MIHyoAMK...'. There could be other failures like JAAS login exception or GSS Exception which will terminate the authentication process. As KerberosRealm can terminate authentication process during context negotiation with some outToken, the header value for `WWW-Authenticate` needs to be preserved. Earlier the behavior was to overwrite all the headers as defined in authentication failure handler in my last commit. Negotiate does maintain kind of state over HTTP and so we have to handle this in a special way. For this, I have added a special check for if exception has header 'WWW-Authenticate' with 'Negotiate ' scheme and token, it will not be overwritten. We want Kerberos to be a platinum feature, so it is not included as part of standard types similar to SAML. TODO: Support for user lookup from other realms like AD/LDAP. Authorizing realms feature is work in progress, once completed I will add the support to KerberosRealm. I have a TODO note in source code.
This commit adds support for removing realm name from the Kerberos principal name. The principal names in Kerberos are in the form primary/instance@realm. Since we will be supporting user lookups and depending on the scenario we may want to remove the REALM part and use the username for lookup or role mapping. This change adds a new setting with the default value false to control removing of realm name. Modified tests to randomly use this setting during testing.
This commit adds the rest client integration test for Kerberos. This uses existing krb5kdc-fixture, which makes use of MIT Kerberos. Added support to create principals with password in krb5kdc-fixture. The rest test demonstrates the following: - Use of rest client to invoke Elasticsearch APIs authenticating using spnego mechanism, example showing what customizations we need to do to build the rest client. - test for login by keytab for user principal - test for login by username password for user principal
This commit does some refactoring to remove support package and move classes to kerberos package. That was the only class in that package, so no need for it to be in separate package. Changes done to use default values for jaas configuration options for the ones which we can use defaults. Fix couple of random failures in tests. Modified `refreshKrb5Config` to use default value `false` in KerberosTicketValidator. If the krb5.conf file is modified then we will need to restart JVM as the config will not be refreshed. For testing, `refreshKrb5Config` is set to `true` as we keep changing the kdc port. This is set in SpnegoClient and only for tests.
The exception was being sent twice due to incorrect handling of conditional statements causing multiple authentication_failed events in audit logs.
From 5.0 onwards use of few characters will not be allowed, one of them is ':'. This commit removes that character. Also add dependency for copy task on creation of principal names which caused problems with clean test runs.
bizybot
added
>feature
review
v7.0.0
:Security/Authentication
|
Pinging @elastic/es-security |
|
@bizybot given that all the other commits will be squashed down when merging this, we really need a better description. There is a lot in this PR, eg how we test, multiple auth challenges, etc. |
|
Yes, expanded the commit message. I was going to keep the commit history after squashing but this works too. Thanks. |
|
Thank you for updating the description but I still think we can do better. The description jumps all over the place when it should be easy to read. Think about this as telling someone about all the work that went into this so that they can understand the what and why without digging into the code |
|
Thanks, I have updated the description. Hope this is helpful. |
tvernum
approved these changes
Jul 24, 2018
There was a problem hiding this comment.
LGTM.
I think the commit message is still too long, and contains details that are more applicable for the docs than a commit message, but I don't want to hold up the feature on that basis alone.
I would strongly recommend that you go back through the message and ask "If another engineer was trying to understand the purpose of my change, would they want this information?"
With any luck you should be able to cut about half of it out.
Don't explain kerberos, don't repeat things that are better handled in the javadocs, don't explain how things used to work, don't talk about things that aren't relevant to this change.
|
Thanks, Tim. I updated the description with reduced information. Yes the commit message was too big as I focused more on the without digging into the code part and got carried away. |
|
I updated the commit message and will merge/backport this. |
jaymode
pushed a commit
that referenced
this pull request
Jul 24, 2018
This commit adds support for Kerberos authentication with a platinum license. Kerberos authentication support relies on SPNEGO, which is triggered by challenging clients with a 401 response with the `WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide a Kerberos ticket in the `Authorization` header. The tickets are validated using Java's built-in GSS support. The JVM uses a vm wide configuration for Kerberos, so there can be only one Kerberos realm. This is enforced by a bootstrap check that also enforces the existence of the keytab file. In many cases a fallback authentication mechanism is needed when SPNEGO authentication is not available. In order to support this, the DefaultAuthenticationFailureHandler now takes a list of failure response headers. For example, one realm can provide a `WWW-Authenticate: Negotiate` header as its default and another could provide `WWW-Authenticate: Basic` to indicate to the client that basic authentication can be used in place of SPNEGO. In order to test Kerberos, unit tests are run against an in-memory KDC that is backed by an in-memory ldap server. A QA project has also been added to test against an actual KDC, which is provided by the krb5kdc fixture. Closes #30243
dnhatn
added a commit
that referenced
this pull request
Jul 25, 2018
* 6.x: Security: revert to old way of merging automata (#32254) Fix a test bug in RangeQueryBuilderTests introduced in the field aliases backport. Introduce Application Privileges with support for Kibana RBAC (#32309) Undo a debugging change that snuck in during the field aliases merge. [test] port linux package packaging tests (#31943) Painless: Update More Methods to New Naming Scheme (#32305) Tribe: Add error with secure settings copied to tribe (#32298) Add V_6_3_3 version constant Add ERR to ranking evaluation documentation (#32314) [DOCS] Added link to 6.3.2 RNs [DOCS] Updates 6.3.2 release notes with PRs from ml-cpp repo (#32334) [Kerberos] Add Kerberos authentication support (#32263) [ML] Extract persistent task methods from MlMetadata (#32319) Backport - Add Snapshots Status API to High Level Rest Client (#32295) Make release notes ignore the `>test-failure` label. (#31309) [DOCS] Adds release highlights for search for 6.4 (#32095) Allow Integ Tests to run in a FIPS-140 JVM (#32316) Add support for field aliases to 6.x. (#32184) Register ERR metric with NamedXContentRegistry (#32320) fixes broken build for third-party-tests (#32315) Relates #31918 / Closes infra/issues/6085 [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280) Rest HL client: Add put watch action (#32026) (#32191) Add WeightedAvg metric aggregation (#31037) Consistent encoder names (#29492) Switch monitoring to new style Requests (#32255) specify subdirs of lib, bin, modules in package (#32253) Rename ranking evaluation `quality_level` to `metric_score` (#32168) Add new permission for JDK11 to load JAAS libraries (#32132) Switch x-pack:core to new style Requests (#32252) Watcher: Store username on watch execution (#31873) Silence SSL reload test that fails on JDK 11 Painless: Clean up add methods in PainlessLookup (#32258) CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185) Fail shard if IndexShard#storeStats runs into an IOException (#32241) Fix `range` queries on `_type` field for singe type indices (#31756) (#32161) AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated Add new fields to monitoring template for Beats state (#32085) (#32273) [TEST] improve REST high-level client naming conventions check (#32244) Check that client methods match API defined in the REST spec (#31825)
dnhatn
added a commit
that referenced
this pull request
Jul 25, 2018
* master: Security: revert to old way of merging automata (#32254) Networking: Fix test leaking buffer (#32296) Undo a debugging change that snuck in during the field aliases merge. Painless: Update More Methods to New Naming Scheme (#32305) [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests Ingest: Support integer and long hex values in convert (#32213) Introduce fips_mode setting and associated checks (#32326) Add V_6_3_3 version constant [DOCS] Removed extraneous callout number. Rest HL client: Add put license action (#32214) Add ERR to ranking evaluation documentation (#32314) Introduce Application Privileges with support for Kibana RBAC (#32309) Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240) [Kerberos] Add Kerberos authentication support (#32263) [ML] Extract persistent task methods from MlMetadata (#32319) Add Restore Snapshot High Level REST API Register ERR metric with NamedXContentRegistry (#32320) fixes broken build for third-party-tests (#32315) Allow Integ Tests to run in a FIPS-140 JVM (#31989) [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280) awaitsfix testRandomClusterStateUpdates [TEST] add version skip to weighted_avg tests Consistent encoder names (#29492) Add WeightedAvg metric aggregation (#31037) Switch monitoring to new style Requests (#32255) Rename ranking evaluation `quality_level` to `metric_score` (#32168) Fix a test bug around nested aggregations and field aliases. (#32287) Add new permission for JDK11 to load JAAS libraries (#32132) Silence SSL reload test that fails on JDK 11 [test] package pre-install java check (#32259) specify subdirs of lib, bin, modules in package (#32253) Switch x-pack:core to new style Requests (#32252) awaitsfix SSLConfigurationReloaderTests Painless: Clean up add methods in PainlessLookup (#32258) Fail shard if IndexShard#storeStats runs into an IOException (#32241) AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated Remove unnecessary warning supressions (#32250) CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185) Add new fields to monitoring template for Beats state (#32085)
89 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
WWW-Authenticate: Negotiateheader. A SPNEGO client will then providea Kerberos ticket in the
Authorizationheader. The tickets arevalidated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.
In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
WWW-Authenticate: Negotiateheader as its default and another couldprovide
WWW-Authenticate: Basicto indicate to the client that basicauthentication can be used in place of SPNEGO.
In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.
Closes #30243