[Feature Request] Support for FIPS 140-2 enforced mode.

[thunderstumpges]

In some customer service contracts (especially Government contracts), there is a requirement to run all services in FIPS 140-2 compliant/enforced mode.

Elasticsearch supports this currently in their xpack security plugin.

We, and I imagine others could benefit greatly with the ability to run OpenSearch in a FIPS 140-2 enforced mode JVM.

I see there was also an OpenDistro community post asking for this feature as well.

[saratvemulapalli]

Thanks for reaching out.
I'll move this to our security plugin team.

Also @setiah probably you might interested!

[CEHENKLE]

@davidlago You might be interested in this as well :)

[tdurden82]

FIPS 140-2 Mode would be an excellent feature

[setiah]

A related discussion around making the crypto provider in security plugin configurable. https://github.com/opensearch-project/OpenSearch/issues/1487 As part of this, it should also enable users to provide a FIPS compliant provider.

[dblock]

Please consider how/whether/when this will work on Windows, which has its own world of cryptographic providers.

[bhupendra-mskhatri]

@davidlago any timeline/roadmap on FIPS compliance for opensearch on windows?
Our customers want our application to be FIPS compliant by mid this year and since we use opensearch we would like to see this soon.

[davidlago]

Thanks for the interest, @bhupendra-mskhatri! We are not actively working on this issue at the moment, though it is a contribution we'd most definitely welcome.

[bhupendra-mskhatri]

Thankyou @davidlago for your response.
Can you or someone from your team give a basic idea of what changes might be required in opensearch to make it FIPS compliant. This would help me understand the contribution estimate.

[stephen-crawford]

Hi @bhupendra-mskhatri, I tried to gather some information for you to be able to gauge the scope.

For starters, we are going to have to rework the use of Bouncy Castle. If you look at their website you can see that while they do have FIPS certified APIs for Java, they are only certified for 1.7, 1.8, and 1.11. This means that the 2.x line and main which supports 1.17 will have to be run with 1.11 in order to be compliant. This is probably acceptable if a little bit confusing (there will be OpenSearch versions that are only compliant in one version configuration).

You can see the Bouncy Castle FIPS versions here and some implementation details here. In effect, you would minimally need to replace any non-compliant algorithms with their certified counterparts from one of the linked Bouncy Castle API versions and make sure that the software then met the guidelines of the documentation.

[rathiyajiv]

Thanks, @scrawfor99.
Our application bundles OpenSearch as a repository to persist some related settings. In order to make our application FIPS compliant we are running this whole setup under the FIPS setup.

OpenSearch crashes when it runs with FIPS on the mode by saying -

Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
at sun.security.util.DerInputStream.getLength(DerInputStream.java:606) ~[?:?]
at sun.security.util.DerValue.init(DerValue.java:390) ~[?:?]
at sun.security.util.DerValue.(DerValue.java:331) ~[?:?]
at sun.security.util.DerValue.(DerValue.java:344) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1996) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1479) ~[?:?]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365) ~[?:?]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313) ~[?:?]
at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55) ~[?:?]
at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278) ~[?:?]
at org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:151) ~[?:?]
at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68) ~[?:?]
at org.elasticsearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:136) ~[?:?]
at org.elasticsearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:145) ~[?:?]
at org.elasticsearch.index.reindex.ReindexSslConfig.(ReindexSslConfig.java:115) ~[?:?]
at org.elasticsearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:91) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$15(Node.java:553) ~[elasticsearch-7.10.2.jar:7.10.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:557) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:0-SNAPSHOT]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:0-SNAPSHOT]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:0-SNAPSHOT]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:0-SNAPSHOT]
... 7 more

** @davidlago @saratvemulapalli, Please if anyone came across with similar situation?**

FIPS settings turned on the windows box -
OS settings

1. In Control Panel, click Administrative Tools, and then double-click Local Security Policy.
2. In Security Settings, expand Local Policies and then click Security Options.
3. Under Policy in the right pane, double-click System cryptography: Use FIPS-compliant algorithms for encryption, hashing, and signing, and then click enable.

Java settings
Update the precedence of Sun’s crypto modules JCE & JSSE to 1 & 2 under java

jdk-11.0.11.9-hotspot\conf\security\java.security
security.provider.1=SunJCE
security.provider.2=SunJSSE
.
.
security.provider.3=SUN

[peternied]

@rathiyajiv That error message is based on the certificate being loaded, is it possible the cert isn't in PKCS12 format?

Relevant StackOverflow question/answers https://stackoverflow.com/a/57683701/533057

[rathiyajiv]

@rathiyajiv That error message is based on the certificate being loaded, is it possible the cert isn't in PKCS12 format?

Relevant StackOverflow question/answers https://stackoverflow.com/a/57683701/533057

Thanks @peternied, the observation here is an error occurs only when FIPS settings are on. Does PKCS12 format a must for FIPS mode from the opeansearch perspective?

[peternied]

Referencing the bouncy castle guide, PKCS12 is not supported in approved-mode. More details from chapter 7 Key Stores https://downloads.bouncycastle.org/fips-java/BC-FJA-UserGuide-1.0.2.pdf

After switch to a supported keystore this should get you around that error.

[stephen-crawford]

Hi @rathiyajiv, thank you for following up. I see Peter already provided some more information to help you along your way, but I wanted to address your question about the format. PKCS12 is not a must that I am aware of. That being said, there are several factors that go into FIPS compliance which I am not an expert on.

You will notice on the intial RFC that the requirements are listed for authentication as:

FIPS does not have any specific guidelines for the type of algorithm used so long as it is compatible with these requirements. Instead you can use

The annex of this draft has a list of some of the approved options for implementation. If there is a modification you wish to make to your local OpenSearch configuration in order to be able to run the enforced mode, I would encourage you to do so.

If you were then able to let us know what you did, we may be able to make it a configurable option on the plugin so that it was easier to set up in the future.

[rathiyajiv]

Thanks @peternied , @scrawfor99 -

There are some traces around the FIPS in the code @ below
https://github.com/opensearch-project/OpenSearch/tree/main/buildSrc/src/main/resources

All the version of security file fips_java_xx_x.security contains the security.provider.1 which points to the bouncycastle jcajce & jsse modules. While the java's /conf/security/java.security refers to SunJCE & SunJSSE.

Can anyone suggest where to find the fips_java_bcjsse_11.security under Opensearch's deployed folder structure? We are on Opensearch version 2.4.1

[iainjacksonsas]

I've been having a look at the security plugin, and I think the hard-coded dependency on the BouncyCastle provider will need to be broken for this to work. I think is mentioned on another related ticket about abstracting this particular dependency.

So far from my digging, I see the security plugin uses BouncyCastle directly for:

• Security Provider itself which is hard-coded to be loaded on start-up
• OpenBSD BCrypt for authentication
• Blake2b for data anonymization
• ASN.1 decoding
• Hex and Base64 encoding/decoding

If we can replace these with some non-hardcoded implementations or other libraries. Then it should be possible to try and run OpenSearch with another security provider - like the BouncyCastle FIPS provider.

[szwlhd]

I've been having a look at the security plugin, and I think the hard-coded dependency on the BouncyCastle provider will need to be broken for this to work. I think is mentioned on another related ticket about abstracting this particular dependency.

So far from my digging, I see the security plugin uses BouncyCastle directly for:

• Security Provider itself which is hard-coded to be loaded on start-up
• OpenBSD BCrypt for authentication
• Blake2b for data anonymization
• ASN.1 decoding
• Hex and Base64 encoding/decoding

If we can replace these with some non-hardcoded implementations or other libraries. Then it should be possible to try and run OpenSearch with another security provider - like the BouncyCastle FIPS provider.

As you said BouncyCastle is hard coded means if we use BouncyCastle provider in JVM then Opensearch will Run in Fips mode automatically as BouncyCastle is by default provider

[stephen-crawford]

Hi @iainjacksonsas,

Thanks for the details on this. I know you are taking a further look, but to clarify, the major change to make OpenSearch FIPS compliant is swapping out Bouncy Castle use? If changes were made to make another library an option for running things, would that be enough to be considered compliant? Or would further changes on top of that be required?

Thank you.

[iainjacksonsas]

Yes, looking at the usage of the BouncyCastle provider is a starting point. The key requirement is to run on a FIPS enforcing platform (e.g. RHEL 8) using FIPS certified cryptography.

The major issue stems from OpenSearch Security plugin's direct use of classes in the BouncyCastle provider, which would need replaced with the FIPS certified version and run on a certified JDK (e.g. Java 11). However you cannot have both the FIPS and non-FIPS BouncyCastle providers on the classpath at the same time as they interfere with one another. You must use one or the other.

I currently have a version of the OpenSearch Security plugin locally which does not have any direct dependencies on BouncyCastle. I did some code surgery to take from BouncyCastle just what was needed. This is not a solution, just working through what changes could and need to be made to make this work.

I am currently working through trying to get it to start-up on a FIPS enforcing platform. Still not there, but i'm getting closer.

[szwlhd]

Any update on FIPS implementation? I can see when I am using BC provide in java.security as below

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:HYBRID;ENABLE{All};
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS

opensearch is starting fine . Can we assume its running in FIPS mode.

[iaindjackson]

I'm still pulling together my notes. But to answer your question, no it's not quite running in FIPS mode.

The JVM - particularly on RedHat 8 - behaves differently when the kernel is set to FIPS mode. Also, to be running in a FIPS-compliant mode we can't use certain algorithms (e.g. BCrypt).

I'm hoping to get a ticket raised this week with all the details on my investigation.

[szwlhd]

Thanks for update.
I am using RedHat 8 with kernel is set to FIPS mode true and opensearch is running fine with above mentioned java.security setting.

[bhupendra-mskhatri]

I'm still pulling together my notes. But to answer your question, no it's not quite running in FIPS mode.

The JVM - particularly on RedHat 8 - behaves differently when the kernel is set to FIPS mode. Also, to be running in a FIPS-compliant mode we can't use certain algorithms (e.g. BCrypt).

I'm hoping to get a ticket raised this week with all the details on my investigation.

Hi @iainjacksonsas
Is there a checklist we can go against to verify and conclude that after the changes we make for FIPS compliance and enabling FIPS at OS level, opensearch is fully FIPS compliant or is not?

e.g. to @szwlhd query you responded that though after his changes opensearch started successfully, but it isn't running in FIPS mode. What are those references against which you cross checked to identify whether opensearch is running in FIPS mode or not.

[iaindjackson]

There isn't a checklist to verify. There's nothing you can simply check other than checking the output of 'fips-mode-setup --check' on RedHat to see if it's enabled.

So far I've found for RedHat 8 that you should be using the JDK provided by the OS and not the one that ships with OpenSearch. The one that ships doesn't do anything special on FIPS mode - unless you start configuring it. It doesn't block the non-FIPS implementations which is what starts to cause issues.

[szwlhd]

Hi @iainjacksonsas,
I tried in RedHat8 kernal fips enabled machine with default JDK yes opensearch is not running because default JDK is using SunPKCS11.

But when I use my java not packaged with Opensearch with provider as BouncyCastleFipsProvider its working fine.

Today going one step ahead set up the code locally for opensearch security plugin and changed the provider from
BouncyCastleProvider to BouncyCastleFipsProvider in OpenSearchSecurityPlugin.java class and other places. After that also opensearch is starting fine. Can we say now this time its running in FIPS mode ?

Code which I changed.
if(Security.getProvider("BC") == null) {
Security.addProvider(new BouncyCastleProvider()); // removed this line
Security.addProvider(new BouncyCastleFipsProvider()) // added this line
}

[iaindjackson]

I initially started working through those steps, but hit into continual roadblocks (cryptography, Java security manager, etc). @szwlhd
could you share more details on your environment?

Here's a summary of the environment I have setup and what I have been working with and where I've got to so far:

• RedHat Enterprise Linux 8.3 in FIPS-mode. The VM I'm working with was setup a while ago.
• OpenSearch 2.6 (Haven't updated it yet to 2.7)
• Installed the following libraries into the opensearch/lib directory: bc-fips-1.0.2.3.jar, bcpkix-fips-1.0.7.jar, bctls-fips-1.0.14.1.jar
• JDK from RedHat 8.3, JAVA_HOME=/usr/lib/jvm/jre-11. I also delete the jdk directory when I untar the OpenSearch distribution to ensure I don't use the built-in JDK.

I have the following additions in the security policy and properties files that I used with my no BouncyCastle implementation. I can get OpenSearch to boot with this and it have it running on TLS, but some of the code is still half-assembled and likely doesn't work.

Security Policy

grant {
    //io.netty.handler.codec.DecoderException
    permission java.lang.RuntimePermission "accessClassInPackage.sun.security.internal.spec";
     
    //java.security.InvalidAlgorithmParameterException: Cannot process GCMParameterSpec
    permission java.lang.RuntimePermission "accessDeclaredMembers";
     
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";   
    permission org.bouncycastle.crypto.CryptoServicesPermission "exportSecretKey";
    permission org.bouncycastle.crypto.CryptoServicesPermission "exportPrivateKey";
 
    permission java.security.SecurityPermission "getProperty.keystore.type.compat";
    permission java.security.SecurityPermission "getProperty.jdk.certpath.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.jdk.tls.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.jdk.tls.ephemeralDHKeySize";
    permission java.security.SecurityPermission "getProperty.jdk.tls.server.defaultDHEParameters";
    permission java.security.SecurityPermission "getProperty.jdk.tls.server.enableCAExtension";
     
    permission java.io.FilePermission "${javax.net.ssl.trustStore}", "read";
    permission java.io.FilePermission "${java.home}/lib/security/jssecacerts", "read";
    permission java.io.FilePermission "${java.home}/lib/security/cacerts", "read";
}

Security Properties

fips.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
fips.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
fips.provider.3=SUN
fips.provider.4=SunEC
fips.provider.5=SunPKCS11 ${java.home}/conf/security/nss.fips.cfg
fips.provider.6=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS-FIPS
fips.keystore.type=jks
ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX

The RedHat JDK has a separate fips.xxx namespace in the security properties for the FIPS-settings which separates them from the non-FIPS settings. These are used instead on a FIPS-enabled system.

Issues Found

The major issues are the following:

• The RedHat JDK is FIPS-enforcing, so code which accesses non-FIPS cryptography via the usual interfaces gets blocked. This is unless you plug the missing cryptography with BC FIPS (or something similar) you run into issues. You run into issues like the keystore can't be created and basic algorithms are missing.

This come baked in with all the FIPS restrictions on what cryptography you can and cannot use. So most of the policy configuration is done. All that remains is to plugin in the BC FIPS provider and get it wired up. If we can get OpenSearch to run in this very strict environment it should prove well for running in others (e.g. Windows, Mac).

• The security plugin also includes BCPROV and you can't have both the FIPS and non-FIPS versions of BouncyCastle on the classpath at the same time even if you have only 1 configured. At the moment it doesn't even get this far with OpenSearch as the JarHell detector picks up on it before it even loads the plugin.

Caused by: java.lang.IllegalStateException: jar hell! class: org.bouncycastle.LICENSE jar1: /home/xxxxxx/opensearch/default/opensearch-2.6.0/plugins/opensearch-security/bcprov-jdk15on-1.67.jar jar2: /home/xxxxxx/opensearch/default/opensearch-2.6.0/lib/bc-fips-1.0.2.3.jar

Trying to have both on the classpath results in weird errors like missing keys, etc when it's initializing as it has the data from both providers from my testing. It's not a valid configuration.

• The Performance Analyzer plugin also includes BCPROV which conflicts with switching to BC FIPS. This also triggers the JarHell on start-up as well. I removed the plugin from my setup for now.

• The BCPROV used by the OpenSearch security plugin has hard coded references throughout the security plugin code. Not just to cryptographic or digest functions (e.g. BCrypt, Blake2b) it also uses a bunch of utility classes (Hex, Base64, ASN.1).

Most of these are replaceable using the BC FIPS library instead - except BCrypt and Blake2b (see below).

Switching to BC FIPS also means requiring the BC TLS FIPS and BC PKIX FIPS libraries too. We could ship all of them with the plugin, but ideally you would want to configure this at a JVM level and not through the plugin since these would only get instantiated when the plugin loads.

• The BouncyCastle FIPS library doesn't have BCrypt or Blake2b implementations which are direct references to BouncyCastle in OpenSearch Security.

You can't have both FIPS and non-FIPS dependencies, so if you switch the compilation dependencies to BC FIPS (which is an option) you will be missing a number of classes. I have switched them out locally with other implementations to plug the gap to work through that issue later when investigating.

However, to be running in a FIPS-compatible mode OpenSearch must only use cryptographic algorithms which have been FIPS-certified. So these two algorithms would need to be swapped for an implementation supplied by a FIPS-compliant module e.g. PBKDF2. At the moment there is no method to configure the default password hashing or data anonymization algorithms via settings - this is something that is missing from OpenSearch.

New settings are really needed for:

• Is FIPS Compatible mode enabled? To ensure we don't use non-FIPS algorithms.
• Password Hashing Algorithm. To re-configure the password hashing to use an algorithm from a FIPS-compliant module.
• Data Anonymization Algorithm. Same for the data anonymization algorithm.

Libraries I've been looking at to plug the gaps between BC FIPS and BC:

• https://mvnrepository.com/artifact/at.favre.lib/bcrypt
• https://mvnrepository.com/artifact/com.rfksystems/blake2b

The ASN.1 parsing also needs to be implemented using another library. I've had no luck yet in tracking one down that's usable.

[szwlhd]

Hi @iaindjackson

Thanks for putting all the details. I completely agree with you we can use BC FIPS and BCPROV together. So I tried by removing BCPROV and including BC-FIPS. After that opensearch started but could not able to run https://localhost:9200
as getting java.lang.NoClassDefFoundError: org/bouncycastle/crypto/generators/OpenBSDBCrypt.

So I am at the place where you mentioned we have to replace the OpenBSDBCrypt algorithms.

I was looking to BC FIPS documentation (https://www.bouncycastle.org/fips-java/BCFipsIn100.pdf) and thinking to implement something below. As of now I am not aware where it can be done or not need to evaluate. But please guide me if investigating in this direction worth it.

Example 6 – ECB Mode Encryption
public static byte[] ecbEncrypt(SecretKey key, byte[] data)
throws GeneralSecurityException
{
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding", "BCFIPS");
cipher.init(Cipher.ENCRYPT_MODE, key);
return cipher.doFinal(data);
}
public static byte[] ecbDecrypt(SecretKey key, byte[] cipherText)
throws GeneralSecurityException
{
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding", "BCFIPS");
cipher.init(Cipher.DECRYPT_MODE, key);
return cipher.doFinal(cipherText);
}

Meanwhile I will look into these what you have suggested.
https://mvnrepository.com/artifact/at.favre.lib/bcrypt
https://mvnrepository.com/artifact/com.rfksystems/blake2b

My environment is

RedHat Enterprise Linux 8.3 in FIPS-mode.
OpenSearch 0-SNAPSHOT
Installed the following libraries into the opensearch/lib directory: bc-fips-1.0.2.3.jar, bcpkix-fips-1.0.7.jar, bctls-fips-1.0.14.1.jar
My own JDK 11, JAVA_HOME=/home/user/java11.
I also delete the jdk directory when I untar the OpenSearch distribution..

I my java.security which is at location /home/user/java11/conf/security
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:HYBRID;ENABLE{All};
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJSSE
security.provider.7=SunJCE
security.provider.8=SunJGSS
security.provider.9=SunSASL
security.provider.10=XMLDSig
security.provider.11=SunPCSC
security.provider.12=JdkLDAP
security.provider.13=JdkSASL
security.provider.14=SunMSCAPI
security.provider.15=SunPKCS11
ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX
keystore.type=pkcs12

Security policy -- there other permission as well but mainly I have added these for BC fips
permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";
permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
permission java.security.SecurityPermission "insertProvider";

[szwlhd]

Hi @iaindjackson,

Today I implemented these two libraries
https://mvnrepository.com/artifact/at.favre.lib/bcrypt
https://mvnrepository.com/artifact/com.rfksystems/blake2b
code changed as below -

  public boolean passwordMatchesHash(String hash, char[] array) {
   return OpenBSDBCrypt.checkPassword(hash, array); // old code
    BCrypt.Result result = BCrypt.verifyer().verify(array, hash.getBytes(StandardCharsets.UTF_8)); // newly added
    return result.verified;
}

public static String hash(final char[] clearTextPassword) {
final byte[] salt = new byte[16];
new SecureRandom().nextBytes(salt);
// final String hash = OpenBSDBCrypt.generate((Objects.requireNonNull(clearTextPassword)), salt, 12);
final String hash= Bytes.wrap(BCrypt.withDefaults().hash(6,salt,new String(Objects.requireNonNull(clearTextPassword)).getBytes(StandardCharsets.UTF_8))).toString();
Arrays.fill(salt, (byte) 0);
Arrays.fill(clearTextPassword, '\0');
return hash;
}

private byte[] blake2bHash(byte[] in) {
MessageDigest digest =null;
try {
// final Blake2bDigest hash = new Blake2bDigest(null, 32, null, defaultSalt);
// digest = MessageDigest.getInstance("SHA3-224", "BCFIPS");
digest = MessageDigest.getInstance(Blake2b.BLAKE2_B_256);
// hash.update(in, 0, in.length);
digest.update(in);
// final byte[] out = new byte[hash.getDigestSize()];
final byte[] out = new byte[64];
// hash.doFinal(out, 0);
// return digest.digest();
// return Hex.encode(out);
} catch (NoSuchAlgorithmException e){
System.out.println("Error message "+e);
}
return digest.digest();
}

code compiled successfully and able to start the opensearch even able to hit
https://localhost:9200 and able to login with credential admin/admin

But then getting below error
: "javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at

full error is as below.

Any guidance ?

[szwlhd]

Could you please guide how to debug security plugin code in intellij IDEA ?

[trevor-vaughan]

This blog post may help in your BouncyCastle journey.

[dVenkatNaveen]

Any update by when we can expect opensearch o FIPS enable ?

[szwlhd]

Hi @iaindjackson
I am able to run opensearch in FIPS mode with BCFKS keystore.
Now got stuck in RestClient . Trying communicate with opensearch with restclient but getting error BCFKS not found
Any suggestion
#########################################
Exception in thread "main" java.security.KeyStoreException: BCFKS not found
at java.base/java.security.KeyStore.getInstance(KeyStore.java:878)
at hello.RESTClientSample.main(RESTClientSample.java:55)
Caused by: java.security.NoSuchAlgorithmException: BCFKS KeyStore not available
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
at java.base/java.security.Security.getImpl(Security.java:700)
at java.base/java.security.KeyStore.getInstance(KeyStore.java:875)
... 1 more
######################################
Process finished with exit code 1

[stephen-crawford]

Hi @szwlhd and @iaindjackson, just checking in on your efforts here. Is there a branch you are working on I could checkout or do you have your code set as private for the timing being. If you are able to share the stpes you have taken to reach your current state, I may have the bandwidth to contribute some when you run into a block.

[szwlhd]

Hi @scrawfor99
Thanks for comments.

Please find latest code below with my updates:
https://github.com/dVenkatNaveen/OpenSearch.git
https://github.com/dVenkatNaveen/security.git

This is not a production code but just implemented to see if OpenSearch is working in FIPS-mode.

1. I Changed provider from BouncyCastle to BouncyCastleFipsProvider. For that I changed in OpenSearchSecurityPlugin.java
2. To make this I removed bcprov-jdk15on to bc-fips jar. All the relevant changes can be find in build.gradle.
3. Changed the certificate from JKS to BCFKS.
4. I implemented these two libraries as OpenBSDBCrypt is not supported by bc-fips:
   https://mvnrepository.com/artifact/at.favre.lib/bcrypt
   https://mvnrepository.com/artifact/com.rfksystems/blake2b
5. Made Changes in java.security and java.policy file. Will Attach that for your reference.
   security.zip

My environment is
RedHat Enterprise Linux 8.3 in FIPS-mode.
OpenSearch 0-SNAPSHOT
Installed the following libraries into the opensearch/lib directory: bc-fips-1.0.2.3.jar, bcpkix-fips-1.0.7.jar, bctls-fips-1.0.14.1.jar
My own JDK 11, JAVA_HOME=/home/user/java11.
I also delete the jdk directory when I untar the OpenSearch distribution..

I my java.security which is at location /home/user/java11/conf/security
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:HYBRID;ENABLE{All};
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJSSE
security.provider.7=SunJCE
security.provider.8=SunJGSS
security.provider.9=SunSASL
security.provider.10=XMLDSig
security.provider.11=SunPCSC
security.provider.12=JdkLDAP
security.provider.13=JdkSASL
security.provider.14=SunMSCAPI
security.provider.15=SunPKCS11
ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX
keystore.type=BCFKS

Let me know if you need running instance of opensearch in FIPS mode as well so that you can try the things quickly.
After these changes I am getting error when trying to connect with opensearch with opensearch-java client. Details will post you in next comments.

[szwlhd]

Now using below code to connect with OpenSearch but getting error
jakarta.json.stream.JsonParsingException: Jackson exception: Unrecognized token 'Unauthorized': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (ByteArrayInputStream); line: 1, column: 13]

Code used is as below

Security.addProvider(new BouncyCastleFipsProvider());
System.setProperty("javax.net.ssl.trustStore", "D:\SpectrumGit23.1\platform\dist\dev-combined\target\container\index\config\certs\spectrum-truststore.BCFKS");
System.setProperty("javax.net.ssl.trustStorePassword", "password");
String certFileLocation = (new File("D:\SpectrumGit23.1\platform\dist\dev-combined\target\container\index\config\certs")).getCanonicalPath();
Path trustStorePath = Paths.get(certFileLocation + "/" + "spectrum-keystore.BCFKS", new String[0]);
KeyStore truststore = KeyStore.getInstance("BCFKS", "BCFIPS");

        InputStream is = Files.newInputStream(trustStorePath, new java.nio.file.OpenOption[0]);
        truststore.load(is, "password".toCharArray());
        final HttpHost host = new HttpHost("https", "localhost", 9200);
        final BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        // Only for demo purposes. Don't specify your credentials in code.
        credentialsProvider.setCredentials(new AuthScope(host), new UsernamePasswordCredentials("admin", "password".toCharArray()));
        final SSLContext sslcontext = SSLContextBuilder
                .create()
                .loadTrustMaterial(truststore, (chains, authType) -> true)
                .build();
        final ApacheHttpClient5TransportBuilder builder = ApacheHttpClient5TransportBuilder.builder(host);
        builder.setHttpClientConfigCallback(httpClientBuilder -> {
            final TlsStrategy tlsStrategy;
            try {
                tlsStrategy = ClientTlsStrategyBuilder.create()
                        .setSslContext(SSLContextBuilder.create().build())
                        // See https://issues.apache.org/jira/browse/HTTPCLIENT-2219
                        .setTlsDetailsFactory(new Factory<SSLEngine, TlsDetails>() {
                            @Override
                            public TlsDetails create(final SSLEngine sslEngine) {
                                return new TlsDetails(sslEngine.getSession(), sslEngine.getApplicationProtocol());
                            }
                        })
                        .build();
            } catch (NoSuchAlgorithmException e) {
                throw new RuntimeException(e);
            } catch (KeyManagementException e) {
                throw new RuntimeException(e);
            }

            final PoolingAsyncClientConnectionManager connectionManager = PoolingAsyncClientConnectionManagerBuilder
                    .create()
                    .setTlsStrategy(tlsStrategy)
                    .build();

            return httpClientBuilder
                    .setDefaultCredentialsProvider(credentialsProvider)
                    .setConnectionManager(connectionManager);
        });
    final OpenSearchTransport transport = ApacheHttpClient5TransportBuilder
            .builder(host)
            .setMapper(new JacksonJsonpMapper())
            .build();

    OpenSearchClient client = new OpenSearchClient(transport);
    String index = "sample-index";
    CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder().index(index).build();
        CreateIndexResponse createIndexResponse = client.indices().create(createIndexRequest);
        boolean acknowledged = createIndexResponse.acknowledged();

[szwlhd]

@scrawfor99 Did you get the change to look into my FIPS changes.
I can be found at
https://mvnrepository.com/artifact/at.favre.lib/bcrypt
https://mvnrepository.com/artifact/com.rfksystems/blake2b

[iaindjackson]

I'm back to having a look at this with some of the team here. From looking into the changes, there's an important I think needs to be made with the usage of BC and BCFIPS.

Do we switch to BCFIPS-only or remove BouncyCastle entirely?

We need to decide whether we simply switch the BouncyCastle libraries for their FIPS counterparts in the code.

1. If we switch to BCFIPS-only this would ensure for BouncyCastle that we are only using FIPS-compliant algorithms. This would not negate the need to have an option in OpenSearch to prevent the software from using non-FIPS options (e.g. in-built BCrypt and Blake2b implementations) and use other options via JCA.

2. The other option is to remove all references to BouncyCastle from the Security Plugin and any other plugins using it (e.g. Performance Analyzer) and access the algorithms largely by JCA. If we have in-built BCrypt and Blake2b implementations, we could still use this but they would need to be disabled in a FIPS mode.

My major concern with the first option is that would also require bringing in the BC-TLS and PKIX libraries as well. These should really be configured at the JVM level for the running application, not injected directly by applications. There is also an advantage that we could also continue to use the utility functions provided by BCFIPS and not have to replace them with other implementations.

The second option frees OpenSearch to just use standard Java cryptography methods, however we need to replace all usage of BouncyCastle utility classes with another implementation. We can ship some in-built algorithms, but the JVM could be re-configured to use any other cryptography library - it's not just a choice between BC and BCFIPS. The other advantage is that since BC would not longer be part of the distribution, and it would be one less library to maintain in the face of CVEs and other security vulnerabilities.

There is a half-way house where we make BC a runtime dependency rather than compile, and load it if it's present. This would allow us to preserve some of the same behavior we have for deployments right now - but provide the option to remove BC if necessary and switch to BCFIPS or another.

[trevor-vaughan]

As a user, I would prefer having instructions to configure BCFIPS. Then it is my responsibility to "do the right thing" if necessary. That said, there should definitely be acceptance tests to ensure that the software works when correctly configured.

👍 I also definitely like the option to use any crypto library.

[yanivpaz]

This blog post may help in your BouncyCastle journey.

link is broken @trevor-vaughan

[trevor-vaughan]

@yanivpaz And that's what I get for trying to improve my pages :-D.

Updated Link