MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

kellymclaughlin · 2020-03-18T22:28:18Z

The heart of this problem is that we need a way when using AES/CTR mode to increment the counter that's used along with the nonce or initialization vector (IV) as input to the decryption process based on the block that the bytes returned from the Range header request belong to. Presently, our solution to that is the calls to Cipher.update, but this is an expensive method when really all we need is to increment a counter to reflect the current block being decrypted. So instead what we can do is to take the IV and increment that value manually (I could not find any built-in Java API for this purpose) and then we can proceed to decrypt the block just like before. Calculating this block-targeted IV value is very cheap compared to the update calls.

I created a test program that riffs on the Range header example already in the repo. It uploads an object to Manta and then downloads 1024 bytes chunks from the beggining, middle, and end of the object and reports the time it took and if the downloaded data matched what we expected. Here is the full code I used:

/*
 * Copyright 2020 Joyent, Inc. All rights reserved.
 *
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
 */

import com.joyent.manta.client.MantaClient;
import com.joyent.manta.client.MantaObjectResponse;
import com.joyent.manta.config.ChainedConfigContext;
import com.joyent.manta.config.ConfigContext;
import com.joyent.manta.config.DefaultsConfigContext;
import com.joyent.manta.config.EncryptionAuthenticationMode;
import com.joyent.manta.config.EnvVarConfigContext;
import com.joyent.manta.config.MapConfigContext;
import com.joyent.manta.http.MantaHttpHeaders;

import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Scanner;
import java.util.Arrays;

/*
* Usage: set the mantaUserName, privateKeyPath, and publicKeyId with your own values.
 */
public class ClientEncryptionRangeDownload {

    public static void main(String... args) throws IOException {
        String mantaUserName = "kelly";
        String privateKeyPath = "/home/kelly/.ssh/id_rsa";
        String publicKeyId = "39:98:02:54:3f:d5:28:df:09:76:39:62:94:5f:c6:49";

        ConfigContext config = new ChainedConfigContext(
                new DefaultsConfigContext(),
                new EnvVarConfigContext(),
                new MapConfigContext(System.getProperties()))
                .setMantaURL("https://manta.coal.joyent.us")
                // If there is no subuser, then just use the account name
                .setMantaUser(mantaUserName)
                .setMantaKeyPath(privateKeyPath)
                .setMantaKeyId(publicKeyId)
                .setClientEncryptionEnabled(true)
                .setEncryptionAlgorithm("AES256/CTR/NoPadding")
                .setEncryptionAuthenticationMode(EncryptionAuthenticationMode.Optional)
                .setPermitUnencryptedDownloads(false)
                .setEncryptionKeyId("simple/example")
                .setEncryptionPrivateKeyBytes(Base64.getDecoder().decode("RkZGRkZGRkJEOTY3ODNDNkM5MUUyMjIyMTExMTIyMjI="));

        try (MantaClient client = new MantaClient(config)) {
            String mantaPath = MantaClient.SEPARATOR + mantaUserName + "/stor/foo";

            byte[] buf = new byte[100000000];
            Arrays.fill(buf, 0, 55555000, (byte)'a');
            Arrays.fill(buf, 55555000, 99998976, (byte)'b');
            Arrays.fill(buf, 99998976, 100000000, (byte)'c');

            String plaintext = new String(buf, StandardCharsets.UTF_8.name());

            MantaObjectResponse response = client.put(mantaPath, plaintext);

            // Read the first chunk of bytes from the uploaded file
            MantaHttpHeaders headers = new MantaHttpHeaders();
            headers.setByteRange(0L, 1023L);

            long start1 = System.currentTimeMillis();

            byte[] readBuf = new byte[1024];

            byte[] expectedBytes1 = new byte[1024];
            Arrays.fill(expectedBytes1, (byte)'a');
            byte[] expectedBytes2 = new byte[1024];
            Arrays.fill(expectedBytes2, (byte)'b');
            byte[] expectedBytes3 = new byte[1024];
            Arrays.fill(expectedBytes3, (byte)'c');

            String readData = new String("");

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end1 = System.currentTimeMillis();
            long timeElapsed = end1 - start1;
            System.out.println("Time to fetch first chunk: " + timeElapsed);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes1, readBuf)) {
                System.out.println("First chunk of data matched expected data");
            }

            readData = "";

            // Read a middle chunk of the file
            headers.setByteRange(55555000L, 55556023L);

            long start2 = System.currentTimeMillis();

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end2 = System.currentTimeMillis();
            long timeElapsed2 = end2 - start2;
            System.out.println("Time to fetch middle chunk: " + timeElapsed2);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes2, readBuf)) {
                System.out.println("Middle chunk of data matched expected data");
            }

            readData = "";


            //Read last chunk of the file
            headers.setByteRange(99998976L, 99999999L);

            long start3 = System.currentTimeMillis();

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end3 = System.currentTimeMillis();
            long timeElapsed3 = end3 - start3;
            System.out.println("Time to fetch last chunk: " + timeElapsed3);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes3, readBuf)) {
                System.out.println("Last chunk of data matched expected data");
            }
        }
    }
}

Here is a run of the test program without the changes for this PR (i.e. It is using the Cipher.update method to increment the AES/CTR counter):

[kelly@mantadev java-manta-cse-test]$ java -cp ./target/cse-test-1.0-SNAPSHOT.jar ClientEncryptionRangeDownload
16:19:20.903 [main] DEBUG com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Bouncy Castle provider was not loaded, adding to providers
16:19:20.919 [main] INFO com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Security provider chosen for CSE: BC version 1.61 
16:19:20.920 [main] DEBUG com.joyent.manta.client.MantaClient - Preferred Security Provider: BC version 1.61
16:19:20.993 [main] WARN com.joyent.manta.http.MantaSSLConnectionSocketFactory - Configuration: tlsInsecure is true.  ALL TLS VERIFICATION IS DISABLED!
16:19:21.599 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Secret key id: simple/example
16:19:21.600 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption type: client/1
16:19:21.600 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption cipher: AES256/CTR/NoPadding
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - IV: e0c84bb6823ea2fc0fa44622ba073086
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - HMAC algorithm: HmacMD5
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Plaintext content-length: 100000000
16:19:21.602 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata IV: 39911fce41ae93ad5b1d92a85ec8bac6
16:19:21.604 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata plaintext:
e-content-type: text/plain; charset=UTF-8

16:19:21.604 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata ciphertext: OZEfzkGuk61bHZKoXsi6xg==
16:19:21.606 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata HMAC: Rq81Kksb9ObxyZ9lmUlBDg==
16:19:21.606 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled TLS protocols: TLSv1.2
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported TLS protocols: TLSv1.2
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:19:30.026 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo response [204] No Content 
16:19:30.182 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch first chunk: 166
First chunk of data matched expected data
16:19:30.341 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch middle chunk: 1331
Middle chunk of data matched expected data
16:19:31.668 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch last chunk: 2208
Last chunk of data matched expected data

As has been reported the time complete the request for each file chunk increases as the offset into the object increases.

Now here is a run of the program with the changes from this PR:

[kelly@mantadev java-manta-cse-test]$ java -cp ./target/cse-test-1.0-SNAPSHOT.jar ClientEncryptionRangeDownload
16:17:48.319 [main] DEBUG com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Bouncy Castle provider was not loaded, adding to providers
16:17:48.338 [main] INFO com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Security provider chosen for CSE: BC version 1.61 
16:17:48.339 [main] DEBUG com.joyent.manta.client.MantaClient - Preferred Security Provider: BC version 1.61
16:17:48.417 [main] WARN com.joyent.manta.http.MantaSSLConnectionSocketFactory - Configuration: tlsInsecure is true.  ALL TLS VERIFICATION IS DISABLED!
16:17:49.032 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Secret key id: simple/example
16:17:49.033 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption type: client/1
16:17:49.033 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption cipher: AES256/CTR/NoPadding
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - IV: 13ac4a9d39378fc7f16c2ef8458db5d8
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - HMAC algorithm: HmacMD5
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Plaintext content-length: 100000000
16:17:49.035 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata IV: 888b6ee6ed4f95526da61bc07a8911ed
16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata plaintext:
e-content-type: text/plain; charset=UTF-8

16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata ciphertext: iItu5u1PlVJtphvAeokR7Q==
16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata HMAC: LiGvRuT8fCf7OYL4zj08TA==
16:17:49.036 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo
16:17:49.146 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled TLS protocols: TLSv1.2
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported TLS protocols: TLSv1.2
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:17:55.535 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo response [204] No Content 
16:17:55.697 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch first chunk: 172
First chunk of data matched expected data
16:17:55.856 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch middle chunk: 149
Middle chunk of data matched expected data
16:17:55.996 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch last chunk: 142
Last chunk of data matched expected data

The reported times for each chunk request are now within the same order of magnitude which is exactly what we want while the data for each chunk still matches our expected data indicating successful decryption.

…e start is big.

indianwhocodes

It might be prudent to have a dependent test-case reproducing the conditions that were explained earlier in the PR's premise and making comparative evaluations accordingly

...t-unshaded/src/main/java/com/joyent/manta/client/crypto/MantaEncryptedObjectInputStream.java

...-manta-client-unshaded/src/main/java/com/joyent/manta/client/crypto/AesCtrCipherDetails.java

indianwhocodes · 2020-03-20T22:30:53Z

All tests passed:

[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta 3.4.2-SNAPSHOT .......................... SUCCESS [  5.250 s]
[INFO] java-manta-client-unshaded ......................... SUCCESS [06:48 min]
[INFO] java-manta-client .................................. SUCCESS [  8.782 s]
[INFO] java-manta-client-kryo-serialization ............... SUCCESS [ 39.553 s]
[INFO] java-manta-cli ..................................... SUCCESS [02:16 min]
[INFO] java-manta-it ...................................... SUCCESS [19:45 min]
[INFO] java-manta-benchmark ............................... SUCCESS [02:12 min]
[INFO] java-manta-examples 3.4.2-SNAPSHOT ................. SUCCESS [  0.528 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:57 min
[INFO] Finished at: 2020-03-20T15:21:01-07:00
[INFO] ------------------------------------------------------------------------

indianwhocodes · 2020-03-20T22:10:28Z

...-manta-client-unshaded/src/main/java/com/joyent/manta/client/crypto/AesCtrCipherDetails.java

+        final long startingBlock = position / blockSize;
+        final byte[] counter = ByteBuffer.allocate(16).putLong(startingBlock).array();
+        final BigInteger ivBigInt = new BigInteger(iv);
+        byte[] updatedIV = Arrays.copyOf(ivBigInt.add(BigInteger.valueOf(startingBlock)).toByteArray(), 16);


Can we potentially replace 16 with getIVLengthInBytes() ?

This corrects a bug introduced in #562 that prevented correct decryption of some objects encrypted using AES CTR mode. The objects susceptible to this bug are ones where IV values generated during the encryption process are interpreted as negative BigInteger values. The calculation of applying the CTR mode offset to the initial, randomly-generated IV and the conversion back to a byte array yielded incorrect results in such cases and prevented correct decryption of the associated objects. This PR changes the method used to determine the IV value to use for decrypting the request to instead use operations directly on the byte array rather than converting the values to BigInteger values. This PR also adds testing to ensure that the IV calculation when using an offset of zero results in an IV that is identical to the input IV. It also expands the canRandomlyReadPlaintextPositionFromCiphertext testing to exercise both the former and current methods of advancing the Cipher to the requested cipher block. The testing compares each decryption result to the expected plaintext as well as compares the results of each method to one another.

MANTA-5083 MANTA JAVA client is taking too long to get file when rang…

e026a1b

…e start is big.

kellymclaughlin requested a review from indianwhocodes March 18, 2020 22:28

Address some test failures

8b54521

indianwhocodes reviewed Mar 20, 2020

View reviewed changes

indianwhocodes suggested changes Mar 20, 2020

View reviewed changes

...t-unshaded/src/main/java/com/joyent/manta/client/crypto/MantaEncryptedObjectInputStream.java Outdated Show resolved Hide resolved

...-manta-client-unshaded/src/main/java/com/joyent/manta/client/crypto/AesCtrCipherDetails.java Outdated Show resolved Hide resolved

indianwhocodes assigned kellymclaughlin Mar 20, 2020

indianwhocodes added the enhancement label Mar 20, 2020

Updates based on review feedback

a24d468

indianwhocodes approved these changes Mar 20, 2020

View reviewed changes

kellymclaughlin merged commit 10e9cba into master Mar 20, 2020

kellymclaughlin deleted the MANTA-5083 branch March 20, 2020 22:47

indianwhocodes changed the title ~~MANTA-5083 MANTA JAVA client is taking too long to get file when range start is big.~~ MANTA-5083 JAVA client is taking too long to get file when range start is big. Mar 26, 2020

kellymclaughlin mentioned this pull request Apr 14, 2020

MANTA-5143 Data integrity issue with Java SDK 3.4.2 #568

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

kellymclaughlin commented Mar 18, 2020

indianwhocodes left a comment

indianwhocodes commented Mar 20, 2020

indianwhocodes Mar 20, 2020

MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

Conversation

kellymclaughlin commented Mar 18, 2020

indianwhocodes left a comment

Choose a reason for hiding this comment

indianwhocodes commented Mar 20, 2020

indianwhocodes Mar 20, 2020

Choose a reason for hiding this comment