Multipart Upload default Part Size

[patrickscottbest]

Hi, i tried to run with and without specifying multipart upload size. Both cases do not work. Specifying a manually chosen power of 2 works.

-p, --partsize [Integer]                sets the size of each part for multipart uploads (default: 10485760)

com.amazonaws.services.glacier.model.InvalidParameterValueException: Invalid part size: 10485760. Part size must not be null, must be a power of two and be between 1048576 and 4294967296 bytes. (Service: AmazonGlacier; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 

These are the nearest powers of two, which works when specified manually.

2^23 8388608
2^24 16777216
2^25 33554432

Also wanted to mention, that an auto-part-size-option would be nice, where a file is evaluated for sizing options and then offered a suggested range.

[patrickscottbest]

Wow! To my surprise, you are also the MVN keeper for the component required to populate the default value 👍

UploadMultipartArchiveCommand.java

import de.kopis.glacier.parsers.GlacierUploaderOptionParser;
    @Override
    public void exec(OptionSet options, GlacierUploaderOptionParser optionParser) {
       ...
        final Long partSize = options.valueOf(optionParser.partSize);
       ... 

Where can that code be reviewed?

Would you be OK with a GitHub PR that overrides a non- ^2 number with, say, 16777216 ? Or can you fix or show me how i can review the .jar artifact?

    <parent>
        <groupId>de.kopis.glacier</groupId>
        <artifactId>glacieruploader-parent</artifactId>
        <version>0.1.2-SNAPSHOT</version>
        <relativePath>..</relativePath>
    </parent>

[MoriTanosuke]

I'm not sure if I understood you correctly, but the tool glacieruploader lives completely in this repository. The module you mentioned is the root module of this project, see

glacieruploader/pom.xml

Line 5 in e39b71d

<artifactId>glacieruploader-parent</artifactId>

I will look into this and make sure that the default value is still valid with the current version of the amazon SDK.

[MoriTanosuke]

Strange, the default value for the partsize option is 16777216, see

glacieruploader/glacieruploader-parsers/src/main/java/de/kopis/glacier/parsers/GlacierUploaderOptionParser.java

Lines 244 to 251 in e39b71d

	private ArgumentAcceptingOptionSpec<Long> parsePartSize(final Configuration config) {
	return acceptsAll(Arrays.asList("partsize", "p"),
	"sets the size of each part for multipart uploads (must be a power of 2)")
	.withRequiredArg()
	.ofType(Long.class)
	.defaultsTo((long) Math.pow(4096, 2));
	// 16 MB.
	}

[MoriTanosuke]

@patrickscottbest can you provide the logfile from your failed run? It should contain a line like:

Multipart uploading FILENAME (TOTALSIZE) to vault XXX with part size 123456789 (123.4)

You can clean information about the filename and the vault, I'm mostly interested in the reported multipart size and the following logmessages.

[MoriTanosuke]

I found an issue with the command detection in the current code which prevented the UploadMultipartArchiveCommand to trigger when the user did not specify the partsize option. I was able to upload an archive without changing the default value though:

# java -jar glacieruploader-impl\target\glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar --region XXX -v XXX--multipartupload glacieruploader-impl\target\glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar
INFO  Using region: XXX
INFO  Multipart uploading glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar (117.41MB) to vault XXX with part size 16777216 (16.00MB).
...
INFO  Local Checksum: 99cea888d76b150b922a67491f71268e0fa4f990e50e1c3b75d2caacde13db81
INFO  Remote Checksum: 99cea888d76b150b922a67491f71268e0fa4f990e50e1c3b75d2caacde13db81
INFO  Checksums are identical, upload succeeded.

I was also able to upload when manually configuring --partsize 16777216:

java -jar glacieruploader-impl\target\glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar --region XXX -v XXX --multipartupload glacieruploader-impl\target\glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar --partsize 16777216
INFO  Using region: XXX
INFO  Multipart uploading glacieruploader-impl-0.1.2-SNAPSHOT-jar-with-dependencies.jar (117.41MB) to vault XXX with part size 16777216 (16.00MB).
INFO  Upload ID (token): XXX
INFO  Part 1/8 (bytes 0-16777215/*) uploaded, checksum: dd367b9aa66f0c6a72b34092d6c71264bdfb9547718a7cb64b3763c66f5270b3, retries: 1
INFO  Part 2/8 (bytes 16777216-33554431/*) uploaded, checksum: db3eb893db37a8f01c4419d03338a3e301aae021430299378d413b87fb8cd534, retries: 1
INFO  Part 3/8 (bytes 33554432-50331647/*) uploaded, checksum: df03efff6eea3cf548f467fa3b013d835d1d053f8ee95e096ee9502e7907ac71, retries: 1
INFO  Part 4/8 (bytes 50331648-67108863/*) uploaded, checksum: 6c555dbe99cdad598a5c76ea3d6be17ce0296cbb77203eaf597c85ee33c8dd28, retries: 1
INFO  Part 5/8 (bytes 67108864-83886079/*) uploaded, checksum: 2716b1e840bcc092f98d555318a24a980333a78ea637ca0b8f68370688b8ebbf, retries: 1
INFO  Part 6/8 (bytes 83886080-100663295/*) uploaded, checksum: 13dd3832ed510e9ac71eeb32b5d8b38840635e58ad18fabb21054a1006eeec53, retries: 1
INFO  Part 7/8 (bytes 100663296-117440511/*) uploaded, checksum: bf3a211b6bb11731c802e42dc03e1d7bfab4ad3f0461d02767159e07af3976ff, retries: 1
INFO  Part 8/8 (bytes 117440512-123104293/*) uploaded, checksum: 4b3a36a5ba02b1f14bb18c0a16c0d5f0dfd87d1b9e5d3e70c2a6b871223ce070, retries: 1
INFO  Uploaded Archive ID: XXX
INFO  Local Checksum: 99cea888d76b150b922a67491f71268e0fa4f990e50e1c3b75d2caacde13db81
INFO  Remote Checksum: 99cea888d76b150b922a67491f71268e0fa4f990e50e1c3b75d2caacde13db81
INFO  Checksums are identical, upload succeeded.

[MoriTanosuke]

I've uploaded branch bugfix/74 with my fixes for the partsize option.

[patrickscottbest]

Thank you! It's a great tool. I personally have no problem because I calculate my own preferred size and supply it to the command line every time, except the first time I used the app!