Implement new Chunking Algorithm

[dragotin]

There is a new chunking algorithm described here:
https://dragotin.wordpress.com/2015/06/22/owncloud-chunking-ng/
which is implemented for the server here:
owncloud/core#20118

This needs to be implemented in the client.

[moscicki]

Hi, other options expressed in the comments on the blog post are still considered?

[dragotin]

According to #4342 this is a gold ticket.

[moscicki]

@dragotin: for how long old chunking will be supported for compatibility with older server versions?

Also, I think it would make sense to write up the details in the style of https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md so that we can catch any potential inconsistencies or issues before it is actually fully implemented. What do you think about it? Would you be willing to start this doc (can be in a separate file)?

[dragotin]

Is not going to happen in 2.2.0 as the server API is not merged.

[guruz]

Tagging as "Performance" because we will implement automatic chunk size scaling for #4354

[dragotin]

Also consider to do dynamic chunk sizes as described in #4875 and #4354

[guruz]

This was done as part of #5102

[guruz]

I wonder if it not just makes sense to merge the server side PR to check the checksums while uploading. But it is closed server side -> owncloud/core#26582

We don't have good test coverage for this currently..

So I still vote for having chunking ng only if the server checks checksums.. :/

FYI @ogoffart @SamuAlfageme @PVince81

[guruz]

The situation looks better now as @ogoffart @ckamm and @jturcotte tested chunking NG a lot.
@ogoffart is in progress of running the smashbox tests
And @IljaN is implementing checksum verification for 10.0 :)

[SamuAlfageme]

@guruz we can leave this one open until both, the smashbox results and the checksum capabilities, are available since those would contribute to "formally" prove chunking is working. For now, as said, I didn't noticed any bugs regarding this.