Chunked upload / partial updates to object store

[matthiasharrer]

Hi,

I was just looking through the object store interfaces, evaluating it for my use cases.
One thing I am missing is a partial update of an object. A bit like multipart upload to S3 and the like.
As my understanding is, that objects are basically just streams this should technically not be an issue right?

For my use case, which I am not sure if it's an appropriate use case for object store in general:

Users are uploading large files (100 MBs to multiple GBs) using https://tus.io/ for increasing upload reliability, where users upload the file in chunks and in case of any failure can restart the upload where it was left off.
Failures might be the users internet connection, but also for example server restarts / upgrades.

So I think the basic requirement for using NATS as a store would be to add new chunks at the end of a file and to somehow mark a file as incomplete / complete (which could be done as metadata I guess)

Does such functionality already exist?
Is this a proper use case for the object store?

[aricart]

this actually doesn't work with ObjectStore - the reason being that each chunk is considered to be in-order in the stream. So it is not possible to just update chunk #123 and have it be at the right offset. With that said, there could be different strategies that could be used to implement this with a little more work from the client. You could think of the asset as being made of 100 logical entries on your side (regardless what the storage chunks are), and you could update each of these entries as you see fit. ObjectStore will ensure that the chunks are correct, but you could simply replace the entry that failed as a whole. - IE store large.1, large.2..., large.N - your code can then simply request each of these objects and then put it back together yourself.

[matthiasharrer]

Hey thanks for the quick answer!

Actually for me it would be totally ok if I could just add new chunks at the end of the file / stream (I think .. )
But I guess that's also not curretnly possible with the ObjectStore.
Maybe I will continue thinking about a more bespoke solution as you mentioned.

[gedw99]

@matthiasharrer

Does this example of client code help ?

[matthiasharrer]

Which example code do you mean?

[gedw99]

https://github.com/mprimi/nasefa

sorry I am idiot - did not paste URL in.

the code works very well.

there are 2 thing am am adding btw:

1. Client chunks to local disk and then into NATS Server so I get resumable uploads
2. Client agent chunks into NATS . This I think is what you also need right ?

here is a sample via Makefile.

1:
	# upload
	#nasefa send -bundleName tax_documents tax-return.pdf tax-summary.xls receipts.zip
	#$(REPO_NAME) send -h
	$(CMD) send -bundleName $(EX_BUNDLE_NAME) $(USER_01)/$(EX_FILE_NAME)
2:
	# download
	# nasefa receive ~/Desktop tax_documents
	mkdir -p $(USER_02)
	$(REPO_NAME) receive $(USER_02) $(EX_BUNDLE_NAME)

3:
	# list
	# nasefa list
	$(REPO_NAME) list

4:
	# send tagged
	# nasefa send -bundleName app-foo-binaries -to cluster-bar app-foo.exe
	# Any hosts tagged cluster-bar will auto-download the latest version of app-foo.exe into ~/bin.
	#$(REPO_NAME) send -h
	$(REPO_NAME) send -bundleName $(EX_BUNDLE_NAME) -to $(EX_TAG_NAME) $(USER_01)/$(EX_FILE_NAME)

5:
	# review tagged automatically.
	# OWN TERMNAL
	# nasefa auto-receive ~/bin app-foo cluster-bar host-baz
	#$(REPO_NAME) auto-receive -h
	$(REPO_NAME) auto-receive $(USER_02) $(EX_TAG_NAME) 

6:
	# web GUI
	# OWN TERMNAL
	# nasefa web -bindAddr :8080
	$(REPO_NAME) web -bindAddr :8080
	# http://localhost:8080

[gedw99]

Buzz me if your interested in working on this...