upload_file() fails when file_or_path is open file larger than 5GB and obj_name is specified

[frankoid]

If file_or_path is an open file then its filename is used to name the created object, but the passed obj_name is used in the call to self.get_object() at the end of the method. This results in the following error:

Traceback (most recent call last):
  File "./manage", line 14, in <module>
    execute_manager(settings)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/bright/asset-bank-tools/core/management/commands/migrate.py", line 99, in handle
    delete=options['delete'])
  File "/home/bright/asset-bank-tools/core/migrate.py", line 175, in migrate
    ltr.migrate()
  File "/home/bright/asset-bank-tools/core/migrate.py", line 32, in migrate
    self.migrate_asset(asset)
  File "/home/bright/asset-bank-tools/core/migrate.py", line 51, in migrate_asset
    self.migrate_asset_file(asset, location)
  File "/home/bright/asset-bank-tools/core/migrate.py", line 67, in migrate_asset_file
    self.move_file(asset, location)
  File "/home/bright/asset-bank-tools/core/migrate.py", line 139, in move_file
    self.get_remote_key(asset, location), f)
  File "/home/bright/asset-bank-tools/core/container.py", line 295, in put_object_from_file
    self.backend.upload_file(fp, key)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/pyrax/cf_wrapper/container.py", line 168, in upload_file
    content_length=content_length)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/pyrax/cf_wrapper/client.py", line 68, in _wrapped
    ret = fnc(self, *args, **kwargs)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/pyrax/cf_wrapper/client.py", line 805, in upload_file
    return self.get_object(container, obj_name)
  File "/home/bright/env/asset-bank-tools/lib/python2.6/site-packages/pyrax/cf_wrapper/client.py", line 87, in _wrapped
    bad_object.groups()[0])
pyrax.exceptions.NoSuchObject: Object 'large.mov' doesn't exist

[frankoid]

I'll submit a pull request to fix this soon...

[EdLeafe]

I'm not following the disconnect here. What was the full path? If it ended in 'large.mov', that should be the name of the object via the manifest.

[frankoid]

I'd rather not say what the exact full path is because it's client data, but it was contained a directory path as well as a filename (N.B. large.mov isn't actually the real filename either, I changed it too for confidentiality - I don't think that makes any difference but sorry if it caused confusion).

The problem is, if you do:

f = open("/home/francis/foo.mov", "rb")
client.upload_file(cont, f)

...then pyrax creates an object in cont called /home/francis/foo.mov. It should create one called foo.mov (IMHO).

If you were to say "when you pass a file object in then the object name is the full path, if you want to specify the object name then you need to explicitly pass an obj_name parameter" then that would be fair enough - but then an alternative fix would be needed for the NoSuchObject exception thrown by the get_object call at the end of upload_file:

return self.get_object(container, obj_name)

To be honest I'm not quite sure what's going on here - I'm not sure why the get_object call fails...

Not quite sure how obj_name gets assigned the value foo.mov when fname is /home/francis/foo.mov.

[frankoid]

Actually I think that our code that calls pyrax might be doing something weird here - it's probably best if you hold off on this until I've investigated the non-pyrax code!

[EdLeafe]

OK, I see the problem now. And I don't care about the actual names; I just needed to see the difference between what the file path was and what was being stored.

[frankoid]

Nope - our code looks sane. I thought it might be missing out the container argument to upload_file, but in fact it is calling pyrax.cf_wrapper.container.Container.upload_file which fills in the container argument.

However, our code does pass obj_name, so I think that my pyrax change in this pull request doesn't fix the right problem. The problem is that for multi-part uploads pyrax uses fname to name the created object when it should be using the obj_name that it was passed.

[frankoid]

OK - so the key piece of information that was missing from my original report was that our code is passing obj_name to pyrax.

My original "fix" - use basename when deriving object name from file objects - would not fix this issue at all but I still think it's an improvement in behaviour so I'll submit a pull request for it.

I now have another change that I think does fix this actual issue. It currently has the 'use basename' commit as its parent commit, but I can unpick it if you want the fix for this usse without the 'use basename' change.