Consider a GenericStorage or SettingStorage

[fission6]

First off awesome work with django-storages its very very handy.

One issue I've run into is easily managing different storages on a model field level per environment. When I run locally everything can use the default storage but in production we need different subclasses of s3boto3storages. This makes it really tricky to manage as we flip back and forth between environments. Perhaps I am missing a better way to go about things but I whipped up this code which may be worth considering.

The concept is introduce a more Generic or Settings driven Storage class and leave the configuration of which storage backends to use in a typical settings file.

from django.conf import settings
from django.utils.module_loading import import_string
from django.utils.deconstruct import deconstructible

def get_storage_class(import_path=None):
    return import_string(import_path or settings.DEFAULT_FILE_STORAGE)

@deconstructible
class GenericStorage(object):
    """
    Generic storage class which takes in a key and looks up a storage defined
    in STORAGES in django settings. If setting does not exist then use the default
    storage.
    """
    def __init__(self, storage_key):
        self.storage_key = storage_key
        self._storage = self.build_storage()

    def __getattr__(self, name):
        """Proxy to underlying storage class"""
        return getattr(self._storage, name)

    def get_storage_from_settings(self):
        """Fetch backend from settings"""
        storage_key = self.storage_key
        storage_backend = None

        if storage_key and hasattr(settings, 'STORAGES'):
            try:
                storage_entry = settings.STORAGES[self.storage_key]
            except KeyError:
                msg = 'Could not find storage {}'.format(self.storage_key)
                raise KeyError(msg)
            storage_backend = storage_entry.get('BACKEND')

        return storage_backend

    def build_storage(self):
        """Construct Storage"""
        storage_backend = self.get_storage_from_settings()
        return get_storage_class(storage_backend)()

-----
settings

STORAGES = {
    'public-media': {
        'BACKEND': 'whatever.storages.backends.s3boto3.PublicMediaStorage',
    },
    'private-media': {
        'BACKEND': 'whatever.storages.backends.s3boto3.PrivateMediaStorage',
    }
}

----
storages
from storages.backends.s3boto3 import S3Boto3Storage


class PublicMediaStorage(S3Boto3Storage):
    """S3 Public Storage"""
    bucket_name = 'media.whatever.com'
    default_acl = 'private' # made public through aws bucket policy
    file_overwrite = False
    location = 'media'
    querystring_auth = False
    region_name = 'us-west-2'


class PrivateMediaStorage(S3Boto3Storage):
    """S3 Private Storage"""
    bucket_name = 'media.whatever.com'
    default_acl = 'private'
    file_overwrite = False
    location = 'media'
    querystring_expire = 60 * 60
    region_name = 'us-west-2'


-----
example

document = models.FileField(upload_to,...,storage=GenericStorage('public-media'))

[sww314]

@fission6 Do you envision this like the database router?

I am interested in that idea and see how it could useful for any sites that have a mix of public and private files.

[fission6]

Similar to a router, yes, it can be controlled through settings file so it can change environment to environment

and it came about for the reason you pointed out mix of public and private files AND things changing environment to environment

[jschneier]

So I think #524 is basically what you want. And then there is a minor feature request on top of that I think.

[fission6]

@jschneier i really can't make immediate sense of whats going on in #524 at first glance, there is def a relationship between it and what I am proposing, but whats offered in this issue is what has proven to be a very flexible way to configure storages, similar to how you'd look to configure multiple databases as well as the luxury to easily change across staging environment

perhaps @jdufresne can offer some thoughts too or find areas to squeeze the best from both sides

[bennettrogers]

@fission6 – thank you for this snippet! I was struggling to figure out a good way to select different storage backends for ImageFields based on the DEBUG setting, dev/production environments, etc without generating unnecessary migrations. Your approach solves this very elegantly, and I can see it being useful for enabling complete control of dynamic storage backend selection in the future.

[jschneier]

We went through with something like this.