S3: An error occurred (403) when calling the HeadObject operation: Forbidden

[th3hamm0r]

We've upgraded django-storages in a wagtail installation from 1.11.1 to 1.12.3, which seems to break the image admin when a thumbnail/rendition does not exist.
Maybe this has something to do with #1084/#1085 or #938?

Downgrading to 1.11.1 fixes the issue for now.

2022-01-17 10:41:11 ERROR django.request: Internal Server Error: /admin/content/images/8/ (log.py:224)
Traceback (most recent call last):
  File "/***/.venv/lib/python3.8/site-packages/wagtail/images/models.py", line 307, in get_rendition
    rendition = self.renditions.get(
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
mysite.base.models.images.ExtendedRendition.DoesNotExist: ExtendedRendition matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
mysite.base.models.images.ExtendedRendition.DoesNotExist: ExtendedRendition matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/***/.venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/***/.venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 204, in _get_response
    response = response.render()
  File "/***/.venv/lib/python3.8/site-packages/wagtail/admin/auth.py", line 189, in overridden_render
    return render()
  File "/***/.venv/lib/python3.8/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/***/.venv/lib/python3.8/site-packages/django/template/response.py", line 83, in rendered_content
    return template.render(context, self._request)
  File "/***/.venv/lib/python3.8/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/***/.venv/lib/python3.8/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/***/.venv/lib/python3.8/site-packages/wagtail/images/templatetags/wagtailimages_tags.py", line 107, in render
    rendition = get_rendition_or_not_found(image, self.filter)
  File "/***/.venv/lib/python3.8/site-packages/wagtail/images/shortcuts.py", line 13, in get_rendition_or_not_found
    return image.get_rendition(specs)
  File "/***/.venv/lib/python3.8/site-packages/wagtail/images/models.py", line 351, in get_rendition
    rendition, created = self.renditions.get_or_create(
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 683, in get_or_create
    return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
    for sql, params in self.as_sql():
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
    value_rows = [
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 302, in pre_save
    file.save(file.name, file.file, save=False)
  File "/***/.venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/***/.venv/lib/python3.8/site-packages/django/core/files/storage.py", line 53, in save
    name = self.get_available_name(name, max_length=max_length)
  File "/***/.venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 605, in get_available_name
    return super().get_available_name(name, max_length)
  File "/***/.venv/lib/python3.8/site-packages/django/core/files/storage.py", line 87, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "/***/.venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 469, in exists
    self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
  File "/***/.venv/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/***/.venv/lib/python3.8/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

[andreasschmitz]

I've encountered a similar problem. Our stack trace looks basically identical, including the line numbers. I also encountered: An error occurred (403) when calling the HeadObject operation: Forbidden

For us the problem was solved by properly configuring the bucket policy as described in https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#iam-policy. Since we are using a S3 compatible API and not AWS S3, we had to slightly alter the policy by separating the bucket and bucket object permissions into two separate statements instead of configuring them as a single one:

    {
            "Sid": "userCRUDObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "<your user here>"
                ]
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name-here>/*"
            ]
        },
        {
            "Sid": "userListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "<your user here>"
                ]
            },
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name-here>"
            ]
        }

Before, we were using the bucket without the s3:ListBucket permission, because of a misconfiguration.

Hope this helps.

However, I am a bit confused why this only happened in 1.12.3, but not in 1.11.1.
I would love to find out what causes this. Does anyone know?

[th3hamm0r]

@andreasschmitz Thanks a lot for sharing this information, we will also check our S3 configs.

+1 for finding the related change in the library ;)

[HenrichHanusovsky]

@th3hamm0r it's because of changes in this commit:
10be72f

For full context, they are using Head object request to find out whether or not object exists in S3 bucket (more on this request here).

It's response depends on the fact if file exists and what permissions you have:

• 200 if file exists
• 404 if file exists and you have s3:ListBucket permission
• 403 if file does not exits and you don't have s3:ListBucket permission (because it's unable to list all files and see if the requested one really does not exist - I assume)

In the mentioned comment, they changed the behavior (to correct one I must say), for checking if file exists:

• previously, if response was not 404, they returned yes, file exists
• now, when response is 200 they return yes, file exists, when it is 404 specifically they return no file does not exist and if any other exception occur (including the 403 response mentioned above), they re-raise the exception

although this is correct, I think some more log output or more specific error would be great, something like:
"We failed to check if file exists, it might be caused by 403 returned by AWS in case of missing permissions"

[GitRon]

I encountered the same issue and ensuring that the IAM user has these two permissions fixed the problem:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Action":[
            "s3:"
         ],
         "Resource":"arn:aws:s3:::your-bucket",
         "Effect":"Allow"
      },
      {
         "Effect":"Allow",
         "Action":"s3:",
         "Resource":[
            "arn:aws:s3:::your-bucket/*"
         ]
      }
   ]
}