ETag/If-None-Match doesn't work as expected on production

[sergeysolovev]

swapi.co doesn't return 304 (Not Modified) on subsequent requests with If-None-Match request header set to the same value as ETag from the server response

Steps to reproduce (Chrome)
Make sure that cache is not disabled in dev tools. Request the following url twice directly from the browser: http://swapi.co/api/?format=json, or execute fetch('http://swapi.co/api/?format=json') from console.

The first response from the server would have ETag header like this:

Etag:W/"1f7a4766c9ebf66cdb1ddb85d5cc6f2f"

The second request to the server would have If-None-Match header with the same checksum:

If-None-Match:W/"1f7a4766c9ebf66cdb1ddb85d5cc6f2f"

It is expected that the second response would have status code 304 (Not Modified) with empty body, since the checksum hasn't changed. But the actual status code is 200 with the same body as the first request's.

Dev environment
The thing is, it works as expected in dev environment, given that it's being served over HTTP/1.1.

Possible reasons
The first notable difference between prod and dev is that prod responds with weak ETag in the form W/"<string>". It might have something to do with cloudflare, since they support ETag. I haven't investigated that further though.

That I am sure about is that it has to do with weak ETags. Try to make the second request, mentioned above, through curl:

curl -I 'http://swapi.co/api/?format=json' -H 'Origin: null' \
-H 'If-None-Match: W/"1f7a4766c9ebf66cdb1ddb85d5cc6f2f"'

outputs

HTTP/1.1 200 OK
...

Then try to manually remove W/ part from the checksum:

curl -I 'http://swapi.co/api/?format=json' -H 'Origin: null' \
-H 'If-None-Match: "1f7a4766c9ebf66cdb1ddb85d5cc6f2f"'

outputs

HTTP/1.1 304 NOT MODIFIED
...

It would be nice to have it working!

[sergeysolovev]

Found the reason and possible fixes.

If the client sends the header Accept-Encoding with non-empty value, e.g. gzip, deflate, br (and it usually does), the production reverse proxy compresses response body, received from django, and sends back to the client. In this case the reverse proxy prepends an ETag strong validator, generated by django, with W/ prefix, to make it a weak validator and meet https://tools.ietf.org/html/rfc7232#section-2.1. That's how nginx works and that's right.

One possible solution is to avoid sending Accept-Encoding from the client. But it's not always possible. For example, this header is a forbidden header name for fetch. The same applies for XMLHttpRequest. It means that this solution won't work for JavaScript clients.

Another solution is to make compression on the upstream side with django.middleware.gzip.GZipMiddleware and remove the suffix ;gzip from generated ETag value, like described here. In this case the reverse proxy keeps ETag untouched.

To see how it works, request this url twice from the browser:

https://swapi.now.sh/api/?format=json

The second request should return 304 NOT MODIFIED, as expected. See my previous message for more details on how to test it.

PR is coming.

[reaper8055]

May I please take a look on your nginx config, I am struggling configuring if-none-match. I am new to this so it would be really helpful if I can compare the proxy_cache_key config part.

Thank you