S3Boto3Storage.open(name, "r") doesn't handle newlines properly

[craigds]

With S3Boto3Storage, opening a file in text mode ("r") doesn't properly convert windows newlines ("\r\n") to "\n" as you'd expect. Neither does it handle \r as \n

[jschneier]

Thanks for opening an issue. When you say "as you expect" can you link me to docs where the expected behavior is to change the file contents in that way?

[craigds]

Thanks for your reply.

This is the default behaviour of text-mode files in Python: https://docs.python.org/3/library/functions.html#open

newline determines how to parse newline characters from the stream. It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

• When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

• When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '' or '\n', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

Django's builtin storage classes also adhere to this API, which is how we discovered this discrepancy - when changing from a django-builtin storage backend to a django-storages one, the newlines in one of our tests started playing up.