Cannot run with python 3.12 on macOS sonoma #1343

patrick-amuse · 2023-10-19T13:46:41Z

I tried to install s3mcd on my mac with zsh.
Since it didn't work there i spun up an ubuntu container and I got it working there. I verified i had the same s3cmd config for both.
Note: i had different versions of python on my ubuntu container and my mac zsh.
In mac i had 3.12.0, and in ubuntu i had 3.8. So it could be a python version problem and not a mac problem.

When trying to run the test i got the following error.

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
ERROR: Test failed: sequence item 1: expected str instance, bytes found

When i tried to run s3cmd la I got a more verbose error:

pat@Patricks-MacBook-Pro dbMigration % s3cmd la         

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /opt/homebrew/bin/s3cmd la
Problem: <class 'TypeError: sequence item 1: expected str instance, bytes found
S3cmd:   2.3.0
python:   3.12.0 (main, Oct  2 2023, 12:03:24) [Clang 15.0.0 (clang-1500.0.40.1)]
environment LANG=en_CA.UTF-8

Traceback (most recent call last):
  File "/opt/homebrew/bin/s3cmd", line 3286, in <module>
    rc = main()
         ^^^^^^
  File "/opt/homebrew/bin/s3cmd", line 3183, in main
    rc = cmd_func(args)
         ^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/s3cmd", line 186, in cmd_all_buckets_list_all_content
    response = s3.list_all_buckets()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/S3.py", line 327, in list_all_buckets
    response["list"] = getListFromXml(response["data"], "Bucket")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 277, in getListFromXml
    tree = getTreeFromXml(xml)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 263, in getTreeFromXml
    xml, xmlns = stripNameSpace(encode_to_s3(xml))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 255, in stripNameSpace
    xml = RE_XML_NAMESPACE.sub("\\1\\2", xml, 1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected str instance, bytes found

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The text was updated successfully, but these errors were encountered:

collinhundley · 2023-10-19T21:36:51Z

I'm having a similar issue with Python 3.12 - did you find a solution?

Here's my error:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ImportError trying to import dateutil.parser.
Please install the python dateutil module:
$ sudo apt-get install python-dateutil
  or
$ sudo yum install python-dateutil
  or
$ pip install python-dateutil
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

s3cmd installed via Homebrew on macOS Ventura. The issue seems to be that the python dependency in Homebrew was just updated to 3.12, as it was working fine with python3.11 a couple days ago.

fviard · 2023-10-20T00:06:36Z

Thank you for your reports. It should now be fixed in Master, and I will try to do the next release very soon.

sveltema · 2023-11-01T02:01:49Z

Workaround until the next version if you don't want to mess about with installing from master:
Edit /opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py
and make the change from commit 9ecb2d8

changing

xml = RE_XML_NAMESPACE.sub("\\1\\2", xml, 1)

to

xml = RE_XML_NAMESPACE.sub(b"\\1\\2", xml, 1)

pragneshpj · 2023-11-08T12:14:05Z

@sveltema after changing xml variable, still not working. :(

fviard · 2023-11-08T12:42:52Z

Sadly it is a known bug with python 3.12. If you can use the GitHub master version, this issue is already fixed. Otherwise a new release will arrive soon with that included. Le mer. 8 nov. 2023 à 13:14, Pragnesh ***@***.***> a écrit :

…

@sveltema <https://github.com/sveltema> after changing xml variable, still not working. :( — Reply to this email directly, view it on GitHub <#1343 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIBJJF72WJDQEZNJ2ANYZDYDNZRRAVCNFSM6AAAAAA6HHWV5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBRG43TCMZUGM> . You are receiving this because you were assigned.Message ID: ***@***.***>

Lipemenezes · 2023-11-29T13:06:11Z

Workaround until the next version if you don't want to mess about with installing from master: Edit /opt/homebrew/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py and make the change from commit 9ecb2d8

changing
xml = RE_XML_NAMESPACE.sub("\\1\\2", xml, 1)
to
xml = RE_XML_NAMESPACE.sub(b"\\1\\2", xml, 1)

Thanks, that fixed it for me, except for the path of the file that was a bit different:
/opt/homebrew/Cellar/s3cmd/2.3.0/lib/python3.12/site-packages/S3/BaseUtils.py

jawys · 2023-11-29T14:49:59Z

Otherwise a new release will arrive soon with that included.

Can you please please please just release a patch version rather than waiting for 2.4.0 milestone to complete?

3 more weeks have passed since your comment @fviard and the issue is now open for 42 days

fviard · 2023-11-29T15:11:58Z

@jawys Indeed, seeing the high impact of this issue, i'm thinking about releasing right now without completing everything that was planned for the milestone.

bestgopher · 2024-02-01T07:22:47Z

Has there been any progress on this issue？

fviard · 2024-02-06T22:42:24Z

@bestgopher
I'm sorry that I did not put a comment here also, but the release with the fix is out v2.4.0:
https://github.com/s3tools/s3cmd/releases/tag/v2.4.0
It is also available through pypi and brew updates.

TheRed86 · 2024-04-03T12:34:17Z

Still an issue on MAcOs Sonoma V14.1...

command:
s3cmd -c ~/.s3cfg -r get s3://endpointaddress ...

Same error as above:

Problem: <class 'KeyError: 'ETag'
S3cmd:   2.4.0
python:   3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]
environment LANG=NOTSET

Traceback (most recent call last):
  File "/opt/homebrew/bin/s3cmd", line 3627, in <module>
    rc = main()
         ^^^^^^
  File "/opt/homebrew/bin/s3cmd", line 3524, in main
    rc = cmd_func(args)
         ^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/s3cmd", line 548, in cmd_object_get
    remote_list, exclude_list, remote_total_size = fetch_remote_list(
                                                   ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.4.0_1/libexec/lib/python3.12/site-packages/S3/FileLists.py", line 515, in fetch_remote_list
    objectlist, tmp_total_size = _get_filelist_remote(uri, recursive = True)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/s3cmd/2.4.0_1/libexec/lib/python3.12/site-packages/S3/FileLists.py", line 478, in _get_filelist_remote
    'md5' : object['ETag'].strip('"\''),
            ~~~~~~^^^^^^^^
KeyError: 'ETag'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

fviard · 2024-04-13T00:42:39Z

@TheRed86 Your error is curious, and I don't think that it is related to the other errors, but I might be wrong.
Which S3 service are you using? AWS?

I see that you say that you use the following command:
s3cmd -c ~/.s3cfg -r get s3://endpointaddress ...
"endpointaddress" is supposed only be the "bucket" name, like s3://mybucket/folder1/file.txt .
The issue is not coming from the fact that you are putting a domain name or ip there?

fviard closed this as completed in 9ecb2d8 Oct 20, 2023

fviard self-assigned this Oct 20, 2023

fviard added the to-be-fixed label Oct 20, 2023

fviard added this to the 2.4.0 milestone Oct 20, 2023

kvibber mentioned this issue Nov 10, 2023

ls command fails while parsing response #1348

Closed

bryansray mentioned this issue Nov 17, 2023

Issue with running s3cmd info? #1354

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot run with python 3.12 on macOS sonoma #1343

Cannot run with python 3.12 on macOS sonoma #1343

patrick-amuse commented Oct 19, 2023 •

edited

Loading

collinhundley commented Oct 19, 2023

fviard commented Oct 20, 2023

sveltema commented Nov 1, 2023

pragneshpj commented Nov 8, 2023

fviard commented Nov 8, 2023 via email

Lipemenezes commented Nov 29, 2023

jawys commented Nov 29, 2023

fviard commented Nov 29, 2023

bestgopher commented Feb 1, 2024

fviard commented Feb 6, 2024

TheRed86 commented Apr 3, 2024 •

edited

Loading

fviard commented Apr 13, 2024

Cannot run with python 3.12 on macOS sonoma #1343

Cannot run with python 3.12 on macOS sonoma #1343

Comments

patrick-amuse commented Oct 19, 2023 • edited Loading

collinhundley commented Oct 19, 2023

fviard commented Oct 20, 2023

sveltema commented Nov 1, 2023

pragneshpj commented Nov 8, 2023

fviard commented Nov 8, 2023 via email

Lipemenezes commented Nov 29, 2023

jawys commented Nov 29, 2023

fviard commented Nov 29, 2023

bestgopher commented Feb 1, 2024

fviard commented Feb 6, 2024

TheRed86 commented Apr 3, 2024 • edited Loading

fviard commented Apr 13, 2024

patrick-amuse commented Oct 19, 2023 •

edited

Loading

TheRed86 commented Apr 3, 2024 •

edited

Loading