Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scan_module(): ValueError: bad marshal data (unknown type code) #1257

Closed
jwilk opened this issue Jan 23, 2018 · 26 comments · Fixed by CTPUG/pygame_cffi#97
Closed

scan_module(): ValueError: bad marshal data (unknown type code) #1257

jwilk opened this issue Jan 23, 2018 · 26 comments · Fixed by CTPUG/pygame_cffi#97

Comments

@jwilk
Copy link

jwilk commented Jan 23, 2018

This happens with CPython from git master (e768c86ef442ef89004089a8a34ce5909ffb90f2):

Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    install_requires=["appdirs"],
  File ".../python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File ".../python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File ".../python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

I believe this is because PEP 552 has been implemented, which adds extra 4 bytes to the .pyc header; but setuptools is not aware of that yet.
@jwilk jwilk mentioned this issue Jan 23, 2018
Closed
@benoit-pierre
Copy link
Member

See #1263.

@benoit-pierre benoit-pierre mentioned this issue Jan 25, 2018
Merged
@jaraco
Copy link
Member

jaraco commented Feb 3, 2018

Release on v38.4.1 pending...

@jaraco jaraco closed this as completed Feb 3, 2018
kryskool added a commit to mgasystem/giraumon that referenced this issue Feb 5, 2018
@kryskool
Update .travis.yml
aed8366 
Fix nigthly failing see pypa/setuptools#1257
@jenisys
Copy link

jenisys commented Feb 5, 2018

Same problem occurred w/ Python 3.7.0a4+ on Travis CI (reported as: travis-ci/travis-ci#9181), probably w/o the fix.

@oremanj oremanj mentioned this issue Feb 8, 2018
Merged
@smagafurov
Copy link

@jaraco Same problem
https://travis-ci.org/aio-libs/aiopg/jobs/341326946

@smagafurov smagafurov mentioned this issue Feb 14, 2018
Merged
@benoit-pierre
Copy link
Member

@smagafurov : did you update setuptools?

@smagafurov
Copy link

setuptools (38.5.1)

@benoit-pierre
Copy link
Member

@smagafurov: I'm sorry but I don't see the setuptools' update in your Travis log and the traceback line numbers don't match 38.5.1. Can you add a call to pip freeze --all before to make sure the right version is used? Just to be clear, you're not relying on setup_requires to update setuptools, right?

@smagafurov
Copy link

@benoit-pierre: I repeat this problem locally

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
(venv) sergey@magafurov:~/projects/aiopg [master]
$ pip list | grep tool
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
prompt-toolkit (1.0.15)
setuptools (38.5.1)

@smagafurov
Copy link

smagafurov commented Feb 14, 2018 •
edited
Loading

@benoit-pierre: How to repeat:

git clone git@github.com:aio-libs/aiopg.git
cd aiopg
python3.7 -m venv ./venv
source ./venv/bin/activate
sudo apt-get install libpq-dev
pip install -U setuptools
python setup.py install

@benoit-pierre
Copy link
Member

I can't reproduce the issue:

> git clone https://github.com/aio-libs/aiopg.git
Cloning into 'aiopg'...
remote: Counting objects: 3267, done.
remote: Total 3267 (delta 0), reused 0 (delta 0), pack-reused 3267
Receiving objects: 100% (3267/3267), 666.68 KiB | 2.22 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.
> cd aiopg
> python3.7 -m venv venv
> ./venv/bin/python --version
Python 3.7.0a4
> ./venv/bin/pip install -U pip wheel setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-38.5.1-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.5.1 wheel-0.30.0
> ./venv/bin/python setup.py -q bdist_egg                                                        
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'tests/**'
  File "build/bdist.linux-x86_64/egg/aiopg/utils.py", line 18
    ensure_future = asyncio.async
                                ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/aiopg/connection.py", line 109
    self._conn = psycopg2.connect(dsn, async=True, **kwargs)
                                           ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...

@benoit-pierre
Copy link
Member

And setuptools' master is still ok on Travis with nightly.

@smagafurov
Copy link

@benoit-pierre: |'m sorry. Please clone this instead:

git clone git@github.com:smagafurov/aiopg.git

I am trying to add python3.7 support to 'aio-libs/aiopg.git'

@benoit-pierre
Copy link
Member

I don't think the clone URL matters (works fine with either one). Can you patch setuptools.commands.bdist_egg to show what's being used?

 setuptools/command/bdist_egg.py | 1 +
 1 file changed, 1 insertion(+)

diff --git i/setuptools/command/bdist_egg.py w/setuptools/command/bdist_egg.py
index 423b8187..a66b03c7 100644
--- i/setuptools/command/bdist_egg.py
+++ w/setuptools/command/bdist_egg.py
@@ -417,6 +417,7 @@ def scan_module(egg_dir, base, name, stubs):
         skip = 12  # skip magic & date & file size
     else:
         skip = 16  # skip magic & reserved? & date & file size
+    print(sys.version_info, skip)
     f = open(filename, 'rb')
     f.read(skip)
     code = marshal.load(f)

@smagafurov
Copy link

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

@smagafurov
Copy link 

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 423, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

@smagafurov
Copy link

smagafurov commented Feb 14, 2018 •
edited
Loading

clone URL matters because I fix python3.7 syntax error here 'smagafurov/aiopg.git'

@benoit-pierre
Copy link
Member

That's another issue, irelevant to the setuptools issue.

@jwilk
Copy link
Author

jwilk commented Feb 14, 2018

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

The first version that implemented PEP 552 was v3.7.0a4.
But setuptools currently does:

elif sys.version_info < (3, 7):

@benoit-pierre
Copy link
Member

That's what I was thinking, thanks for confirming.

@smagafurov
Copy link

smagafurov commented Feb 14, 2018 •
edited
Loading

this works for me

elif sys.version_info < (3, 7, 0, 'alpha', 4):

@benoit-pierre
Copy link
Member

I'm not sure it make sense to explicitely test for 3.7.0a4: if you're running with nightly, then problems are to be expected, and you're also expected to keep current, no?

What I don't understand is why it fails on Travis, or is it another hole in our testsuite?

@benoit-pierre
Copy link
Member

@smagafurov: just to be clear, to fix your issues you should:

  • locally: update Python to >=3.7.0a4
  • Travis CI: update setuptools first before installing other requirements (IMHO adding back nightly testing is a good idea)

@benoit-pierre
Copy link
Member

benoit-pierre commented Feb 14, 2018 •
edited
Loading

@smagafurov: https://travis-ci.org/benoit-pierre/aiopg/jobs/341442846

I'm also confused by the fact that python setup.py install is called before pip install -Ur requirements.txt.

@smagafurov
Copy link

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.

This was referenced Feb 19, 2018
Merged
Merged
miso-belica added a commit to miso-belica/sumy that referenced this issue Mar 24, 2018
@miso-belica
Install newest setuptools to pass tests on Py3.7
7b54e2c 
pypa/setuptools#1257
@wiggin15 wiggin15 mentioned this issue Jul 10, 2018
Merged
peterjc pushed a commit to peterjc/backports.lzma that referenced this issue Jul 11, 2018
@peterjc
update setuptools on Travis
b3f5be8 
setuptools on Travis doesn't support Python 3.7 yet
see pypa/setuptools#1257
jakirkham added a commit to jakirkham/drmaa-python that referenced this issue Jul 28, 2018
@jakirkham
Need newer version of setuptools for Python 3.7
0095a33 
ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795
xref: pypa/setuptools#1257
white-gecko added a commit to AKSW/QuitStore that referenced this issue Sep 25, 2018
@white-gecko
Update setuptools on travis.
2c34753 
Because of travis-ci/travis-ci#9181 resp.
pypa/setuptools#1257
jakirkham added a commit to jakirkham/drmaa-python that referenced this issue Sep 25, 2018
@jakirkham
Need newer version of setuptools for Python 3.7
7d9d5a2 
ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795
xref: pypa/setuptools#1257
@Hanaasagi Hanaasagi mentioned this issue Mar 18, 2019
Open
@gautamkrishnar gautamkrishnar mentioned this issue May 19, 2019
Closed
@diricxbart diricxbart mentioned this issue Jan 5, 2020
Closed
@golyalpha
Copy link

golyalpha commented Feb 15, 2020 •
edited
Loading

Having exact same issue.
Python: 3.7.6
Setuptools: 45.2.0 (just tried to update, no luck)

Nevermind, it was an issue with the source release I downloaded containing pre-existing .pyc files.

@LeoniePhiline LeoniePhiline mentioned this issue Mar 15, 2020
Closed
@andy-maier andy-maier mentioned this issue Apr 13, 2020
Closed
@joshiayush joshiayush mentioned this issue Sep 15, 2021
Closed
@Rushi1804
Copy link

2021-12-28 12:47:12.020620: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-12-28 12:47:12.034213: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[facer] : ----------------------------------------------------------------
[facer] : fd = HOG, fr = FaceNet, sc = CosSim(uninorm=1), th = 0.45
[facer] : loading face recognition model FaceNet...
Traceback (most recent call last):
File "main.py", line 46, in
F.loadModel()
File "D:\Project\facer-classroom-master\utils.py", line 81, in loadModel
self.fr_model = load_model(model_path)
File "C:\Users\Rushi\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\Rushi\AppData\Roaming\Python\Python38\site-packages\keras\utils\generic_utils.py", line 792, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Force later setuptools version for nightly build drnlm/pygame_cffi
8 participants
@jwilk @jaraco @jenisys @benoit-pierre @golyalpha @smagafurov @Rushi1804 and others