Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRAFT: Experimental build on Apple M1-based Macs #4330

Closed
wants to merge 2 commits into from
Closed

DRAFT: Experimental build on Apple M1-based Macs #4330

wants to merge 2 commits into from

Conversation

cclauss
Copy link
Contributor

@cclauss cclauss commented Dec 25, 2020 •
edited
Loading

Related to #4386

Steps to perform a experimental build of Open Library on the new Apple M1-based Macs:

  1. Download & install the Docker M1 preview https://docs.docker.com/docker-for-mac/apple-m1
  2. Do get info on your Terminal app and make sure that Open using Rosetta is checked.
  3. git clone this repo
  4. git checkout this branch of the repo
  5. Run the following...
export COMPOSE_FILE="docker-compose.yml:docker-compose.override.yml"
docker build -t openlibrary/olbase:latest -f docker/Dockerfile.olbase .
make git
pushd infogami && git rebase master && popd
docker-compose down && PYENV_VERSION=3.9.1 docker-compose up -d && docker-compose logs -f --tail=10

# In a second terminal tab...
open http://localhost:8080
docker exec -it openlibrary_web_1 /bin/bash

# once inside the container...
make test

Technical

Notes:

  • These changes only work on Python 3, not on Python 2
    • pyenv Python 2.7.6 will NOT install on ARM64 even with Rosetta turned on.
    • pyenv Python 2.7.18 (the final release of legacy Python) will install but pip installing our requirements fails
import platform


def on_apple_silicon():
    return platform.system() == "Darwin" and platform.machine().startswith("arm")


print(f"{platform.system() = }")
print(f"{platform.machine() = }")
print(f"{on_apple_silicon() = }")

Testing

Screenshot

Problem 1: Pyenv Python 2.7.6 

 => ERROR [ 7/30] RUN pyenv update && pyenv install 2.7.6                                                                     45.1s
------
 > [ 7/30] RUN pyenv update && pyenv install 2.7.6:
#10 0.913 Updating /home/openlibrary/.pyenv...
#10 0.913 From https://github.com/pyenv/pyenv
#10 1.597  * branch            master     -> FETCH_HEAD
#10 1.618 Already up-to-date.
#10 1.619 Updating /home/openlibrary/.pyenv/plugins/pyenv-doctor...
#10 1.619 From https://github.com/pyenv/pyenv-doctor
#10 2.182  * branch            master     -> FETCH_HEAD
#10 2.206 Already up-to-date.
#10 2.207 Updating /home/openlibrary/.pyenv/plugins/pyenv-installer...
#10 2.207 From https://github.com/pyenv/pyenv-installer
#10 2.745  * branch            master     -> FETCH_HEAD
#10 2.768 Already up-to-date.
#10 2.769 Updating /home/openlibrary/.pyenv/plugins/pyenv-update...
#10 2.769 From https://github.com/pyenv/pyenv-update
#10 3.323  * branch            master     -> FETCH_HEAD
#10 3.338 Already up-to-date.
#10 3.339 Updating /home/openlibrary/.pyenv/plugins/pyenv-virtualenv...
#10 3.339 From https://github.com/pyenv/pyenv-virtualenv
#10 3.910  * branch            master     -> FETCH_HEAD
#10 3.934 Already up-to-date.
#10 3.935 Updating /home/openlibrary/.pyenv/plugins/pyenv-which-ext...
#10 3.935 From https://github.com/pyenv/pyenv-which-ext
#10 4.488  * branch            master     -> FETCH_HEAD
#10 4.510 Already up-to-date.
#10 4.582 Downloading Python-2.7.6.tgz...
#10 4.582 -> https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz
#10 7.491 Installing Python-2.7.6...
#10 7.496 patching file ./Modules/readline.c
#10 7.496 patching file ./Lib/site.py
#10 7.496 patching file ./Lib/ssl.py
#10 7.496 Hunk #2 succeeded at 430 (offset -5 lines).
#10 7.497 patching file ./Modules/_ssl.c
#10 7.497 Hunk #2 succeeded at 307 (offset -1 lines).
#10 7.497 Hunk #3 succeeded at 1809 (offset -3 lines).
#10 7.498 patching file ./configure
#10 44.40
#10 44.40 BUILD FAILED (Ubuntu 16.04 using python-build 20180424)
#10 44.42
#10 44.42 Inspect or clean up the working tree at /tmp/python-build.20210119181921.156
#10 44.43 Results logged to /tmp/python-build.20210119181921.156.log
#10 44.43
#10 44.43 Last 10 log lines:
#10 44.43     self.build_extension(ext)
#10 44.43   File "./setup.py", line 285, in build_extension
#10 44.43     if not self.configure_ctypes(ext):
#10 44.43   File "./setup.py", line 2033, in configure_ctypes
#10 44.43     exec f in fficonfig
#10 44.43   File "build/temp.linux-aarch64-2.7/libffi/fficonfig.py", line 32, in <module>
#10 44.43     ffi_sources += ffi_platforms['AARCH64']
#10 44.43 KeyError: 'AARCH64'
#10 44.43 Makefile:475: recipe for target 'sharedmods' failed
#10 44.43 make: *** [sharedmods] Error 1
------
executor failed running [/bin/sh -c pyenv update && pyenv install 2.7.6]: exit code: 1

Solution: Upgrade to Python 2.7.18

Problem 2: python2 -m pip install Pillow==6.2.2 

> [22/30] RUN python2 -m pip install --disable-pip-version-check --default-timeout=100 -r requirements_common.txt:

[ ... ]

#26 7.634 Collecting Pillow==6.2.2
#26 7.660   Downloading https://files.pythonhosted.org/packages/b3/d0/a20d8440b71adfbf133452d4f6e0fe80de2df7c2578c9b498fb812083383/Pillow-6.2.2.tar.gz (37.8MB)

[ ... ]

#26 80.95     writing manifest file 'src/Pillow.egg-info/SOURCES.txt'
#26 80.95     running build_ext
#26 80.95
#26 80.95
#26 80.95     The headers or library files could not be found for jpeg,
#26 80.95     a required dependency when compiling Pillow from source.
#26 80.95
#26 80.95     Please see the install instructions at:
#26 80.95        https://pillow.readthedocs.io/en/latest/installation.html
#26 80.95
#26 80.95     Traceback (most recent call last):
#26 80.95       File "<string>", line 1, in <module>
#26 80.95       File "/tmp/pip-install-0PNckY/Pillow/setup.py", line 903, in <module>
#26 80.95         raise RequiredDependencyException(msg)
#26 80.95     __main__.RequiredDependencyException:
#26 80.95
#26 80.95     The headers or library files could not be found for jpeg,
#26 80.95     a required dependency when compiling Pillow from source.
#26 80.95
#26 80.95     Please see the install instructions at:
#26 80.95        https://pillow.readthedocs.io/en/latest/installation.html
#26 80.95
#26 80.95
#26 80.95     ----------------------------------------
#26 81.02 ERROR: Command errored out with exit status 1: /home/openlibrary/.pyenv/versions/2.7.18/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0PNckY/Pillow/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0PNckY/Pillow/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-NhHPX1/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
------
executor failed running [/bin/sh -c python2 -m pip install --disable-pip-version-check --default-timeout=100 -r requirements_common.txt]: exit code: 1

Solution: Comment out the `python2 -m pip install` command... More details at python-pillow/Pillow#5093
  • NOTE:
    • We are installing Pillow v6.2.2 which is the last version to support legacy Python.
    • Updating to the current pip + wheel does not make any improvement.

Problem 3: `python ./scripts/i18n-messages compile` fails 

 > [28/29] RUN make:

[ ... ]

#32 40.81 python ./scripts/i18n-messages compile
#32 40.84 pyenv: version `2.7.6' is not installed (set by /openlibrary/.python-version)
#32 40.84 Makefile:46: recipe for target 'i18n' failed
#32 40.84 make: *** [i18n] Error 1
------
executor failed running [/bin/sh -c make]: exit code: 2

Solution: Change 2.7.6 --> 2.7.18 in `.python-version`

Problem 4: `docker exec -it openlibrary_web_1 /bin/bash` then `make test-py` raises `RuntimeError`

RuntimeError: dictionary changed size during iteration

openlibrary/tests/core/test_cache.py ....F.......                                                                                          [ 65%]

[ ... ]

==================================================================== FAILURES ====================================================================
____________________________________________________ Test_memcache_memoize.test_update_async _____________________________________________________

self = <core.test_cache.Test_memcache_memoize object at 0x40115ec5b0>

    def test_update_async(self):
        m = self.square_memoize()

        m.update_async(20)
>       m.join_threads()

openlibrary/tests/core/test_cache.py:57:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <openlibrary.core.cache.memcache_memoize object at 0x40115ec190>

    def join_threads(self):
        """Waits for all active threads to finish.

        Used only in testing.
        """
>       for name, thread in self.active_threads.items():
E       RuntimeError: dictionary changed size during iteration

openlibrary/core/cache.py:160: RuntimeError
================================================================ warnings summary ================================================================

Solution: Add `list()` in `for name, thread in list(self.active_threads.items()):

Stakeholders

@cclauss cclauss added device-support Theme: Testing Involves work related to testing infrastructure, development, and ops. [managed] Theme: Development Issues related to the developer experience and the dev environment. [managed] labels Dec 25, 2020
@cclauss cclauss self-assigned this Dec 25, 2020
@cclauss cclauss mentioned this pull request Jan 7, 2021
Closed
@cclauss cclauss marked this pull request as ready for review January 19, 2021 11:40
cclauss added a commit that referenced this pull request Jan 21, 2021
@cclauss
cache.py: Avoid dict changed size during iteration
b184513 
Subset of #4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See #4330 Screenshots, problem 4 for details.
@cclauss cclauss mentioned this pull request Jan 21, 2021
Merged
@cclauss
Copy link
Contributor Author

cclauss commented Jan 21, 2021 •
edited
Loading

The Snapshots section above might continue to be of interest.

Closing because these changes should land in #4386 and #4455.

@cclauss cclauss closed this Jan 21, 2021
@cclauss cclauss deleted the Apple-M1 branch January 21, 2021 14:04
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
87e7640 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
cd3aa90 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
753cce2 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
a1674f2 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
baae502 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 15, 2021
@cclauss @Sabreen-Parveen
cache.py: Avoid dict changed size during iteration
f31ecdb 
Subset of internetarchive#4330

This code is ___only used in testing___ but it raises a RuntimeError on Apple Silicon M1 Macs.

See internetarchive#4330 Screenshots, problem 4 for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hornc hornc Awaiting requested review from hornc

@mekarpeles mekarpeles Awaiting requested review from mekarpeles

@cdrini cdrini Awaiting requested review from cdrini

Assignees

@cclauss cclauss

Labels
Theme: Development Issues related to the developer experience and the dev environment. [managed] Theme: Testing Involves work related to testing infrastructure, development, and ops. [managed]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cclauss