feat: make images arm64 compatible

[revant]

• Images are made compatible
• No automated builds yet. User will need to build on their own.
• Reason for no automated builds is: pandas takes 2+ hrs to build on arm64. Travis has job running limit for 50 min.
• DOCKER_REGISTRY_PREFIX can be used as build-arg by forks for automated builds.

Github actions has more timeout limit to run jobs. Project can be moved there for automated builds?
Anyway 2+ hrs to build something (2+ x3 for v13,v12,v11) is not sustainable?

[revant]

@Thunderbottom

• I installed python3-pandas successfully
• I init the env using --system-site-packages and pandas was available in env.

Only problem now is, removing pandas for arm64 and using this package instead, it'll messup with version from requirements.txt

[Thunderbottom]

So the problem is that the pandas version currently being used inside ERPNext is fairly old, and back then they did not have any pre-built wheels. The newer versions are prebuilt and the builds are available on PyPi.

If somehow we could get the pandas version updated on the upstream ERPNext, then we already have a solution for this problem.

[revant]

pandas>=1.0.5 : it needs latest pandas as per requirements

https://github.com/frappe/erpnext/blob/develop/requirements.txt#L5

It should get latest?

[Thunderbottom]

Oh, this must have been updated very recently so I might've missed it. And yes, it should fetch the latest version available, then. There's an issue regarding the v1.2.0 release wheels for ARM: pandas-dev/pandas#37784 (comment)

Maybe we can build the wheel locally, add to this repository, and copy/setup inside the docker image until this issue is sorted out?

[revant]

Any idea how to do that?

I tried, pip install *.whl, pip install -I --no-binary Pillow,psycopg2-binary. It still keeps on building

I tried to do something like this:

RUN python -m venv env \
    && . env/bin/activate \
    && cd apps \
    && git clone --depth 1 -o upstream https://github.com/frappe/frappe --branch ${GIT_BRANCH} \
    # Detect arch to install python wheel
    && if [ `uname -m` = 'aarch64' ]; then \
        pip3 install /opt/wheels/*.whl; \
        export EXTRA_ARGS="-I --no-binary Pillow,psycopg2-binary"; \
    fi \
    && pip3 install --no-cache-dir -e /home/frappe/frappe-bench/apps/frappe ${EXTRA_ARGS}

wheels.zip

[Thunderbottom]

doesn't --no-binary mean that it would build from scratch?

[revant]

I don't know what it means, no download? no compile? don't use downloadable bins?

https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-no-binary

edit: I tried without --no-binary Pillow,psycopg2-binary, still it starts building.

[Thunderbottom]

--no-binary means that it will build the packages from scratch and not use the binaries. You don't want to use that.

[revant]

You don't want to use that.

edit: I tried without --no-binary Pillow,psycopg2-binary, still it starts building.

I tried removing it. It still starts to build.

command to test builds

docker buildx create --use
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx inspect --bootstrap

# from repo root
export DOCKER_REGISTRY_PREFIX=dockerusername

docker buildx build \
  --platform linux/arm64 \
  --push \
  -t ${DOCKER_REGISTRY_PREFIX}/frappe-worker:edge \
  -t ${DOCKER_REGISTRY_PREFIX}/frappe-worker:develop \
  -t ${DOCKER_REGISTRY_PREFIX}/frappe-worker:latest \
  -f build/frappe-worker/Dockerfile .

try copying the whl from wheels.zip attached in previous comment.
you can even try to exec into a slim-buster container and try running commands step by step.

[revant]

@Thunderbottom

We can merge this?

• I'll add wiki page for how to build arm64 images. (no docs because things are not automated like amd64 images)
• when we sort out wheels we'll auto build and make it official.
• For arm64 enthusiasts, they can clone the same repo, build and push arm64 images on any registry for use.

Changes are

• use official python, node, debian-buster images
• use ${DOCKER_REGISTRY_PREFIX} as environment variable. Easy for anyone to build and push referencing their registry
• gcc, g++ added in frappe images, for building wheels on arm

If we merge this i'll propagate all changes develop -> master. The v13beta10 released will get built.

[Thunderbottom]

Can be merged. I am still unsure why it builds, but yeah, good to go!