Using pack manager on arm-based system

[harrrson]

I've tried to install newest pyocd version on armbian, but when trying to install it even with pip, every time installed version of pack manager is 0.1. Is it possible to install newest version of pack manager on arm-based system? sorry for newbie question, but i've tried to google it, without any results.

[harrrson]

Sorry for bothering, i found how to build it by myself. Problem solved.

[theotherjimmy]

Actually, I would like to look into this.

[theotherjimmy]

Dang, not super easy... pypa/manylinux#84

[theotherjimmy]

Also, the code in PEP 571 returns false for non-x86 platforms

[harrrson]

After some tries with compiling (needed to underclock orangepi because of overheating) finally i "finished" compiling a library. Unfortunately, after few hours (idk how long, i left it for night to compile) an error appears:

error: Could not compile `syn`.

Caused by:
  process didn't exit successfully: `rustc --crate-name syn /home/user/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/syn-0.15.34/src/lib.rs --color always --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C debuginfo=2 --cfg 'feature="clone-impls"' --cfg 'feature="default' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="proc-macro2"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=3b725b54ccd4db82 -C extra-filename=-3b725b54ccd4db82 --out-dir /home/user/cmsis-pack-manager/rust/target/release/deps -L dependency=/home/user/cmsis-pack-manager/ust/target/release/deps --extern proc_macro2=/home/user/cmsis-pack-manager/rust/target/release/deps/libproc_macro2-57d57f25912b7e6e.rlib --extern quote=/home/user/cmsis-pack-manager/rust/target/release/deps/libquote-b92e5076b49b28c2.rlib --extern unicode_xid=/home/user/cmsis-pack-manager/rust/target/release/deps/libunicode_xid-e437e5d14509c8d2.rlib --cap-lints allow --cfg syn_can_use_thread_id --cfg syn_can_use_associated_constants --cfg syn_can_call_macro_by_path` (signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: build failed

I hoped i can make it works, but now i know, i cant. Maybe someone, who have more knowledge than me can do it.

Edit#1: grammatical editing :)
Edit #2: Maybe important, maybe not, i used latest nightly version of Rust. I try to compile with version listed in one of commits, but i don't think it should help.

[40Grit]

@theotherjimmy I assume the effect of this would be not being able to install pyocd on beagle bone black, nor raspberry pi?

@Team-EmbeddedPlanet often uses BBB industrial as gdb servers. We are about to switch back to using pyOCD from jlinkgdb server.

[harrrson]

Update: When i downgrade Rust to version nightly from 2018-11-06, I've finally managed to "succesfully" build CPM. Problem is, every compilation gives CPM v0.1.1, not downloaded v0.2.6.

[harrrson]

Maybe it-s important, i don't completely restart compilation from beginning, but i retype same command '''python setup.py -v install''' i've used earlier.

[theotherjimmy]

@40Grit Thanks for the update on EmbeddedPlanet. I think it's becoming a good time to get some sort of pip-installable, arm thing available.

@harrrson That looks like OOM (out of memory), and restarting the build after the OS recollects the memory from the dead process can help. For the "v0.1.1" issue, setup.py detects the version of CPM from git tags. If you're missing tags or you're on the most recent master, then the default is 0.1.1. You can build version 0.2.7 by checking it out.

[harrrson]

@theotherjimmy that sounds logic. On the morning (7:30 pm atm of writing this post) i'd started the compilation with source code from latest release (tar.gz file), but unfornatunately, my orangepi stops responding to vnc remote access. When i retrieve access to it, i'll tell you if it works, or not.

[cederom]

We are also working on that (end-user build with pip) for FreeBSD here #97 :-)

[harrrson]

Sadly, when i restarted compilation of CPM, it starts to "overload" at exactly the same packages (one of them was syn). Because of that i can't connect remotely (again) to orangepi, and needed to restar it manually. Later i tried to compile library on my computer, i had installed aarch64-unknown-linux-gnu version of Rust, but after starting compilation with setup.py script, /home/user/.rustup/toolchains/nightly-aarch64-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory error appears. I've tried everything i can google and understand, but i feel i am reaching an end of my knowledge and googling skill. Maybe someone with more knowledge than me can solve it. At the moment, i am forced to use pyocd 0.19.0 on my orangepi.

[cederom]

Hey @harrrson please follow the Build on FreeBSD thread provided above (#97).. when that works it should work on any other Open-Source platform :-)

Except for some Python PIP dependencies, you will also need to have CARGO and RUST installed on your system.

libdl comes from running binaries that were built/meant for some other specific system configuration, a known libc-glibc mystery for ages unsolved, this is why I don't really like and use Linux anymore - its self-incompatible, quick-win-ditry-mess, and not very well organized.. and worsee of all it does not really care what burden of additional work you will have to maintain your product/code after two months or two years.. anyway you can:

• try linking ln -s libdl.so libdl.so.2 and this may work .
• try installing cargo using your system package manager (apt-get?).
• search for different binaries.
• build the cargo yourself (FreeBSD has Ports that automates that for each package).

You may want to give a try to FreeBSD port to ARM from https://www.freebsd.org/where.html? :-)

[theotherjimmy]

@cederom I think the issue is that rustc consumes all the resources on the rpi/orangepi/other single board computer, so the dependencies are already resolved and using freebsd may not be a solution to the problem. Those are some good instruction on getting it rustc and cargo working on many systems.

@harrrson I'll look into building an arm binary wheel with rust cross-compilation from a large x86_64 box Tuesday or Wednesday.

[cederom]

Sometimes I just wonder.. how did we make it through the 80s with 8-bit machines and 64K of RAM :-) :-) :-)

I missed the instructions link, sorry :-) Looks like something sends SIGKILL to the build process, maybe some sort of timeout/watchdog from build system configuration that can be tuned? Looks like cross-compilation from PC to ARM could be aletrnative solution? I am sure that if we make CPM build on FreeBSD it will work everywhere just with pip (+system dependencies) :-)

[theotherjimmy]

@cederom Agreed, we're in a crazy world now. I'm thinking that the Linux kernel is sending that SIGKILL, due to the fact that the RPI ran out of memory, and that's why it stops at a specific point: when we encounter a particularly big crate.

Yes, we could probably cross-compile and skip this problem on ARM, or other small computers.

[cederom]

I have managed to build CPM and pyOCD on Armbian Linux / Banana Pi M1 using pip install pyocd after CPM 0.2.8 sources were released to PyPI :-)

On Linux some additional development dependencies are required in order to build CPM: rustc, cargo, python3-venv, python3-dev, libffi-dev.. maybe something more.. and it takes more than hour to build.. but it works !! :-)

https://asciinema.org/a/MeYMuwJkUxAFhMG5WwVjJ5aZP

Here is build on FreeBSD-12.0 AMD64:

Here is full reference: pyocd/pyOCD#699

[theotherjimmy]

Ooph, an hour to build is pretty bad. I hope to have some form of cross compile at some point.

[IzidorM]

Hi, is there any solution to this problem available at current time? I need support for STM32F303ZETx which is not supported by the pyocd by default, so on my pc I am using the cmsis packet for it. Now I am trying to port my scripts for automatic flashing to rpi and got stuck at this problem ...

[theotherjimmy]

Hi @IzidorM, I don't have a solution to the OOM in syn just yet. Perhaps you could try building with a cross rustc and copying the built artifacts over?

[flit]

@IzidorM You can manually download the STM32F303 pack and use the --pack argument to pyOCD to point to it. See manual pack usage docs.

[IzidorM]

Thanks both for your help. I went with the flit recommendation and updated my script with:
allProbes = ConnectHelper.get_all_connected_probes()
print("Number of probes found: ", len(allProbes))
self.probe = allProbes[0]
del(allProbes)

self.session = Session(self.probe, options={"pack":"/home/pi/flasher/Keil.STM32F3xx_DFP.2.1.0.pack"}, target_override="STM32F303CCTx")

[cederom]

Hmm, I think I should have a binary built somewhere for ARM Pi, will share during weekend, maybe that helps :-)

[theotherjimmy]

That sounds great. I think I should be looking into the aarch64 travis instances.

[cederom]

Not sure what specific files are required so I have packed all my virtualenv. python3.7 rpi. hope that helps :-)

https://drive.google.com/drive/folders/1eknI93IztWi5Eoarcl145HfnDilVBucn

[towen]

Still an issue 2 years later? :(

Trying to use west flash (Zephyr RTOS) with STM32G474 target from a Raspberry Pi host.
The first problem is, the version of python3-pyocd in raspbian doesn't support stm32g474 devices. No problem: pip3 install --upgrade pyocd
Then, I hit the issue that led me to this thread:

Building wheels for collected packages: cmsis-pack-manager
  Building wheel for cmsis-pack-manager (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /home/pi/.local/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpqrlo2fzh
       cwd: /tmp/pip-install-zfo99ax4/cmsis-pack-manager_82e48ba29c7a4711af6b3ae0ab6bbd26
  Complete output (11 lines):
  running bdist_wheel
  running build
  running build_py
  creating build/lib
  creating build/lib/cmsis_pack_manager
  copying cmsis_pack_manager/_native__ffi.py -> build/lib/cmsis_pack_manager
  copying cmsis_pack_manager/_native.py -> build/lib/cmsis_pack_manager
  copying cmsis_pack_manager/__init__.py -> build/lib/cmsis_pack_manager
  copying cmsis_pack_manager/_version.py -> build/lib/cmsis_pack_manager
  copying cmsis_pack_manager/pack_manager.py -> build/lib/cmsis_pack_manager
  error: [Errno 2] No such file or directory: 'cargo': 'cargo'
  ----------------------------------------
  ERROR: Failed building wheel for cmsis-pack-manager
Failed to build cmsis-pack-manager
ERROR: Could not build wheels for cmsis-pack-manager which use PEP 517 and cannot be installed directly

I have tried downloading the pack file and passing --pack as suggested in this thread, but I get:

$ west debug --pack=/home/pi/Keil.STM32G4xx_DFP.1.4.0.pack 
-- west debug: rebuilding
ninja: no work to do.
-- west debug: using runner pyocd
FATAL ERROR: runner pyocd received unknown arguments: ['--pack=/home/pi/Keil.STM32G4xx_DFP.1.4.0.pack']

[cederom]

Hey @towen you seems to have missing dependency cargo is not installed on your system so you cannot perform build. Thake a loot at setup.cfg to see what are the dependencies. Please install all dependencies and let us know if that helps :-)

@flit is it possible to create ARM Linux and FreeBSD CI build hosts so the binary packages are distributed directly with pip? Will such binary work on a system with no cargo installed anyway?

@flit usually there was the requirements.txt file for easy dependencies setup. Is it possible to easily create this requirements file from setup.cfg ? :-) Having that requirements.txt may simplifi dependencies setup :-)

[towen]

Thanks @cederom - I worked around this in the end with pyocd pack install stm32g474 which successfully added the pack file and worked despite the old version of pyocd.
Thanks for such an excellent and robust tool! :)

Also fyi, I installed cargo and tried to upgrade pyocd again via pip. It got much further but still failed:

...
     Compiling try-lock v0.2.3
     Compiling crc32fast v1.2.1
     Compiling rustc-demangle v0.1.21
     Compiling serde_json v1.0.67
  error[E0658]: use of unstable library feature 'matches_macro'
     --> /home/pi/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rustc-demangle-0.1.21/src/v0.rs:700:20
      |
  700 |                 if matches!((quote, c), ('\'', '"') | ('"', '\'')) {
      |                    ^^^^^^^
      |
      = note: for more information, see https://github.com/rust-lang/rust/issues/65721
  
  error: aborting due to previous error
  
  For more information about this error, try `rustc --explain E0658`.
  error: could not compile `rustc-demangle`.
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  ----------------------------------------
  ERROR: Failed building wheel for cmsis-pack-manager

[cederom]

This looks like a problem with rust compiler this time. CPM is written in this marvelously efficient language that changes more often than Linux Kernel API or JavaScript packages (laugh). Make sure you have latest version of rust installed. On my FreeBSD box I have 1.54.0. Latest release is 1.55.0. Probably you have too old rustc compiler. Please verify and let us know :-)

Also let us know how much RAM do you have on the build ARM machine. Not enough RAM with no Swap can also result in build errors..

[towen]

This is a Raspberry Pi with 4GB RAM, 70MB of which is 'used' (it's not running X11 etc) and it has 100MB of swap, apparently.
rustc version is 1.41.1 (raspbian buster) so it is probably that.

I have now upgraded to raspbian bullseye, and my rustc is now 1.48.0 however when I try pip3 install --upgrade pyocd again, this time it downloads a prebuilt wheel for cmsis-pack-manager, so i'll never know if it builds correctly.

Thanks for your help anyway :)

[flit]

Hi all!

For AArch64, as of a few days ago, we released a new version 0.3.0 of cmsis-pack-manager (CPM) that has binary wheels on PyPI for Linux and includes AArch64, as well as pyocd version 0.32.0 that requires this version of CPM.

For Raspberry Pi, the Bullseye prerelease of Raspberry Pi OS (which is 32-bit, except for a beta 64-bit release) with Python 3.9 should install the cmsis-pack-manager binary wheel. Unfortunately the version of Rust available in Buster is too old to work, and the piwheels folks won't install anything but the Rust Debian package ☹️ (probably reasonable, but annoying in this case).

If you aren't using RPi OS Bullseye, you need a modern Rust toolchain installed, preferably via rustup.

I will also add the CPM wheel for Python 3.7 (RPi OS Buster) to the CPM release on GitHub. This will have to be manually installed though, prior to installing pyocd.

(You can see the piwheels FAQ, as well as their blog, for some more answers about why Linux armv6/7 wheels shouldn't be used on Raspberry Pi and thus why piwheels.org was created.)

I'm testing everything myself as I get time this week, and will try to get the readme and docs updated shortly with the how-to for Raspberry Pi.

It wouldn't hurt if you felt like commenting on the CPM issue for piwheels about uploading the wheel for Buster 😉.

[cederom]

@towen Swap is recommended to have 2 * RAM size.. 1 * RAM at least. This is because when OS runs out of RAM it can still operate. You may try 0.5 * RAM size. The bigger the better because when you want to do some development you may come into a situation where large and memory hungry stuff jumps in like now with rust. You may not need it for "using" the software but for "building" the software RAM can speed things up a bit (CPU is most important here) or return error when compiler / OS runs out of memory. I can tell because on FreeBSD almost no one provides packages (even for pip) so I have to build most of the stuff on my own :-)

Thanks @flit :-) Can you drop me a hint on the CI build so I could setup FreeBSD host for CPM and pyPCD ? This way we could provide out of the box pyOCD and CPM packages to pip :-)

[mathias-arm]

@cederom we are using GitHub Actions (https://github.com/pyocd/cmsis-pack-manager/tree/main/.github/workflows), that does not support FreeBSD. There is this project https://github.com/marketplace/actions/freebsd-vm that you could possibly use.