-
Notifications
You must be signed in to change notification settings - Fork 548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bring ARM support #2130
Comments
@ilgooz I've been doing some experimentation on this one as I was considering to apply for the bounty on it. I've got a good overview of the work, but there is one show-stopper right now, which is the protoc-gen-dart. Nodetime was possible to cross-compile with some libraries installed on Linux, but dart does not have this capability for their executable files (https://dart.dev/tools/dart-compile#exe see the part of about cross-compilation). If Github Actions had ARM versions of mac we might be able to get it to work, but it doesn't. There is a chance that we could build for linux arm64 with qemu, but I haven't tried that yet. What are you thinking here as a solution? protoc-gen-dart doesn't seem to have any prebuilt binaries unfortunately. I could of course build them manually on each architecture by spinning up some VM's here and there, but would violate If you have any Mac M1 GitHub Runners laying around (or willing to create them on scaleway or something) we could use that too of course, but would require a lot more infra setup that I cannot do from this end. An alternative approach would be to build with a kernel build, but that would make dart a requirement for running the generation (since it doesn't output a full self-contained executable). However, if you are generating dart code anyway, it might not be too crazy to assume dart as a required dependency? Edit: After testing more I also realized something was wrong nodetime after upgrading pkg to the newest version. After a lot of digging I found out the problem was this issue: vercel/pkg#1130. The problem has since been fixed but we need a new release from pkg (I have verified this by pkg directly from their main and everything works again). |
Another question: is it desirable to update more libraries (in the gen-nodetime package.json) at the same time here? There is also a new LTS version of node (currently used 14, LTS now 16). I also have a question to understand a bit more context: As part of codegen we also run typescript compilation (tsc from nodetime), but I am a little confused why this is part of codegen as it seems to me to be more of a build step that could be included in the generated vue project instead? |
This is a so valuable investigation! I really appreciate that. About building the Dart plugin for ARM; would it solve if we create a Github Runner from an ARM machine? Do we need a Linux ARM or a Mac ARM or both? We can create machines and add to Starport repo as Github Runners. And we can create some more machines and share the SSH keys with you for your CI testing purposes. Please let me know how do you want to proceed on this, I credit your decision. Which solution we go with, it's better to not require our users to install any dependencies other than the Go installation itself.
This sounds good, if that makes sense we can build it on CI from the source for the time being.
That would be perfect.
JS transpilation planned to be dropped. @marinhoarthur is redesigning our VueJS architecture and I think he was aligned for the same outcome. If you're interested, we can also remove it now. |
@bjaanes you have been assigned to this issue. Thank you for your bounty participation! |
Great! I am all over this (i.e. I confirm my participation). I am finishing up some testing with the hosted GitHub Runners and qemu emulation now to see how far I'll get with those for building the arm binaries without any self-run ones. It is likely not to be excellent for native arm64 for OSX. I have some thoughts on that too, but |
Gjermund, that's perfect! I requested two Mac M1s just in case, to be attached to this repo as a GH runner and one for your testing purposes. They expected to be ready within 48hours. |
I think that is a good idea, using qemu for this is likely to require some hacky OSX images with a bit "shady" licencing.
It looks like it works to build the arm binaries for linux with qemu, but it is a bit slow (7+ minutes). Having separate runners for linux arm64 would also simplify the github tasks a bit to run in a matrix (which is not really great right now).
Great, that should work. I am also reaching to the team there to get an ETA on a new stable release. Looking at their release history it shouldn't be too far into the future.
I will certainly look into it and see if it works well. I think it would make a lot of sense. If there is any context or additional info I should know about just let me know. One last question for right now is about the workflows. The issue mentions "Create a single CI workflow". This is certainly doable, but I the trigger might differ for each part because: Got any thoughts on that? |
Do we specifically need Linux? We're already preparing M1 machines. Which one do you think is more suitable for this? AFAIK, while building ARM binary for MAC through vercel/pkg, it requires an M1 machine. But still we can just benefit from Rosetta for this by using AMD64 binaries and AMD64 Starport CLI. But again AFAIK, if Starport compiled to M1, there was a problem running binaries as child process(e.g. running nodetime through Starport CLI) if these binaries are AMD64.
This is recently removed: #2163
AFAIK We can have multiple CI workflows if that what's needed. |
I can test if the output from the mac will run on linux as well, but they might require different machines, at least for the dart compilation.
For the vercel/pkg compilation it actually works on linux (I've tested in on a cloud-based M1), but it requires some slightly hacky setup to be able to sign the binary. So better to run that on the mac anyway.
Nice, that is excellent! Will look more into that.
Oh, interesting. I will try this out a bit more then and see what I come up with. Thanks for all the answers. I am making good progress and have already created binaries that I've run on M1 and Linux arm64. Should this be compiled to more arm versions than just arm64/aarch64? |
I think arm64 would be enough for now? |
Sounds good. Just FYI: There is a chance I might get somewhat delayed on this (relative to the 10 day deadline) because I got Covid and I am not very productive right now. I'll keep you posted, and also let me know when the M1 become available so I can do some testing with that. |
Sorry to hear that, get well. 🙏 No worries about the deadline. |
@ilgooz Back in action. I've set up the provided M1 machine as a runner and got it to build both amd64 and arm64 for darwin. The dart build unfortunately does not work on linux arm64. I think the only way to get that done is to also get a runner for Linux arm64/aarch64. Just for documentation purposes: The GitHub runner software is not quite yet ready for M1 machines, but there are some workarounds for this until it is released: actions/runner#805 (comment) |
Super cool! We're preparing ARM Linux machines. 👍 |
@ilgooz draft PR up now. There are some things to be sorted out (github runners and practical merge strategy) I left it in merge until we sort those out, but the PR is ready for review otherwise. |
Starport CLI should be able to run on ARM x64 Linux/Darwin platforms.
CLI is written in Go, compiling it to these platforms is pretty straightforward by using the
go build
tool.But CLI has:
binaries embed inside.
These binaries copied to the filesystem to execute their commands when running some certain Starport CLI commands.
For example,
nodetime
is being used:starport g vuex
confio/ts-relayer
as a backend tostarport relayer
command.See related scripts that creates these binaries here (currently, protoc binaries are added manually).
For ARM support, we should also have ARM darwin & linux binaries of these programs embed into Starport CLI.
Make sure that:
nodetime
bundled by usingvercel/pkg
. Use the latest version ofvercel/pkg
to create binaries fornodetime
.To test things make sure that:
starport relayer
commands are running OK.starport g vuex --proto-all-modules
in the root of a blockchain app, it has to be running OK. You should observe changes in thevue/src/store/generated
dir of the blockchain.starport g dart
should be running OK.The text was updated successfully, but these errors were encountered: