-
-
Notifications
You must be signed in to change notification settings - Fork 307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Update build GRASS as macOS application #457
Comments
Agreed. I've been using Apple Disk Utility now for over a year. AFAICT, other packaging approaches require Xcode. Also, we need to be able to sign the code to make it easier for a binary to be opened. OSGEO has signing authority, but I have not been able to figure out how to apply it. Online descriptions of how to do this without moving the entire build to Xcode are pretty opaque and no one on the GRASS team (or OSGEO) responded to my repeated requests for advice last year. Perhaps you have suggestions about how to move forward. |
I don't want to discuss advantages of one over another here, but I think it is important to mention here the packaging efforts for Homebrew and conda. Here is PR for recipe for conda which fails: And here is installation of Homebrew formula which fails too: https://github.com/GRASS-GIS/grass-gis-experimental-ci/tree/homebrew-osgeo4mac I discuss this a little more here: |
If I'm not mistaken, QGIS uses Homebrew for the packaging of the mac app. I could imagine taking a similar approach. It would be really good, if we in the end could create a more or less automatic build and packaging workflow (with code signing). Personally, I have no experience with Homebrew (as I use MacPorts), but that can of course change... An important step towards this is fixing the build of a working GRASS.app (with |
Thanks @neteler for linking the threads. At this moment GRASS is fully working in macOS using OSGeo4Mac tap for Homebrew. The binaries are made available to the community, but one can also locally built from source based on the same formula. However, it’s not a standalone app. AFAIK as for now QGIS app have been based on Homebrew, although it’s probably going to change due to dependencies issues. @PeterPetrik is doing a great effort in this way and could probably be a reference for it (https://github.com/qgis/QGIS-Mac-Packager). |
I've been building this with Anaconda. This has largely been successful until recently. I'm using it to build self-contained apps, which most Mac users expect to have. However, this could also be transformed into an Anaconda recipe for those who want to compile themselves. Eric Hutton of CSDMS developed an initial recipe for early Python 2 versions that could be updated. |
Just to be clear, neither conda nor Homebrew work for me in the macOS virtual machine on GitHub Actions (tested also on Travis in case of Homebrew) and I don't know what to do next (latest email from Rainer M Krug on grass-dev). Please, let me know if you have any suggestions (grass-dev, issues, PR). Homebrew install test (fork here): https://github.com/GRASS-GIS/grass-gis-experimental-ci/runs/487969965?check_suite_focus=true conda recipe test (based on Eric Hutton's work, fork updated version here): https://github.com/wenzeslaus/grass-recipe/runs/486679988?check_suite_focus=true |
@wenzeslaus you are right, I can reproduce it with the code from the email and the latest Homebrew release (with Proj 7.0.0). This should be a new issue though, since the version using Proj 5.2.0 was running fine except for the GUI (not sure about Proj 6.3.1 that was released in february). Oddly, now just executing |
AFAIK, GRASS GIS does not yet support PROJ 7.x, see issue #395 |
According to build log (line 120) python 3.7.6 is used. If this is the case, it is doomed to fail because of changes in that version for ctypes. 3.7.7 should work (or 3.8.2), with latest GRASS 7.8 git-branch version (7a9ce5d is a critical fix) or master. |
FYI, compiling and running on mac (10.14) from terminal is working fine for me with following dependencies:
|
Thanks for this. I was able to compile 7.9 last Friday by downgrading to Python 3.7.4. If I can get it compile consistently, I will happily share my protocol for building a Mac app. The digitizer works for me at last!! Thanks so much to you and Anna for this and other fixes. I will try to also compile 7.8 dev today. |
One annoying problem I am having I could use some advice from anyone on is about Git. I am not writing or modifying code at the moment, just trying to catch up compiling binaries. I would like to regularly update my local repository from the main GRASS site but do not want to push anything out. I do need to make changes to a few files before I compile (Eric Hutton's patches a bit modified), but at the moment want to keep them entirely local. But I can't seem to find anything in Git that works like the old svn up. Ideally, I'd like to update everything except my patches. But if I can't do that, I have no problem redoing the patches before I compile. I've tried to fork my own version (with fetch upstream) and I've tried just to update my repository directly from the osgeo.org repository. I've tried rebase with a stash and even reset --hard. I simply cannot get Git to update if I've made any changes locally. Any suggestions are welcome. Thanks. |
@cmbarton try this:
Subsequent updates simply with |
Thanks. I'll try that.
Michael
… On Mar 31, 2020, at 10:35 AM, Markus Neteler ***@***.***> wrote:
@cmbarton <https://github.com/cmbarton> try this:
git clone curl https://github.com/OSGeo/grass.git
cd grass
git pull
Subsequent updates simply with git pull.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#457 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACENLLY2KJ4OC3CUNBONQDDRKISUJANCNFSM4LSZE6AA>.
|
Hi, it looks like that if LIB environment variable is set, the compilation or gpde fails with "missing symbols |
I think and hope that I may now finally have a consistently reproducible procedure for building self-contained apps again using condo. Here it is in brief schematic. I'm happy to elaborate if anyone is interested in the details.
Currently Anaconda is defaulting to Python 3.7.6, which is the only one that does not currently work with building and running GRASS. While I can create environments for different versions, the miniconda installer defaults to 3.7.6 when installing into the GRASS shell. I've tried various other version in my requirements file (for dependencies), but right now, the only Python version that installs, builds, and runs consistently is 3.6.10. So I'm using that. |
@cmbarton, I don't get past first point on macOS on GitHub Actions [code, workflows]. Suggestions, PRs welcome. If there are is a list of dependencies or local changes you are using, these need to go to OSGeo/grass or another repo. Unfortunately, the versions suggest by @nilason don't work for conda even when I loosen the requirements a bit. I'm getting a lot of decency conflicts basically for all the dependencies. (I got the @nilason, if you want to add MacPorts-based build to https://github.com/GRASS-GIS/grass-gis-experimental-ci/ that would be great. The idea is that once these are work reasonably well, we add them to this repo (OSGeo/grass), so we have the build testing in place. (Speaking about those which can build from the local source code which now not done in GRASS-GIS/grass-gis-experimental-ci since it is another repo and also because existing recipes/packages are used such as the one from homebrew-osgeo4mac.) |
I’ll put together a detailed set of procedures soon for all |
grassmac_supporting_files.zip Using Anaconda to Create a Binary Mac App for GRASSThese instructions are for creating a self-contained Mac app, including all dependencies, with GRASS 7.9 and Anaconda. You can modify them if you don't need some of the steps. Some of the steps you won't need to repeat (e.g., creating an anaconda environment) 1. Create a Mac app shell (I am uploading an example, modified from shells created by Eric Hutton).This can be modified serve for different versions of GRASS by altering the last line of the script that launches GRASS in the app shell. For example, modify the last line of ... ../GRASS-7.9.app/Contents/MacOS/Grass.sh to refer to grass78 or grass76 2. Create an anaconda environment for Python 3.6.10This may not be strictly necessary, but I am doing it make sure that my builds use Python 3 instead of Python 2.7, which is still what ships with the Mac. Anaconda defaults to Python 3.7.6, which will not compile GRASS properly. This makes it more complicated than needed, requiring some workarounds. I've tried this with multiple versions of Python 3 (anaconda has them all the way up to 3.8.2), but the highest version that seems to give consistently successful results is 3.6.10 So I create an anaconda environment for Python 3.6.10
3. Activate the anaconda Python 3.6.10 environment
4. Copy the Mac app shell to /Applications
5. Download the Python 3 miniconda installer and use it to install a minimal anaconda environment into the Mac app shell
6. Export the path to the GRASS Mac app shell so that subsequent operations use the environment of that shell
7. Install GRASS dependencies into Mac app shell, including Python 3.6.10This will downgrade Python from v. 3.7.6 installed by miniconda to the more useable 3.6.10. I use a requirements file for all the dependencies and will upload it along with the app shell
8. Create a symlink to Python so that pythonw can find itGrass launches Python through the pythonw script. But the script does not point to the correct location for Python (which is in fact in 2 different places). This seems to be a bug in Python < 3.7.4 or how anaconda installs it. It puts the python launch file in the wrong place. So a symlink is needed. Hopefully this will go away when we can use Python > 3.7.6, which seems to put python in the correct place, and the place that pythonw points to.
9. cd to the ./grass source folder and apply patches to configure and make filesEric Hutton worked out these patches and I modified one needed to use Python 3. Maybe they can be worked into the source code so that they are no longer needed. I will upload the patches along with the app shell and requirements file.
10. Configure, make, and installI use a configure script that sets flags and prefix. I will include this in my package of uploads
You should now be able to run and test the GRASS-7.9.app in your /Applications folder 11. Clean upThis removes unneeded anaconda package installer files and some other large files that are irrelevant for GRASS
12. Set up folder to make a binary package for distributionPut a copy of the new GRASS app and a symlink to the /Applications folder into a new folder that will become the distribution package
13. Use Disk Utility to make a distribution package
|
@wenzeslaus I am about to finish using workflows for macos CI builds for QGIS, see https://github.com/qgis/QGIS/pull/35407/files#diff-e58291fad8cbc1e6be9085e97679afdd |
I forgot to include my configure script last night. I just now tried to add it to my zip file or separately. For some reason, I could not add it to my comment. So it is here. |
@cmbarton I used your version requirements and I get these two errors (same as before):
The only differences I see, besides shorter list of dependencies, is missing |
I've messed with both of these in several ways and I'm not sure of the impacts. But I sometimes have trouble if I don't include them. So I do. I've all the SKDs back for several years. Originally, I was building against 10.9 or so to try and keep some backward compatibility (though I don't know if it matters or not). Because of all of the difficulty in compiling in the past 9 months, I've moved to just using the current one that comes with Xcode. I guess I'd put it in. The macosx-archs argument was originally to allow for dual 32/64 bit code. With the move to wxPython >2, I was able to drop that and just do 64 bit. Again, I am not sure it is needed or not. As to the proj/gdal error, it looks like it is not looking in the right place for proj. Did you
The configure script I use sets prefix to match what Python sees when pathed to the app. What does echo $PREFIX show? |
Thank you @cmbarton for suggestions. I used --with-macosx-sdk set to
It still fails with
I don't know what you mean by Mac app shell, but I specify my dependencies in meta.yaml and they are installed in the by conda build.
I'm not touching any paths. That's what conda is doing. So, I don't know how GRASS-7.9.app/Contents/Resources/bin plays into that if it has any meaning in conda at all. However, PATH seems to have the $PREFIX as a first entry:
Prefix is set by conda build like this:
Since that would be really inconvenient to see in the outputs, conda replaced that by |
OK. This is a different approach than what I'm doing. What I do is not directly translatable into building this within Anaconda. What might work is the following (maybe you are doing this).
This should create a version of GRASS that launches like the Linux one, but does so in the conda environment you created. You will have to activate that environment to run GRASS. e.g.,
|
That's pretty much what conda build is doing as far as I can tell. It creates environment, actives it and builds GRASS inside. From your Related to that, one question which comes to my mind is what channels you are using. I'm using conda-forge and csdms-stack because the Eric's grass-recipe was for csdms-stack:
|
@PeterPetrik, thanks I'm excited to see this. Please let me know when that's finished. If you have any suggestions how to adapt it for building GRASS in CI, please let me know (build of a standalone app, conda package, Homebrew formula are desired, but any build will do for starters for CI). I looked at the qgis/QGIS-Mac-Packager repo and it was not clear to me how to easily reuse it (without actually using the whole thing), but |
Can you please write me to my email peter.petrik@lutraconsulting.co.uk to discuss the topic, maybe even in the call? |
@wenzeslaus @PeterPetrik Replying to your query from last night. EnvironmentsAnaconda environments are a sort of virtual machine. You can create a specific configuration of anaconda installation and packages. As I suggested, even though Anaconda defaults to Python 2.7 on the current Mac, you can create environments with different versions of Python and other packages. What I was suggesting was to make an Anaconda environment specific to GRASS. Then it could have just the packages and versions it needs to run GRASS. No need to change the main Anaconda installation. To keep it minimal and light, you could do the following
After this, you could configure, build, and install GRASS into the environment
To exit the environment...
ChannelsI am only using the main conda channel and conda-forge. I am not using the csdms channel because I don't think Eric has updated it recently and don't want to inadvertently get unworkable versions of main packages or GRASS dependencies. |
Thanks @cmbarton, @PeterPetrik, email sent. Thanks. |
That's great @wenzeslaus. Let me know if I can help fill in any other details. |
Patches, dependencies, and steps according to @cmbarton in OSGeo/grass#457
I reproduced the configuration for a build according to @cmbarton as a workflow for GitHub Actions. There are some minor changes such as dependencies being in environment.yml file instead of requirements.txt due to how conda works in a non-interactive/script environment and with GitHub Actions. Let me know if you see any differences. The code is in conda-compile branch of GRASS-GIS/grass-gis-experimental-ci. You can open issues or PR against that repo if you want. The build fails in the
I could try to random things like removing the |
One difference is that I hard-coded a path to the SDK. That is not a solution for a generic conda build of course. But you might try that to see if it is the cause of the error. |
I also think this is related to sdk versions. Perhaps |
I'll also copy here what I sent to the dev list. I've got a cleaned up version of instructions for building a binary app using Anaconda. Any thoughts as to where the best place to put it? Wiki? Other? |
The initial intent (and still is) with reporting this issue is to enable --enable-macosx-app to build a mac app bundle. The comments/discussions have though taken a direction into more of packaging/dependency problematics, which is in itself not bad, but just a much complicated matter. I'll take the moment to inform followers of this "thread" of advances in building for mac (using conda) on CI (see successful CI build on https://github.com/nilason/grass-gis-experimental-ci/tree/conda-updates). I have also started work on a script for an all-in-one solution for building from source to creating dmg file with GRASS.app (also using anaconda). It it very reliable for me, but there are surely room for improvement, see https://github.com/nilason/grass-conda. |
Is this issue still valid, since we have been building for macOS in CI and have releases for macOS. Close if already resolved! |
It is unfortunately still valid, the CI build is a classic unix like build and not app bundle (see eg https://en.wikipedia.org/wiki/Bundle_(macOS)). The official release is created with https://github.com/nilason/grass-conda and the procedure is not incorporated in GRASS make files. |
Thanks for the answer! If someone wants to continue on this, but doesn't have access to a macOS to test afterwards, would you be willing to help? The steps in your repo, in their current state, takes approximately how much time to run? Is something based on this suitable to run on CI? Are they other blockers we should be aware of? |
I can test too. |
What type of machine do you have, is it intel or M1-M3, or you have access to both ? |
I have planned all along to add this to GRASS repo myself, but in a more appropriate way. I still have it high on the todo list.
Both the CI build and the bundle build are based on conda dependencies, but there is not much sense to build a bundle just for CI testing.
There is a potential issue for macOS 14 reported with https://lists.osgeo.org/pipermail/grass-dev/2023-September/096038.html, which may need some reworking on the start-up procedure. |
Is it ok with you if I start by including what exists already on your repo (and that builds the releases anyways), and we "build" (as on work) from that to have something better than nothing?
Since this is currently what is used to release the official release, I consider that it is a bigger risk to not follow the health of our "production" build. There's little benefit to
This fall they finally released macOS M1 runners, but they don't offer it free to all. They are part of their "larger" runners, that are pay per use. I kinda understand that they need to have a certain ROI for this, and it was probably really not profitable to acquire the hardware, and find a licensing agreement with Apple, as they usually only allow "renting" a whole system for a complete 24h. And the existing intel-based macOS runners still have a little life left in them ;) |
M2. I might be able to get an intel if needed. But those are now several years out of date. |
Just a note. The current separate conda build (from Nicklaus) I use for my releases is very fast and pretty much error free. |
I have a first draft started at echoix#15, to make sure everything is possible. (I added a job in the macOS workflow that simply clone the grass-conda repo for now, until I'm close enough to integrate directly the functionality). I'm a little stuck, but didn't search for hours neither yet. See this run, where I tried running the print versions script https://github.com/echoix/grass/actions/runs/7318245039/job/19934677740#step:7:1 |
Describe the bug
Building GRASS as a macOS application needs to be updated for Python 3 and long-gone
packagemaker
needs to be replaced.To Reproduce
Steps to reproduce the behavior:
Compiling with --enable-macosx-app using Python 3 fails.
System description (please complete the following information):
The text was updated successfully, but these errors were encountered: