Request: Please make ProtonAOSP manifest more suitable for automated builds

[AnonymousProxy987]

To preface, I am not trying to use this issue as a means to complain, nor to degrade ProtonAOSP. I think this is a very exciting project, and I only seek to improve the build process for other users, so that they may benefit. I am not trying to be negative here.

At the moment, the current manifest is not very suitable for automated builds, or custom builds in general. My goal is to set up my own private build server for ProtonAOSP, that will automatically produce builds once per month, signed with my own keys and with my own private additions. However, there are obstacles in the way of this goal:

(I am not known for my conciseness, TL;DR below)

adevtool

While adevtool is inherently required to build ProtonAOSP for its officially supported devices, adevtool is not included in your manifest. I could, of course, clone it myself, but then there is the issue of the "state file" system; state files for adevtool+ProtonAOSP are not posted publicly, meaning it is up to the user to generate them. However, to generate a single state file requires one to compile an entire ROM, which takes several hours. I could reuse this state file, but as you warn, it will need to be re-generated after major Pixel feature drops. This makes it rather cumbersome to set up an automatic build system that simply pulls your manifest and "gets to building." The state file will need to be regenerated eventually, so some amount of manual intervention will be required at some point. And, if a Pixel feature drop arrives, it would become necessary to compile an entire "dummy" ROM, which would take several hours, for its state file alone. Personally, I do not think 3 hours to produce a .json file is an efficient use of build time.

In other ROM projects, it is customary to include everything needed to build the ROM for its supported devices, right there in the build tree. For ProtonAOSP, this would mean including in the manifest all of the state files you use to create your officially supported modules, a copy of adevtool, and a script to generate the correct module based on $DEVICE and the correct Google build number (more on that below).

An ideal alternative to this would be to simply have users clone your officially supported modules directly, but the size limit makes this unusable for recent devices, thus the need for automatically-reproducible vendor modules. They are also not tagged by ProtonAOSP version or Google build numbers, meaning using them would not result in reproducible builds. Speaking of...

Reproducible Builds

This is mostly an adevtool issue, but I thought it was worth bringing to light in its own section.

Ideally, I would like to produce my own builds that are more or less identical to ProtonAOSP's public builds. However, since there is no published list of corresponding versions to Google build numbers, I can only guess which versions were used. For example, I can guess that ProtonAOSP v12.2.3 uses SQ1D.220205.004, but I cannot be sure. As using the "latest" version is the only current way to automate this, there is a chance I create a build "mismatched" to the Google build number used officially for that ProtonAOSP version.

I believe this can be solved by providing an environment variable in the adevtool script I proposed above - when I pull a manifest tagged with a specific version of ProtonAOSP (i.e. refs/tags/12.2.3), an included script somewhere in tools automatically chooses the correct Google factory image with which to create the vendor module, which would result in a 1:1 reproducible build to the public builds. If you used SQ1D.220205.003, then so should I.

Apart from Google factory image versions, adevtool seems to produce different state files depending on whether or not you specify WITH_GMS=true. A state file generated with this specified seems to include many more Google-specific items. I will not know the ins and outs of adevtool better than you, but does this mean that two state files must be maintained for each release? One for Google-enabled builds and one without?

Releasing

At the moment, the process to sign and release ProtonAOSP seems to be incomplete, or at the very least not fully documented. Your instructions to create factory images do not make any mention of creating signed releases, and the generate-factory-images-package.sh script actually doesn't work as-is; it is a sh script but it attempts to use source (which is undefined behavior, and on my Ubuntu system it simply produces an error), there are no zips prefixed with signed- by this point in the build process, and the RADIO line attempts to run the latter part of the variable as a command, due to the pipe | character. Due to these issues, I am guessing you don't actually use this script for your releases.

CalyxOS and GrapheneOS both provide a release.sh script to automate the process of creating factory images and signing them. As you are maintaining your own releases, I'm guessing you have your own similar script. Could it be incorporated officially into the ProtonAOSP manifest, thus allowing people following your instructions to create their own builds with minimal friction?

TL;DR and closing thoughts:

I don't intend to dump all of this on you as though I'm saying "work for me please;" I know things take time and effort. For as much as I can, I am willing to assist in getting these items included, however some of them are out of my control to provide (maintaining the manifest, generating state files, etc).

With that said, I believe these things could be added rather quickly, and of great benefit to you and those wanting to build ProtonAOSP. You are already publishing your own signed builds, and you already maintain public vendor modules. In theory, this means you already have the state files and scripts I am suggesting you add to the manifest, but are currently manually doing the releasing and signing. I believe that adding these scripts to the manifest will simplify the release process for future builds, perhaps saving you time.

If these things were added, I will absolutely volunteer to update your documentation to reflect the changes, and help you test any changes to the build system along the way.

(TL;DR)

To summarize, here are my suggestions:

• Include adevtool directly into the manifest, especially including the config folder with the .yml files required to create vendor modules
• Include a repository containing pre-generated adevtool state files for each device, on a per-release basis, for adevtool to use
  • Ideally, this would be supplemented by a script, tagged in its own repo by ProtonAOSP version (i.e. refs/tags/12.2.3), that produces the exact same vendor module used for that official build of ProtonAOSP (i.e., if your vendor module was generated with SQ1D.220205.004, then so will mine).
  • If there is a meaningful difference in build output between state files generated with and without WITH_GMS=true, then I suggest to also include one of each for each device, so the build system can use the correct one based on that environment variable.
• Include scripts to automatically handle creating factory images and signing them; the customary name for this in other projects is release.sh

[nicedream06]

Thanks for this write up. I am going to assume that you've been able to build a working image of ProtonAOSP from source, correct?

I tried building from source for Pixel 5 (redfin) according to the guide on the website, but it ultimately was not successful. See here: #25

From what you're describing, it sounds like there's more to the process than the usual repo sync, lunch, and build. Aside from the entire process of running adevtool against official google images, was there anything else that had to be done?

I'm a DBA, not a programmer, but I've been using Linux for 20+ years and playing around with building roms for close to 10. So although I'm not going to be adding new features to roms, I don't really need much hand-holding to build them either. I just need to know the additional steps I have to do to get it working.
I like to build my own roms because there are a few personal tweaks that I like to apply to my phone (the main thing is bringing back the network traffic indicators in the status bar). I'd really like to use ProtonAOSP because it's probably the only rom that's actually making real changes/improvements and not just piling on as many features and tweaks as possible.