x/build: add LUCI aix-ppc64 builder

[dmitshur]

There currently isn't a LUCI builder that tests the aix/ppc64 port (other than the misc-compile builder, which tests only that the port compiles). This is the tracking issue for it.

The next steps that a builder owner will need to follow to make progress here are documented https://go.dev/wiki/DashboardBuilders#luci-builders.

[laboger]

@ayappanec

[ayappanec]

We are working on getting a AIX machine in OSU lab for this purpose.

[dmitshur]

CC @golang/aix.

[pmur]

I'll try poking around with this since I came into access to the VM.

I've chosen the hostname as aix-ppc64-power8, cert attached: aix-ppc64-power8.csr.txt

(edit: fix hostname to match prevailing conventions, edit 2: revert to original naming)

[gopherbot]

Change https://go.dev/cl/600995 mentions this issue: main.star: mark aix-ppc64 as fixed capacity

[dmitshur]

@pmur I see you've edited that post, replacing - with _. You were likely thinking of builder names, where underscores are indeed used to separate the OS version and/or runmods. But for hostnames we do use dashes everywhere. The ideal format is documented at https://go.dev/wiki/DashboardBuilders#how-to-set-up-a-builder as <GOOS>-<GOOARCH>-<GitHub handle of maintainer>. If it's not definitive who the maintainer will be, it's fine to go with just aix-ppc64-power8 as before since this is the first aix/ppc64 LUCI builder.

Are you okay to go back to aix-ppc64-power8 (or aix-ppc64-power8-maintainername) as the hostname?

[pmur]

Oops. I'd prefer to stick with aix-ppc64-power8 for consistency with the other builders I've inherited.

[dmitshur]

Thanks. Here's the certificate: aix-ppc64-power8-1721862086.cert.txt.

Also noting that if the previous coordinator-based builder starts to get in the way of you bringing up the new LUCI builder, you should feel free to stop the previous builder and focus on the new one. It's certainly fine to keep the old one running if it doesn't get in the way. Thanks.

[pmur]

I am still in the process of getting LUCI working. I had to update the OSS software, and that has been rather bumpy. I have the swarming client running, though I suspect I will need to solve more issues once the backend pieces are updated.

This has led to some (hopefully) intermittent failures of the old builder.

[pmur]

The LUCI aix bot is up, though I think something needs to be updated on the backend. It has not run any jobs yet. The cpu and cipd_platform values look suspect.

[dmitshur]

The cpu and cipd_platform values look suspect.

Indeed. The CIPD binaries built for GOOS=aix GOARCH=ppc64 are using "aix-ppc64" (https://chrome-infra-packages.appspot.com/p/infra/experimental/golangbuild/aix-ppc64), so it's intended for the bot to report its cipd_platform dimension as that (instead of the current "aix-00fac25f4b00"). That dimension is also used for match the bot to the builder definitions (e.g., gotip-aix-ppc64). There are zero matches at https://chromium-swarm.appspot.com/botlist?f=cipd_platform%3Aaix-ppc64&f=pool%3Aluci.golang.shared-workers now—hence the bot isn't getting any jobs.

[dmitshur]

The aforementioned issue is fixed in crrev.com/c/5775509, thanks. A build for gotip-aix-ppc64 builder was now able to start at https://ci.chromium.org/b/8739762966790400417, failing with "CIPD error: invalid_version_error with package infra/tools/luci/cas/aix-ppc64, version git_revision:2aba496613f92a5b06d577f82b5d028225d3d577 on path None".

While we do have a aix-ppc64 binary for golangbuild (https://chrome-infra-packages.appspot.com/p/infra/experimental/golangbuild), there isn't one for some other dependencies like cas at https://chrome-infra-packages.appspot.com/p/infra/tools/luci/cas. We either need to get them built and selected, or possibly get by if there's a way this builder can operate in "low dependency" mode without needing them.

[dmitshur]

Other binaries like bb, rdb, result_adapter, and so on are already there. The reason we didn't end up with a cas binary built for aix/ppc64 is because it fails to build:

$ GOOS=aix GOARCH=ppc64 CGO_ENABLED=0 go install go.chromium.org/luci/client/cmd/cas@latest
# github.com/dgraph-io/badger/v3
go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/dir_unix.go:62:13: undefined: unix.Flock

The problem in that particular dependency appears to be reported in dgraph-io/badger#2035, and the x/sys/unix issue #64669 is related.

[pmur]

@dmitshur are there any other dependencies which do not compile for aix?

I have an attempt to get badger (v4) running on aix, but I need to get approval to post and contribute the patch. I am not sure how long that will take.

[dmitshur]

Thanks for working on this Paul. From what I can tell this case (cas failing to build for aix/ppc64) is the only known outstanding issue for this port. It's a good idea to check ahead for possible future problems, and I'm not aware of something more at this time.

Here's a quick check I did just now. Looking at a recent build for another fairly similar port openbsd/ppc64 at https://chromium-swarm.appspot.com/task?id=6bd610f803408d10, there seem to be mentions 14 CIPD packages. Out of those, cas is the only one not resolved for aix-ppc64 platform:

$ export PLATFORM=aix-ppc64
cipd instances "infra/experimental/golangbuild/$PLATFORM" >/dev/null
cipd instances "infra/tools/bb/$PLATFORM" >/dev/null
cipd instances "infra/tools/cipd/$PLATFORM" >/dev/null
cipd instances "infra/tools/cloudtail/$PLATFORM" >/dev/null
cipd instances "infra/tools/git/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci-auth/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/bbagent/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/cas/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/docker-credential-luci/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/git-credential-luci/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/lucicfg/$PLATFORM" >/dev/null
cipd instances "infra/tools/luci/vpython3/$PLATFORM" >/dev/null
cipd instances "infra/tools/prpc/$PLATFORM" >/dev/null
cipd instances "infra/tools/rdb/$PLATFORM" >/dev/null
Error: no such package: infra/tools/luci/cas/aix-ppc64.

[aclements]

FYI, the aix/ppc64 is now passing on the dashboard.

[dmitshur]

@aclements You're referring to the legacy aix/ppc64 builder, right? This issue tracks the new LUCI builder which isn't working yet:

https://ci.chromium.org/ui/p/golang/builders/luci.golang.ci/gotip-aix-ppc64

(It looks like there has been progress since September 4 in that PR dgraph-io/badger#2115 exists to fix the build in a cas dependency, but it hasn't been reviewed yet.)