Web container created with service name wtb instead of web? #5470

deviantintegral · 2023-10-28T15:24:56Z

deviantintegral
Oct 28, 2023

In the last few days, we've started to have an odd error in our GitHub actions running ddev. It's not consistent. For example, we have a matrix build to run automated tests, and all of the code to this point is the same. Yet, some of the jobs will get past the part fine, while others will fail.

The key bit is the container name. Note it's ddev-redacted-wtb not ddev-redacted-web:

2023-10-28T13:19:32.3543568Z Building project images...
2023-10-28T13:20:25.6834691Z �[32mProject images built in 53s.�[0m
2023-10-28T13:20:25.7867556Z  Network ddev-redacted_default  Created
2023-10-28T13:20:25.7882506Z  Volume "ddev-redacted_sqlite_database"  Created
2023-10-28T13:20:25.7892154Z  Volume "ddev-redacted_database"  Created
2023-10-28T13:20:25.8056685Z  Container ddev-redacted-redis  Created
2023-10-28T13:20:25.8062742Z  Container ddev-redacted-wtb  Created
2023-10-28T13:20:25.8063619Z  Container ddev-redacted-db  Created
2023-10-28T13:20:26.1975666Z  Container ddev-redacted-redis  Started
2023-10-28T13:20:26.3890056Z  Container ddev-redacted-wtb  Started
2023-10-28T13:20:26.5696937Z  Container ddev-redacted-db  Started
2023-10-28T13:20:28.2468906Z .........................�[31mFailed to start redacted: composer.json not found in container, stdout='', stderr='': service web does not exist in project redacted (state=doesnotexist); please report this situation, https://github.com/ddev/ddev/issues; probably can be fixed with ddev restart�[0m
2023-10-28T13:20:28.2502647Z ##[error]Process completed with exit code 1.

Our first thought was that this was somehow related to the 1.22.4 release as the timing lines up, but no one has seen this happen on a local.

I searched the ddev code for any possible typos of wtb and didn't find any. Has anyone else seen this come up? I think our next step will be see about downgrading ddev (pending ddev/github-action-setup-ddev#13) to determine if the issue goes away. My other thought was that this could somehow be a bug in a new Docker CE release, but I don't know of any reasonable way for us to test that in Actions.

Error summary

Failed to start redacted: composer.json not found in container, stdout='', stderr='': service web does not exist in project redacted (state=doesnotexist) (container service is called -wtb)
Failed to restart REDACTED: docker-compose build failed: composeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-REDACTED docker-compose -f /home/runner/work/REDACTED/REDACTED/.ddev/.ddev-docker-compose-full.yaml --progress=plain build', action='[]', err='exit status 15', stdout='', stderr='validating /home/runner/work/REDACTED/REDACTED/.ddev/.ddev-docker-compose-full.yaml: services.web Additional property environtent is not allowed', output='', stderr='validating /home/runner/work/REDACTED/REDACTED/.ddev/.ddev-docker-compose-full.yaml: services.web Additional property environtent is not allowed (environment is environtent)
Failed to start REDACTED: yaml: line 126: did not find expected key (line 126 in .ddev/**/*y?ml either doesn't exist, is an empty line, or is a valid line of yaml)

Answered by deviantintegral

Feb 23, 2024

I got to the bottom if this and it's a bug introduced in 1.22.4. PR up at #5893.

View full answer

rfay · 2023-10-28T15:50:56Z

rfay
Oct 28, 2023
Maintainer

This is pretty concerning.

I think the way to test it is by running the test manually using tmate and seeing if you can recreate it there.

I've seen some weird results in the nightly build on https://github.com/ddev/ddev-platformsh recently, but they seem to be intermittent and I havent chased them. But at least one of them looked like data corruption, but IIRC it wasn't anything like what you're seeing here.

Thanks for raising it, happy to help you look, we can do a screenshare and do tmate. If you don't have tmate set up, you'll want it. All the add-ons have it by default and most of ddev's tests have it.

18 replies

deviantintegral Nov 1, 2023
Author

More oddities. Bisecting didn't leave anything conclusive - it said every commit was good except for the 1.22.4 commit. Multiple commits would pass and fail. I've now seen a third error:

Failed to start REDACTED: yaml: line 126: did not find expected key

I downloaded the .ddev directory from the failed build, and there's nothing wrong with line 126 on any yaml file. They all pass yaml-lint too.

I also had a while where I was consistently getting the above error. Then I created a new PR with the same contents... and I didn't, for a while at least.

This has all got me back to your suggestion that this is really an issue with runners. I'm leaning towards that as well. I am now seeing different runner image versions on different job runs (20231025.1.0 vs 20231030.2.1), but both have failures. That got me looking in the runner issues, and there's a few that give me pause:

actions/runner-images#8659
actions/runner-images#8685

I'm concerned to see things like "upgrading core libraries from a PPA" without lots of docs and info around it. I'm thinking this may not be related to the ddev version, but either the environment in which the ddev release was compiled from or some change in the ubuntu-22.04 image. I'm going to try downgrading the ubuntu image to 20.04, while at the same time see about cross-compiling a new build of ddev 1.22.4 on my mac.

For reference, my bisecting procedure was:

From a local clone of ddev, run git bisect ... between the last two releases.
git bisect skip any commits with skip ci since those wouldn't have an artifact to download.
Manually browse to the github commit on that hash, such as https://github.com/ddev/ddev/commit/c6a8acd6762d68626fef95e408413865f76545b5.
Download the build from the associated CI run.
Commit the ddev binary to the repo (I'm never merging this branch, and will delete it so it won't blow up our clone size long-term).
Replaced the install ddev action call with:

   - name: Install ddev from commit
       shell: bash
       run: |
         sudo cp -v ddev-* /usr/local/bin/ddev
         sudo chmod +x /usr/local/bin/ddev
         sudo curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64"
         sudo chmod +x mkcert-v*-linux-amd64
         sudo cp mkcert-v*-linux-amd64 /usr/local/bin/mkcert
         mkcert -install

I also blew out our matrix to 12 jobs, and dropped down from a large to a regular runner, and disabled actually running functional tests. I'm not sure there's a way to truly automate bisect this way, since github generally prevents recursive actions.

deviantintegral Nov 2, 2023
Author

@rfay Ubuntu 20.04 failed with the did not find expected key error as well.

But, my self-compiled version of ddev 1.22.4 has now passed 48 jobs without failure. I compiled on my Mac (I love how easy cross compiling go is, and how you made it even easier for ddev!), so there's huge differences in the build environment. The binary has debug symbols and the like too.

I just don't like that no one else has chimed in with similar issues. Perhaps other users are just rerunning their CI builds, since they do pass most of the time? But if the build environment was somehow broken, wouldn't that also affect regular linux users, or even macos and windows users? I do suppose we are running ddev start way more often in CI due to the number of PRs our project has combined with running tests in parallel shards. It might take a week or two for someone to notice that ddev start is occasionally broken.

I think it's now time to wait on this and see if anyone else reports issues. If they do, let's have them try a separately built copy of the ddev binary. If not, we will try to upgrade our project again whenever the next ddev release is out, and if this is an issue in GitHub's runners then hopefully its fixed by then.

rfay Nov 2, 2023
Maintainer

What go version did you build with? (go version)

DDEV's own tests mostly do their own build, but all the add-ons, which run nightly tests, are using the upstream builds, so should have problems identical to yours. But most would only do a couple of restarts of stable ddev per build. You're not seeing this on the add-ons you maintain right?

deviantintegral Nov 2, 2023
Author

$ go version
go version go1.21.3 darwin/arm64

Looks like I had three failures on HEAD for the playwright addon over the weekend. Last few days have been good though. I was assuming there were changes in the amplitude text that were fixed? https://github.com/deviantintegral/ddev-playwright/actions/runs/6664802645/job/18113113431

Otherwise, that's it.

rfay Nov 2, 2023
Maintainer

The failures on HEAD probably wouldn't be related to build issues with distributed ones, because they (if using standard technique) do a brew install --HEAD which builds locally.

As of v1.22.4, the build was done on github using

      - uses: actions/setup-go@v4
        with:
          go-version: '>=1.20'

Which could possibly have resulted in a variety of go versions since we're now at 1.21.3.

The actual build was in https://github.com/ddev/ddev/actions/runs/6655110294

Linux was done in https://github.com/ddev/ddev/actions/runs/6655110294/job/18084619605

It used go 1.21.3

I did see the

Instrumentation is opted in, but AmplitudeAPIKey is not available. This usually means you have a locally-built ddev binary or one from a PR build. It's not an error. Please report it if you're using an official release build.

I did see the amplitude warning you were seeing in HEAD and fixed it in #5471 - I saw it show up in a few add-ons. However, all tests should be running with amplitude turned off, so this shouldn't show up in the first place. If you're using github-action-add-on-test, this went into https://github.com/ddev/github-action-add-on-test/releases/tag/v1.1.1

deviantintegral · 2023-11-09T23:58:19Z

deviantintegral
Nov 9, 2023
Author

Here's my locally compiled 1.22.4: ddev.tar.gz

0 replies

deviantintegral · 2023-11-17T17:40:53Z

deviantintegral
Nov 17, 2023
Author

We upgraded back to 1.22.4 in CI because we wanted to omit the database container in some jobs and there was a bugfix for that. So far things have been working fine again, though we're now seeing the runner image is version 20231030.2.0 which is somehow in between the versions we had previously seen.

1 reply

rfay Nov 17, 2023
Maintainer

Thanks for chasing and keeping up with this. Sure makes you wonder. They are doing massive maintenance and refactoring on the runner images (all platforms) so I imagine we'll see things happening. They say it will take months to complete.

deviantintegral · 2023-12-06T23:05:56Z

deviantintegral
Dec 6, 2023
Author

We upgraded to 1.22.5 a few days ago and are now seeing the same class of bug, but a different string! It's also not consistent and IMO is somehow an issue with the runners.

stderr='failed to solve: ddev/ddev-webeerver:v1.22.5: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed'

Note webeerver instead of webserver.

5 replies

rfay Dec 6, 2023
Maintainer

So weird. We haven't seen anything like this on our GitHub tests. Remind me which runner you're using. We have mostly Ubuntu 22.04.

deviantintegral Dec 8, 2023
Author

All Ubuntu 22.04. I set up a runner on Linode which is Debian 12 + the Actions Runner, and so far the issue hasn't come up there.

penyaskito Dec 13, 2023
Maintainer

Happened again on the gh hosted runner with ubuntu 22.04, but this time is websesver, not webserver or webeerver

', stderr='failed to solve: ddev/ddev-websesver:v1.22.5: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed'

Note websesver instead of webserver (different typo than previous).

rfay Dec 14, 2023
Maintainer

All of you are using https://github.com/ddev/github-action-setup-ddev ? Could that be part of the equation here?

deviantintegral Dec 14, 2023
Author

We are - and we aren't in our self-hosted runners because they don't get reset between jobs. We're also not using autostart so that's not running either.

There's only a few commands in https://github.com/ddev/github-action-setup-ddev/blob/main/lib/main.js so I would be shocked if it was the issue.

Also for others, @penyaskito is working on the same project as me so we can't treat this as a second independent confirmation of the issue.

deviantintegral · 2023-12-21T19:09:22Z

deviantintegral
Dec 21, 2023
Author

😱 we've now seen this happen in our Linode hosted runners. I guess I can stop being annoyed at GitHub 😅 .

Failed to start PROJECT: docker-compose build failed: composeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-PROJECT docker-compose -f /home/runner/actions-runner/_work/PROJECT/PROJECT/.ddev/.ddev-docker-compose-full.yaml --progress=plain build', action='[]', err='exit status 17', stdout='', stderr='unable to prepare context: path "/home/runner/aciions-runner/_work/PROJECT/PROJECT/.ddev/.webimageBuild" not found', output='', stderr='unable to prepare context: path "/home/runner/aciions-runner/_work/PROJECT/PROJECT/.ddev/.webimageBuild" not found'

We're going to capture the .ddev directory as an artifact. While I can't be 100% sure though (because I'm not taking runners offline), it doesn't appear that the YAML file on disk is corrupt. Next steps I see are:

If we can confirm the docker compose file is fine, then this could be a bug in docker compose itself. Perhaps we can downgrade it and see if that helps.
If the compose file is broken, then I think https://github.com/ddev/ddev/blob/master/pkg/ddevapp/compose_yaml.go is a place to start. Perhaps there's a concurrency bug with the serialize / deserialize yaml dance? I think we could consider adding in some additional debug logs to each step of generating the compose file so we can see where it goes wrong.

1 reply

deviantintegral Dec 22, 2023
Author

I've confirmed the docker compose file is not OK. In this latest error contains:

    web:
        build:
            args:
                BASE_IMAGE: ddev/ddev-websesver:v1.22.5

davereid · 2024-01-04T22:10:36Z

davereid
Jan 4, 2024

Oooooh I have seen this happen randomly in GitHub Actions as well, it's always a one-letter difference in the service name and it won't start correctly.

1 reply

davereid Jan 4, 2024

To clarify: this is completely different project/repo that I was seeing these random startup failure.

deviantintegral · 2024-01-31T20:47:35Z

deviantintegral
Jan 31, 2024
Author

This just happened in a public PR, so at least we can now share everything.

Failed: https://github.com/Lullabot/drainpipe/actions/runs/7727308894/job/21065661055
Rebuild: https://github.com/Lullabot/drainpipe/actions/runs/7727308894/job/21079350697

1 reply

rfay Feb 2, 2024
Maintainer

Well, that's progress.

@davereid I hope you'll provide an example for yours as well.

Someday somebody is going to come up with a repro case. Thanks for staying on this!

rfay · 2024-02-02T20:30:34Z

rfay
Feb 2, 2024
Maintainer

Is it always the docker image name ddev/ddev-webserver that is corrupted?

Is it always the BASE_IMAGE that's broken in .ddev/.ddev-docker-compose-full.yaml?

    web:
        build:
            args:
                BASE_IMAGE: ddev/ddev-webserver:v1.22.6

It would be pretty easy to sprinkle some greps into your test to try to find out where it's getting corrupted. You've probably already done that. But just grep "BASE_IMAGE: ddev/ddev-webserver" .ddev/.ddev-docker-compose-full.yaml would be an interesting test. If not found, the corruption has already happened.

I also note that that line is created from BASE_IMAGE: $DDEV_WEBIMAGE in .ddev/.ddev-docker-compose-base.yaml. So any corruption of the environment variable DDEV_WEBIMAGE would break it all.

DDEV_WEBIMAGE is set here:

ddev/pkg/ddevapp/ddevapp.go

Line 2106 in 94d7750

"DDEV_WEBIMAGE": app.WebImage,

Although it's discouraged, and not even in docs any more I don't think, there is a webimage key in .ddev/config.yaml that can be used to override the webimage.

0 replies

deviantintegral · 2024-02-23T21:42:38Z

deviantintegral
Feb 23, 2024
Author

I got to the bottom if this and it's a bug introduced in 1.22.4. PR up at #5893.

1 reply

rfay Feb 24, 2024
Maintainer

Amazing and congratulations!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DDEV Foundation

Web container created with service name wtb instead of web? #5470

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 28 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

DDEV Foundation

Web container created with service name wtb instead of web? #5470

deviantintegral Oct 28, 2023

Error summary

Replies: 9 comments · 28 replies

rfay Oct 28, 2023 Maintainer

deviantintegral Nov 1, 2023 Author

deviantintegral Nov 2, 2023 Author

rfay Nov 2, 2023 Maintainer

deviantintegral Nov 2, 2023 Author

rfay Nov 2, 2023 Maintainer

deviantintegral Nov 9, 2023 Author

deviantintegral Nov 17, 2023 Author

rfay Nov 17, 2023 Maintainer

deviantintegral Dec 6, 2023 Author

rfay Dec 6, 2023 Maintainer

deviantintegral Dec 8, 2023 Author

penyaskito Dec 13, 2023 Maintainer

rfay Dec 14, 2023 Maintainer

deviantintegral Dec 14, 2023 Author

deviantintegral Dec 21, 2023 Author

deviantintegral Dec 22, 2023 Author

davereid Jan 4, 2024

davereid Jan 4, 2024

deviantintegral Jan 31, 2024 Author

rfay Feb 2, 2024 Maintainer

rfay Feb 2, 2024 Maintainer

deviantintegral Feb 23, 2024 Author

rfay Feb 24, 2024 Maintainer

deviantintegral
Oct 28, 2023

Replies: 9 comments 28 replies

rfay
Oct 28, 2023
Maintainer

deviantintegral Nov 1, 2023
Author

deviantintegral Nov 2, 2023
Author

rfay Nov 2, 2023
Maintainer

deviantintegral Nov 2, 2023
Author

rfay Nov 2, 2023
Maintainer

deviantintegral
Nov 9, 2023
Author

deviantintegral
Nov 17, 2023
Author

rfay Nov 17, 2023
Maintainer

deviantintegral
Dec 6, 2023
Author

rfay Dec 6, 2023
Maintainer

deviantintegral Dec 8, 2023
Author

penyaskito Dec 13, 2023
Maintainer

rfay Dec 14, 2023
Maintainer

deviantintegral Dec 14, 2023
Author

deviantintegral
Dec 21, 2023
Author

deviantintegral Dec 22, 2023
Author

davereid
Jan 4, 2024

deviantintegral
Jan 31, 2024
Author

rfay Feb 2, 2024
Maintainer

rfay
Feb 2, 2024
Maintainer

deviantintegral
Feb 23, 2024
Author

rfay Feb 24, 2024
Maintainer