Upgrading on MacOS is broken: unable to start the Upgrade Watcher

[ycombinator]

For confirmed bugs, please report:

• Version: 8.10.0 / main
• Operating System: MacOS
• Steps to Reproduce:

1. Build Elastic Agent for MacOS from main. This Agent will serve as a our source, i.e. pre-upgrade, Agent.

   DEV=true EXTERNAL=true SNAPSHOT=true PLATFORMS=darwin/arm64 PACKAGES=tar.gz mage package
   

2. Unpack and install this Agent.

3. Create a branch off main so we can bump up the version of Agent and also have a new commit SHA. This Agent will serve as a our target, i.e. post-upgrade, Agent.

   git checkout -b new-version
   

4. Bump up the version of Agent.

   wget -qO- https://github.com/elastic/elastic-agent/files/12295683/new-version.patch | git apply -
   

5. Commit the changes so we have a new commit SHA in the crashing Agent's version. Otherwise the upgrade will not happen.

   git commit -am "[Testing] New version"
   

6. Build Elastic Agent for MacOS from this branch. Make sure not to use EXTERNAL=true since the component artifacts for the bumped up version won't exist in the artifacts repository. Also make sure not to use SNAPSHOT=true otherwise the artifact fetcher will try to fetch from the snapshot repository during the download process.

   DEV=true PLATFORMS=darwin/arm64 PACKAGES=tar.gz mage package
   

7. Attempt to upgrade from the 8.10.0 (installed) Agent to the 8.12.0 Agent.

   sudo elastic-agent upgrade 8.12.0 --skip-verify --source-uri file://$(pwd)/build/distributions
   

   This command will fail with the following error:

   Error: Failed trigger upgrade of daemon: fork/exec /Library/Elastic/Agent/elastic-agent.app/Contents/MacOS/elastic-agent: no such file or directory
   For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html
   

   The Agent's logs give a bit more detail on where this error is coming from and why it's happening:

   {"log.level":"error","@timestamp":"2023-08-08T20:27:41.473Z","log.origin":{"file.name":"upgrade/upgrade.go","file.line":167},"message":"Rolling back: starting watcher failed","log":{"source":"elastic-agent"},"error":{"message":"fork/exec /Library/Elastic/Agent/elastic-agent.app/Contents/MacOS/elastic-agent: no such file or directory"},"ecs.version":"1.6.0"}
   

   I tried reproducing this bug with an 8.9.1-SNAPSHOT version of Agent, built by checking out the 8.9 branch in step 1., and I didn't get the same error from the upgrade command.

   $ sudo elastic-agent version
   Binary: 8.9.1-SNAPSHOT (build: 636eabf7077ef0d55aaa5a30ae042dbf9c9c1d82 at 2023-08-08 21:16:46 +0000 UTC)
   Daemon: 8.9.1-SNAPSHOT (build: 636eabf7077ef0d55aaa5a30ae042dbf9c9c1d82 at 2023-08-08 21:16:46 +0000 UTC)
   $ sudo elastic-agent upgrade 8.12.0 --skip-verify --source-uri file://$(pwd)/build/distributions
   Upgrade triggered to version 8.12.0, Elastic Agent is currently restarting
   

   So I'm fairly confident this bug is likely due to a change introduced for 8.10.0.

Attachments:

• new-version.patch

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[joshdover]

Should we consider this a blocker for 8.10?

[ycombinator]

Should we consider this a blocker for 8.10?

Yes, I would say so.