flatcar-openstack-hostname.service not using config-drive as default for scraping metadata

[alejandro-ripoll]

Description

When booting up a Flatcar instance in OpenStack using Ignition and config-drive, flatcar-openstack-hostname.service is retrieving information from metadata API instead of trying config-drive first.

Impact

Information on config-drive is ignored which might be a problem on environments where metadata API cannot be used.

Environment and steps to reproduce

1. Set-up: Create an OpenStack instance using "Flatcar Container Linux by Kinvolk 3815.2.2 for Openstack" image. Enable config-drive for it and provide any ignition config via userdata.
2. Task: Boot up the instance.
3. Action(s): Boot up the instance.
4. Error: Checking the logs of flatcar-openstack-hostname.service, service is connecting to metadata API instead of using existing config-drive.

Expected behavior

Existing values under /media/configdrive/ should be used. As implemented on coreos/afterburn#462.

Additional information

Service logs:

# journalctl -u flatcar-openstack-hostname.service
May 16 11:29:31 localhost systemd[1]: Starting flatcar-openstack-hostname.service - Flatcar OpenStack Metadata Hostname Agent...
May 16 11:29:31 localhost coreos-metadata[849]: May 16 11:29:31.503 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #1
May 16 11:30:01 localhost coreos-metadata[849]: May 16 11:30:01.504 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:30:02 localhost coreos-metadata[849]: May 16 11:30:02.504 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #2
May 16 11:30:32 localhost coreos-metadata[849]: May 16 11:30:32.504 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:30:34 localhost coreos-metadata[849]: May 16 11:30:34.505 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #3
May 16 11:31:04 localhost coreos-metadata[849]: May 16 11:31:04.505 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:31:08 localhost coreos-metadata[849]: May 16 11:31:08.505 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #4
May 16 11:31:38 localhost coreos-metadata[849]: May 16 11:31:38.506 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:31:43 localhost coreos-metadata[849]: May 16 11:31:43.506 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #5
May 16 11:32:13 localhost coreos-metadata[849]: May 16 11:32:13.506 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:32:18 localhost coreos-metadata[849]: May 16 11:32:18.507 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #6
May 16 11:32:48 localhost coreos-metadata[849]: May 16 11:32:48.507 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:32:53 localhost coreos-metadata[849]: May 16 11:32:53.507 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #7
May 16 11:33:23 localhost coreos-metadata[849]: May 16 11:33:23.508 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:33:28 localhost coreos-metadata[849]: May 16 11:33:28.509 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #8
May 16 11:33:58 localhost coreos-metadata[849]: May 16 11:33:58.509 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:34:03 localhost coreos-metadata[849]: May 16 11:34:03.509 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #9
May 16 11:34:33 localhost coreos-metadata[849]: May 16 11:34:33.510 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:34:38 localhost coreos-metadata[849]: May 16 11:34:38.510 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #10
May 16 11:35:08 localhost coreos-metadata[849]: May 16 11:35:08.510 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:13 localhost coreos-metadata[849]: May 16 11:35:13.510 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #11
May 16 11:35:43 localhost coreos-metadata[849]: May 16 11:35:43.511 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:43 localhost coreos-metadata[849]: Error: failed to run
May 16 11:35:43 localhost coreos-metadata[849]: Caused by:
May 16 11:35:43 localhost coreos-metadata[849]:     0: writing hostname
May 16 11:35:43 localhost coreos-metadata[849]:     1: maximum number of retries (10) reached
May 16 11:35:43 localhost coreos-metadata[849]:     2: failed to fetch
May 16 11:35:43 localhost coreos-metadata[849]:     3: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:43 localhost coreos-metadata[849]:     4: operation timed out
May 16 11:35:43 localhost systemd[1]: flatcar-openstack-hostname.service: Main process exited, code=exited, status=1/FAILURE
May 16 11:35:43 localhost systemd[1]: flatcar-openstack-hostname.service: Failed with result 'exit-code'.
May 16 11:35:43 localhost systemd[1]: Failed to start flatcar-openstack-hostname.service - Flatcar OpenStack Metadata Hostname Agent.

Config drive exists:

# ls -ltr /media/configdrive/openstack/latest/meta_data.json
-r--r--r--. 1 root root 1981 May 16 11:29 /media/configdrive/openstack/latest/meta_data.json

Ignition used:

# cat /etc/.ignition-result.json
{
  "provisioningBootID": "4963079c-b3fb-4ebe-a983-110b783c2c80",
  "provisioningDate": "2024-05-16T11:35:53Z",
  "userConfigProvided": true
}

Unsure whether this is relevant, but https://github.com/flatcar/bootengine/blob/flatcar-master/dracut/30ignition/flatcar-openstack-hostname.service#L17 is using openstack-metadata instead of openstack.

[tormath1]

Hello @alejandro-ripoll and sorry for the delay. After some archeology work, I found that this change was making sense: https://github.com/flatcar/bootengine/blob/8da532c809c89a9c434ada0fa9532a1c1bf49f4c/dracut/30ignition/flatcar-openstack-hostname.service#L16-L17 because at this time, an old Afterburn version was still being used.

Would you be interested to contribute to Flatcar to fix the file you mentioned? It's quite easy: fork the https://github.com/flatcar/bootengine repo, then update the bootengine ebuild to target your fork with your changes:

• https://github.com/flatcar/scripts/blob/7c3a3d384f3d3f75e94fbf44583e246b72b7b718/sdk_container/src/third_party/coreos-overlay/sys-kernel/bootengine/bootengine-9999.ebuild#L5
• https://github.com/flatcar/scripts/blob/7c3a3d384f3d3f75e94fbf44583e246b72b7b718/sdk_container/src/third_party/coreos-overlay/sys-kernel/bootengine/bootengine-9999.ebuild#L13

Once done, I'll build an OpenStack image that you could test to see if it solves the issue. Let me know!

[alejandro-ripoll]

Thanks a lot for your response @tormath1.

I've forked both bootengine and flatcar-scripts and applied my changes on:

• https://github.com/alejandro-ripoll/bootengine
• https://github.com/alejandro-ripoll/scripts

Not sure what the next step would be. Should I raise a PR for bootengine?

Thanks again.

[tormath1]

Perfect thanks. You can now open a pull request for bootengine and scripts for reviewing the changes, I'll start a Flatcar build to try your changes and assert that it does not bring regression on OpenStack platform.

[alejandro-ripoll]

Hi @tormath1.
I've raised the following PRs:

• OpenStack: Use config-drive by default and metadata API as fallback bootengine#96
• Test migration to openstack afterburn provider scripts#2009

Thanks for the help and patience.