Add Azurestack Support #561

patrickdillon · 2021-03-08T15:42:33Z

Updated as this is no longer WIP.

This PR:

adds a bare-bones azurestack provider
copies all of the logic from the azure provider
adjusts the logic behind hostname fetching to query the identity metadata service (not the new instance metadata endpoint)
removes support for attributes (using the logic of azure resulted in incorrect IP addresses)

I tested this and it pulled unique hostnames for a bootstrap node and 3 masters. The SSH keys also appear in the login screen.

Note: because this uses the identity metadata service, this will work in any azure stack environment.

Further todo on this PR:

remove changes to dracut/30afterburn/afterburn-hostname.service
Go through all of the logic from the azure provider and determine whether it is necessary; strip it out if we don't need it
Determine whether we want to query both the compute endpoint and the identity endpoint, falling back to one if the other doesn't work
Clean up commits (better messages, squash FIXUPs)

Regarding the compute endpoint:
From a running VM (in the lab environment) I am able to query it:

[core@ash-bootstrap ~]$ curl -H Metadata:true --noproxy "*" http://169.254.169.254/metadata/instance?api-version=2019-03-11 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1484  100  1484    0     0   9160      0 --:--:-- --:--:-- --:--:--  9217
{
  "compute": {
    "azEnvironment": "AzureStack",

But when I hit that same endpoint during afterburn I get an error. Because I was able to get the vm name from the identity endpoint, I prioritized that. as future work we could add more support for the instance metadata service.

patrickdillon · 2021-03-08T15:43:41Z

/hold
WIP
/cc @jlebon

dracut/30afterburn/afterburn-hostname.service

jlebon · 2021-03-08T16:07:16Z

src/providers/azurestack/crypto/mod.rs

@@ -0,0 +1,90 @@
+// Copyright 2017 CoreOS, Inc.


2021

This came up in discussion in another project as well. For this project at least, we should probably be able to drop all these blocks and just rely on the Cargo.toml license field. (To be clear, I mean this as a follow-up, not something you need to do here.)

jlebon · 2021-03-08T16:17:40Z

But when I hit that same endpoint from ignition, I get a 400. Any ideas?

You mean when running Afterburn from the initrd, right? Hmm, not sure offhand. Can you paste the logs?

patrickdillon · 2021-03-08T16:42:03Z

You mean when running Afterburn from the initrd, right?

Yes, afterburn! I keep mistakenly saying "from ignition".

Hmm, not sure offhand. Can you paste the logs?

Looks like I did not save the logs. I remember there was minimal info in the error, with a 400 status code. I will run again in order to capture. Let me know if you have any suggestions for getting a more detailed debug output.

jlebon · 2021-03-08T17:16:56Z

I remember there was minimal info in the error, with a 400 status code.

That's odd, I don't know why it would give a 400 in the initrd, but work in the real root. Hmm, wonder if it wants the node to check in first?

If you have interactive access to the console BTW, you can try booting with rd.break (which itself might not be possible), and then run commands in the initrd shell. (E.g. I'd start with doing the same curl command to make sure that it's not a difference between how curl and afterburn do the request, then try doing a checkin first, etc...)

patrickdillon · 2021-03-23T18:12:31Z

/retest

jlebon · 2021-03-23T19:24:24Z

src/providers/azurestack/mod.rs

+        let instance_metadata = AzureStack::fetch_identity();
+        Ok(Some(instance_metadata.unwrap().vm_name))


The linter is indirectly saying to use Err instead of unwrap(), e.g.:

Suggested change

let instance_metadata = AzureStack::fetch_identity();

Ok(Some(instance_metadata.unwrap().vm_name))

let instance_metadata = AzureStack::fetch_identity()

.context("fetching instance identity")?;

Ok(Some(instance_metadata.vm_name))

Is there a way I should change fetch_identity: https://github.com/coreos/afterburn/pull/561/files#diff-34cbce01c9aa769653cb8f06a598911f28d90de5710e9b602560660669d702d1R172-R183 where I already have context?

The fetch_identity function looks good. The context stuff is like a human-friendly backtrace where you can add more error-prefixing as the error bubbles up the stack to make it easier to tell where the error came from. It's up to you when to add a .context(...)? vs just ?. If you think e.g. it's repetitive with the context already added by the callee, you can omit it in the caller. In this case for example, yeah it's probably superfluous given that my suggestion is more or less repeating the context provided by fetch_identity, and our caller ultimately adds the context that the failure happened while fetching the hostname:

// write hostname if configured to do so self.hostname_file .map_or(Ok(()), |x| metadata.write_hostname(x)) .context("writing hostname")?;

So you can probably just swap out .context("fetching instance identity")? with just ? in my suggestion above.

patrickdillon · 2021-03-25T20:23:43Z

/retest

patrickdillon · 2021-03-25T20:27:26Z

@jlebon All work on this is done and tested. Can you give it a more thorough review or suggest how to get more eyes on it?

I have logs from testing master and bootstrap nodes if that will help.

Not sure why continuous-integration/jenkins/pr-merge test is failing. Perhaps this is a test ifnra problem? I cannot replicate locally. Also can you tip me on how to retrigger this test? I can't figure out a way. Do I have to push a new empty commit?

patrickdillon · 2021-03-25T23:04:25Z

The merge test is failing with:

[2021-03-25T20:16:35.302Z] + make
[2021-03-25T20:16:35.302Z] sed -e 's,@DEFAULT_INSTANCE@,'core',' < systemd/afterburn-sshkeys@.service.in > systemd/afterburn-sshkeys@.service.tmp && mv systemd/afterburn-sshkeys@.service.tmp systemd/afterburn-sshkeys@.service
[2021-03-25T20:16:35.302Z] cargo build --release
[2021-03-25T20:16:35.302Z] make: cargo: No such file or directory
[2021-03-25T20:16:35.302Z] make: *** [Makefile:23: all] Error 127

script returned exit code 2

This looks like a problem running cargo so a problem with the test itself.

jlebon · 2021-03-26T16:01:47Z

Yeah, CI was hitting issues yesterday. It should be fine now, so I've restarted another run.

jlebon · 2021-03-26T16:38:47Z

So in a first pass, there seems to be a lot of duplication between azure and azurestack. From what I can tell, all the crypto and goalstate stuff is exactly the same. We should try to dedupe this. Hmm, how about re-arranging things like this:

src/providers/azure/mod.rs
src/providers/azure/goalstate.rs
src/providers/azure/crypto/...
src/providers/azure/azure/mod.rs
src/providers/azure/azure/mock_test.rs
src/providers/azure/azurestack/mod.rs
src/providers/azure/azurestack/mock_test.rs

So then the top-level providers::azure module would just re-export the azure and azurestack (using pub mod) so that they could be used in the top-level match in metadata.rs. WDYT?

src/providers/azurestack/mod.rs

Prep for adding an `azurestack` provider which shares a lot with `azure`.

jlebon · 2021-03-31T18:52:47Z

OK, updated this based on #561 (comment). The only change was that I made the top-level module microsoft instead of azure to make it less confusing (because otherwise we'd have azure::azure).

Still need to test this.

Adds support and tests for the Azure Stack platform. Sets the hostname based on the vm name. Obtains SSH keys. Checks in to finish the boot process. Unlike Azure, does not offer any attributes. Follows the same pattern as Azure with some notable differences: The hostname is obtained by parsing the vm name from the identity metadata service, similar to the WALinuxAgent. Early attempts were made to obtain this vm name from the compute instance metadata service which was available in a pre-release lab environment, but the endpoint was not reachable pre-check in. Consdering the vm name was available from the identity endpoint, accessing the instance metadata service can be saved for future work. SSH key functionality is the same as Azure. Regarding attributes, obtaining the IP addresses in the same manner as Azure led to incorrect values, so they were removed as well as vmsize for simplicity. Therefore no attributes are available. Check in is the same process as Azure. Co-authored-by: Ben Howard <ben.howard@redhat.com> Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>

jlebon · 2021-03-31T20:06:44Z

Now with authorship actually preserved! :)

lucab · 2021-04-01T09:24:51Z

@jlebon there seems to be still quite a bit of overlap between azure and azurestack code (at least in constants/macros/helpers/tests). Is the plan to keep polishing here, or is that for later?

ashcrow · 2021-04-06T13:33:46Z

Other than possible overlap of constants/etc.. is there anything else holding this PR up?

jlebon · 2021-04-06T14:32:31Z

@jlebon there seems to be still quite a bit of overlap between azure and azurestack code (at least in constants/macros/helpers/tests). Is the plan to keep polishing here, or is that for later?

Other than possible overlap of constants/etc.. is there anything else holding this PR up?

To answer both of you at the same time: this PR is being tested in ASH, which is painful to do. If testing comes back green, my suggestion is to get this in as is and tweak things in follow-ups.

patrickdillon · 2021-04-07T16:44:41Z

I tested this in Azure Stack and everything looked good. I did not test ssh key fetching which I am not familiar with. But happy to help further.

jlebon

I've also finally been able to successfully test this in ASH, including SSH key fetching (which matters more for FCOS):

[root@jlebon-vm-bootstrap ~]# journalctl -u afterburn-sshkeys@core.service
-- Logs begin at Thu 2021-04-08 01:43:58 UTC, end at Thu 2021-04-08 03:27:48 UTC. --
Apr 08 02:00:22 jlebon-vm-bootstrap systemd[1]: Starting Afterburn (SSH Keys)...
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.046 WARN Failed to get fabric address from DHCP: maximum number of retries (60) reached
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.046 INFO Using fallback address
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.055 INFO Fetching http://168.63.129.16/?comp=versions: Attempt #1
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.062 INFO Fetch successful
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.062 INFO Fetching http://168.63.129.16/machine/?comp=goalstate: Attempt #1
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.219 INFO Fetch successful
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.270 INFO Fetching http://168.63.129.16/machine/70f676d8-fb84-4921-9716-96cc629c05a7/...
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: Apr 08 02:00:51.577 INFO Fetch successful
Apr 08 02:00:51 jlebon-vm-bootstrap afterburn[1929]: wrote ssh authorized keys file for user: core
Apr 08 02:00:51 jlebon-vm-bootstrap systemd[1]: afterburn-sshkeys@core.service: Succeeded.
Apr 08 02:00:51 jlebon-vm-bootstrap systemd[1]: Started Afterburn (SSH Keys).
[root@jlebon-vm-bootstrap ~]# cat /home/core/.ssh/authorized_keys.d/afterburn
ssh-rsa AAA...

So my vote is to get this in and we can do more cleanups afterwards. @lucab Would appreciate a stamp to sanity-check there isn't anything majorly wrong.

jlebon · 2021-04-09T15:11:02Z

One thing worth noting is that in the base case, ASH support also requires coreos/fedora-coreos-config#935 so it has networking in the initrd. For OKD/OCP it works without it, because we use pointer Ignition configs and so Ignition itself needs networking and requests it.

jlebon reviewed Mar 8, 2021

View reviewed changes

patrickdillon force-pushed the azurestack branch from b9b0da4 to e3c7e0f Compare March 23, 2021 18:55

jlebon reviewed Mar 23, 2021

View reviewed changes

patrickdillon force-pushed the azurestack branch 2 times, most recently from 5d8f3f2 to ccaaa2d Compare March 25, 2021 20:13

patrickdillon changed the title ~~WIP Add Azurestack Support~~ Add Azurestack Support Mar 25, 2021

patrickdillon force-pushed the azurestack branch from ccaaa2d to 436b818 Compare March 25, 2021 20:51

jlebon reviewed Mar 26, 2021

View reviewed changes

src/providers/azurestack/mod.rs Outdated Show resolved Hide resolved

src/providers/azurestack/mod.rs Outdated Show resolved Hide resolved

providers: move azure into microsoft module

7a50d0b

Prep for adding an `azurestack` provider which shares a lot with `azure`.

jlebon force-pushed the azurestack branch from 436b818 to 06832c4 Compare March 31, 2021 18:50

jlebon force-pushed the azurestack branch from 06832c4 to 3a9f3c0 Compare March 31, 2021 20:06

jlebon approved these changes Apr 8, 2021

View reviewed changes

lucab approved these changes Apr 8, 2021

View reviewed changes

lucab enabled auto-merge April 8, 2021 11:50

lucab merged commit 1c8780f into coreos:master Apr 8, 2021

jlebon mentioned this pull request Apr 9, 2021

platforms: investigate support for Azure Stack Hub (azurestack) coreos/fedora-coreos-tracker#476

Closed

jlebon mentioned this pull request Apr 9, 2021

35coreos-network: enable initrd networking on azurestack coreos/fedora-coreos-config#935

Merged

prestist mentioned this pull request Jan 12, 2023

providers: add a new "azurestack" platform (client logic for AzureStackHub) #441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Azurestack Support #561

Add Azurestack Support #561

patrickdillon commented Mar 8, 2021 •

edited

Loading

patrickdillon commented Mar 8, 2021

jlebon Mar 8, 2021

jlebon commented Mar 8, 2021

patrickdillon commented Mar 8, 2021 •

edited

Loading

jlebon commented Mar 8, 2021

patrickdillon commented Mar 23, 2021

jlebon Mar 23, 2021

patrickdillon Mar 23, 2021

jlebon Mar 23, 2021

patrickdillon commented Mar 25, 2021

patrickdillon commented Mar 25, 2021

patrickdillon commented Mar 25, 2021 •

edited

Loading

jlebon commented Mar 26, 2021

jlebon commented Mar 26, 2021

jlebon commented Mar 31, 2021

jlebon commented Mar 31, 2021

lucab commented Apr 1, 2021

ashcrow commented Apr 6, 2021

jlebon commented Apr 6, 2021

patrickdillon commented Apr 7, 2021

jlebon left a comment

jlebon commented Apr 9, 2021

		let instance_metadata = AzureStack::fetch_identity();
		Ok(Some(instance_metadata.unwrap().vm_name))

Add Azurestack Support #561

Add Azurestack Support #561

Conversation

patrickdillon commented Mar 8, 2021 • edited Loading

patrickdillon commented Mar 8, 2021

jlebon Mar 8, 2021

Choose a reason for hiding this comment

jlebon commented Mar 8, 2021

patrickdillon commented Mar 8, 2021 • edited Loading

jlebon commented Mar 8, 2021

patrickdillon commented Mar 23, 2021

jlebon Mar 23, 2021

Choose a reason for hiding this comment

patrickdillon Mar 23, 2021

Choose a reason for hiding this comment

jlebon Mar 23, 2021

Choose a reason for hiding this comment

patrickdillon commented Mar 25, 2021

patrickdillon commented Mar 25, 2021

patrickdillon commented Mar 25, 2021 • edited Loading

jlebon commented Mar 26, 2021

jlebon commented Mar 26, 2021

jlebon commented Mar 31, 2021

jlebon commented Mar 31, 2021

lucab commented Apr 1, 2021

ashcrow commented Apr 6, 2021

jlebon commented Apr 6, 2021

patrickdillon commented Apr 7, 2021

jlebon left a comment

Choose a reason for hiding this comment

jlebon commented Apr 9, 2021

patrickdillon commented Mar 8, 2021 •

edited

Loading

patrickdillon commented Mar 8, 2021 •

edited

Loading

patrickdillon commented Mar 25, 2021 •

edited

Loading