add oracle ipv6 single stack imds functionality #5785

a-dubs · 2024-10-04T21:08:12Z

Proposed Commit Message

N/A

We rebasing 😎

Additional Context

This is to enable future functionality on the Oracle Cloud and will have no immediate ramifications or generally available use cases for the meantime.

My personal acceptance criteria for this MP is that I maintain or improve code coverage for any code I touch, which in this case is cloudinit/sources/DataSourceOracle.py and cloudinit/net/ephemeral.py. For the ephemeral networking changes I added, I made sure that I added sufficient unit tests so that the EphemeralIPNetwork class and subsequent helper functions have 100% code and branch coverage. For the Oracle DS, I improved code coverage from 91% to 94% while adding nearly 40 statements and over 20 branches that would need covered.

Test Steps

These changes were manually tested on both jammy (22.04) and noble (24.04), on Oracle Platform Ubuntu images for testing ipv4 and dual stack, and then on custom built images for ipv6 single stack testing. For this manual validation, the following things were checked:

No new warnings logged vs unmodified cloud-init package
No boot speed slowdowns introduced vs unmodified cloud-int package

The new oracle single-stack ipv6 integration tests was run against an existing instance with the following command:

CLOUD_INIT_EXISTING_INSTANCE_ID="ocid1.instance............" \
CLOUD_INIT_PLATFORM="oci" \
CLOUD_INIT_RUN_UNSTABLE="true" \
CLOUD_INIT_KEEP_INSTANCE="true" \
tox -e integration-tests -- \
tests/integration_tests/datasources/test_oracle_ipv6.py

Testing Backwards compatability:

Launched oracle instance
Installed custom cloud-init deb onto instance
ran cloud-init clean and rebooted and pulled logs and verified everything was good.
Repeat steps 1-3 for each instance type being tested: Native VM, Paravirtualized VM, and BM.

Integration testing:

Ran full suite of integration tests against oracle using my custom ezdeb package and 5 tests failed, but they were all explainable / expected / unrelated to my changes.

On focal (20.04), 11 tests failed due to running 24.X on focal using my ezdeb package.

On noble (24.04), only 5 tests failed. And they occurred both with and without my custom ezdeb package. so I feel confident in saying that no integration tests have been broken by my changes.

the tests that failed:

FAILED tests/integration_tests/datasources/test_caching.py::test_no_cache_network_only
FAILED tests/integration_tests/datasources/test_caching.py::test_no_cache_with_fallback
FAILED tests/integration_tests/datasources/test_none.py::test_datasource_none_discovery
FAILED tests/integration_tests/modules/test_ubuntu_drivers.py::test_ubuntu_drivers_installed
FAILED tests/integration_tests/modules/test_ansible.py::test_ansible_pull_pip

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

TheRealFalcon

This isn't a full review, but I'm commenting on what I think are the big issues to be addressed.

I think you may be trying to make the EphemeralIPNetwork class do more than it needs to. The original purpose of the connectivity url was to see if we could connect to the IMDS BEFORE setting up our ephemeral connection. If we have connectivity due to something outside of cloud-init (i.e., klibc in initramfs on OCI), then cloud-init doesn't need to do the ephemeral network setup. However, you've modified it to attempt to setup the v6 address, and then check for connectivity afterwards. I don't think there's value added by adding the connectivity check after. If we've setup a connection (either ipv4, ipv6, or both), we can assume we have the connectivity we need.

Additionally, the context manager in EphemeralIPNetwork is now attempting to signal to the caller (via ipv6_reached_at_url) if it should be using ipv6 or not. This isn't really the job of EphemeralIPNetwork though, and it's currently preventing v4 from being setup if v6 works, and preventing v6 from being setup if no v6 callback was provided. This is a fairly large change in behavior that would break EC2 as implemented.

I think that the only changes that were needed to ephemeral.py are around allowing adding support for multiple connectivity urls and updating EphemeralIPNetwork to do the connectivity check early in the context manager (see my inline comment for more details).

cloudinit/sources/DataSourceOracle.py

cloudinit/net/ephemeral.py

cloudinit/sources/DataSourceOracle.py

cloudinit/net/ephemeral.py

cloudinit/sources/DataSourceOracle.py

a-dubs · 2024-11-08T17:21:19Z

@TheRealFalcon all changes have been made to allow for a cleaner happy eyeballs ish approach for ephemeral networking. ready for re-review!

TheRealFalcon

I left a number of inline comments, but nothing super major this time.

I haven't looked at tests yet, but I also remember you saying you didn't plan on updating tests until the approach is solid.

Also, can you remind me what local testing has looked like so far? Were you able to test on both native and paravirtualized instances? Are both ipv6-only and dual stack both currently supported? Were there cases that didn't include a pre-existing initramfs network?

TheRealFalcon · 2024-11-08T20:15:36Z

cloudinit/net/ephemeral.py

+        ][0]
+        return headers
+
+    def _perform_connectivity_check(


We now have two functions doing the same thing. Any reason we can't merge them? This one could just become a generic function in this file and we could remove the other one.

@TheRealFalcon I understand the sentiment but I feel like they are slightly different functions. Previously, I tried using the existing has_url_connectivity and it did not seem to fit my needs. Would like to discuss this further with you.

TheRealFalcon · 2024-11-08T20:16:33Z

cloudinit/net/ephemeral.py

+                    url_data["url"] for url_data in self.connectivity_urls_data
+                ],
+                headers_cb=self._headers_cb,
+                timeout=0.1,  # keep really short for quick failure path


The connectivity_url structure can specify a timeout. We should use that if specified. Also, 0.1 feels a bit short here. I don't think it's uncommon that a standard GET request can take longer than that, even over link local.

TheRealFalcon · 2024-11-08T20:19:27Z

cloudinit/net/ephemeral.py

@@ -456,5 +465,135 @@ def __enter__(self):
            raise exceptions[0]
        return self

+    def _do_ipv4(


I think we could use a slightly more descriptive name here. Something like _obtain_ephemeral_v4_address maybe? Same for the _do_ipv6 below.

yeah... _do_ipv4 and _do_ipv6 definitely leave something to be desired... lol

TheRealFalcon · 2024-11-08T20:22:35Z

cloudinit/net/ephemeral.py

@@ -456,5 +465,135 @@ def __enter__(self):
            raise exceptions[0]
        return self

+    def _do_ipv4(
+        self, ephemeral_obtained, exceptions


These parameters don't seem necessary. Mutable parameters make the state harder to follow, and in this case since we're returning them too, there doesn't seem to be a need to pass anything. Both v4 and v6 functions can instead return their ephemeral_obtained value along with the exception (if it exists), and the caller should be able to know what to do with them.

TheRealFalcon · 2024-11-08T20:24:52Z

cloudinit/net/ephemeral.py

+            exceptions.append(e)
+        return ephemeral_obtained, exceptions
+
+    def _do_ipv6(


This method seems really similar to the _do_ipv4 method...so much so that I think they could really be one method that takes an option telling them whether to do ipv4 vs ipv6.

hmmm interesting. I do agree. will have to look at the ramifications of doing so for unit test mocking.

TheRealFalcon · 2024-11-08T22:08:46Z

cloudinit/sources/DataSourceOracle.py

    start_time = time.monotonic()
-    instance_url, instance_response = wait_for_url(
-        urls,
+    url_that_worked, instance_response = wait_for_url(


This is where we want our happy eyeballs, not in the latter call. We should use a connect_synchronously=False here along with no specified sleep time.

Once we know this worked, we can use the url_that_worked to know which url to use for all future calls.

Also, nit, but url_that_worked implies that the others the others don't. Let's see if we can come up with a better name.

oh shoot good catch on the happy eyeballs.

Also, I agree that url_that_worked is not great but am unsure if i can come up with something that much better. i shall see.

TheRealFalcon · 2024-11-08T22:11:12Z

cloudinit/sources/DataSourceOracle.py

-    if not instance_url:
-        LOG.warning("Failed to fetch IMDS metadata!")
-        return None
+    if not url_that_worked:


No need to do an if not... along with an else. If we swap the blocks we can have if url_that_worked: with an else instead.

TheRealFalcon · 2024-11-08T22:17:07Z

cloudinit/sources/DataSourceOracle.py

+            self.metadata_address = _get_versioned_metadata_base_url(
+                url=url_that_worked
+            )
+            if _is_ipv4_metadata_url(self.metadata_address):


I think these logs are redundant. Didn't we already log our IMDS address in read_opc_metadata()?

TheRealFalcon · 2024-11-08T22:27:16Z

cloudinit/sources/DataSourceOracle.py

-            `fetch_vnics_data` is True, else None
-        or None if fetching metadata failed
-
+        A tuple containing:


The reason for defining the OpcMetadata named tuple was to simplify what this function returns. If we're returning a bunch of things at once, the other options are kind of ugly.

E.g.,

( version, instance_data, vnics_data ) = read_opc_metadata(...)

or

metadata = read_opc_metadata(...) ... data = self._crawled_metadata = metadata[1] self._vnics_data = fetched_metadata[2]

A named tuple simplifies the typing as well as the call and usage semantics.

All that to say, instead of changing the return type, can we add what you want to return into the named tuple?

yeah wait a minute... good point. 🤦 will do.

TheRealFalcon · 2024-11-08T22:29:42Z

cloudinit/sources/DataSourceOracle.py

        for index, vnic_dict in enumerate(vnics_data):
            is_primary = set_primary and index == 0
            mac_address = vnic_dict["macAddr"].lower()
+            is_ipv6_only = vnic_dict.get(


Is dual-stack not an option? The either/or here feels odd to me.

dual stack IS an option. but if it is dual stack, the primary network will be ipv4. i.e. it is either ipv6 only (ipv6 single stack) OR it is not ipv6 only (either ipv4 single stack or ipv4 and ipv6 dual stack).

a-dubs

responding to @TheRealFalcon comments. will begin making changes and applying feedback now.

a-dubs · 2024-11-11T14:27:47Z

cloudinit/net/ephemeral.py

+        self.connectivity_urls_data = connectivity_urls_data
+
+        # will be updated by the context manager
+        self.imds_reached_at_url: Optional[str] = None


good catch. this was still left over from when this value was set from the _do_ipv4 and _do_ipv6 helper functions!

a-dubs · 2024-11-11T14:28:12Z

cloudinit/net/ephemeral.py

-                        self.distro,
-                        self.interface,
-                    )
+        self.imds_reached_at_url = self._perform_connectivity_check()


good suggestion! definitely more readable.

a-dubs · 2024-11-11T14:28:43Z

cloudinit/net/ephemeral.py

@@ -456,5 +465,135 @@ def __enter__(self):
            raise exceptions[0]
        return self

+    def _do_ipv4(


yeah... _do_ipv4 and _do_ipv6 definitely leave something to be desired... lol

cloudinit/net/ephemeral.py

a-dubs · 2024-11-11T14:51:29Z

cloudinit/net/ephemeral.py

+        ][0]
+        return headers
+
+    def _perform_connectivity_check(


@TheRealFalcon I understand the sentiment but I feel like they are slightly different functions. Previously, I tried using the existing has_url_connectivity and it did not seem to fit my needs. Would like to discuss this further with you.

a-dubs · 2024-11-11T14:53:59Z

cloudinit/net/ephemeral.py

+            exceptions.append(e)
+        return ephemeral_obtained, exceptions
+
+    def _do_ipv6(


hmmm interesting. I do agree. will have to look at the ramifications of doing so for unit test mocking.

a-dubs · 2024-11-11T15:05:45Z

cloudinit/sources/DataSourceOracle.py

+                    ipv4=True,
+                    connectivity_urls_data=CONNECTIVITY_URLS_DATA,
+                )
+            except Exception:


@TheRealFalcon hm..... good point. might need to discuss this a little further with you. at the very least logging a warning would be important. This is meant to catch any of the exceptions purposefully raised by EphemeralIPNetwork, which should be ProcessExecutionError and NoDHCPLeaseError. So maybe just catching for those would be best.

a-dubs · 2024-11-11T15:10:10Z

cloudinit/sources/DataSourceOracle.py

        for index, vnic_dict in enumerate(vnics_data):
            is_primary = set_primary and index == 0
            mac_address = vnic_dict["macAddr"].lower()
+            is_ipv6_only = vnic_dict.get(


dual stack IS an option. but if it is dual stack, the primary network will be ipv4. i.e. it is either ipv6 only (ipv6 single stack) OR it is not ipv6 only (either ipv4 single stack or ipv4 and ipv6 dual stack).

a-dubs · 2024-11-11T15:11:48Z

cloudinit/sources/DataSourceOracle.py

-            `fetch_vnics_data` is True, else None
-        or None if fetching metadata failed
-
+        A tuple containing:


yeah wait a minute... good point. 🤦 will do.

a-dubs · 2024-11-11T15:13:43Z

cloudinit/sources/DataSourceOracle.py

    start_time = time.monotonic()
-    instance_url, instance_response = wait_for_url(
-        urls,
+    url_that_worked, instance_response = wait_for_url(


oh shoot good catch on the happy eyeballs.

Also, I agree that url_that_worked is not great but am unsure if i can come up with something that much better. i shall see.

TheRealFalcon · 2024-11-12T21:21:37Z

@holmanb , do you mind reviewing the changes to ephemeral.py here?

TheRealFalcon

@a-dubs , I'm trying to give feedback quickly, so here's a few more comments I had

TheRealFalcon · 2024-11-11T22:33:41Z

cloudinit/net/ephemeral.py

+        if _check_connectivity_to_imds(self.connectivity_urls_data):
+            LOG.debug("We already have connectivity to IMDS, skipping DHCP.")
+            return self
+        else:


Since the if condition returned, we can remove this else entirely.

TheRealFalcon · 2024-11-11T22:34:48Z

cloudinit/net/ephemeral.py

-                    )
+        # short-circuit if we already have connectivity to IMDS
+        if _check_connectivity_to_imds(self.connectivity_urls_data):
+            LOG.debug("We already have connectivity to IMDS, skipping DHCP.")


I like that the log on line 302 logs the address used. We should do that here too.

the _check_connectivity_to_imds already logs the url so it felt redundant but might as well add since we are already logging something anyways.

TheRealFalcon · 2024-11-11T23:03:59Z

cloudinit/sources/DataSourceOracle.py


-OpcMetadata = namedtuple("OpcMetadata", "version instance_data vnics_data")
+ReadOpcMetadataResponse = namedtuple(


Since we're updaing this anyway and you're adding typing, we should probably convert it to the newer style:

class ReadOpcMetadataResponse(NamedTuple): version: int instance_data: Dict[str, Any] vnics_data: Optional[Dict[str, Any]] imds_url_used: str

I'm definitely down. But, while we're at it, shouldn't we just do a dataclass then? since that is python 3.7+ and we are on 3.8 now? I much prefer that over a named tuple tbh.

Eh, I figured it's easier to still keep the same data types, but I'm not against it. I think I still slightly prefer named tuples though if there's no need for a class. They're immutible, hashable, and implemented in c so theoretically faster. Why would you rather use dataclasses?

Fair enough! I'll keep it as a named tuple then. It's just personal preference for data classes. But for something this simple, I agree named tuples fit the job better.

TheRealFalcon · 2024-11-11T23:09:28Z

cloudinit/sources/DataSourceOracle.py

+
+V2_HEADERS = {"Authorization": "Bearer Oracle"}
+
+CONNECTIVITY_URLS_DATA = [


We're leaning towards keeping less running during import time, just because it increases the load time of cloud-init. If it's also used in unit tests, it's fine as-is...though for unit tests, I also like to hard code the full URLs to ensure we're actually hitting the endpoints that we expect.

TheRealFalcon · 2024-11-12T21:11:51Z

cloudinit/net/ephemeral.py

+                    exceptions.append(ipv6_exception)
+
+            # need to set this if we only have ipv6 ephemeral network
+            if not self.ipv4 or not ipv4_ephemeral_obtained:


I think at this point this should be if ipv6_ephemeral_obtained and not ipv4_ephemeral_obtained.

hmmmm... fair enough. i shall try that out.

TheRealFalcon · 2024-11-12T21:25:04Z

Also, let's not forget to remove the original function

TheRealFalcon

The rest of my comments

TheRealFalcon · 2024-11-12T22:01:23Z

cloudinit/sources/DataSourceOracle.py

-                    "headers": V2_HEADERS,
-                },
+            nic_name = net.find_fallback_nic()
+            # try:


Commented code here and a few lines lower.

TheRealFalcon · 2024-11-12T22:16:53Z

cloudinit/sources/DataSourceOracle.py

    ]
+
+    url_that_worked = None


This line shouldn't be needed.

TheRealFalcon · 2024-11-13T14:03:26Z

tests/unittests/net/test_dhcp.py

@@ -815,16 +815,19 @@ def test_multiple_files(self):


 @pytest.mark.usefixtures("disable_netdev_info")
+@mock.patch("cloudinit.net.ephemeral._check_connectivity_to_imds")


Are the changes in this file necessary? What was wrong with the responses code?

TheRealFalcon · 2024-11-13T14:14:13Z

tests/integration_tests/datasources/test_oracle_ipv6.py

+    # assert r.ok
+    # sleep for 10s to allow cloud-init clean to run
+    # oracle takes > 10s to boot so this is not slowing us down
+    time.sleep(10)


If you remove the reboot part of cloud-init clean, client.restart() should work here and take care of any waiting necessary.

TheRealFalcon · 2024-11-13T14:33:02Z

tests/integration_tests/datasources/test_oracle_ipv6.py

@@ -0,0 +1,186 @@
+import time


This test is a great start, but I think we also need some testing beyond IMDS reachability. In particular:

No long waits or timeouts when fetching the data

Network rendered properly

No errors or warnings in the logs

I think it's important to verify these things on both native and paravirtualized instances. I also realize some of this isn't really viable using our current integration testing tools and that this functionality isn't widely available yet, but I do think that means we also need some manual checks in the meantime.

holmanb

@a-dubs I just gave a first pass on the ephemeral code. No major objections, just a minor comment and a question.

cloudinit/net/ephemeral.py

holmanb · 2024-11-15T02:56:43Z

cloudinit/net/ephemeral.py

+            the following keys:
+            - "url" (str): The URL to check connectivity for.
+            - "headers" (dict, optional): Headers to include in the request.
+            - "timeout" (int, optional): Timeout for the request in seconds.


Is this key used in any callsite?

Yes the timeouts are used to find the max timeout value specified and then they are passed to wait for url

holmanb

@a-dubs I think we should remove the dead code (either in a seperate commit in this PR or in a followup PR), and you mentioned having testing data to share. Once we have this in place I'm happy to merge this.

holmanb · 2024-11-18T20:41:14Z

cloudinit/net/ephemeral.py

        self.distro = distro
        self.interface_addrs_before_dhcp = netinfo.netdev_info()

    def __enter__(self):
        """Setup sandboxed dhcp context, unless connectivity_url can already be
        reached."""
-        if self.connectivity_url_data:
-            if net.has_url_connectivity(self.connectivity_url_data):


This function is now dead code. Can you please remove it in a separate commit or a followup PR?

TheRealFalcon

Things look good to me, but I also want to see the testing results before we merge. Thanks for all the work here!

…onnectivity_to_imds()

a-dubs changed the title ~~Cpc 3194 oracle ipv6 single stack~~ add oracle ipv6 single stack imds functionality Oct 4, 2024

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 11 times, most recently from 3d1941e to f207b2a Compare October 10, 2024 22:07

a-dubs marked this pull request as ready for review October 10, 2024 22:15

blackboxsw self-assigned this Oct 14, 2024

TheRealFalcon requested changes Oct 17, 2024

View reviewed changes

TheRealFalcon assigned TheRealFalcon and unassigned blackboxsw Oct 17, 2024

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 2 times, most recently from 16d2927 to 042cca9 Compare October 22, 2024 17:10

a-dubs marked this pull request as draft October 23, 2024 13:36

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 2 times, most recently from 12c26f4 to 697b4bd Compare October 24, 2024 17:47

TheRealFalcon added this to the cloud-init-24.4 milestone Nov 7, 2024

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 4 times, most recently from b3f8d71 to ccdaf55 Compare November 8, 2024 17:18

a-dubs marked this pull request as ready for review November 8, 2024 17:21

a-dubs requested a review from TheRealFalcon November 8, 2024 17:21

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 2 times, most recently from 9b25497 to b47f284 Compare November 8, 2024 20:09

TheRealFalcon requested changes Nov 8, 2024

View reviewed changes

a-dubs commented Nov 11, 2024

View reviewed changes

a-dubs requested a review from TheRealFalcon November 12, 2024 20:59

TheRealFalcon reviewed Nov 12, 2024

View reviewed changes

TheRealFalcon reviewed Nov 13, 2024

View reviewed changes

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 2 times, most recently from a4a904a to bcbc950 Compare November 13, 2024 19:44

a-dubs requested review from TheRealFalcon and holmanb November 13, 2024 22:10

holmanb reviewed Nov 15, 2024

View reviewed changes

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch from d71305a to 0b2890a Compare November 18, 2024 14:40

holmanb approved these changes Nov 18, 2024

View reviewed changes

TheRealFalcon approved these changes Nov 18, 2024

View reviewed changes

a-dubs added 2 commits November 18, 2024 17:33

feat(ephemeral): refactor ephemeralIP and add ipv6 connectivity check

63dd3be

feat(oracle): add true single stack ipv6 support

acbbb3b

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch 3 times, most recently from 70247b1 to ebea3f7 Compare November 19, 2024 21:24

feat(ephemeral): replace old has_url_connectivity() with new _check_c…

efe56a1

…onnectivity_to_imds()

a-dubs force-pushed the CPC-3194-oracle-ipv6-single-stack branch from ebea3f7 to efe56a1 Compare November 19, 2024 21:42

TheRealFalcon merged commit 1f520d9 into canonical:main Nov 19, 2024
22 checks passed


		OpcMetadata = namedtuple("OpcMetadata", "version instance_data vnics_data")
		ReadOpcMetadataResponse = namedtuple(


		V2_HEADERS = {"Authorization": "Bearer Oracle"}

		CONNECTIVITY_URLS_DATA = [

		@@ -815,16 +815,19 @@ def test_multiple_files(self):


		@pytest.mark.usefixtures("disable_netdev_info")
		@mock.patch("cloudinit.net.ephemeral._check_connectivity_to_imds")

add oracle ipv6 single stack imds functionality #5785

add oracle ipv6 single stack imds functionality #5785

Conversation

a-dubs commented Oct 4, 2024 • edited Loading

Proposed Commit Message

Additional Context

Test Steps

Testing Backwards compatability:

Integration testing:

Merge type

TheRealFalcon left a comment • edited Loading

Choose a reason for hiding this comment

a-dubs commented Nov 8, 2024

TheRealFalcon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

a-dubs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealFalcon commented Nov 12, 2024

TheRealFalcon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealFalcon commented Nov 12, 2024

TheRealFalcon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealFalcon Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holmanb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holmanb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealFalcon left a comment

Choose a reason for hiding this comment

a-dubs commented Oct 4, 2024 •

edited

Loading

TheRealFalcon left a comment •

edited

Loading

TheRealFalcon Nov 13, 2024 •

edited

Loading