Cloud support #521
Merged
Cloud support #521
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make public IP configurable for peering, instead of depending on discovery via interface list.
kentquirk
reviewed
Sep 19, 2022
MikeGoldsmith
added
type: enhancement
kentquirk
approved these changes
Sep 27, 2022
|
@Baliedge does this supersede the the etcd PR? |
No. I would say that's a separate effort. We decided to stick to Redis mode for now. Moving to etcd was determined to be a nice to have. We may decide to develop that later if necessary. |
LokeshOpsramp
pushed a commit
to opsramp/tracing-proxy
that referenced
this pull request
Jul 5, 2024
## Which problem is this PR solving? When hosting Refinery in a cloud environment, certain network assumptions are no longer true. Issues arise from these assumptions while hosting in our cloud environment using [Nomad](https://www.nomadproject.io/) as container orchestration. Terms, as used in Nomad: - Task: Single deployed container. - Group: Group of tasks running as a scalable service (e.g. Refinery). This assumption is not necessarily true under Nomad: - The IP assigned to a task's `eth0` can be accessed by other tasks in the group. Redis peering expects peers to talk to each other using their `eth0` IP. This behavior can be overridden by specifying `RedisIdentifier` in config. This value will be advertised as the peer's IP in the peer list. However, on startup, peering uses its `eth0` IP to cross-reference in the peers list found in Redis. Since the internal `eth0` IP is not the true public IP (set by `RedisIdentifier`), this check will always fail and the container exits in error. ## Short description of the changes A workaround is to treat `RedisIdentifier` as the true public IP address of the task. Thus, it should be one of the "self" IPs when checking the peers list. This behavior is reinforced in existing code in [`internal/peer/redis.go`](https://github.com/honeycombio/refinery/blob/main/internal/peer/redis.go#L291-L298) where `RedisIdentifier` is used to create a URL with the value being the host.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which problem is this PR solving?
When hosting Refinery in a cloud environment, certain network assumptions are no longer true.
Issues arise from these assumptions while hosting in our cloud environment using Nomad as container orchestration.
Terms, as used in Nomad:
This assumption is not necessarily true under Nomad:
eth0can be accessed by other tasks in the group.Redis peering expects peers to talk to each other using their
eth0IP. This behavior can be overridden by specifyingRedisIdentifierin config. This value will be advertised as the peer's IP in the peer list.However, on startup, peering uses its
eth0IP to cross-reference in the peers list found in Redis. Since the internaleth0IP is not the true public IP (set byRedisIdentifier), this check will always fail and the container exits in error.Short description of the changes
A workaround is to treat
RedisIdentifieras the true public IP address of the task. Thus, it should be one of the "self" IPs when checking the peers list.This behavior is reinforced in existing code in
internal/peer/redis.gowhereRedisIdentifieris used to create a URL with the value being the host.