ref(project): Treat invalid project states as pending and unify state types #3770

jjbayer · 2024-06-27T14:15:19Z

We currently deal with different envelope states inconsistently: Because the state of a project is encoded into many separate fields and types, it's hard to enforce that the config is validated correctly everywhere that it is used.

For example, invalid (i.e. unparsable) project configs are sometimes treated the same as pending, sometimes the same as disabled:

relay/relay-server/src/services/project.rs

Lines 308 to 312 in b561ec3

    
           // if we recorded an invalid project state response from the upstream (i.e. parsing 
        
           // failed), discard the event with a state reason. 
        
           if self.invalid() { 
        
               return Err(DiscardReason::ProjectState); 
        
           }

relay/relay-server/src/services/project.rs

Lines 852 to 855 in b561ec3

    
           // If the state is still invalid, return back the taken channel and schedule state update. 
        
           if state.invalid() { 
        
               self.state_channel = Some(channel); 
        
               let attempts = self.backoff.attempt() + 1;

New types

classDiagram

Project "1" *-- "1" ProjectFetchState

  ProjectState <|-- Enabled
  ProjectState <|-- Disabled
  ProjectState <|-- Pending

  Enabled "1" *-- "1" ProjectInfo

ProjectInfo "1" *-- "1" ProjectConfig

  ProjectFetchState *-- ProjectState
  ProjectFetchState: -last_fetch

  ProjectFetchState --> "computes" ExpiryState
  ExpiryState --> "Exposes" ProjectState

Future Work

Replace GetOrFetch by ProjectState.
Related to the above: Pending does not currently guarantee that another "fetch" is scheduled. We might be able to enforce this by typing Pending as Pending(StateReceiver).
Replace "tight" retry loop for pending with the same backoff mechanism we use for unparsable projects. This might require setting a custom intitial backoff.
Introduce an explicit ProjectState::Allowed state to special case proxy relays.

Fixes https://github.com/getsentry/team-ingest/issues/364.

…eanup

jjbayer · 2024-07-16T13:24:08Z

relay-server/src/endpoints/common.rs

            .enter(
                envelope,
                state.outcome_aggregator().clone(),
                state.test_store().clone(),
                group,
            )
            .map_err(BadStoreRequest::QueueFailed)?;
+        envelope.scope(scoping);


Previously the split-out envelope was scoped by a call to check_envelope in handle_processing. That function takes a valid ProjectInfo now, so we have to do it here.

jjbayer · 2024-07-16T14:02:11Z

relay-server/src/endpoints/project_configs.rs

+    #[serde(with = "LimitedProjectInfo")]
+    #[serde(flatten)]
+    info: ProjectInfo,
+}


I introduced a helper struct ParsedProjectState to allow parsing the wire format in different places. Ideally this would be hidden away in the Serialize / Deserialize implementation, but there is one location (load_local_states) where we actually need to be able to parse a disabled state plus project keys.

jjbayer · 2024-07-16T14:04:05Z

relay-server/src/envelope.rs

+            )
+        })?;
+        self.dsc().map(|dsc| dsc.public_key)
+    }


I needed a queue_key() utility, so I decided to make sampling_key() a method as well, instead of a standalone function.

jjbayer · 2024-07-16T14:06:28Z

relay-server/src/services/processor.rs

@@ -74,7 +74,7 @@ use crate::service::ServiceError;
 use crate::services::global_config::GlobalConfigHandle;
 use crate::services::outcome::{DiscardReason, Outcome, TrackOutcome};
 use crate::services::processor::event::FiltersStatus;
-use crate::services::project::ProjectState;
+use crate::services::project::ProjectInfo;


ProjectState is now an enum. ProjectInfo now contains most of the old fields of the old ProjectState struct, minus disabled.

jjbayer · 2024-07-16T14:08:43Z

relay-server/src/services/processor/dynamic_sampling.rs

@@ -279,7 +279,7 @@ mod tests {
    use crate::services::processor::{
        ProcessEnvelope, ProcessingExtractedMetrics, ProcessingGroup, SpanGroup,
    };
-    use crate::services::project::ProjectState;
+    use crate::services::project::ProjectInfo;


ProjectState is now an enum. ProjectInfo contains the fields that ProjectState used to have (project_id, config, etc.), but is only accessible if the state is Enabled.

jjbayer · 2024-07-16T14:58:00Z

relay-server/src/services/project_cache.rs

+                        // Unspooled envelopes need to be checked, just like we do on the fast path.
+                        if let Ok(CheckedEnvelope { envelope: Some(envelope), rate_limits: _ }) = metric!(timer(RelayTimers::ProjectCacheTaskDuration), task = "handle_check_envelope", {
+                            broker.handle_check_envelope(CheckEnvelope::new(managed_envelope))
+                        }) {


This is new.

relay-server/src/services/project_cache.rs

jjbayer · 2024-07-16T15:00:31Z

relay-server/src/services/project_cache.rs

-        let ValidateEnvelope { envelope: context } = message;
+        // TODO: check_envelope here?
+
+        let ValidateEnvelope {


All the project checks are now done in handle_validate_envelope. There is no more redundant check in handle_processing.

jjbayer · 2024-07-16T15:24:04Z

relay-server/src/services/project_local.rs

-            states.insert(key.public_key, Arc::new(sanitized.clone()));
+            let mut state = state.clone();
+            state.info.public_keys = smallvec::smallvec![key.clone()];
+            states.insert(key.public_key, ProjectState::from(state).sanitize());


This makes it hard to make ParsedProjectState a private concern of the (de)serializer: We need to peek into the config to get the project key.

relay-server/src/services/project_upstream.rs

Dav1dde

Do we have an integration test for invalid?.
Don't forget to update the PR title.
I think we should have a changelog entry for this, quite a big change.

relay-server/src/envelope.rs

relay-server/src/endpoints/project_configs.rs

Dav1dde · 2024-07-17T11:56:47Z

relay-server/src/services/project/state.rs

+    /// - The upstream returned "pending" for this project (see [`crate::services::project_upstream`]).
+    /// - The upstream returned an unparsable project so we have to try again.
+    /// - The project has expired and must be treated as "has not been fetched".
+    Pending,


Wonder if Missing would be a better name, e.g. when a project is expired you return Pending, but there is no gurantuee it's actually pending, it's just missing.

I'm a bit hesitant to call it "missing" because when a projectconfig response does not contain a key, we treat the project as disabled and we previously called this state "missing", see

relay/relay-server/src/services/project.rs

Lines 122 to 127 in c9bd1f8

/// Project state for a missing project.

pub fn missing() -> Self {

ProjectState {

project_id: None,

last_change: None,

disabled: true,

Dav1dde · 2024-07-17T11:57:36Z

relay-server/src/services/project.rs

-                self.project_key
-            );
+        let is_enabled = match self.current_state() {
+            ProjectState::Enabled(info) => info.has_feature(Feature::CustomMetrics),


I think it makes sense to implement has_feature on the ProjectState. This match + check happens at least twice already.

I like the idea, but how would you handle the Pending variant? In this file alone we have two different behaviors (treat as enabled vs. early return).

Fair point, I didn't look that deep, I assumed pending = false. If that is just becomes a footgun, better not do it.

relay-server/src/services/project.rs

Dav1dde · 2024-07-17T12:13:39Z

relay-server/src/services/project/state.rs

+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ParsedProjectState {


Can we make this private and have the (de)-serialize impl of ProjectState use this?

Possibly tricky with the limited versions though

I had it hidden in a mod that containted the (De)Serialize implementations, but unfortunately, project_local needs to look inside the config even for disabled states. See https://github.com/getsentry/relay/pull/3770/files/3452ab1d4af042327565421a8297cfa4afbb83e4#r1679462820

Dav1dde · 2024-07-17T12:15:52Z

relay-server/src/endpoints/project_configs.rs

@@ -39,6 +39,15 @@ struct VersionQuery {
    version: u16,
 }

+#[derive(Debug, Clone, Serialize)]
+#[serde(rename_all = "camelCase", remote = "ParsedProjectState")]
+struct LimitedParsedProjectState {


Should we move this to where ParsedProjectState lives?

It's only required for serialization in the endpoint, so it should in theory be private. But it makes sense to keep the Limited* types next to their unlimited types to keep them in sync, so 👍.

Dav1dde · 2024-07-17T12:18:08Z

relay-server/src/services/project/state/fetch_state.rs

+    /// This state is used for forwarding in Proxy mode.
+    pub fn allowed() -> Self {
+        Self::enabled(ProjectInfo {
+            project_id: None,


Does it make sense to eliminate this in a follow-up, I think a ProjectInfo with a non optional project_id and organization_id could reduce some cruft and possibly also help solve our problem where feature flags in proxy mode are treated as disabled.

Good idea, will note this as a follow-up.

Dav1dde · 2024-07-17T12:19:14Z

relay-server/src/services/project/state/fetch_state.rs

+    }
+
+    /// Sanitizes the contained project state. See [`ProjectState::sanitize`].
+    pub fn sanitize(self) -> Self {


Suggested change

pub fn sanitize(self) -> Self {

pub fn sanitized(self) -> Self {

Seems more like a transition than a mutation

relay-server/src/services/project_upstream.rs

This reverts commit 96edc4e.

…eanup

jjbayer · 2024-07-18T09:19:48Z

Do we have an integration test for invalid?

@Dav1dde there is test_unparsable_project_config, but on closer look I don't understand why it passed before on master, will investigate.

Don't forget to update the PR title.

Any suggestions? Something like "Treat invalid project states as pending"? Or rather "ProjectState enum"?

Dav1dde

Dav1dde · 2024-07-18T09:53:43Z

Any suggestions? Something like "Treat invalid project states as pending"? Or rather "ProjectState enum"?

Maybe something like: ref(project): Treat invalid project states as pending and unify project states. The functional change and the motivation.

jjbayer · 2024-07-18T11:57:52Z

tests/integration/test_projectconfigs.py

-        }
-    } == data
+    data = request_config(relay, packed, signature, version="3").json()
+    assert data == {"configs": {}, "pending": [public_key]}


Relay was already configured to buffer envelopes for unparsable projects, but downstream relays would get them as disabled and drop data anyway. This is now fixed.

This started buffering all envelopes to memory on S4S, need to investigate why. Reverts #3770.

… types (#3770) We currently deal with different envelope states inconsistently: Because the state of a project is encoded into many separate fields and types, it's hard to enforce that the config is validated correctly everywhere that it is used. For example, invalid (i.e. unparsable) project configs are sometimes treated the same as pending, sometimes the same as disabled.

Fixes and reapplies #3770. Above PR had a bug that created a tight spool-unspool loop for disabled projects.

jjbayer added 18 commits June 18, 2024 16:58

state -> info

dc191af

Split into ProjectFetchState and ProjectInfo

23819cd

wip

6354f02

wip

b1b551e

Merge remote-tracking branch 'origin/master' into ref/server-state-cl…

9eb1a3c

…eanup

fix: limited config

c75f596

wip

2fe1ef8

wip

d8e4661

wip

d97864e

wip

f54cc25

ref: check_envelope

f144966

fix

c26ab17

cargo check passes

41d5305

it compiles

6aadc85

Merge remote-tracking branch 'origin/master' into ref/server-state-cl…

6e8730a

…eanup

fix: lint

0f5a299

fix: Initialize project with expired state.

ad97f80

Merge remote-tracking branch 'origin/master' into ref/server-state-cl…

0fefe75

…eanup

jjbayer changed the title ~~Ref/server state cleanup~~ Unify project states Jul 9, 2024

jjbayer mentioned this pull request Jul 9, 2024

ref(server): Better state transitions in project::State #3728

Closed

jjbayer added 10 commits July 10, 2024 11:19

get rid of non_expired_state

72321f8

simplify

d9cb3dc

rm unused function

da5a98e

lint

ab3384d

fix: expiry

8218dfc

fix: Parse disabled

5abb665

ref: rm handle_processing

90dffe2

fix: Outcomes for disabled state

946939b

Merge remote-tracking branch 'origin/master' into ref/server-state-cl…

80f9ef8

…eanup

fix: Check envelope

97f67c1

jjbayer added 3 commits July 16, 2024 14:24

doc

8cbf021

lint

5c50153

Little things

3452ab1

jjbayer commented Jul 16, 2024

View reviewed changes

jjbayer marked this pull request as ready for review July 16, 2024 15:42

jjbayer requested a review from a team as a code owner July 16, 2024 15:42

feat: Use existing backoff loop for pending

96edc4e

Dav1dde reviewed Jul 17, 2024

View reviewed changes

jjbayer added 9 commits July 17, 2024 16:19

Revert "feat: Use existing backoff loop for pending"

c49469d

This reverts commit 96edc4e.

Use backoff loop for invalid

018e356

doc

c3ac129

Merge remote-tracking branch 'origin/master' into ref/server-state-cl…

e5e0b8a

…eanup

changelog

d5e984d

ref: QueueKey

4baf239

ref: Move limited

f192372

ref: review comments

1510b07

lint: doc

8ffb253

jjbayer requested a review from Dav1dde July 18, 2024 09:10

Dav1dde approved these changes Jul 18, 2024

View reviewed changes

jjbayer commented Jul 18, 2024

View reviewed changes

jjbayer changed the title ~~Unify project states~~ ref(project): Treat invalid project states as pending and unify state types Jul 18, 2024

jjbayer merged commit 87ceb2c into master Jul 18, 2024
24 checks passed

jjbayer deleted the ref/server-state-cleanup branch July 18, 2024 12:01

jjbayer mentioned this pull request Jul 18, 2024

fix(server): Revert project state refactor #3831

Merged

jjbayer added a commit that referenced this pull request Jul 18, 2024

fix(server): Revert project state refactor (#3831)

f9c8227

This started buffering all envelopes to memory on S4S, need to investigate why. Reverts #3770.

jjbayer mentioned this pull request Jul 23, 2024

ref(server): Fix and re-apply #3770 #3854

Merged

jjbayer added a commit that referenced this pull request Jul 24, 2024

ref(server): Fix and re-apply #3770 (#3854)

441d178

Fixes and reapplies #3770. Above PR had a bug that created a tight spool-unspool loop for disabled projects.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref(project): Treat invalid project states as pending and unify state types #3770

ref(project): Treat invalid project states as pending and unify state types #3770

jjbayer commented Jun 27, 2024 •

edited

Loading

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

jjbayer Jul 16, 2024

Dav1dde left a comment

Dav1dde Jul 17, 2024

jjbayer Jul 18, 2024

Dav1dde Jul 17, 2024

jjbayer Jul 18, 2024

Dav1dde Jul 18, 2024

Dav1dde Jul 17, 2024

jjbayer Jul 18, 2024

Dav1dde Jul 17, 2024

jjbayer Jul 17, 2024

Dav1dde Jul 17, 2024

jjbayer Jul 18, 2024

Dav1dde Jul 17, 2024

jjbayer commented Jul 18, 2024

Dav1dde left a comment

Dav1dde commented Jul 18, 2024

jjbayer Jul 18, 2024

	// if we recorded an invalid project state response from the upstream (i.e. parsing
	// failed), discard the event with a state reason.
	if self.invalid() {
	return Err(DiscardReason::ProjectState);
	}

	// If the state is still invalid, return back the taken channel and schedule state update.
	if state.invalid() {
	self.state_channel = Some(channel);
	let attempts = self.backoff.attempt() + 1;

	/// Project state for a missing project.
	pub fn missing() -> Self {
	ProjectState {
	project_id: None,
	last_change: None,
	disabled: true,

	pub fn sanitize(self) -> Self {
	pub fn sanitized(self) -> Self {

ref(project): Treat invalid project states as pending and unify state types #3770

ref(project): Treat invalid project states as pending and unify state types #3770

Conversation

jjbayer commented Jun 27, 2024 • edited Loading

New types

Future Work

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dav1dde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjbayer commented Jul 18, 2024

Dav1dde left a comment

Choose a reason for hiding this comment

Dav1dde commented Jul 18, 2024

Choose a reason for hiding this comment

jjbayer commented Jun 27, 2024 •

edited

Loading