feat: enable user feedback feature #3193

aldy505 · 2024-07-10T00:39:39Z

This is based on https://develop.sentry.dev/feedback-architecture/ and some other stuff that I've used since User Feedback is in beta.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

codecov · 2024-07-10T01:02:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.02%. Comparing base (cd7c460) to head (ef7bb23).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3193      +/-   ##
==========================================
- Coverage   99.01%   98.02%   -0.99%     
==========================================
  Files           3        3              
  Lines         203      203              
==========================================
- Hits          201      199       -2     
- Misses          2        4       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bruno-garcia · 2024-07-10T14:16:51Z

Thanks @aldy505 !

aliu39 · 2024-07-10T18:50:23Z

Thanks @aldy505! This relates to a PR @hubertdeng123 and I were working on a while back: #3060. We had some problems getting relay to accept the envelopes when manually testing, something CORS related.

You'll also need the option in #3061, though we can do that in a separate PR

aldy505 · 2024-07-11T00:33:49Z

Thanks @aldy505! This relates to a PR @hubertdeng123 and I were working on a while back: #3060. We had some problems getting relay to accept the envelopes when manually testing, something CORS related.

@aliu39 How did you guys encounter the CORS issue?

You'll also need the option in #3061, though we can do that in a separate PR

I don't quite get it why it's not 1.0 yet, since it's GA already...

aliu39 · 2024-07-11T01:08:14Z

Thanks @aldy505! This relates to a PR @hubertdeng123 and I were working on a while back: #3060. We had some problems getting relay to accept the envelopes when manually testing, something CORS related.

@aliu39 How did you guys encounter the CORS issue?

You'll also need the option in #3061, though we can do that in a separate PR

I don't quite get it why it's not 1.0 yet, since it's GA already...

It's been a while so I don't remember the details.. cc @hubertdeng123. I can look into it again later this week

It is 1.0 for our SaaS site, but is 0 for self-hosted since we SH doesn't have the topic which your PR adds.

hubertdeng123 · 2024-07-11T17:18:44Z

Thanks for putting this PR up @aldy505. I'll work with @aliu39 and look into this to get this enabled

hubertdeng123 · 2024-07-11T23:34:44Z

Testing this locally, but encountering this error:
getsentry/sentry#67412

It appears that hitting http://127.0.0.1:9000/organizations/sentry/feedback/?feedbackSlug=${project_slug}:${feedback_event_id} works.

However, http://127.0.0.1:9000/organizations/sentry/feedback/ does not.

aldy505 · 2024-07-12T00:17:28Z

Testing this locally, but encountering this error: getsentry/sentry#67412

It appears that hitting http://127.0.0.1:9000/organizations/sentry/feedback/?feedbackSlug=${project_slug}:${feedback_event_id} works.

However, http://127.0.0.1:9000/organizations/sentry/feedback/ does not.

It's an old known bug. Do you want to wait for the fix for it before we add the feature into SH?

aliu39 · 2024-07-12T02:38:32Z

feedback-ingest
feedback-post-process-group
Found out these flags are needed for issues platform and should be added to this PR.

We still need to some time to debug the new topic, but it's not needed to enable this feature. To add this feature, technically all we need are the feature flags (can keep the option at 0.0).

It'd definitely be nice to fix the frontend bug. We're making progress! I think you can merge this PR independent of that, though.

aldy505 · 2024-07-12T03:56:23Z

feedback-ingest feedback-post-process-group Found out these flags are needed for issues platform and should be added to this PR.

Are you sure you're specifying the right flag? There's no such flag as that on features/temporary.py and features/permanent.py. The feature is working as intended on my side, testing it on my development environment (an actual server) using JS SDK v8.

We still need to some time to debug the new topic, but it's not needed to enable this feature. To add this feature, technically all we need are the feature flags (can keep the option at 0.0).

The topic should be created automatically though, see #3121

aliu39 · 2024-07-12T05:01:58Z

Yeah the topic is created but we were having some problems ingesting thru it.

The flags are right, they are programmatically generated by a feature handler

hubertdeng123 · 2024-07-12T15:33:16Z

It'd definitely be nice to fix the frontend bug. We're making progress! I think you can merge this PR independent of that, though.

@aldy505 Did you encounter the bug that I was facing when testing the feature? I'm wondering if all users running user feedback are experiencing the issue or just me. If all users are facing the issue, then I'd like to avoid merging for now.

aldy505 · 2024-07-12T16:05:59Z

It'd definitely be nice to fix the frontend bug. We're making progress! I think you can merge this PR independent of that, though.

@aldy505 Did you encounter the bug that I was facing when testing the feature? I'm wondering if all users running user feedback are experiencing the issue or just me. If all users are facing the issue, then I'd like to avoid merging for now.

@hubertdeng123 Yeah I'm encountering it. But since I kind of want the feature and ingestion is not a problem, so I think it's still acceptable.

TheDevMinerTV · 2024-07-13T20:30:37Z

From my tests, using the embed (triggered by Sentry.showReportDialog()) works, but using Sentry.captureFeedback() doesn't. There's nothing being logged that it isn't working, but the User Feedback tab in an issue also doesn't show anything.

aldy505 · 2024-07-14T02:16:42Z

An update: This PR didn't work. Seems like we need to postpone this one a bit.😢

@TheDevMinerTV said on Discord, that using Sentry.showReportDialog() from the JS SDK works. This function doesn't submit the feedback event through the regular envelope. Submitting through the envelope via Sentry.captureFeedback() and Sentry.feedbackIntegration() didn't work. I only tested with Sentry.captureFeedback() and Sentry.feedbackIntegration(), both didn't work.

What works so far:

The topic is created as expected. So no issue on Kafka topic creation.
The ingest-feedback consumer group is correctly consuming ingest-feedback-events topic.
Relay correctly produces feedback event to the ingest-feedback-events topic.

I can't check whether the events are correctly ingested to Postgres/Clickhouse since I don't know how to check that.

The feature flags organizations:user-feedback-spam-filter-ingest and organizations:user-feedback-spam-filter-actions turns out calls to a function/service that requires LLM object to be set up. I think we need to remove these. Exception from the worker container (thanks @TheDevMinerTV):

worker-1 | Traceback (most recent call last):
worker-1 |   File "/usr/src/sentry/src/sentry/feedback/usecases/create_feedback.py", line 197, in create_feedback_issue
worker-1 |     is_message_spam = is_spam(event["contexts"]["feedback"]["message"])
worker-1 |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 |   File "/usr/src/sentry/src/sentry/utils/metrics.py", line 229, in inner
worker-1 |     return f(*args, **kwargs)
worker-1 |            ^^^^^^^^^^^^^^^^^^
worker-1 |   File "/usr/src/sentry/src/sentry/feedback/usecases/spam_detection.py", line 36, in is_spam
worker-1 |     response = complete_prompt(
worker-1 |                ^^^^^^^^^^^^^^^^
worker-1 |   File "/usr/src/sentry/src/sentry/llm/usecases/__init__.py", line 75, in complete_prompt
worker-1 |     usecase_config = get_usecase_config(usecase.value)
worker-1 |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 |   File "/usr/src/sentry/src/sentry/llm/usecases/__init__.py", line 90, in get_usecase_config
worker-1 |     raise InvalidUsecaseError(
worker-1 | sentry.llm.exceptions.InvalidUsecaseError: LLM usecase options not found. please check llm.usecases.options
worker-1 | 19:17:48 [ERROR] sentry.feedback.usecases.create_feedback: Error checking if message is spam

Removing both feedback-spam flags didn't fix the issue that feedback isn't showing though.

What me and @TheDevMinerTV tried:

Setting SENTRY_OPTIONS["feedback.ingest-topic.rollout-rate"] to 0 instead of 1.0. Didn't work.

@aliu39 @hubertdeng123 Let me know if you guys want to create a debug session with us for this. (Although I know every one of us live on a very different timezones)

aliu39 · 2024-07-15T21:38:56Z

Thanks so much for debugging in-depth @aldy505 @TheDevMinerTV. Good call on removing the spam flags! There are extra settings for that feature not supported in SH yet.

Hubert and I are in US Pacific time (PST). I'll try the tests you mentioned and see if I reproduce the same result.

aldy505 · 2024-07-20T02:31:05Z

I'm going to prioritize something else, until we can finally have the time to debug all this.

bruno-garcia · 2024-07-25T19:23:19Z

Related?

[self-hosted] User feedback: Something went horribly wrong rendering this page sentry#70136

aldy505 · 2024-07-26T03:13:22Z

Since getsentry/sentry@ee95b68 is included in 24.7.1, I'm gonna debug everything this weekend. Go to #self-hosted channel on Discord to see me (potentially) spamming everything about User Feedback.

aldy505 · 2024-07-29T06:43:43Z

Debugging update (not really an update since this is not a progress at all):

So everything is correctly working during:

Event sent to Relay
Relay then published to "ingest-feedback-events"
A Sentry consumer eats "ingest-feedback-events", convert the payload into "occurrences" object and throw it back to Kafka to "ingest-occurrences" (this bit is on the create_feedback_issue function)
Then another Sentry consumer should eat "ingest-occurrences", and push it to Arroyo, then it's pushed to some topic that I don't know, consumed again by Snuba and finally insert into Clickhouse to some table that I don't know. At this point, I'm in the blind.

This is the Kafka payload on ingest-occurrences topic: https://pastebin.com/ywnedhwi

I have no idea on how this ingest-occurrences consumer works. But I can see my event_id of 62386ca702324ad7b9fada27c4b20379 on outcomes_raw_local table.

SELECT *
FROM outcomes_raw_local
WHERE (project_id = 49) AND (event_id = '62386ca7-0232-4ad7-b9fa-da27c4b20379')

┌─org_id─┬─project_id─┬─key_id─┬───────────timestamp─┬─category─┬─outcome─┬─reason─┬─quantity─┬─event_id─────────────────────────────┬─size─┐
│      1 │         49 │   ᴺᵁᴸᴸ │ 2024-07-29 05:45:48 │       14 │       0 │ ᴺᵁᴸᴸ   │        1 │ 62386ca7-0232-4ad7-b9fa-da27c4b20379 │    0 │
└────────┴────────────┴────────┴─────────────────────┴──────────┴─────────┴────────┴──────────┴──────────────────────────────────────┴──────┘

So on Clickhouse, that project_id + event_id combination only exists on the outcomes_raw_local table. I don't know whether this is a good sign or not.

And... now I'm stuck.

aliu39 · 2024-07-30T21:31:12Z

Hi @aldy505 , I'm looking into this now. The issue platform system isn't owned by us so my knowledge goes about as far as you, between ingest-occurrences and Clickhouse. I can try asking some people from that team though.

Just curious, why are you looking at the outcomes_raw table and how did you find it?

aliu39 · 2024-07-30T22:57:36Z

The correct table is search_issues_local, though it's interesting you got the same event id in outcomes 🤔🤔. I'm able to repro all the steps you did but my search_issues_local table ends up empty. So seems it seems like something's broken in the occurrence consumer.

Btw, I sent you a friend request on discord, feel free to reach out there if it's more convenient!

aliu39 · 2024-07-30T23:09:58Z

feedback-ingest
feedback-post-process-group
Found out these flags are needed for issues platform and should be added to this PR.

Also this didn't solve the issue for me, but remember to add these flags, the first is especially important. Sorry I missed this for the develop docs

aldy505 · 2024-07-30T23:12:29Z

The correct table is search_issues_local, though it's interesting you got the same event id in outcomes 🤔🤔.

Ok so weird thing is.. the data showed up search_issues_v2_local..

aldy505 · 2024-07-31T01:35:05Z

Okay so it finally works on my instance. Can you guys also try it out?

aliu39

Working on my end too!

TheDevMinerTV

Seems to work fine for me as well (can't approve because of GitHub permissions), just the frontend is crashing sometimes, but that doesn't belong here.

hubertdeng123

code looks fine to me, although I have not tested this code locally

aliu39 · 2024-08-02T01:32:09Z

Seems to work fine for me as well (can't approve because of GitHub permissions), just the frontend is crashing sometimes, but that doesn't belong here.

@TheDevMinerTV what kind of crashes are you experiencing?

Is it related to getsentry/sentry#70136 at all? (We think this issue has been closed by a recent PR)

Ref getsentry/sentry#66100 The topic and consumer has been deployed to all regions, and this option is 1.0 everywhere: https://github.com/search?q=repo%3Agetsentry%2Fsentry-options-automator%20topic.rollout&type=code. After getsentry/self-hosted#3193 was merged, this is true for self-hosted as well. We don't have to worry about a migration there because this is the first PR enabling user feedback in SH. Follow-up: remove from - getsentry - self-hosted - finally, options-automator

feat: enable user feedback feature

db0fbd2

aldy505 requested review from bruno-garcia, hubertdeng123 and JoshFerge July 10, 2024 00:39

chore: remove trailing whitespace

b852158

aldy505 mentioned this pull request Jul 10, 2024

Release 24.7.0 #3140

Closed

17 tasks

bruno-garcia requested review from cmanallen and aliu39 July 10, 2024 14:16

BYK approved these changes Jul 10, 2024

View reviewed changes

feat: add 'feedback.ingest-topic.rollout-rate' on SENTRY_OPTIONS

12349c6

aliu39 mentioned this pull request Jul 11, 2024

feat(feedback): create kafka topic and consumer for ingest-feedback #3060

Closed

aliu39 assigned aliu39 and hubertdeng123 Jul 11, 2024

aldy505 marked this pull request as draft July 20, 2024 02:30

Merge branch 'master' into feat/user-feedback

21d3a52

aldy505 added 2 commits July 31, 2024 08:24

feat(user-feedback): use feature flags required by issues platform

3efbbeb

Merge branch 'master' into feat/user-feedback

ef7bb23

aldy505 marked this pull request as ready for review July 31, 2024 01:35

aliu39 approved these changes Jul 31, 2024

View reviewed changes

aliu39 mentioned this pull request Jul 31, 2024

chore(feedback): Update info on self hosted flags and support getsentry/sentry-docs#10942

Merged

TheDevMinerTV reviewed Aug 1, 2024

View reviewed changes

hubertdeng123 approved these changes Aug 1, 2024

View reviewed changes

hubertdeng123 merged commit 0b11564 into getsentry:master Aug 1, 2024
11 of 12 checks passed

aldy505 deleted the feat/user-feedback branch August 3, 2024 08:33

bruno-garcia mentioned this pull request Aug 6, 2024

User feedback page sometimes crashes #3240

Closed

aliu39 mentioned this pull request Aug 8, 2024

ref(feedback): clean up feedback topic rollout option getsentry/relay#3916

Merged

github-actions bot locked and limited conversation to collaborators Aug 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: enable user feedback feature #3193

feat: enable user feedback feature #3193

aldy505 commented Jul 10, 2024

codecov bot commented Jul 10, 2024 •

edited

Loading

bruno-garcia commented Jul 10, 2024

aliu39 commented Jul 10, 2024 •

edited

Loading

aldy505 commented Jul 11, 2024

aliu39 commented Jul 11, 2024

hubertdeng123 commented Jul 11, 2024

hubertdeng123 commented Jul 11, 2024

aldy505 commented Jul 12, 2024

aliu39 commented Jul 12, 2024

aldy505 commented Jul 12, 2024

aliu39 commented Jul 12, 2024

hubertdeng123 commented Jul 12, 2024

aldy505 commented Jul 12, 2024 •

edited

Loading

TheDevMinerTV commented Jul 13, 2024

aldy505 commented Jul 14, 2024 •

edited

Loading

aliu39 commented Jul 15, 2024

aldy505 commented Jul 20, 2024

bruno-garcia commented Jul 25, 2024 •

edited

Loading

aldy505 commented Jul 26, 2024

aldy505 commented Jul 29, 2024

aliu39 commented Jul 30, 2024

aliu39 commented Jul 30, 2024

aliu39 commented Jul 30, 2024 •

edited

Loading

aldy505 commented Jul 30, 2024

aldy505 commented Jul 31, 2024

aliu39 left a comment

TheDevMinerTV left a comment

hubertdeng123 left a comment

aliu39 commented Aug 2, 2024 •

edited

Loading

feat: enable user feedback feature #3193

feat: enable user feedback feature #3193

Conversation

aldy505 commented Jul 10, 2024

Legal Boilerplate

codecov bot commented Jul 10, 2024 • edited Loading

Codecov Report

bruno-garcia commented Jul 10, 2024

aliu39 commented Jul 10, 2024 • edited Loading

aldy505 commented Jul 11, 2024

aliu39 commented Jul 11, 2024

hubertdeng123 commented Jul 11, 2024

hubertdeng123 commented Jul 11, 2024

aldy505 commented Jul 12, 2024

aliu39 commented Jul 12, 2024

aldy505 commented Jul 12, 2024

aliu39 commented Jul 12, 2024

hubertdeng123 commented Jul 12, 2024

aldy505 commented Jul 12, 2024 • edited Loading

TheDevMinerTV commented Jul 13, 2024

aldy505 commented Jul 14, 2024 • edited Loading

aliu39 commented Jul 15, 2024

aldy505 commented Jul 20, 2024

bruno-garcia commented Jul 25, 2024 • edited Loading

aldy505 commented Jul 26, 2024

aldy505 commented Jul 29, 2024

aliu39 commented Jul 30, 2024

aliu39 commented Jul 30, 2024

aliu39 commented Jul 30, 2024 • edited Loading

aldy505 commented Jul 30, 2024

aldy505 commented Jul 31, 2024

aliu39 left a comment

Choose a reason for hiding this comment

TheDevMinerTV left a comment

Choose a reason for hiding this comment

hubertdeng123 left a comment

Choose a reason for hiding this comment

aliu39 commented Aug 2, 2024 • edited Loading

codecov bot commented Jul 10, 2024 •

edited

Loading

aliu39 commented Jul 10, 2024 •

edited

Loading

aldy505 commented Jul 12, 2024 •

edited

Loading

aldy505 commented Jul 14, 2024 •

edited

Loading

bruno-garcia commented Jul 25, 2024 •

edited

Loading

aliu39 commented Jul 30, 2024 •

edited

Loading

aliu39 commented Aug 2, 2024 •

edited

Loading