Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit logs with workflow execution tag for timedout frontend requests #4379

Merged
merged 2 commits into from
Aug 24, 2021

Conversation

yycptt
Copy link
Contributor

@yycptt yycptt commented Aug 17, 2021

What changed?

  • Emit log with domain/workflow/runID tag for timedout frontend requests
  • For worker related APIs, count the request toward the domain level rate limiter

Why?

  • Make investigation of request timeout easier

How did you test it?

  • Unit test

Potential risks

  • Pressure on logging
  • Customer may likely encounter more service busy errors

Release notes

Documentation Changes

@yycptt yycptt requested a review from a team August 17, 2021 17:22
}
// Count the request in the RPS towards the requested domain,
// but we still accept it even if RPS is exceeded
wh.allow(domainWrapper)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously this wh.allow(nil) which won't consume RPS token for the requested domain, only token for the global RPS is consumed.

@coveralls
Copy link

coveralls commented Aug 17, 2021

Pull Request Test Coverage Report for Build b7be0815-effb-405a-afa6-a559b7981e09

  • 215 of 462 (46.54%) changed or added relevant lines in 1 file are covered.
  • 181 unchanged lines in 24 files lost coverage.
  • Overall coverage decreased (-0.03%) to 56.467%

Changes Missing Coverage Covered Lines Changed/Added Lines %
service/frontend/workflowHandler.go 215 462 46.54%
Files with Coverage Reduction New Missed Lines %
client/history/client.go 2 41.25%
client/history/metricClient.go 2 47.05%
common/persistence/executionManager.go 2 75.5%
common/persistence/statsComputer.go 2 95.71%
common/quotas/multistageratelimiter.go 2 90.91%
service/history/handler.go 2 48.15%
common/persistence/dataManagerInterfaces.go 3 62.54%
common/persistence/nosql/nosqlplugin/cassandra/workflow.go 3 50.23%
common/task/fifoTaskScheduler.go 3 82.47%
service/history/queue/timer_gate.go 3 95.83%
Totals Coverage Status
Change from base Build 98dbb8b1-620b-4037-869f-02b0390f9c76: -0.03%
Covered Lines: 78924
Relevant Lines: 139771

💛 - Coveralls

@yycptt yycptt merged commit 76ec20a into cadence-workflow:master Aug 24, 2021
@yycptt yycptt deleted the log-frontend-timeout branch August 24, 2021 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants