BUG - sev-2 - All - Claims status - (Anecdotally) slower loading times, more frequent errors with LH than EVSS

[TKDickson]

What happened?

When my staging users are flipped to the Lighthouse claims status APIs, loading times are taking longer (list of claims/index, and individual claims details/{id}) and I'm more often running into generic 500 errors, which Andrew investigated and look like they're timeouts coming from upstream.

He's pinging the benefits team for more context.

Specs:

• Device:
• OS:
• User Account:
• Accessibility State:

Steps to Reproduce

Desired behavior

Acceptance Criteria

Bug Severity - BE SURE TO ADD THE SEVERITY LABEL

See Bug Tracking for details on severity levels

• Impact: High Low
• Frequency: High Low

• 3 - Low
• 2 - High
• 1 - Critical

Linked to Story

Found testing #4745

Screen shot(s) and additional information

Ticket Checklist

• Steps to reproduce are defined
• Desired behavior is added
• Labels added (front-end, back-end, relevant feature, accessibility, etc)
• Estimate of 1 added (for front-end tickets)
• Added either to the relevant feature epic (if found during new feature implementation), or the bug epic

[TKDickson]

https://dsva.slack.com/archives/C02CQP3RFFX/p1684952308474089

[aherzberg]

There does look to be some high error rates on LH claims. LH is not seeing any issues on their side so it's hard to tell the source of the issue. also can't tell if the error rates are coming from the index or detail endpoint, though anecdotally it sounds like it's the index. We are doing some more complex logic here where we call the claims index endpoint in parallel (async) with the appeals. this may be causing the issue?

[TKDickson]

@aherzberg is this testable? Are you planning on monitoring to see if the 500s go down, and then moving it into the with QA column if the answer is 'yes'? (Or some other plan?)

[aherzberg]

yes, I was going to move into QA column once I saw the error rate went down to normal levels to ensure there isn't a behavioral difference between our BE tester scripts and the real deal. It'll only be testable in the sense that you anecdotally will see a lot less errors when testing on the app.

[aherzberg]

Had some issues getting logs from our mobile tester. merged in a PR today to fix that. Still unsure of the solution to this issue as it's a timeout in the upstream service but it seems to only happen for mobile but I can't confirm this since the benefits team doesn't have a dashboard of error rates.

[aherzberg]

Putting in blocked until Benefits team puts this in prod. Once it's in prod I will test a percentage of users on the mobile side in prod to see if this is a staging only issue.

[aherzberg]

still waiting on benefits team to put in prod.

[aherzberg]

Benefits team is still working out prod issues today, moving back to blocked until resolved.

[jperk51]

@aherzberg Any updates on this?

[aherzberg]

Just talked to Jerek. Still 1% on production on their side.

[aherzberg]

They are also seeing some timeouts on their side. so makes me think the issue is not mobile specific but we're just hitting our endpoints a lot more in staging due to our mobile tester.

[jperk51]

@kellylein What's your opinion here? Seems like this is beyond our control plus Jerek and crew are aware of it. Should we close this and move forward with the phased rollout on our side? it does seem like this would occur in prod, but who knows how often. We can switch on X% and monitor this. If it's happening too often we can turn it back off and readdress it with the LH and claims teams.

[kellylein]

I'd like to keep this around just until VA.gov ups their rollout percentage enough that we can confirm they are seeing the same thing. If it isn't just us then it's clearly at the LH layer and we will close this.

[jperk51]

Jacob Worrell let us know they are also seeing this issue on their side which means it's an issue upstream of us and LH is already aware and trying to fix it. Closing this ticket.