Solr performance issues

[seabelis]

Multiple folks have noted solr performance issues as noted below.

From community call notes...
Autocomplete

• adding authors
• Languages
• Changing work IDs for moving a work
• Subjects
• Searching by ISBN; there’s a delay before the preview appears (main nav search box)

Consistent in the past ~month
Might be everything in this autocomplete

Lisa: Searching frequently produces no results but then searching the same terms again produces results
+Onno
Mark: First time I load OL in the day doesn’t work
-Lisa

Evidence / Screenshot (if possible)

Relevant url?

Steps to Reproduce

1. Go to ...any edit form
2. Do ...try adding an author or language

• Actual: delay is longer than expected and longer than it has been in the past
• Expected:

Details

• Logged in (Y/N)?
• Browser type/version?
• Operating system?
• Environment (prod/dev/local)? prod

Proposal & Constraints

Related files

Stakeholders

@cdrini

[mekarpeles]

Similar to #6377 epic

[tfmorris]

Performance is erratic and miserable. I'd suggest adding latency metrics and monitoring them so that you don't have to wait for users to spend a month being pissed off.

Some of the symptoms I've seen include:

• autocomplete taking several seconds

• page renders failing with obscure errors (e.g. "plural must be integer, not None") because solr delivered information isn't available
  

• author merge page rendering with no works for any author (but also no error message)

• author merge pages rendering with no works for some of the authors - makes me wonder if the missing works will fail to get moved if I don't notice and refresh the page before merging

• page renders failing with "connection retry count exceeded" trying to connect to solr backend

• author search page listing authors with none of their works (and a count of 1 - another longstanding bug)

[cdrini]

Howdy folks I've patch deployed a number of solr config improvements which should address these last night. Please let me know if things look better or if they do not!

[cdrini]

Howdy folks, our sentry metrics are looking muuuuch better; way fewer errors!

This should address a lot of the "things appearing with 0 results", "got NoneType", etc.

And search load times are stable in our grafana. Closing this for now since it's been a week and things look better.

Does autocomplete perf look better, @seabelis @tfmorris ?

[tfmorris]

Yes, the site is usable again now. I miss having the updated info available immediately in search after edits, but it's only a minute or two delayed, and better than having the site be unusable.

Since there are apparently metrics in place already (based on the graph above), I'd suggest setting an alarm threshold so that devs get notified when they break something.