503s Solr Performance Issues

[mekarpeles]

Summary

• What is wrong?

Since ~Jan 1, 2025, the website has been seeing lots of 503s and slow performance

• Evidence

(Annotations needed on each)

Sentry performance shows increase in request load time

Digging into book page, solr seems to be the bottleneck

Solr strain evident on CPU charts

Increase in errors (related to solr 503s in sentry):

Load times correlated with decrease in page views and error response codes:

• What caused it?

A part of our solr re-indexing flow is to run an index optimize. If we don't rebuild the frequently enough, the current index may become fragmented.

Possible a lack of optimization step on our index is partially responsible for the problem.

There was also low free disk space

• What fixed it?

• What was the impact?

Degraded website performance over multiple days as solr traffic increased

Followup / What could have gone better

Monitoring of:

• solr disk space
• solr total # requests
• solr average response time
• stats.inc every time solr restarted

More routine:

• docker prune (to be handled by Add ol-solr0 to deploy flow #10239

• running solr index optimize (and possibly automated monthly re-indexing)

• Followup actions:

Steps to close

1. Assignment: Is someone assigned to this issue? (notetaker, responder)
2. Labels: Is there an Affects: label applied?
3. Diagnosis: Add a description and scope of the issue
4. Updates: As events unfold, is notable provenance documented in issue comments? (i.e. useful debug commands / steps / learnings / reference links)
5. "What caused it?" - please answer in summary
6. "What fixed it?" - please answer in summary
7. "Followup actions:" actions added to summary

[mekarpeles]

Optimize took ~2.5h, the last re-index was ~July 2024 (so ~6 months)

[mekarpeles]

Example of optimize command:

openlibrary/scripts/solr_builder/Jenkinsfile

Lines 316 to 324 in e8215dc

	stage('Optimize indices') {
	// This is kind of like a disk defrag, but it's ESSENTIAL after a big import. Shouldn't
	// really be needed after that.
	steps {
	dir(env.HOST_SOLR_BUILDER_DIR) {
	sh "docker compose exec -T solr curl -s 'http://localhost:8983/solr/openlibrary/update?optimize=true&maxSegments=1'"
	}
	}
	}

[tfmorris]

Anecdotally, the site was completely unusable while the index optimization task was (presumably) being run yesterday, with a pasted OL..A identifier taking multiple seconds to resolve in the work edit author autocomplete. If possible, it would be good to schedule future reindexing/index optimization runs for off-peak times.

Also, it seems a little strange that 6 months worth of index fragmentation would suddenly become a huge issue in the last two weeks, so I would keep an eye out for indications that it wasn't the (only) problem.