Add solr restarter service to autoheal solr

[cdrini]

Closes #5343

This service responds to errors on the production endpoint, and restarts solr accordingly. This is ideally code we shouldn't have to write, but couldn't find how to make this work with docker's healthcheck -- it seemed like that was hard blocked for docker-compose users: https://forums.docker.com/t/unhealthy-container-does-not-restart/105822/3 . Since our healthcheck involves processing some JS as well, created a whole script for it :(

Technical

• https://github.com/internetarchive/olsystem/pull/146 Add .env file to olsystem

Testing

• Tested a near-identical version on prod solr for the past few weeks, and it's been working well!
• Pull on ol-solr0 and try running
• To test locally:

1. Modify docker-compose.production.yml, and comment out env_file and SEND_SLACK_MESSAGE=true
2. Run COMPOSE_FILE="docker-compose.yml;docker-compose.production.yml" docker-compose --profile ol-solr0 up -d
3. Everything should start
4. Check the logs of solr_restarter ; should be saying "healthy".
5. Modify the TEST_URL to be q=asdflkjaslkefj (some gibberish)
6. up again
7. Observer it restarts after 3 health fails

Screenshot

Stakeholders