Slow performance after docker upgrade

[mekarpeles]

Summary

• What is wrong?

tl;dr: apt-get install apparmor

Site is extremely slow, solr auto-restarting frequently, 503s for merge queue (EDIT: some of these issues preceded docker upgrade)

We checked on

• What caused it?

At ~11am PT @cclauss performed a docker upgrade on ol-home0 re: #7626 (comment). The upgrade was completed successfully however no containers were running.

Initially, we suspected ~performance issues with ol-www0? We ran a sudo docker restart openlibrary_web_nginx_1 openlibrary_web_haproxy_1 on ol-www0 which seemed to work for a moment.

• What fixed it?

We ssh'ed to ol-home0 and noticed no containers running via docker ps. Identified this was related to the recent docker upgrade. When we tried to manually restart containers:

cd /opt/openlibrary
export COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml"
export    HOSTNAME="$HOSTNAME"
docker compose --profile ol-home0 up -d

And failed with error:

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply AppArmor profile: AppArmor failed to apply profile: write /proc/self/attr/apparmor/exec: no such file or directory: unknown

We asked our newest team member ChatGPT for help and didn't get great suggestions however, we found a hint in docker/for-linux#1199 which suggested apt install AppArmor. After a restart, and re-running of the commands above, the systems were back up!

• Meta Problems

• Notifications when big production changes to #openlibrary
• Attempt install in dev environment prior to production
• Upgrade

• Followup actions:

https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture#performing-upgrades

Steps to close

1. Assignment: Is someone assigned to this issue? (notetaker, responder)
2. Labels: Is there an Affects: label applied?
3. Diagnosis: Add a description and scope of the issue
4. Updates: As events unfold, is notable provenance documented in issue comments? (i.e. useful debug commands / steps / learnings / reference links)
5. "What caused it?" - please answer in summary
6. "What fixed it?" - please answer in summary
7. "Followup actions:" actions added to summary