Week 08 routine

[kiwixbot]

Infra

• Check nodes free space

df -h / && df -h /data

• Nodes and worker system upgrades

apt update && apt upgrade

• Ensure all borg repositories are being updated
• Check Pod errors

k get pods -A -o wide|grep Error

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• Upgrade k8s if applicable

Stats

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Mirrorbrain dashboard is normal
• There is no abnormal behaviors on cluster resources consumption

Projects

• UptimeRobot has no alert
• youzim.it backlog is reasonable
• No systematic failure in tasks.
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.7G	27G	25%	+0.1G
stats	233G	102G	119G	47%	+7G, +4%
services	456G	217G	216G	51%	+17G, +4%
storage	33T	17T	15T	54%	-

• stats: anyone has an idea of what happened?

• services: weird than k8s cleanup was not triggered, limit is a 50%, let's see next week

• update nodes packages:

  • stats:
    • bind9 (1:9.16.48-1) over (1:9.16.44-1~deb11u1)
    • bind9-libs:amd64 (1:9.16.48-1) over (1:9.16.44-1~deb11u1)
    • bind9-utils (1:9.16.48-1) over (1:9.16.44-1~deb11u1)
    • bind9-host (1:9.16.48-1) over (1:9.16.44-1~deb11u1)
  • services et storage:
    • idem
    • bind9-dnsutils (1:9.16.48-1) over (1:9.16.44-1~deb11u1)

Upgrade k8s cluster to 1.29.1 : TODO

Zimit

• 5x Python 403 (www.blablacar.ru, www.zillow.com, www.coinbase.com, www.minecraft.wiki + 1 real 403)
• 2x Python read timeout (www.lowes.com, www.yellowpages.com.au)
• 1x Python name resolution error (www.uticaschools.org)
• 2x Python 404 (ageofempires.fandom.com + really wrong URL ...)
• 1x docker container not found (https://farm.youzim.it/pipeline/af8c0659-7671-4232-934c-0b112163be9b/debug)

[benoit74]

We might want to wait for an answer on kubernetes/kube-state-metrics#2323 before upgrading k8s

[benoit74]

k8s upgrade will be for later for sure now that we are in W09 ^^