ENH: Fix caching-related warnings in GHA `build-test-publish` CI #93

Job	Run time
Build & inspect package	30s
test (3.9, latest, fast)	7m 21s
test (3.10, latest, fast)	8m 20s
test (3.11, latest, fast)	7m 12s
test (3.11, pre, fast)	7m 26s
test (3.12, latest, fast)	8m 13s
test (3.12, pre, fast)	8m 31s
test (3.13, latest, fast)	7m 14s
test (3.13, pre, fast)	7m 35s
test (3.9, min, fast)	8m 21s
test (3.9, latest, slow)	6m 27s
test (3.12, latest, veryslow)	5m 37s
Publish released package to pypi.org	0s
	1h 22m 47s

Provide feedback