ENH: Fix caching-related warnings in GHA `build-test-publish` CI #96

Job	Run time
Build & inspect package	16s
test (3.9, latest, fast)	7m 15s
test (3.10, latest, fast)	8m 20s
test (3.11, latest, fast)	7m 53s
test (3.11, pre, fast)	7m 25s
test (3.12, latest, fast)	8m 28s
test (3.12, pre, fast)	8m 39s
test (3.13, latest, fast)	7m 40s
test (3.13, pre, fast)	8m 9s
test (3.9, min, fast)	8m 17s
test (3.9, latest, slow)	14m 52s
test (3.12, latest, veryslow)	14m 17s
Publish released package to pypi.org	0s
	1h 41m 31s

Provide feedback