Build and Push Manifest: ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'cachecontrol'

[jwmatthews]

Automatic builds via GitHub Actions are broken as per below:

[Build and Push Manifest](https://github.com/konveyor/kai/actions/runs/10221583766/job/28284424125#step:2:1263)
Error: buildah exited with code 1
time="2024-08-02T20:22:51Z" level=warning msg="Failed to decode the keys [\"machine\"] from \"/usr/share/containers/containers.conf\"."
time="2024-08-02T20:22:51Z" level=warning msg="Failed to decode the keys [\"machine\"] from \"/usr/share/containers/containers.conf\"."
Trying to pull registry.access.redhat.com/ubi9/python-312:latest...
Getting image source signatures
Copying blob sha256:a5fd5316d18892c6c60f0aeebc417cfe742bf9d5724ac45d941dbd046bd7dc7d
Copying blob sha256:cc296d75b61273dcb0db7527435a4c3bd03f7723d89a94d446d3d52849970460
Copying blob sha256:db22e630b1c7cf081461536c489254a8d1b39ceda32f8f3025314f032860d984
Copying blob sha256:9f2fbf79f82326e925352def0dbdccc800cb9da2fe0125c4d1c33a9cbfc1b629
Copying blob sha256:cc296d75b61273dcb0db7527435a4c3bd03f7723d89a94d446d3d52849970460
Copying blob sha256:db22e630b1c7cf081461536c489254a8d1b39ceda32f8f3025314f032860d984
Copying blob sha256:9f2fbf79f82326e925352def0dbdccc800cb9da2fe0125c4d1c33a9cbfc1b629
Copying blob sha256:a5fd5316d18892c6c60f0aeebc417cfe742bf9d5724ac45d941dbd046bd7dc7d
Copying config sha256:f8e346731464740d6670df9d94a183dfe9cdea93af6076d7a2e052f3e2ee55aa
Writing manifest to image destination
Storing signatures
time="2024-08-02T20:23:07Z" level=warning msg="Failed to decode the keys [\"machine\"] from \"/usr/share/containers/containers.conf\"."
time="2024-08-02T20:23:08Z" level=warning msg="Failed to decode the keys [\"machine\"] from \"/usr/share/containers/containers.conf\"."
time="2024-08-02T20:23:22Z" level=warning msg="Failed to decode the keys [\"machine\"] from \"/usr/share/containers/containers.conf\"."
  Running command git clone --filter=blob:none --quiet https://github.com/fabianvf/vcrpy.git /tmp/pip-install-a7se61_2/vcrpy_c45e141a65254375911380ec69988a43
  Running command git checkout -b httpx-async-threadpool --track origin/httpx-async-threadpool
  Switched to a new branch 'httpx-async-threadpool'
  branch 'httpx-async-threadpool' set up to track 'origin/httpx-async-threadpool'.
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'cachecontrol'

WARNING: Ignoring invalid distribution ~ip (/opt/app-root/lib64/python3.12/site-packages)
WARNING: Ignoring invalid distribution ~ip (/opt/app-root/lib64/python3.12/site-packages)
error building at STEP "RUN pip install --no-cache -r /kai/requirements.txt": error while running runtime: exit status 1
time="2024-08-02T20:23:57Z" level=error msg="exit status 1"

[jmontleon]

Seems to be a pip bug that is expose by using fuse-overlay.

pypa/pip#11368 (comment)

containers/fuse-overlayfs#332 (comment)

The workaround looks simple enough to implement and seems to work. The failure is just no access to push from my forked repo.
https://github.com/jmontleon/kai/actions/runs/10250193756/job/28355348302