[BUG] ModuleNotFoundError: No module named 'distutils.sysconfig'

[KyleHunter]

Describe the bug
On my test project that has been working for a while, I now get this error on trying to produce schematic outputs.

-c kibot_files/schematic-files.kibot.yaml -d KiBotOutput -b kicad_proj/test_kibot.kicad_pcb -e kicad_proj/test_kibot.kicad_sch Traceback (most recent call last): File "/usr/local/bin/kibot", line 16, in <module> main() File "/usr/local/lib/python3.9/dist-packages/kibot/__main__.py", line [25](https://github.com/MicroType-Engineering/test_kibot/runs/6969702244?check_suite_focus=true#step:9:26)9, in main load_actions() File "/usr/local/lib/python3.9/dist-packages/kibot/kiplot.py", line 83, in load_actions from kibot.mcpyrate import activate File "/usr/local/lib/python3.9/dist-packages/kibot/mcpyrate/activate.py", line 31, in <module> from .importer import source_to_xcode, path_xstats File "/usr/local/lib/python3.9/dist-packages/kibot/mcpyrate/importer.py", line 7, in <module> import distutils.sysconfig ModuleNotFoundError: No module named 'distutils.sysconfig'

To Reproduce

kibot:
  version: 1

global:
  # Sets the default output filename for all generated files. You likely don't have to change this.
  output: "%i.%x"

preflight:
  sch_replace:
    date_command: git log -1 --format='%as' -- $KIBOT_PCB_NAME
    replace_tags:
    - tag: "version"
      command: echo $version
outputs:
  - name: "print_sch"
    comment: "PDF - print schematic"
    type: pdf_sch_print
    dir: "Schematic-Files"

  - name: "standard_bom"
    comment: "Standard csv BOM for upload to Octopart or similar"
    type: bom
    dir: "Schematic-Files"
    options:
      blacklist: "DNF*"
      csv:
        hide_pcb_info: true
        hide_stats_info: true
        quote_all: true
      columns:
        - Part
        - field: manf
          name: Manufacturer
        - References
        - Quantity Per PCB

Environment (please complete the following information):
Where are you running KiBot:

• Github Actions
• v1_k6 and v2_k6. Both show it

Additional context
Something odd (I don't have much experience with Github actions) is that this isn't the first bug that a new version of Kibot is caused, yet using my fork of a known working version on Github will also show the error. It's like even when using a fork, it still references back to your branch.

[set-soft]

Hi @KyleHunter!

Are you sure this problem is actually happening? Are you sure this wasn't in the dev images?

When I switched the dev images to use kicad_auto instead of kicad_debian (See 70e01b3) I had a glitch because the Dockerfile now uninstalls KiBot (dpkg --remove kicost kibot) and then did apt-get -y autoremove which removed distutils. But it was fixed. I tried the images and they are working.

[KyleHunter]

Hi @KyleHunter!

Are you sure this problem is actually happening? Are you sure this wasn't in the dev images?

When I switched the dev images to use kicad_auto instead of kicad_debian (See 70e01b3) I had a glitch because the Dockerfile now uninstalls KiBot (dpkg --remove kicost kibot) and then did apt-get -y autoremove which removed distutils. But it was fixed. I tried the images and they are working.

Uh, I'm not sure I follow. It shows that error and fails on every single repo I have that uses Kibot. All of which worked some weeks ago.
All of the following will show this error:

# Run KiBot to generate all the files.
- name: Generate schematic outputs
  uses: INTI-CMNB/KiBot@v1_k6

# Run KiBot to generate all the files.
- name: Generate schematic outputs
  uses: INTI-CMNB/KiBot@v2_k6

# Run KiBot to generate all the files.
- name: Generate schematic outputs
  uses: MicroType-Engineering/KiBot@stable

Again, I'm really fuzzy on Github actions and the whole Dockerfile deal. The issue for us, is that even when pointing to a nice stable commit on our local branch, it still errors. Last time when there was a similar error on your branch, it did the same thing to our stable fork.

So it's becoming tough to rely on Kibot for work, since at any time, something from the latest version has the ability to break all of our current and past projects...

[set-soft]

Hi @KyleHunter !

So it's becoming tough to rely on Kibot for work, since at any time, something from the latest version has the ability to break all of our current and past projects...

1. You must choose between simple mechanism of use and automagic upgrades and reproducibility. You are using the most simple and automagic mechanism: GitHub action. So you can't spec to get a 100% reproducible result.
2. I think your problem was a temporal glitch and you didn't retry latter. I triggered a redo of the demo and it worked, see: https://github.com/INTI-CMNB/kibot_variants_arduprog/actions/runs/2536121121
3. In order to get a 100% reproducible result you must use your own docker image, or one that is old. These glitches are rare and doesn't last long because as soon as some of the tests fail the problem gets fixed. After the problem is fixed you just need to re-run the workflow, that's all, no need to upgrade anything or change your repo, just ask GitHub to run the workflow again.
4. The v* tags are just API tags, they make reference to different Github action APIs for KiBot. What you get from them is advancing from time to time. The KiBot you get is stable and passes more than 400 tests. The image is built using a stable KiBot (the v2_dk6 and v2_d are testing versions, but even those tags passes the more than 400 tests). The images evolve, so you get newer tools and newer KiCad versions. We don't keep every single image because they are more than 1 GiB each (and you have one for KiCad 5 and another for KiCad 6). But you can find some old images at docker hub. To use them you must avoid using GitHub actions.

Again: the most simple way to use KiBot has advantages and disadvantages. If a workflow fails with some error you must report it as soon as possible.

[KyleHunter]

@set-soft Hmm That makes sense for the most part. But the error is certainly still there for all of our repo's:

and

[set-soft]

Hi @KyleHunter !
Do you have a link to a public repo with this problem? Some repo that I can fork to find where is the problem.

[KyleHunter]

@set-soft Yeah, here's a test repo I've been using: https://github.com/MicroType-Engineering/test_kibot

[set-soft]

Hi @KyleHunter !
I think I found the problem: You are using a forked repo:

# Run KiBot to generate all the files.
      - name: Generate schematic outputs
        uses: MicroType-Engineering/KiBot@v2_k6

This tag points to a repo with an old Dockerfile that uses the development images:

FROM setsoft/kicad_auto:dev_k6

And which uses an old mechanism that no longer works:

apt-get -y autoremove && \

The use of development images was a temporal hack that I disabled after releasing KiBot 1.0.0, to make the results more stable, so now you must explicitly ask for the last code using v2_dk6.

Your fork is before 1.0.0 release: https://github.com/MicroType-Engineering/KiBot/blob/08b1b8bdfe276ba5fe92e3091f4b0627913b9fa2/kibot/__main__.py#L70

So you are always using the development code. I simplified the way docker images are generated to reduce errors (70e01b3 and 287d9c7). Now the four main GitHub actions tags use the same base. As a result all development images doesn't support running:

apt-get -y autoremove && \

This is because now the development images uses the regular images as base, but removing the KiBot and KiCost packages and installing git snap-shots. Your forked code can be adapted to the new images removing it. But I think you should avoid a forked repo for the GitHub actions. You can also change your forked repo to use the last stable image ki6 and not dev_k6. You can change it in:

FROM setsoft/kicad_auto:dev_k6

If you need an image that won't change you can use ki6.0.4_Debian this is currently available at Docker Hub but I can promise they'll keep it forever, the service is free and they are very generous giving to us many GiB and bandwidth, so they could remove old images at any time. Just take a look at the images they are storing and how many downloads these images have.

Note that a forked repo for a GitHub action won't use the code in the repo, just the Dockerfile and what it says to use as base.

Let me know if you can fix the problem so I can close the issue.

[KyleHunter]

@set-soft Yes! That did indeed fix it. So then going forward I'll stop using a forked branch and instead do:
uses: INTI-CMNB/KiBot@v2_k6

Thank you so much for your help as always!