1.33 regression: dist-git-source behaves differently in different provisions

[LecrisUT]

The issue occurs in https://src.fedoraproject.org/rpms/python-scikit-build-core/pull-request/59

There I am using:

discover:
  how: fmf
  dist-git-source: true
  dist-git-extract: scikit_build_core-*/
/example:
  summary:
    Documentation examples
  discover+:
    how: fmf
    filter: "tag: examples"
    path: docs
  execute:
    how: tmt

In 1.32 (2 weeks ago) the tests were still ok, but now I am getting issues that the tmt tree is not found:

output

/plans/examples
summary: Documentation examples
    discover
        how: fmf
        directory: docs
        Tests will be discovered after dist-git patching in prepare.
        summary: 0 tests selected
    provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: connect
        primary address: 3.17.12.31
        topology address: 3.17.12.31
        port: 22
        key: /etc/citool.d/id_rsa_artemis
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 41 (Cloud Edition Prerelease)
        kernel: 6.9.0-64.fc41.x86_64
        package manager: dnf5
        selinux: yes
        is superuser: yes
    
        summary: 1 guest provisioned
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
        queued prepare task #2: requires (dist-git) on default-0
        queued prepare task #3: requires on default-0
        queued prepare task #4: recommends (dist-git) on default-0
        
        prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
        how: distgit
        name: Prepare dist-git sources (buildrequires, patches, discovery...)
        path: docs
        fail: Metadata tree path 'docs' not found.
    finish
    
        summary: 0 tasks completed

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    prepare step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Metadata tree path 'docs' not found.

What is confusing is that when I try it locally with provision.how = container and the tmt-reproducer, this seems to go through:

provision container

$ tmt -c arch=x86_64 -c distro=fedora-rawhide -c initiator=fedora-ci -c trigger=commit run -a --verbose -e @tmt-environment-plans-examples.yaml provision --how container --image fedora:rawhide plan --name ^/plans/examples$
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
        queued prepare task #2: requires (dist-git) on default-0
        queued prepare task #3: requires on default-0
        queued prepare task #4: recommends (dist-git) on default-0
        
        prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
        how: distgit
        name: Prepare dist-git sources (buildrequires, patches, discovery...)
        path: docs
        filter: tag: examples
        summary: 7 tests selected
            /examples/downstream/pybind11_example
            /examples/getting_started/abi3
            /examples/getting_started/c
            /examples/getting_started/cython
            /examples/getting_started/fortran
            /examples/getting_started/pybind11
            /examples/getting_started/swig

I would try the full reproducer but I am getting another issue which might be unrelated (ssh timeout): #2937. After managing to make it work on F39/F40, I am indeed reproducing the issue with provision.how: virtual

Update: Seems like F40 PR works, something weird is going on. Sort of works. It worked on Zuul, but failed on Fedora-CI. Might be having a flaky response. This seems to be very flaky when it works or not 😟

[LecrisUT]

Could be related to RPM 4.20 🤔

[LecrisUT]

Ok, so in 1.34 this issue is resolved (afaiu it was RPM 4.20 issue), but then there is a more specific issue with the tmt-reproducer where the dist-git prepare step is not performed when it is split up in steps:

Output

$ tmt --root . -c arch=x86_64 -c distro=fedora-rawhide -c trigger=commit run --until provision --verbose -e @tmt-environment-plans-examples.yaml provision --how virtual --image Fedora-Rawhide plan --name ^/plans/examples$
/var/tmp/tmt/run-019
Found 1 plan.

/plans/examples
summary: Documentation examples
    discover
        how: fmf
        directory: docs
        warn: Sources will not be extracted, prepare step is not enabled.
        Tests will be discovered after dist-git patching in prepare.
        summary: 0 tests selected
    provision
        queued provision.provision task #1: default-0
$ tmt --root . run --last --since prepare
/var/tmp/tmt/run-019

/plans/examples
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: requires on default-0
        
        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        where: default-0
        package: /usr/bin/flock
    
        queued pull task #1: pull from default-0
        
        pull task #1: pull from default-0
    
        summary: 1 preparation applied
    execute

@happz @lukaszachy Any ideas what caused this regression? Afaict #2641 caused this workflow to break. Seems like the prepare steps are not saved when running the discover step (or rather when running every phase)

[lukaszachy]

Discover prints this warn: Sources will not be extracted, prepare step is not enabled.

There is no way to extract sources now without running the provision & prepare steps. With your rfe #2994 it might be possible, I'm not sure if the 'prepare.distgit' isn't enabled only in runtime.

How big problem is this for you?

[LecrisUT]

How big problem is this for you?

Only affects tmt-reproducer so not urgent for me, but others trying to run this might find this surprise and be less clear why it doesn't work.

One patch could be to instead of splitting the tmt-reproducer steps, to add the guest-setup-0.sh as a prepare step, e.g.

$ tmt run prepare --insert --how shell --script "./guest-setup-0.sh"

It seems to work on my side, but not sure about what order it should have. @thrix thoughts?

There is no way to extract sources now without running the provision & prepare steps. With your rfe #2994 it might be possible, I'm not sure if the 'prepare.distgit' isn't enabled only in runtime.

The issue is rather that the prepare step is not saved, e.g. in run-xxx/**/prepare/step.yaml it only contains a prepare.shell step (not sure why, would have expected an install) after the first half command is executed.

[lukaszachy]

AFAIK the 'install' step injection (require, recommend) happens as 'runtime' only.