Prepare dist-git sources using rpmbuild -bp

[lukaszachy]

I had to split DiscoverFMF.go() to have something to call from prepare step, could be a conflict with #2448.

Also I've duplicated GuestPackageManager into PrepareDistGit so this shouldn't be blocked by #2557 - Once that PR is merged I will change the code to be nicer but lets get this working ...

Feedback wanted mostly for:

• Added discover.post_dist_git method as way to run test discovery from prepare
• What can / should be done about printing accurate numbers of discovered tests as we know the number only in prepare step (no way to update previous step)

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[lukaszachy]

Note: when squashing - reference #1482

[lukaszachy]

@happz finally I'm down to single mypy complain. But I can't fix it, any help greatly appreciated

[lukaszachy]

Rebased to the current main (mainly to have PROVISION_HOW available)

[psss]

Seems that el9 packages failed to be built and core test plan failed.

[lukaszachy]

issues should be fixed now

[lukaszachy]

I've omitted @tmt.steps.provides_method and now PrepareDistGit is not available in --help menu.
I believe this isn't step one should run manually.

[lukaszachy]

I believe it is a good to go now (feature PoV). I'll start updating docs...

[lukaszachy]

Should dist-git-install-builddeps be true by default?

[lukaszachy]

Investigatig /tests/execute/upgrade failure... Error message is misleading - the missing path is Path('/var/tmp/tmt/run-333/plan/execute/upgrade-discover-remote/tests/var/tmp/tmt/run-333/plan/execute/upgrade-discover/tests') -- I need to find out why it is duplicated after dist-git changes in fmf.go()

[psss]

Thanks a lot for implementing this! Looks good. Added some comments and suggestions.

[lukaszachy]

package-manager PR was merged so I'm going to incorporate it here.

[psss]

One typo.

[lukaszachy]

Using package_manager now and added coverage for 'install-builddeps' showed its implementation was buggy.

[lukaszachy]

Hit #2744 again.

[lukaszachy]

@psss Added test discovery: 2 tests selected it appears in prepare, example tmt run (no verbose):

/plan/fmf
    discover
        how: fmf
        directory: /home/lzachar/Doing/tmt-srpm-discovery
        Tests will be discovered after dist-git patching in prepare.
        summary: 0 tests selected
    provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: container
        image: builder-fedora
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 39 (Container Image)
    
        summary: 1 guest provisioned
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
        queued prepare task #2: requires on default-0
        
        prepare task #1: Prepare dist-git sources (buildrequires, patches, discovery...) on default-0
            how: distgit
            name: Prepare dist-git sources (buildrequires, patches, discovery...)
            test discovery: 2 tests selected
        
        prepare task #2: requires on default-0
        how: install
        summary: Install required packages

[lukaszachy]

pre-commit find issue which will happen after the rebase. fixing.

[lukaszachy]

That package_manager fact is really flaky. dist-git patch requires 'dnf' (or 'dnf5') so the test failed because

        distro: Fedora Linux 39 (Cloud Edition)
        kernel: 6.5.6-300.fc39.x86_64
        package manager: yum

And later test failed on package_manager None which is the same issue - fact gathering failed (the order is dnf->yum) just for dist-git case the 'yum' probe was successful but in /prepare/ansible even this probe failed...

[happz]

That package_manager fact is really flaky. dist-git patch requires 'dnf' (or 'dnf5') so the test failed because

        distro: Fedora Linux 39 (Cloud Edition)
        kernel: 6.5.6-300.fc39.x86_64
        package manager: yum

And later test failed on package_manager None which is the same issue - fact gathering failed (the order is dnf->yum) just for dist-git case the 'yum' probe was successful but in /prepare/ansible even this probe failed...

Something fishy in the world of virtual... #2744 (comment)

[psss]

Thanks for all the adjustments! Looks good. Added some last fine tuning changes we discussed on the chat in 8047fe6.

[psss]

Hm, just tried a full test using the following plan in the tmt fedora distgit:

discover:
    how: fmf
    dist-git-source: true
    test: /tests/core/ls
execute:
    how: tmt

And it seems that the require is not properly handled:

prepare task #2: requires on default-0
how: install
summary: Install required packages
name: requires
order: 70
where: default-0
package: 1 package requested
    /usr/bin/flock

The test requires tmt package, but it is not included :-(

[lukaszachy]

Good catch, working on it:

• Fix issue
• Add require so it is tested in distgit.sh as well

[psss]

Thanks, now works fine, just one suggestion/question.

[lukaszachy]

Thanks for the call, as mentioned I'll fix the 'list(set)' to make dependencies unique.
In the next release the refactor as #2784 will follow.