add new command modules test #1526

mirpedrol · 2022-04-25T10:33:36Z

Add a new command nf-core modules test to run module tests locally.

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

fabianegli

Hi Júlia, it is great that you are working on an easy CLI for running tests! I made some comments and hope they help. Some are a bit out of scope for this PR (especially the comment about the path handling), but maybe they are still useful to you. Cheers and happy programming :-)

nf_core/modules/module_test.py

fabianegli · 2022-04-26T08:49:57Z

nf_core/modules/module_test.py

+        self.module_dir = os.path.join("modules", *self.module_name.split("/"))
+        self.module_test_main = os.path.join("tests", "modules", *self.module_name.split("/"), "main.nf")


I wonder if this kind of module structure/path handling could be done as part of ModulesRepo or in an util function. I think I saw a lot of such path handling in too many places.

…o modules_test

nf_core/modules/module_test.py

codecov · 2022-04-28T08:05:39Z

Codecov Report

Merging #1526 (aca376d) into dev (6052563) will decrease coverage by 0.33%.
The diff coverage is 49.29%.

@@            Coverage Diff             @@
##              dev    #1526      +/-   ##
==========================================
- Coverage   64.94%   64.61%   -0.34%     
==========================================
  Files          53       54       +1     
  Lines        6105     6178      +73     
==========================================
+ Hits         3965     3992      +27     
- Misses       2140     2186      +46

Impacted Files	Coverage Δ
nf_core/modules/module_test.py	`48.27% <48.27%> (ø)`
nf_core/__main__.py	`51.36% <50.00%> (-0.04%)`	⬇️
nf_core/modules/__init__.py	`100.00% <100.00%> (ø)`
nf_core/utils.py	`79.75% <0.00%> (-3.25%)`	⬇️
nf_core/modules/create.py	`79.27% <0.00%> (ø)`
nf_core/modules/install.py	`59.09% <0.00%> (+0.95%)`	⬆️
nf_core/modules/lint/main_nf.py	`81.76% <0.00%> (+1.76%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6052563...aca376d. Read the comment docs.

fabianegli · 2022-04-28T17:41:42Z

nf_core/modules/module_test.py

+        self._check_inputs()
+        self._set_profile()
+        self._run_pytests()
+
+    def _check_inputs(self):
+        """Do more complex checks about supplied flags."""
+
+        # Get the tool name if not specified
+        if self.module_name is None:
+            if self.no_prompts:
+                raise UserWarning(
+                    "Tool name not provided and prompts deactivated. Please provide the tool name as TOOL/SUBTOOL or TOOL."
+                )
+            modules_repo = ModulesRepo()
+            modules_repo.get_modules_file_tree()
+            self.module_name = questionary.autocomplete(
+                "Tool name:",
+                choices=modules_repo.modules_avail_module_names,
+                style=nf_core.utils.nfcore_question_style,
+            ).ask()
+        module_dir = Path("modules") / self.module_name
+
+        # First, sanity check that the module directory exists
+        if not module_dir.is_dir():
+            raise UserWarning(f"Cannot find directory '{module_dir}'. Should be TOOL/SUBTOOL or TOOL")


This suggestion makes the code more expressive, merges the two separate User warnings into one, hopefully more useful one, and removes all path handling from this file (the Path import at the top can be removed, too). I am missing all the nice editor sugar here in GitHubs suggestion mode and hope it's still acceptable for black etc.

Suggested change

self._check_inputs()

self._set_profile()

self._run_pytests()

def _check_inputs(self):

"""Do more complex checks about supplied flags."""

# Get the tool name if not specified

if self.module_name is None:

if self.no_prompts:

raise UserWarning(

"Tool name not provided and prompts deactivated. Please provide the tool name as TOOL/SUBTOOL or TOOL."

)

modules_repo = ModulesRepo()

modules_repo.get_modules_file_tree()

self.module_name = questionary.autocomplete(

"Tool name:",

choices=modules_repo.modules_avail_module_names,

style=nf_core.utils.nfcore_question_style,

).ask()

module_dir = Path("modules") / self.module_name

# First, sanity check that the module directory exists

if not module_dir.is_dir():

raise UserWarning(f"Cannot find directory '{module_dir}'. Should be TOOL/SUBTOOL or TOOL")

self._check_module()

self._set_profile()

self._run_pytests()

def _check_module(self):

"""Ensure the module corresponding to the supplied module name is available."""

modules_repo = ModulesRepo()

modules_repo.get_modules_file_tree()

if self.module_name in modules_repo.modules_avail_module_names:

return

if self.no_prompts:

raise UserWarning(

f"The module name '{self.module_name}' is invalid. "

f"Valid module names are {modules_repo.modules_avail_module_names}."

)

self.module_name = questionary.autocomplete(

f"Chose a module:",

choices=modules_repo.modules_avail_module_names,

style=nf_core.utils.nfcore_question_style,

).ask()

mahesh-panchal · 2022-04-29T08:06:46Z

I opened this in Gitpod and just ran a test:

$ nf-core modules test

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Tool name: fastqc
CRITICAL Cannot find directory 'modules/fastqc'. Should be TOOL/SUBTOOL or TOOL

but it failed to find fastqc from the root of the directory.

mirpedrol · 2022-04-29T09:00:30Z

I opened this in Gitpod and just ran a test:

$ nf-core modules test

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Tool name: fastqc
CRITICAL Cannot find directory 'modules/fastqc'. Should be TOOL/SUBTOOL or TOOL

but it failed to find fastqc from the root of the directory.

Could it be because the command needs to be run from nf-core/modules ?
I'm sorry I am not familiar with the use of Gitpod so not sure how to test this command from the modules folder, but it's working for me locally though.

mahesh-panchal · 2022-04-29T09:08:39Z

Ah, that was a silly mistake on my part. You're right. I need to clone modules into the repo too...

$ cd ..
$ git clone https://github.com/nf-core/modules.git
Cloning into 'modules'...
remote: Enumerating objects: 31564, done.
remote: Total 31564 (delta 0), reused 0 (delta 0), pack-reused 31564
Receiving objects: 100% (31564/31564), 97.04 MiB | 13.22 MiB/s, done.
Resolving deltas: 100% (16501/16501), done.
$ cd modules
$ nf-core modules test

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Tool name: fastqc
? Choose software profile Docker
INFO     Setting environment variable '$PROFILE' to 'docker'                                                                                                                                                                                                                                        module_test.py:111
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Running pytest for module 'fastqc'                                                                                                                                                                                                                                                         module_test.py:124
================================================================================================================================================ test session starts =================================================================================================================================================
platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0
rootdir: /workspace/modules, configfile: pytest.ini
plugins: datafiles-2.0, cov-3.0.0, workflow-1.6.0
collecting ... 
collected 777 items                                                                                                                                                                                                                                                                                                  

fastqc single-end:
        command:   nextflow run ./tests/modules/fastqc/ -entry test_fastqc_single_end -c ./tests/config/nextflow.config -c ./tests/modules/fastqc/nextflow.config -c ./tests/modules/fastqc/nextflow.config
        directory: /tmp/pytest_workflow_ri5hifnx/fastqc_single-end
        stdout:    /tmp/pytest_workflow_ri5hifnx/fastqc_single-end/log.out
        stderr:    /tmp/pytest_workflow_ri5hifnx/fastqc_single-end/log.err
'fastqc single-end' done.

fastqc paired-end:
        command:   nextflow run ./tests/modules/fastqc/ -entry test_fastqc_paired_end -c ./tests/config/nextflow.config -c ./tests/modules/fastqc/nextflow.config -c ./tests/modules/fastqc/nextflow.config
        directory: /tmp/pytest_workflow_ri5hifnx/fastqc_paired-end
        stdout:    /tmp/pytest_workflow_ri5hifnx/fastqc_paired-end/log.out
        stderr:    /tmp/pytest_workflow_ri5hifnx/fastqc_paired-end/log.err
'fastqc paired-end' done.

tests/test_versions_yml.py sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..ssssssssss [ 35%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 74%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                                                                                               [ 98%]
tests/modules/fastqc/test.yml ........                                                                                                                                                                                                                                                                         [100%] Keeping temporary directories and logs. Use '--kwd' or '--keep-workflow-wd' to disable this behaviour.


================================================================================================================================================== warnings summary ==================================================================================================================================================
../../opt/conda/lib/python3.9/site-packages/pytest_workflow/plugin.py:122: 489 warnings
  /opt/conda/lib/python3.9/site-packages/pytest_workflow/plugin.py:122: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to YamlFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return YamlFile.from_parent(parent, fspath=path)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================================================================================== 10 passed, 767 skipped, 489 warnings in 37.74s ===================================================================================================================================

mahesh-panchal · 2022-04-29T09:22:26Z

Very well done. These commands are working nicely:

$ nf-core modules test fastqc

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Choose software profile Conda
...

nf-core modules test -p fastqc

$ nf-core modules test fastqc

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Choose software profile Docker
...

$ nf-core modules test --help

A comment on this one though:

$ nf-core modules test fastqc

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.4.dev0 - https://nf-co.re


INFO     Press enter to use default values (shown in brackets) or type your own responses                                                                                                                                                                                                            module_test.py:60
? Choose software profile Singularity
INFO     Setting environment variable '$PROFILE' to 'singularity'                                                                                                                                                                                                                                   module_test.py:111
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Running pytest for module 'fastqc'                                                                                                                                                                                                                                                         module_test.py:124
================================================================================================================================================ test session starts =================================================================================================================================================
platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0
rootdir: /workspace/modules, configfile: pytest.ini
plugins: datafiles-2.0, cov-3.0.0, workflow-1.6.0
collecting ... 
collected 777 items                                                                                                                                                                                                                                                                                                  

fastqc single-end:
        command:   nextflow run ./tests/modules/fastqc/ -entry test_fastqc_single_end -c ./tests/config/nextflow.config -c ./tests/modules/fastqc/nextflow.config -c ./tests/modules/fastqc/nextflow.config
        directory: /tmp/pytest_workflow_ihqvoq40/fastqc_single-end
        stdout:    /tmp/pytest_workflow_ihqvoq40/fastqc_single-end/log.out
        stderr:    /tmp/pytest_workflow_ihqvoq40/fastqc_single-end/log.err
'fastqc single-end' done.

fastqc paired-end:
        command:   nextflow run ./tests/modules/fastqc/ -entry test_fastqc_paired_end -c ./tests/config/nextflow.config -c ./tests/modules/fastqc/nextflow.config -c ./tests/modules/fastqc/nextflow.config
        directory: /tmp/pytest_workflow_ihqvoq40/fastqc_paired-end
        stdout:    /tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/log.out
        stderr:    /tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/log.err
'fastqc paired-end' done.

tests/test_versions_yml.py sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssFFssssssssss [ 35%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 74%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                                                                                               [ 98%]
tests/modules/fastqc/test.yml FFFFFFFF                                                                                                                                                                                                                                                                         [100%] Keeping temporary directories and logs. Use '--kwd' or '--keep-workflow-wd' to disable this behaviour.


====================================================================================================================================================== FAILURES ======================================================================================================================================================
__________________________________________________________________________________________________________________________________ test_ensure_valid_version_yml[fastqc single-end] __________________________________________________________________________________________________________________________________

workflow_dir = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_single-end')

    @pytest.mark.workflow(*_get_workflow_names())
    def test_ensure_valid_version_yml(workflow_dir):
        workflow_dir = Path(workflow_dir)
        software_name = workflow_dir.name.split("_")[0].lower()
        try:
            versions_yml_file = workflow_dir / f"output/{software_name}/versions.yml"
>           versions_yml = versions_yml_file.read_text()

tests/test_versions_yml.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml'), encoding = None, errors = None

    def read_text(self, encoding=None, errors=None):
        """
        Open the file in text mode, read it, and close the file.
        """
>       with self.open(mode='r', encoding=encoding, errors=errors) as f:

/opt/conda/lib/python3.9/pathlib.py:1266: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml'), mode = 'r', buffering = -1, encoding = None, errors = None, newline = None

    def open(self, mode='r', buffering=-1, encoding=None,
             errors=None, newline=None):
        """
        Open the file pointed by this path and return a file object, as
        the built-in open() function does.
        """
>       return io.open(self, mode, buffering, encoding, errors, newline,
                       opener=self._opener)

/opt/conda/lib/python3.9/pathlib.py:1252: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml'), name = '/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml', flags = 524288, mode = 438

    def _opener(self, name, flags, mode=0o666):
        # A stub for the opener argument to built-in open()
>       return self._accessor.open(self, flags, mode)
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml'

/opt/conda/lib/python3.9/pathlib.py:1120: FileNotFoundError

During handling of the above exception, another exception occurred:

workflow_dir = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_single-end')

    @pytest.mark.workflow(*_get_workflow_names())
    def test_ensure_valid_version_yml(workflow_dir):
        workflow_dir = Path(workflow_dir)
        software_name = workflow_dir.name.split("_")[0].lower()
        try:
            versions_yml_file = workflow_dir / f"output/{software_name}/versions.yml"
            versions_yml = versions_yml_file.read_text()
        except FileNotFoundError:
>           raise AssertionError(
                dedent(
                    f"""\
                    `versions.yml` not found in the output directory.
                    Expected path: `{versions_yml_file}`
    
                    This can have multiple reasons:
                    * The test-workflow failed before a `versions.yml` could be generated.
                    * The workflow name in `test.yml` does not start with the tool name.
                    """
                )
            )
E           AssertionError: `versions.yml` not found in the output directory.
E           Expected path: `/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/versions.yml`
E           
E           This can have multiple reasons:
E           * The test-workflow failed before a `versions.yml` could be generated.
E           * The workflow name in `test.yml` does not start with the tool name.

tests/test_versions_yml.py:33: AssertionError
__________________________________________________________________________________________________________________________________ test_ensure_valid_version_yml[fastqc paired-end] __________________________________________________________________________________________________________________________________

workflow_dir = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end')

    @pytest.mark.workflow(*_get_workflow_names())
    def test_ensure_valid_version_yml(workflow_dir):
        workflow_dir = Path(workflow_dir)
        software_name = workflow_dir.name.split("_")[0].lower()
        try:
            versions_yml_file = workflow_dir / f"output/{software_name}/versions.yml"
>           versions_yml = versions_yml_file.read_text()

tests/test_versions_yml.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml'), encoding = None, errors = None

    def read_text(self, encoding=None, errors=None):
        """
        Open the file in text mode, read it, and close the file.
        """
>       with self.open(mode='r', encoding=encoding, errors=errors) as f:

/opt/conda/lib/python3.9/pathlib.py:1266: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml'), mode = 'r', buffering = -1, encoding = None, errors = None, newline = None

    def open(self, mode='r', buffering=-1, encoding=None,
             errors=None, newline=None):
        """
        Open the file pointed by this path and return a file object, as
        the built-in open() function does.
        """
>       return io.open(self, mode, buffering, encoding, errors, newline,
                       opener=self._opener)

/opt/conda/lib/python3.9/pathlib.py:1252: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml'), name = '/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml', flags = 524288, mode = 438

    def _opener(self, name, flags, mode=0o666):
        # A stub for the opener argument to built-in open()
>       return self._accessor.open(self, flags, mode)
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml'

/opt/conda/lib/python3.9/pathlib.py:1120: FileNotFoundError

During handling of the above exception, another exception occurred:

workflow_dir = PosixPath('/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end')

    @pytest.mark.workflow(*_get_workflow_names())
    def test_ensure_valid_version_yml(workflow_dir):
        workflow_dir = Path(workflow_dir)
        software_name = workflow_dir.name.split("_")[0].lower()
        try:
            versions_yml_file = workflow_dir / f"output/{software_name}/versions.yml"
            versions_yml = versions_yml_file.read_text()
        except FileNotFoundError:
>           raise AssertionError(
                dedent(
                    f"""\
                    `versions.yml` not found in the output directory.
                    Expected path: `{versions_yml_file}`
    
                    This can have multiple reasons:
                    * The test-workflow failed before a `versions.yml` could be generated.
                    * The workflow name in `test.yml` does not start with the tool name.
                    """
                )
            )
E           AssertionError: `versions.yml` not found in the output directory.
E           Expected path: `/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/versions.yml`
E           
E           This can have multiple reasons:
E           * The test-workflow failed before a `versions.yml` could be generated.
E           * The workflow name in `test.yml` does not start with the tool name.

tests/test_versions_yml.py:33: AssertionError
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/test_fastqc.html' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_single-end/output/fastqc/test_fastqc.zip' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'fastqc single-end' exited with exit code '1' instead of '0'.
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/test_1_fastqc.html' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/test_2_fastqc.html' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/test_2_fastqc.zip' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'/tmp/pytest_workflow_ihqvoq40/fastqc_paired-end/output/fastqc/test_1_fastqc.zip' does not exist while it should
____________________________________________________________________________________________________________________________________________________ test session ____________________________________________________________________________________________________________________________________________________
'fastqc paired-end' exited with exit code '1' instead of '0'.
================================================================================================================================================== warnings summary ==================================================================================================================================================
../../opt/conda/lib/python3.9/site-packages/pytest_workflow/plugin.py:122: 489 warnings
  /opt/conda/lib/python3.9/site-packages/pytest_workflow/plugin.py:122: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to YamlFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return YamlFile.from_parent(parent, fspath=path)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================================================================== short test summary info ===============================================================================================================================================
FAILED tests/test_versions_yml.py::test_ensure_valid_version_yml[fastqc single-end] - AssertionError: `versions.yml` not found in the output directory.
FAILED tests/test_versions_yml.py::test_ensure_valid_version_yml[fastqc paired-end] - AssertionError: `versions.yml` not found in the output directory.
FAILED tests/modules/fastqc/test.yml::fastqc single-end::output/fastqc/test_fastqc.html::should exist
FAILED tests/modules/fastqc/test.yml::fastqc single-end::output/fastqc/test_fastqc.zip::should exist
FAILED tests/modules/fastqc/test.yml::fastqc single-end::exit code should be 0
FAILED tests/modules/fastqc/test.yml::fastqc paired-end::output/fastqc/test_1_fastqc.html::should exist
FAILED tests/modules/fastqc/test.yml::fastqc paired-end::output/fastqc/test_2_fastqc.html::should exist
FAILED tests/modules/fastqc/test.yml::fastqc paired-end::output/fastqc/test_2_fastqc.zip::should exist
FAILED tests/modules/fastqc/test.yml::fastqc paired-end::output/fastqc/test_1_fastqc.zip::should exist
FAILED tests/modules/fastqc/test.yml::fastqc paired-end::exit code should be 0
=================================================================================================================================== 10 failed, 767 skipped, 489 warnings in 14.39s ===================================================================================================================================

Is it possible to get nicer output than the stack trace? Something that points to the actual issue instead perhaps (in this case singularity is not installed)?

Can one pass the profile with pytest args? It's not clear.

mirpedrol · 2022-04-29T13:14:56Z

Is it possible to get nicer output than the stack trace? Something that points to the actual issue instead perhaps (in this case singularity is not installed)?

This is more on the pytest side, but I have added a check to have a nicer error when the profile is not available.

Can one pass the profile with pytest args? It's not clear.

As far as I know, it's not possible to select the profile with arguments, it's selected with the env variable PROFILE.

mahesh-panchal · 2022-04-29T13:28:56Z

Is it possible to generally suppress the stack trace?
This time I made a syntax error ( reads -> read in the input line ) in the fastqc module and ran it.

nf-core modules test fastqc

It prints out a stack trace that could be distracting.

It may be another issue, but if it's related, then perhaps point the user to look at the logs for the error, rather than displaying the symptom (e.g. FAILED filesX should exist).

ggabernet · 2022-04-29T14:04:52Z

Hi @asthara10, I've just tested the command and it worked great for me, also the check for singularity being installed worked great for me now. I was just wondering if the test results could be displayed in a bit nicer summarized way in a similar way as for the nf-core modules lint test results:

Or maybe this is hard here because it runs the pytests?

mirpedrol · 2022-04-29T14:07:24Z

It is possible to reduce traceback and other reporting info from pytest with some arguments, for example --tb=no, --disable-warnings or --no-summary. I personally wouldn't add them by default, I think it's easier for the user to provide them as extra arguments if the long output is not needed, rather than having to go to the logs.

mirpedrol · 2022-04-29T14:09:56Z

Regarding the nicer summary, right now it runs the pytest command, so the output comes from pytest. I am not sure if it can be modified afterwards, will have a look. Any ideas on how to do this are welcome :)

mahesh-panchal

Looks awesome to me.
I think generally this quite functional, so 👍🏽 from me.
If it's feasible to have prettier output, then please try.

nf_core/modules/module_test.py

Co-authored-by: Gisela Gabernet <gisela.gabernet@gmail.com>

Co-authored-by: Fabian Egli <fabianegli@users.noreply.github.com>

nf_core/modules/module_test.py

fabianegli

Thank you for all your hard work @asthara10 !

ewels

Great stuff! Haven't tried running it but the code reads well. Couple of minor comments but nothing substantial.

ewels · 2022-05-02T11:29:38Z

nf_core/modules/module_test.py

+
+    ...
+
+    Attributes


If I'm being picky, these are for the __init__() def really, so you could probably move them there. Unless this is some kind of Pythonic way of documenting stuff that I'm not aware of. Low prio though.

nf_core/modules/module_test.py

Co-authored-by: Phil Ewels <phil.ewels@scilifelab.se>

ewels · 2022-05-02T12:16:42Z

remove unnecessary libraryes

Would be good to add isort to the CI to handle these, been meaning to for a little while now. Would be a nice mini-PR follow on if you fancy it some time.

Júlia Mir Pedrol added 3 commits April 25, 2022 11:40

add new command modules test

54d218f

run tests with pytest

5097f04

remove test workflow check

3227499

fabianegli reviewed Apr 26, 2022

View reviewed changes

Júlia Mir Pedrol and others added 6 commits April 26, 2022 13:10

add code tests

2e2ea58

Merge branch 'dev' into modules_test

e763b96

fix linting

c0b29f5

apply comments from PR review

99b5170

Merge branch 'modules_test' of https://github.com/asthara10/tools int…

57c7541

…o modules_test

fix linting

b80fd62

fabianegli reviewed Apr 26, 2022

View reviewed changes

nf_core/modules/module_test.py Show resolved Hide resolved

fabianegli reviewed Apr 26, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

fabianegli reviewed Apr 26, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

apply comments from PR

6c006fd

fabianegli reviewed Apr 27, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

fabianegli reviewed Apr 27, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

Júlia Mir Pedrol added 4 commits April 27, 2022 12:48

fix test

0e4a835

always set profile

a808853

do not abbreviate env var

4165a9f

add new test

b431f98

fabianegli reviewed Apr 27, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

Júlia Mir Pedrol added 2 commits April 27, 2022 16:53

update CHANGELOG

58df639

specify test files as test_*.py in pytest.ini

530a383

mirpedrol marked this pull request as ready for review April 28, 2022 08:02

apply comments from PR review

b885e5f

fabianegli reviewed Apr 28, 2022

View reviewed changes

Júlia Mir Pedrol and others added 2 commits April 29, 2022 14:12

check if profile is available

f49af39

modify linting

f33f1fb

mahesh-panchal approved these changes Apr 29, 2022

View reviewed changes

fabianegli reviewed Apr 29, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

fabianegli reviewed Apr 29, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

ggabernet reviewed May 2, 2022

View reviewed changes

nf_core/modules/module_test.py Show resolved Hide resolved

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

mirpedrol and others added 2 commits May 2, 2022 12:42

Add nf-core/modules folder warning

b732386

Co-authored-by: Gisela Gabernet <gisela.gabernet@gmail.com>

Add valid profile check

c182ef3

Co-authored-by: Fabian Egli <fabianegli@users.noreply.github.com>

fabianegli reviewed May 2, 2022

View reviewed changes

nf_core/modules/module_test.py Outdated Show resolved Hide resolved

mirpedrol added 2 commits May 2, 2022 13:05

fix linting

69247e7

avoid using shell=True when checking profile

1d8e567

fabianegli approved these changes May 2, 2022

View reviewed changes

ewels approved these changes May 2, 2022

View reviewed changes

mirpedrol and others added 2 commits May 2, 2022 13:54

Apply suggestions from code review

8e33499

Co-authored-by: Phil Ewels <phil.ewels@scilifelab.se>

remove unnecessary libraryes

aca376d

mirpedrol merged commit e76ddeb into nf-core:dev May 2, 2022

mirpedrol mentioned this pull request May 3, 2022

add isort workflow #1538

Merged

4 tasks

mirpedrol mentioned this pull request May 13, 2022

modify modules test documentation nf-core/website#1158

Merged

mirpedrol deleted the modules_test branch May 17, 2022 12:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new command modules test #1526

add new command modules test #1526

mirpedrol commented Apr 25, 2022 •

edited

Loading

fabianegli left a comment

fabianegli Apr 26, 2022

codecov bot commented Apr 28, 2022 •

edited

Loading

fabianegli Apr 28, 2022

mahesh-panchal commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022 •

edited

Loading

ggabernet commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal left a comment

fabianegli left a comment

ewels left a comment

ewels May 2, 2022

ewels commented May 2, 2022

		self.module_dir = os.path.join("modules", *self.module_name.split("/"))
		self.module_test_main = os.path.join("tests", "modules", *self.module_name.split("/"), "main.nf")

add new command modules test #1526

add new command modules test #1526

Conversation

mirpedrol commented Apr 25, 2022 • edited Loading

PR checklist

fabianegli left a comment

Choose a reason for hiding this comment

fabianegli Apr 26, 2022

Choose a reason for hiding this comment

codecov bot commented Apr 28, 2022 • edited Loading

Codecov Report

fabianegli Apr 28, 2022

Choose a reason for hiding this comment

mahesh-panchal commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal commented Apr 29, 2022 • edited Loading

ggabernet commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mirpedrol commented Apr 29, 2022

mahesh-panchal left a comment

Choose a reason for hiding this comment

fabianegli left a comment

Choose a reason for hiding this comment

ewels left a comment

Choose a reason for hiding this comment

ewels May 2, 2022

Choose a reason for hiding this comment

ewels commented May 2, 2022

mirpedrol commented Apr 25, 2022 •

edited

Loading

codecov bot commented Apr 28, 2022 •

edited

Loading

mahesh-panchal commented Apr 29, 2022 •

edited

Loading