Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nf-core Launch web GUI #638

Merged
merged 17 commits into from
Jun 30, 2020
Merged

nf-core Launch web GUI #638

merged 17 commits into from
Jun 30, 2020

Conversation

ewels
Copy link
Member

@ewels ewels commented Jun 26, 2020

New code for using a nice web graphical user interface when running nf-core launch. Should work with any nextflow pipeline.

Bunch of other small tweaks and improvements to nf-core launch and nf-core schema too.

Tests haven't been updated so will probably get a lot of failures. Will fix these next week. In the mean time, any code review and / or testing the tools are very welcome!

Works in tandem with changes to the nf-core website, see nf-core/website#400

See Slack #json-schema for screen capture demos: https://nfcore.slack.com/archives/CUC8PPDL2/p1593212486143900

PR checklist

  • This comment contains a description of changes (with reason)
  • If you've fixed a bug or added code that should be tested, add tests!
  • Documentation in docs is updated
  • CHANGELOG.md is updated
  • README.md is updated

ewels added 5 commits June 24, 2020 17:57
Moved code into nf_core.utils as reusable functions, so that they can be used for other stuff aside from nf-core schema build
Also means some reduction in code duplication
* Use pre-release version of PyInquirer - capture keyboard interruptions, set defaults for lists
* Make booleans use a True / False select list instead of confirm prompt
* Various tweaks and changes / updates to make launch work with new web launch GUI
* Fix for schema website response type
* Fix for launch web filtering for booleans
@ewels ewels added the WIP Work in progress label Jun 26, 2020
setup.py Outdated
'PyInquirer',
# 'PyInquirer>1.0.3',
# Need the new release of PyInquirer, see nf_core/launch.py for details
'PyInquirer @ git+ssh://git@github.com/CITGuru/PyInquirer.git@master#egg=PyInquirer',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just like you I am working with PyInquirer and need the latest release.

Do you think that the solution here is suitable for PyPI?
pypa/pip#4187 (comment)

Or are you for now just hoping that there will be a new release soonish?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right.

So I will go with https://github.com/tmbo/questionary .
Seems to be much more up to date (tech wise) and more actively maintained.
Looks like it supports pretty much the same stuff as PyInquirer.

Might be of interest to you?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I've looked over questionary, there was another one mentioned in the threads too. I wasn't too keen on rewriting the code for a different library though, especially when the two features I needed are already merged into PyInquirer. I'm hoping that he does a new minor release soon. If we get close to doing a new release of nf-core/tools and it's still not out then I'll come back to this and look into questionary.

It's kind of ironic, as apparently PyInquirer was forked from another previous project because of the lack of updates 😅

@codecov
Copy link

codecov bot commented Jun 29, 2020

Codecov Report

Merging #638 into dev will increase coverage by 0.57%.
The diff coverage is 77.67%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #638      +/-   ##
==========================================
+ Coverage   75.45%   76.02%   +0.57%     
==========================================
  Files          11       11              
  Lines        2159     2269     +110     
==========================================
+ Hits         1629     1725      +96     
- Misses        530      544      +14     
Impacted Files Coverage Δ
nf_core/launch.py 70.00% <72.84%> (+5.93%) ⬆️
nf_core/schema.py 89.82% <86.20%> (+1.53%) ⬆️
nf_core/utils.py 90.67% <88.63%> (-1.22%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81ca05b...b1d18d9. Read the comment docs.

@ewels ewels marked this pull request as ready for review June 29, 2020 15:43
@ewels ewels removed the WIP Work in progress label Jun 29, 2020
@ggabernet
Copy link
Member

I've tested the Launch web GUI and it looks really nice and it's very intuitive and easy to use! Great job!

@ewels
Copy link
Member Author

ewels commented Jun 29, 2020

Brilliant - thanks @ggabernet! 😊 To confirm, did you combine it with running the command line tool, and it worked?

@ggabernet
Copy link
Member

ggabernet commented Jun 29, 2020

Yes, it worked! I installed your latest version of the launch-web-gui branch, and tested it with the atacseq dev release

@ggabernet
Copy link
Member

ggabernet commented Jun 29, 2020

Ah, however using only the cli, without the web GUI does not work:

$ nf-core launch atacseq -r dev

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 1.10.dev0


INFO: This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles


INFO: Using local workflow: nf-core/atacseq (dev - 12b8d0b)

INFO: [✓] Pipeline schema looks valid

Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?

? Choose launch method  Command line

Nextflow command-line flags - General Nextflow flags to control how the pipeline runs.
These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.
? Nextflow command-line flags  -name

-name - Unique name for this nextflow run
Traceback (most recent call last):
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/bin/nf-core", line 492, in <module>
    nf_core_cli()
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/bin/nf-core", line 143, in launch
    if launcher.launch_pipeline() == False:
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/nf_core/launch.py", line 137, in launch_pipeline
    self.prompt_schema()
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/nf_core/launch.py", line 345, in prompt_schema
    answers.update(self.prompt_group(param_id, param_obj))
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/nf_core/launch.py", line 431, in prompt_group
    answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required, answers))
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/nf_core/launch.py", line 368, in prompt_param
    answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/PyInquirer/prompt.py", line 67, in prompt
    application = getattr(prompts, type).question(message, **_kwargs)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/PyInquirer/prompts/input.py", line 46, in question
    return create_prompt_application(
TypeError: create_prompt_application() got an unexpected keyword argument 'raise_keyboard_interrupt'

I guess because of the PyInquirer issue?

@ewels
Copy link
Member Author

ewels commented Jun 30, 2020

Yup - if you re-run the installation (pip install -e . if the code is checked out locally) it should hopefully work.

@ewels
Copy link
Member Author

ewels commented Jun 30, 2020

To clarify - I guess that you installed this when you first tested, then just pulled the updated code? If that's the case you'll need to rerun the installation as in my comment above.

If you're installing like that and you're still getting that error then it's potentially an installation error problem I guess?

@ewels ewels linked an issue Jun 30, 2020 that may be closed by this pull request
@ggabernet
Copy link
Member

Hi Phil, I initially tested it by installing your branch of nf-core tools directly from git with:

pip install git+https://github.com/ewels/nf-core-tools.git@launch-web-gui

Now, I've also tested it with cloning your fork, checking out to the launch-web-gui branch and installing it as you suggested:

pip install -e .

That is the log obtained from the installation, in case it helps for debugging:

Obtaining file:///Users/gisela/Projects/Pipelines/ewels-tools
Requirement already satisfied: cookiecutter in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (1.7.2)
Requirement already satisfied: click in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (7.1.2)
Requirement already satisfied: GitPython in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (3.1.3)
Requirement already satisfied: jinja2 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (2.11.2)
Requirement already satisfied: jsonschema in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (3.2.0)
Requirement already satisfied: PyInquirer@ https://github.com/CITGuru/PyInquirer/archive/master.zip from https://github.com/CITGuru/PyInquirer/archive/master.zip in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (1.0.3)
Requirement already satisfied: pyyaml in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (5.3.1)
Requirement already satisfied: requests in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (2.24.0)
Requirement already satisfied: requests_cache in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (0.5.2)
Requirement already satisfied: tabulate in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from nf-core==1.10.dev0) (0.8.7)
Requirement already satisfied: six>=1.10 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (1.15.0)
Requirement already satisfied: jinja2-time>=0.2.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (0.2.0)
Requirement already satisfied: binaryornot>=0.4.4 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (0.4.4)
Requirement already satisfied: MarkupSafe<2.0.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (1.1.1)
Requirement already satisfied: poyo>=0.5.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (0.5.0)
Requirement already satisfied: python-slugify>=4.0.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from cookiecutter->nf-core==1.10.dev0) (4.0.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from GitPython->nf-core==1.10.dev0) (4.0.5)
Requirement already satisfied: attrs>=17.4.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from jsonschema->nf-core==1.10.dev0) (19.3.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from jsonschema->nf-core==1.10.dev0) (0.16.0)
Requirement already satisfied: setuptools in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from jsonschema->nf-core==1.10.dev0) (47.3.1.post20200616)
Requirement already satisfied: regex>=2016.11.21 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from PyInquirer@ https://github.com/CITGuru/PyInquirer/archive/master.zip->nf-core==1.10.dev0) (2020.6.8)
Requirement already satisfied: Pygments>=2.2.0 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from PyInquirer@ https://github.com/CITGuru/PyInquirer/archive/master.zip->nf-core==1.10.dev0) (2.6.1)
Requirement already satisfied: prompt-toolkit==1.0.14 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from PyInquirer@ https://github.com/CITGuru/PyInquirer/archive/master.zip->nf-core==1.10.dev0) (1.0.14)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from requests->nf-core==1.10.dev0) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from requests->nf-core==1.10.dev0) (2020.6.20)
Requirement already satisfied: idna<3,>=2.5 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from requests->nf-core==1.10.dev0) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from requests->nf-core==1.10.dev0) (3.0.4)
Requirement already satisfied: arrow in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from jinja2-time>=0.2.0->cookiecutter->nf-core==1.10.dev0) (0.15.7)
Requirement already satisfied: text-unidecode>=1.3 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from python-slugify>=4.0.0->cookiecutter->nf-core==1.10.dev0) (1.3)
Requirement already satisfied: smmap<4,>=3.0.1 in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->GitPython->nf-core==1.10.dev0) (3.0.4)
Requirement already satisfied: wcwidth in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from prompt-toolkit==1.0.14->PyInquirer@ https://github.com/CITGuru/PyInquirer/archive/master.zip->nf-core==1.10.dev0) (0.2.5)
Requirement already satisfied: python-dateutil in /Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages (from arrow->jinja2-time>=0.2.0->cookiecutter->nf-core==1.10.dev0) (2.8.1)
Installing collected packages: nf-core
  Running setup.py develop for nf-core
Successfully installed nf-core

So it seems that the installation was successful.

But I got the same error when using the nf-core launch cli:

nf-core launch atacseq

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 1.10.dev0


INFO: This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles


INFO: Using local workflow: nf-core/atacseq (dev - 12b8d0b)

INFO: [✓] Pipeline schema looks valid

Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?

? Choose launch method  Command line

Nextflow command-line flags - General Nextflow flags to control how the pipeline runs.
These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.
? Nextflow command-line flags  -name

-name - Unique name for this nextflow run
Traceback (most recent call last):
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/bin/nf-core", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/gisela/Projects/Pipelines/ewels-tools/scripts/nf-core", line 492, in <module>
    nf_core_cli()
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/gisela/Projects/Pipelines/ewels-tools/scripts/nf-core", line 143, in launch
    if launcher.launch_pipeline() == False:
  File "/Users/gisela/Projects/Pipelines/ewels-tools/nf_core/launch.py", line 137, in launch_pipeline
    self.prompt_schema()
  File "/Users/gisela/Projects/Pipelines/ewels-tools/nf_core/launch.py", line 345, in prompt_schema
    answers.update(self.prompt_group(param_id, param_obj))
  File "/Users/gisela/Projects/Pipelines/ewels-tools/nf_core/launch.py", line 431, in prompt_group
    answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required, answers))
  File "/Users/gisela/Projects/Pipelines/ewels-tools/nf_core/launch.py", line 368, in prompt_param
    answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/PyInquirer/prompt.py", line 67, in prompt
    application = getattr(prompts, type).question(message, **_kwargs)
  File "/Users/gisela/anaconda3/envs/nfcorelaunchtest/lib/python3.8/site-packages/PyInquirer/prompts/input.py", line 46, in question
    return create_prompt_application(
TypeError: create_prompt_application() got an unexpected keyword argument 'raise_keyboard_interrupt'

@drpatelh
Copy link
Member

It is slightly troubling in the way that PyInquirer is breaking stuff here...

@ewels
Copy link
Member Author

ewels commented Jun 30, 2020

Hmm, ok - so pip thinks you already have the required version. If you force a reinstall it should work:

pip install --upgrade git+ssh://git@github.com/CITGuru/PyInquirer.git@master#egg=PyInquirer

I'm pretty sure that this is only an issue because you had already installed PyInquirer, should hopefully work ok in fresh installs.

@ewels
Copy link
Member Author

ewels commented Jun 30, 2020

@drpatelh yes - I'm hoping that there will be a new PyPI release before we push this new version of tools. See CITGuru/PyInquirer#90 (comment)

If we get close to a release and this is the only thing missing then I'll downgrade the code so that it works with the existing latest version - it's only a couple of minor features that I'm using here. It'd be a shame, but it's doable. Alternatively I could rewrite to use a different library if needed.

I'll make an issue about bumping this version and add it to the release milestone so that we don't forget.

@ggabernet
Copy link
Member

@ewels yes, this solved the issue, thanks!

@ewels ewels mentioned this pull request Jun 30, 2020
Copy link
Member

@ggabernet ggabernet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I can understand completely what is going on in the code, but the functional tests looked good to me!

@ewels
Copy link
Member Author

ewels commented Jun 30, 2020

Great! The launch_pipeline() function has the main logic if you're curious. That's what's triggered when you run the command and goes through the various steps, calling sub-functions at each stage to "do stuff".

@ewels ewels merged commit d821b81 into nf-core:dev Jun 30, 2020
@ewels ewels deleted the launch-web-gui branch June 30, 2020 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Launch - make a web GUI for setting parameters
4 participants