snakemake.exceptions.WorkflowError: Config file is not valid JSON or YAML

[mattdoug604]

Hello,
I am trying to run MAVIS and I am getting error.
-bash-4.2$ snakemake -j 1 --configfile mavis.cfg -s Snakefile
Traceback (most recent call last):
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1604, in _load_configfile
return yaml.load(f, Loader=OrderedLoader)
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/init.py", line 114, in load
return loader.get_single_data()
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/composer.py", line 39, in get_single_node
if not self.check_event(StreamEndEvent):
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/yaml/parser.py", line 171, in parse_document_start
raise ParserError(None, None,
yaml.parser.ParserError: expected '', but found ''
in "mavis.cfg", line 2, column 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/u/home/m/mdistler/project-jflint/anaconda3/bin/snakemake", line 33, in
sys.exit(load_entry_point('snakemake==6.15.5', 'console_scripts', 'snakemake')())
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/init.py", line 2757, in main
success = snakemake(
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/init.py", line 500, in snakemake
update_config(overwrite_config, load_configfile(f))
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1617, in load_configfile
config = _load_configfile(configpath)
File "/u/home/m/mdistler/project-jflint/anaconda3/lib/python3.8/site-packages/snakemake/io.py", line 1606, in _load_configfile
raise WorkflowError(
snakemake.exceptions.WorkflowError: Config file is not valid JSON or YAML. In case of YAML, make sure to not mix whitespace and tab indentation.

Thank you for help

[543090lee]

[metadata]
name = mavis
version = 3.0.0
url = https://github.com/bcgsc/mavis.git
download_url = https://github.com/bcgsc/mavis/archive/v2.2.10.tar.gz
description = A Structural Variant Post-Processing Package
author_email = creisle@bcgsc.ca
author = Caralyn Reisle
maintainer_email = mavis@bcgsc.ca
maintainer = mavis
long_description = file: README.md
long_description_content_type = text/markdown
license_file = LICENSE
project_urls = mavis = http://mavis.bcgsc.ca

[bdist_wheel]
universal = 1

[pycodestyle]
ignore = E501
W503
E203
statistics = True

[flake8]
ignore = E501,W503,E203

[options]
packages = find:
package_dir =
= src
python_requires = >=3.7
dependency_links = []
include_package_data = True
install_requires =
biopython>=1.70, <1.78
braceexpand==0.1.2
colour
Distance>=0.1.3
mavis_config>=1.1.0, <2.0.0
networkx>=2.5,<3
numpy>=1.13.1
pandas>=1.1, <2
pysam
Shapely>=1.6.4.post1
shortuuid>=0.5.0
svgwrite
typing_extensions>=4
setup_requires =
pip>=9.0.0
setuptools>=36.0.0

[options.packages.find]
exclude = tests
where = src

[options.extras_require]
doc =
mkdocs>=1.1.2
markdown-refdocs
mkdocs-material>=5.4.0
markdown-include
mkdocs-simple-hooks>=0.1.2
mkdocs-awesome-pages-plugin
test =
timeout-decorator>=0.3.3
coverage>=4.2
pycodestyle>=2.3.1
pytest
pytest-cov
dev =
black
flake8
twine
wheel
timeout-decorator>=0.3.3
coverage>=4.2
pycodestyle>=2.3.1
pytest
pytest-cov
pytest-xdist
mkdocs>=1.1.2,<2
markdown-refdocs
mkdocs-material>=5.4.0
markdown-include
mkdocs-simple-hooks>=0.1.2
types-setuptools>=57.4.7, <58
deploy =
twine
wheel
tools =
pyensembl
simplejson
requests

[options.entry_points]
console_scripts =
mavis = mavis.main:main
calculate_ref_alt_counts = tools.calculate_ref_alt_counts:main

[zhemingfan]

Hi @543090lee and @mattdoug604, the files you posted are not MAVIS config files.

MAVIS configuration files are used to specify the specific parameters you want to use for each step of the MAVIS pipeline. This file needs to be a json. An example of a config can be found at mini-tutorial.config.json. The full list of customizable settings can be found in our separate mavis_config repository.

[543090lee]

I tried running the mini-tutorial.config.json and I followed all the instructions on mavis website, and got error below.
(base) [seichang@login3 project-zarlab]$ snakemake -j 1 --configfile=tests/mini-tutorial.config.json -s Snakefile
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Traceback (most recent call last):
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/init.py", line 785, in snakemake
keepmetadata=keep_metadata,
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/workflow.py", line 1060, in execute
logger.run_info("\n".join(dag.stats()))
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/dag.py", line 2191, in stats
yield tabulate(rows, headers="keys")
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 2049, in tabulate
tabular_data, headers, showindex=showindex
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 1471, in _normalize_tabular_data
rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 1471, in
rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/tabulate/init.py", line 107, in _is_separating_line
(len(row) >= 1 and row[0] == SEPARATING_LINE)
File "/u/home/s/seichang/project-zarlab/anaconda3/lib/python3.7/site-packages/snakemake/rules.py", line 1138, in eq
return self.name == other.name and self.output == other.output
AttributeError: 'str' object has no attribute 'name'

Thank you for your help

[zhemingfan]

Hi @543090lee, thank you for the question. The issue stems from the tabulate dependency included in Snakemake. We currently have a PR that's targeted to fix this. Once its merged, could you re-install and try it again?

[543090lee]

Hello,
Thank you for the comment.
Are there any updates on the PR that's targeted to fix the issue stemming from the tabulate dependency issue?

Thank you for your help.

[zhemingfan]

Hi @543090lee, try out the latest develop_v3 branch. It should be working now after the merge from the latest PR #337

[ziphra]

I had the same problem than @543090lee, I installed the latest develop_v3 branch with git clone -b develop_v3 https://github.com/bcgsc/mavis.git but it didnt work.

However, after reinstalling with pip and doing pip3 install tabulate==0.8.9 the mini tutorial worked

[543090lee]

Thank you for your help @ziphra and @zhemingfan
I followed the above instruction, and I run the mini tutorial and still got the following error.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job stats:
job count min threads max threads

---

all 1 1 1
annotate 7 1 1
cluster 2 1 1
convert 1 1 1
copy_config 1 1 1
init_config 1 1 1
pairing 1 1 1
summary 1 1 1
validate 7 1 1
total 22 1 1

Select jobs to execute...

[Thu Nov 3 16:26:44 2022]
rule copy_config:
output: output_dir/config.raw.json
log: output_dir/logs/copy_config.snakemake.log.txt
jobid: 8
resources: tmpdir=/tmp, time_limit=57600, mem_mb=4000, cpus=1, log_dir=output_dir/logs

[Thu Nov 3 16:26:46 2022]
Finished job 8.
1 of 22 steps (5%) done
Select jobs to execute...

[Thu Nov 3 16:26:46 2022]
rule init_config:
input: output_dir/config.raw.json
output: output_dir/config.json
log: output_dir/logs/init_config.snakemake.log.txt
jobid: 7
resources: tmpdir=/tmp, time_limit=57600, mem_mb=16000, cpus=1, log_dir=output_dir/logs

/usr/bin/bash: mavis: command not found
[Thu Nov 3 16:26:46 2022]
Error in rule init_config:
jobid: 7
output: output_dir/config.json
log: output_dir/logs/init_config.snakemake.log.txt (check log file(s) for error message)
shell:
mavis setup --config output_dir/config.raw.json --outputfile output_dir/config.json
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /u/project/zarlab/seichang/mavis/.snakemake/log/2022-11-03T162643.244861.snakemake.log

Thank you

[creisle]

Singularity containers: ignored

@543090lee from your log above it looks like you are not using the container for the workflow is that correct?

Because of this, the workflow is looking for mavis to be available on the command line which is why you are seeing the error below

/usr/bin/bash: mavis: command not found

We should have added that to the documentation. I will make an update to the docs. You have 2 options.

1. add the --use-singularity flag to the last command in the tutorial
2. install mavis via pip as well as the aligners (see user manual) to be available on the default PATH

[creisle]

I had the same problem than @543090lee, I installed the latest develop_v3 branch with git clone -b develop_v3 https://github.com/bcgsc/mavis.git but it didnt work.

However, after reinstalling with pip and doing pip3 install tabulate==0.8.9 the mini tutorial worked

Release v3.1.0 should fix the tabulate error

[543090lee]

Hello,
I installed the new version 3.1.0, and ran the mini tutorial and got this error.
(base) [seichang@login4 mavis]$ snakemake -j 1 --configfile=tests/mini-tutorial.config.json -s Snakefile
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job stats:
job count min threads max threads

---

all 1 1 1
annotate 7 1 1
cluster 2 1 1
convert 1 1 1
copy_config 1 1 1
init_config 1 1 1
pairing 1 1 1
summary 1 1 1
validate 7 1 1
total 22 1 1

Select jobs to execute...

[Mon Nov 28 16:23:43 2022]
rule copy_config:
output: output_dir/config.raw.json
log: output_dir/logs/copy_config.snakemake.log.txt
jobid: 8
resources: tmpdir=/tmp, time_limit=57600, mem_mb=4000, cpus=1, log_dir=output_dir/logs

[Mon Nov 28 16:23:44 2022]
Finished job 8.
1 of 22 steps (5%) done
Select jobs to execute...

[Mon Nov 28 16:23:44 2022]
rule init_config:
input: output_dir/config.raw.json
output: output_dir/config.json
log: output_dir/logs/init_config.snakemake.log.txt
jobid: 7
resources: tmpdir=/tmp, time_limit=57600, mem_mb=16000, cpus=1, log_dir=output_dir/logs

/usr/bin/bash: mavis: command not found
[Mon Nov 28 16:23:44 2022]
Error in rule init_config:
jobid: 7
output: output_dir/config.json
log: output_dir/logs/init_config.snakemake.log.txt (check log file(s) for error message)
shell:
mavis setup --config output_dir/config.raw.json --outputfile output_dir/config.json
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /u/project/zarlab/seichang/mavis/.snakemake/log/2022-11-28T162342.743481.snakemake.log

Thank you

[zhemingfan]

Hi @543090lee , what was the error in the log?