Matplotlib errors #1263

ggabernet · 2020-07-31T18:36:32Z

Description of bug:
I have an error with MultiQC v1.9 and the fastqc module that seems to be related to #973.

MultiQC Error log:

  [INFO   ]         multiqc : This is MultiQC v1.9
  [INFO   ]         multiqc : Template    : default
  [INFO   ]         multiqc : Searching   : /Users/gisela/Projects/Pipelines/test_bcellmagic/work/97/18f57abe660bdce1c7dd963d09a599
  [INFO   ]         multiqc : Only using modules custom_content, fastqc
  [INFO   ]  custom_content : nf-core-bcellmagic-summary: Found 1 sample (html)
  [INFO   ]  custom_content : software_versions: Found 1 sample (html)
  [INFO   ]          fastqc : Found 16 reports
  [ERROR  ]         multiqc : Oops! The 'fastqc' MultiQC module broke...
    Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
    If possible, please include a log file that triggers the error - the last file found was:
      ./fastqc/Sample2_Sample2_test_R1_fastqc.zip
  ============================================================
  Module fastqc raised an exception: Traceback (most recent call last):
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 178, in plot
      return get_template_mod().bargraph(plotdata, plotsamples, pconfig)
  AttributeError: module 'multiqc.templates.default' has no attribute 'bargraph'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/multiqc.py", line 569, in run
      output = mod()
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/modules/fastqc/fastqc.py", line 109, in __init__
      self.read_count_plot()
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/modules/fastqc/fastqc.py", line 344, in read_count_plot
      plot = bargraph.plot(pdata, pcats, pconfig)
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 190, in plot
      matplotlib_bargraph(plotdata, plotsamples, pconfig)
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 495, in matplotlib_bargraph
      fig.savefig(img_buffer, format='png', bbox_inches='tight')
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2180, in savefig
      self.canvas.print_figure(fname, **kwargs)
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2060, in print_figure
      bbox_extra_artists=bbox_artists)
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2361, in get_tightbbox
      artists = self.get_default_bbox_extra_artists()
    File "/opt/conda/envs/nf-core-bcellmagic-1.3.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2332, in get_default_bbox_extra_artists
      bbox_artists.remove(self.patch)
  ValueError: list.remove(x): x not in list
  ============================================================
  [INFO   ]         multiqc : Compressing plot data
  [INFO   ]         multiqc : Report      : multiqc_report.html
  [INFO   ]         multiqc : Data        : multiqc_data
  [INFO   ]         multiqc : Plots       : multiqc_plots
  [INFO   ]         multiqc : MultiQC complete

File that triggers the error:

MultiQC run details (please complete the following):

Command used to run MultiQC: multiqc -f .
MultiQC Version: MultiQC v1.9
Operating System: nf-core base container v1.10.2
Python Version: Python 3.6.11
Method of MultiQC installation: conda (inside container)

Additional context

I solved it by pinning matplotlib=3.0.3 in the environment.yml, as was suggested already then.

The text was updated successfully, but these errors were encountered:

ewels · 2020-07-31T21:16:43Z

Hmm, not great. Could you please do a conda env dump / pip freeze in an environment where you had this error?

ggabernet · 2020-08-04T16:56:11Z

Hi Phil, sure, here is the conda env dump. It's not urgent at all though, just wanted to report it!

env_dump.txt

zxl124 · 2020-08-19T19:39:33Z

I am experiencing similar errors using module mirtrace with MultiQC v1.9. This error only occurs if -p is called, therefore the error must occurred during exporting of plots. matplotlib version when this occurred was 3.3.1.

[ERROR  ]         multiqc : Oops! The 'mirtrace' MultiQC module broke...
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
  If possible, please include a log file that triggers the error - the last file found was:
    ./mirtrace-stats-mirna-complexity.tsv

Module mirtrace raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 178, in plot
    return get_template_mod().bargraph(plotdata, plotsamples, pconfig)
AttributeError: module 'multiqc_zymo.templates.zymo' has no attribute 'bargraph'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 325, in to_rgba_array
    result = np.array([to_rgba(cc, alpha) for cc in c])
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 325, in <listcomp>
    result = np.array([to_rgba(cc, alpha) for cc in c])
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 189, in to_rgba
    rgba = _to_rgba_no_colorcycle(c, alpha)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 260, in _to_rgba_no_colorcycle
    raise ValueError(f"Invalid RGBA argument: {orig_c!r}")
ValueError: Invalid RGBA argument: '('

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/usr/local/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 89, in __init__
    plot = self.mirtrace_contamination_check()
  File "/usr/local/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 320, in mirtrace_contamination_check
    return bargraph.plot(self.contamination_data, keys, config)
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 190, in plot
    matplotlib_bargraph(plotdata, plotsamples, pconfig)
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 436, in matplotlib_bargraph
    linewidth = pconfig.get('borderWidth', 0)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 2631, in barh
    align=align, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/__init__.py", line 1438, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 2443, in bar
    color = itertools.chain(itertools.cycle(mcolors.to_rgba_array(color)),
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 330, in to_rgba_array
    "'rgb'. Note also that the latter is deprecated." % c) from err
ValueError: 'rgb(255,255,179)' is neither a valid single color nor a color sequence consisting of single character color specifiers such as 'rgb'. Note also that the latter is deprecated.

I tried changing matplotlib version to 3.0.3. There was still an error with slightly different error message.

[ERROR  ]         multiqc : Oops! The 'mirtrace' MultiQC module broke...
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
  If possible, please include a log file that triggers the error - the last file found was:
    ./mirtrace-stats-mirna-complexity.tsv

Module mirtrace raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 178, in plot
    return get_template_mod().bargraph(plotdata, plotsamples, pconfig)
AttributeError: module 'multiqc_zymo.templates.zymo' has no attribute 'bargraph'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/multiqc/multiqc.py", line 546, in run
    output = mod()
  File "/usr/local/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 89, in __init__
    plot = self.mirtrace_contamination_check()
  File "/usr/local/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 320, in mirtrace_contamination_check
    return bargraph.plot(self.contamination_data, keys, config)
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 190, in plot
    matplotlib_bargraph(plotdata, plotsamples, pconfig)
  File "/usr/local/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 436, in matplotlib_bargraph
    linewidth = pconfig.get('borderWidth', 0)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 2597, in barh
    align=align, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/__init__.py", line 1601, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 2386, in bar
    color = itertools.chain(itertools.cycle(mcolors.to_rgba_array(color)),
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 294, in to_rgba_array
    result[i] = to_rgba(cc, alpha)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 177, in to_rgba
    rgba = _to_rgba_no_colorcycle(c, alpha)
  File "/usr/local/lib/python3.6/site-packages/matplotlib/colors.py", line 233, in _to_rgba_no_colorcycle
    raise ValueError("Invalid RGBA argument: {!r}".format(orig_c))
ValueError: Invalid RGBA argument: '('

JoseEspinosa · 2021-12-01T11:34:12Z

I also found this (or a similar) error when running multiqc for the new implementation of the nf-core/smrnaseq:

  /// MultiQC 🔍 | v1.11

|           multiqc | Search path : /nfs/users2/cn/jespinosa/scratch/90/0a5c64bf7933ead50b5c8b55c62be9
|    custom_content | software_versions: Found 1 sample (html)
|    custom_content | nf-core-smrnaseq-summary: Found 1 sample (html)
╭──────────────── Oops! The 'mirtrace' MultiQC module broke... ────────────────╮
│ Please copy this log and report it at                                        │
│ https://github.com/ewels/MultiQC/issues                                      │
│ Please attach a file that triggers the error. The last file found was:       │
│ ./mirtrace-stats-mirna-complexity.tsv                                        │
│                                                                              │
│ Traceback (most recent call last):                                           │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/plots/bargraph.py", l │
│     return get_template_mod().bargraph(plotdata, plotsamples, pconfig)       │
│ AttributeError: module 'multiqc.templates.default' has no attribute 'bargrap │
│                                                                              │
│ During handling of the above exception, another exception occurred:          │
│                                                                              │
│ Traceback (most recent call last):                                           │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/multiqc.py", line 624 │
│     output = mod()                                                           │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/modules/mirtrace/mirt │
│     plot=self.mirtrace_contamination_check(),                                │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/modules/mirtrace/mirt │
│     return bargraph.plot(self.contamination_data, keys, config)              │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/plots/bargraph.py", l │
│     matplotlib_bargraph(plotdata, plotsamples, pconfig)                      │
│   File "/usr/local/lib/python3.9/site-packages/multiqc/plots/bargraph.py", l │
│     axes.barh(                                                               │
│   File "/usr/local/lib/python3.9/site-packages/matplotlib/axes/_axes.py", li │
│     patches = self.bar(x=left, height=height, width=width, bottom=y,         │
│   File "/usr/local/lib/python3.9/site-packages/matplotlib/__init__.py", line │
│     return func(ax, *map(sanitize_sequence, args), **kwargs)                 │
│   File "/usr/local/lib/python3.9/site-packages/matplotlib/axes/_axes.py", li │
│     color = itertools.chain(itertools.cycle(mcolors.to_rgba_array(color)),   │
│   File "/usr/local/lib/python3.9/site-packages/matplotlib/colors.py", line 3 │
│     raise ValueError("Using a string of single character colors as "         │
│ ValueError: Using a string of single character colors as a color sequence is │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
|            mirtop | Found 28 reports
|          samtools | Found 56 stats reports
|            fastqc | Found 28 reports
|           multiqc | Compressing plot data
|           multiqc | Report      : multiqc_report.html
|           multiqc | Data        : multiqc_data
|           multiqc | Plots       : multiqc_plots
|           multiqc | MultiQC complete`, size: 3620 (max: 255)

ewels · 2021-12-01T17:29:39Z

Ok, so I don't think that this is module specific - it's something funky going on in the core code when custom bargraph category colours are passed. They work fine for the interactive plots, but if Matplotlib is invoked due to large sample size or exported plots, it throws an error.

It clearly also has something to do with Matplotlib syntax as @ggabernet was able to solve the issue by downgrading matplotlib from 3.1.0 to 3.0.3.

This is backed up by the code google found for me here (weirdly doesn't seem to correspond to the source on GitHub?)

    # Convert one at a time.
    if isinstance(c, str):
        # Single string as color sequence.
        # This is deprecated and will be removed in the future.
        try:
            result = np.array([to_rgba(cc, alpha) for cc in c])
        except ValueError as err:
            raise ValueError(
                "'%s' is neither a valid single color nor a color sequence "
                "consisting of single character color specifiers such as "
                "'rgb'. Note also that the latter is deprecated." % c) from err
        else:
            cbook.warn_deprecated(
                "3.2", message="Using a string of single character colors as "
                "a color sequence is deprecated since %(since)s and will be "
                "removed %(removal)s. Use an explicit list instead.")
            return result

Right, so the actual cause for the exception still seems obscure to me (getting only '(' for the colour string) but that is irrelevant as the method of invoking the custom colour that MultiQC is using is deprecated.

ewels · 2021-12-01T17:36:45Z

Fix will be to change the core code in MultiQC to pass lists of rgb values instead of strings. Somewhere around.... here:

https://github.com/ewels/MultiQC/blob/947eeb9d2bdbda72c72593e44d6b641f314157b5/multiqc/plots/bargraph.py#L165-L166

Note: Maybe not exactly here, as we don't want to break the interactive charts

Huh, looks like the default colours we're giving to matplotlib are hex strings:
https://github.com/ewels/MultiQC/blob/947eeb9d2bdbda72c72593e44d6b641f314157b5/multiqc/plots/bargraph.py#L336-L339

So if that's not also about to be deprecated imminently another option could be to convert the rgb strings to hex codes. We're already using the spectra library to do a bunch of this in mqc_colour.py. Could also go the other way and use Spectra to convert the hex codes into a list of rgb values, might be safer.

ewels · 2021-12-01T17:38:49Z

I really don't understand why this hasn't come up more already.. Version 3.1 of matplotlib was released way back in Feb 2019... I feel like I must be missing something 🤔

…essage. See #1263

See #1263

ewels · 2022-01-26T15:46:56Z

Ok, managed to replicate this locally now. It's actually quite difficult to get the unhandled traceback above - MultiQC needs to run with normal interactive plots (not --flat or many samples) but with config.export_plots/-p set. Using flat plots (--flat) also hits the same error but it's handled with a nice error message instead and falls back to interactive. I've added the comparable nicer error handling for exporting plots in bf40904

Anyway, using verbose logging (-v) still shows the same exception either way. It looks like these rgb errors are being triggered because the mirtrace module supplies string rgb values to the plot:

https://github.com/ewels/MultiQC/blob/bf40904180f7865fb9e410708595e1b46dd45b1a/multiqc/modules/mirtrace/mirtrace.py#L284-L310

Switching these to Hex codes instead seems to fix the issue. That's done in a1b4cdf - I think this fixes the errors seen by @JoseEspinosa and @zxl124

However, your original error @ggabernet is a bit different. I can't replicate this one, and it's now on a fairly dated version of MultiQC. As I can't replicate it I'm just going to close this issue and ignore it for now if that's ok 😅 If anyone else hits this error or is able to replicate the same matplotlib crash let me know, and we can reopen.

Thanks all,

Phil

JoseEspinosa · 2022-01-26T17:40:15Z

Awesome @ewels , thanks a lot!
I will give it a try!

JoseEspinosa · 2022-01-26T22:03:43Z

Just tried and solved the problem! 🎉 Are you planning to release a new version including these changes soon?
Thanks again! 😄

ewels · 2022-01-27T05:44:53Z

Great! Yes release v. soon. Maybe today, we'll see..

ggabernet · 2022-01-27T07:45:09Z

Thanks for looking into the issue again, I think now with DSL2 and the new version of MultiQC I should not have this issue any more.

See MultiQC/MultiQC#1263

ewels added the core: software-packaging label Dec 28, 2020

ewels changed the title ~~Error with module Fastqc~~ Matplotlib errors Dec 28, 2020

ewels mentioned this issue Dec 1, 2021

mirTRace module error when export_plots enabled #1581

Closed

ewels added bug: core Bug in the main MultiQC code priority: high and removed core: software-packaging labels Dec 1, 2021

MatthiasZepper self-assigned this Jan 8, 2022

ewels added a commit that referenced this issue Jan 26, 2022

Handle matplotlib exceptions when exporting plots with a nice error m…

bf40904

…essage. See #1263

ewels added a commit that referenced this issue Jan 26, 2022

miRTrace: Replace rgb colours with hex.

a1b4cdf

See #1263

ewels closed this as completed Jan 26, 2022

ewels added a commit to MultiQC/test-data that referenced this issue Jan 27, 2022

miRTrace: log from nf-core/smrnaseq

85a1d5f

See MultiQC/MultiQC#1263

apeltzer mentioned this issue Feb 7, 2022

Update MultiQC to 1.12 planned release before releasing DSLv2 version nf-core/smrnaseq#128

Closed

vladsavelyev pushed a commit to vladsavelyev/MultiQC_TestData that referenced this issue Apr 16, 2022

miRTrace: log from nf-core/smrnaseq

0ab9506

See MultiQC/MultiQC#1263

apeltzer mentioned this issue Apr 12, 2023

miRTrace broken for some samples #1901

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Matplotlib errors #1263

Matplotlib errors #1263

ggabernet commented Jul 31, 2020 •

edited

Loading

ewels commented Jul 31, 2020

ggabernet commented Aug 4, 2020

zxl124 commented Aug 19, 2020 •

edited by ewels

Loading

JoseEspinosa commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Jan 26, 2022

JoseEspinosa commented Jan 26, 2022

JoseEspinosa commented Jan 26, 2022

ewels commented Jan 27, 2022

ggabernet commented Jan 27, 2022

Matplotlib errors #1263

Matplotlib errors #1263

Comments

ggabernet commented Jul 31, 2020 • edited Loading

ewels commented Jul 31, 2020

ggabernet commented Aug 4, 2020

zxl124 commented Aug 19, 2020 • edited by ewels Loading

JoseEspinosa commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Dec 1, 2021

ewels commented Jan 26, 2022

JoseEspinosa commented Jan 26, 2022

JoseEspinosa commented Jan 26, 2022

ewels commented Jan 27, 2022

ggabernet commented Jan 27, 2022

ggabernet commented Jul 31, 2020 •

edited

Loading

zxl124 commented Aug 19, 2020 •

edited by ewels

Loading