[ci] fix current `master` fails with graphviz-related error #5068

StrikerRUS · 2022-03-11T01:53:24Z

To workaround the following error on Windows:

__________________________ test_create_tree_digraph ___________________________
cmd = ['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv'], input_lines = None
encoding = None, capture_output = True, quiet = False
kwargs = {'cwd': None, 'startupinfo': <subprocess.STARTUPINFO object at 0x000000F539A7A988>, 'stderr': -1, 'stdout': -1}
proc = CompletedProcess(args=['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv'], returncode=1, stdout=b'', stderr=b'Warning: Could n... json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg tif tiff tk vml xdot xdot1.2 xdot1.4 xdot_json\r\n')
    def run_check(cmd: typing.Sequence[typing.Union[os.PathLike, str]], *,
                  input_lines: typing.Optional[BytesOrStrIterator] = None,
                  encoding: typing.Optional[str] = None,
                  capture_output: bool = False,
                  quiet: bool = False,
                  **kwargs) -> subprocess.CompletedProcess:
        """Run the command described by ``cmd``
            with ``check=True`` and return its completed process.
    
        Raises:
            CalledProcessError: if the returncode of the subprocess is non-zero.
        """
        log.debug('run %r', cmd)
    
        cmd = list(map(_compat.make_subprocess_arg, cmd))
    
        if not kwargs.pop('check', True):  # pragma: no cover
            raise NotImplementedError('check must be True or omited')
    
        if capture_output:  # Python 3.6 compat
            kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE
    
        if encoding is not None:
            kwargs['encoding'] = encoding
    
        kwargs.setdefault('startupinfo', _compat.get_startupinfo())
    
        try:
            if input_lines is not None:
                assert kwargs.get('input') is None
                assert iter(input_lines) is input_lines
                proc = _run_input_lines(cmd, input_lines, kwargs=kwargs)
            else:
                proc = subprocess.run(cmd, **kwargs)
        except OSError as e:
            if e.errno == errno.ENOENT:
                raise ExecutableNotFound(cmd) from e
            raise
    
        if not quiet and proc.stderr:
            _write_stderr(proc.stderr)
    
        try:
>           proc.check_returncode()
C:\Miniconda3-x64\envs\test-env\lib\site-packages\graphviz\backend\execute.py:95: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = CompletedProcess(args=['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv'], returncode=1, stdout=b'', stderr=b'Warning: Could n... json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg tif tiff tk vml xdot xdot1.2 xdot1.4 xdot_json\r\n')
    def check_returncode(self):
        """Raise CalledProcessError if the exit code is non-zero."""
        if self.returncode:
            raise CalledProcessError(self.returncode, self.args, self.stdout,
>                                    self.stderr)
E           subprocess.CalledProcessError: Command '['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv']' returned non-zero exit status 1.
C:\Miniconda3-x64\envs\test-env\lib\subprocess.py:444: CalledProcessError
During handling of the above exception, another exception occurred:
breast_cancer_split = [array([[1.881e+01, 1.998e+01, 1.209e+02, ..., 1.294e-01, 2.567e-01,
        5.[737](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42862161/job/h6ne7ffui870w4fs#L737)e-02],
       [1.366e+01, 1.515e+01,...      0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1,
       1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1])]
    @pytest.mark.skipif(not GRAPHVIZ_INSTALLED, reason='graphviz is not installed')
    def test_create_tree_digraph(breast_cancer_split):
        X_train, _, y_train, _ = breast_cancer_split
    
        constraints = [-1, 1] * int(X_train.shape[1] / 2)
        gbm = lgb.LGBMClassifier(n_estimators=10, num_leaves=3, verbose=-1, monotone_constraints=constraints)
        gbm.fit(X_train, y_train)
    
        with pytest.raises(IndexError):
            lgb.create_tree_digraph(gbm, tree_index=83)
    
        graph = lgb.create_tree_digraph(gbm, tree_index=3,
                                        show_info=['split_gain', 'internal_value', 'internal_weight'],
                                        name='Tree4', node_attr={'color': 'red'})
>       graph.render(view=False)
..\tests\python_package_test\test_plotting.py:172: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Miniconda3-x64\envs\test-env\lib\site-packages\graphviz\_tools.py:172: in wrapper
    return func(*args, **kwargs)
C:\Miniconda3-x64\envs\test-env\lib\site-packages\graphviz\rendering.py:119: in render
    rendered = self._render(*args, **kwargs)
C:\Miniconda3-x64\envs\test-env\lib\site-packages\graphviz\_tools.py:172: in wrapper
    return func(*args, **kwargs)
C:\Miniconda3-x64\envs\test-env\lib\site-packages\graphviz\backend\rendering.py:320: in render
    capture_output=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cmd = ['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv'], input_lines = None
encoding = None, capture_output = True, quiet = False
kwargs = {'cwd': None, 'startupinfo': <subprocess.STARTUPINFO object at 0x000000F539A7A988>, 'stderr': -1, 'stdout': -1}
proc = CompletedProcess(args=['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv'], returncode=1, stdout=b'', stderr=b'Warning: Could n... json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg tif tiff tk vml xdot xdot1.2 xdot1.4 xdot_json\r\n')
    def run_check(cmd: typing.Sequence[typing.Union[os.PathLike, str]], *,
                  input_lines: typing.Optional[BytesOrStrIterator] = None,
                  encoding: typing.Optional[str] = None,
                  capture_output: bool = False,
                  quiet: bool = False,
                  **kwargs) -> subprocess.CompletedProcess:
        """Run the command described by ``cmd``
            with ``check=True`` and return its completed process.
    
        Raises:
            CalledProcessError: if the returncode of the subprocess is non-zero.
        """
        log.debug('run %r', cmd)
    
        cmd = list(map(_compat.make_subprocess_arg, cmd))
    
        if not kwargs.pop('check', True):  # pragma: no cover
            raise NotImplementedError('check must be True or omited')
    
        if capture_output:  # Python 3.6 compat
            kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE
    
        if encoding is not None:
            kwargs['encoding'] = encoding
    
        kwargs.setdefault('startupinfo', _compat.get_startupinfo())
    
        try:
            if input_lines is not None:
                assert kwargs.get('input') is None
                assert iter(input_lines) is input_lines
                proc = _run_input_lines(cmd, input_lines, kwargs=kwargs)
            else:
                proc = subprocess.run(cmd, **kwargs)
        except OSError as e:
            if e.errno == errno.ENOENT:
                raise ExecutableNotFound(cmd) from e
            raise
    
        if not quiet and proc.stderr:
            _write_stderr(proc.stderr)
    
        try:
            proc.check_returncode()
        except subprocess.CalledProcessError as e:
>           raise CalledProcessError(*e.args)
E           graphviz.backend.execute.CalledProcessError: Command '['dot', '-Kdot', '-Tpdf', '-O', 'Tree4.gv']' returned non-zero exit status 1. [stderr: b'Warning: Could not load "C:\\Miniconda3-x64\\envs\\test-env\\Library\\bin\\gvplugin_pango.dll" - It was found, so perhaps one of its dependents was not.  Try ldd.\r\nWarning: Could not load "C:\\Miniconda3-x64\\envs\\test-env\\Library\\bin\\gvplugin_pango.dll" - It was found, so perhaps one of its dependents was not.  Try ldd.\r\nFormat: "pdf" not recognized. Use one of: bmp canon cmap cmapx cmapx_np dot dot_json emf emfplus eps fig gd gif gv imap imap_np ismap jpe jpeg jpg json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg tif tiff tk vml xdot xdot1.2 xdot1.4 xdot_json\r\n']

Linking

Why `libxml2`?

Failing build:

## Package Plan ##
  environment location: C:\Miniconda3-x64\envs\test-env
  added / updated specs:
    - python-graphviz
The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    cairo-1.16.0               |    hb19e0ff_1008         2.3 MB  conda-forge
    expat-2.4.7                |       h39d44d4_0         347 KB  conda-forge
    font-ttf-dejavu-sans-mono-2.37|       hab24e00_0         388 KB  conda-forge
    font-ttf-inconsolata-3.000 |       h77eed37_0          94 KB  conda-forge
    font-ttf-source-code-pro-2.038|       h77eed37_0         684 KB  conda-forge
    font-ttf-ubuntu-0.83       |       hab24e00_0         1.9 MB  conda-forge
    fontconfig-2.13.96         |       hce3cb01_2         201 KB  conda-forge
    fonts-conda-ecosystem-1    |                0           4 KB  conda-forge
    fonts-conda-forge-1        |                0           4 KB  conda-forge
    fribidi-1.0.10             |       h8d14728_0          63 KB  conda-forge
    getopt-win32-0.1           |       h8ffe710_0          19 KB  conda-forge
    gettext-0.19.8.1           |    ha2e2712_1008         4.9 MB  conda-forge
    graphite2-1.3.13           |             1000          97 KB  conda-forge
    graphviz-2.50.0            |       hefbd956_1         1.1 MB  conda-forge
    gts-0.7.6                  |       h7c369d9_2         188 KB  conda-forge
    harfbuzz-3.1.1             |       hc601d6f_0         921 KB  conda-forge
    libffi-3.4.2               |       h8ffe710_5          41 KB  conda-forge
    libgd-2.3.3                |       h8bb91b0_0         459 KB  conda-forge
    libglib-2.70.2             |       h3be07f2_4         3.1 MB  conda-forge
    libiconv-1.16              |       he77[452](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42862161/job/h6ne7ffui870w4fs#L452)2_0         680 KB  conda-forge
    pango-1.48.10              |       h33e[477](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42862161/job/h6ne7ffui870w4fs#L477)9_2         735 KB  conda-forge
    pcre-8.45                  |       h0e60522_0         518 KB  conda-forge
    pixman-0.40.0              |       h8ffe710_0         482 KB  conda-forge
    python-graphviz-0.19.1     |     pyhaef67bd_0          34 KB  conda-forge
    xorg-kbproto-1.0.7         |    hcd874cb_1002          28 KB  conda-forge
    xorg-libice-1.0.10         |       hcd874cb_0         181 KB  conda-forge
    xorg-libsm-1.2.3           |    hcd874cb_1000          96 KB  conda-forge
    xorg-libx11-1.7.2          |       hcd874cb_0         966 KB  conda-forge
    xorg-libxext-1.3.4         |       hcd874cb_1         282 KB  conda-forge
    xorg-libxpm-3.5.13         |       hcd874cb_0         236 KB  conda-forge
    xorg-libxt-1.2.1           |       hcd874cb_2         811 KB  conda-forge
    xorg-xextproto-7.3.0       |    hcd874cb_1002          28 KB  conda-forge
    xorg-xproto-7.0.31         |    hcd874cb_1007          74 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        21.7 MB

https://ci.appveyor.com/project/guolinke/lightgbm/builds/42862161/job/h6ne7ffui870w4fs#L431

Last successful:

## Package Plan ##
  environment location: C:\Miniconda3-x64\envs\test-env
  added / updated specs:
    - python-graphviz
The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    cairo-1.16.0               |    hb19e0ff_1008         2.3 MB  conda-forge
    expat-2.4.6                |       h39d44d4_0         346 KB  conda-forge
    font-ttf-dejavu-sans-mono-2.37|       hab24e00_0         388 KB  conda-forge
    font-ttf-inconsolata-3.000 |       h77eed37_0          94 KB  conda-forge
    font-ttf-source-code-pro-2.038|       h77eed37_0         684 KB  conda-forge
    font-ttf-ubuntu-0.83       |       hab24e00_0         1.9 MB  conda-forge
    fontconfig-2.13.96         |       h1989[441](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42803460/job/bu9ht0yo0292bxx5#L441)_0         2.0 MB  conda-forge
    fonts-conda-ecosystem-1    |                0           4 KB  conda-forge
    fonts-conda-forge-1        |                0           4 KB  conda-forge
    fribidi-1.0.10             |       h8d14728_0          63 KB  conda-forge
    getopt-win32-0.1           |       h8ffe710_0          19 KB  conda-forge
    gettext-0.19.8.1           |    ha2e2712_1008         4.9 MB  conda-forge
    graphite2-1.3.13           |             1000          97 KB  conda-forge
    graphviz-2.50.0            |       hefbd956_1         1.1 MB  conda-forge
    gts-0.7.6                  |       h7c369d9_2         188 KB  conda-forge
    harfbuzz-3.1.1             |       hc601d6f_0         921 KB  conda-forge
    libffi-3.4.2               |       h8ffe710_5          41 KB  conda-forge
    libgd-2.3.3                |       h8bb91b0_0         459 KB  conda-forge
    libglib-2.70.2             |       h3be07f2_4         3.1 MB  conda-forge
    libiconv-1.16              |       he77[452](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42803460/job/bu9ht0yo0292bxx5#L452)2_0         680 KB  conda-forge
    libxml2-2.9.12             |       hf5bbc77_1         3.4 MB  conda-forge
    pango-1.48.10              |       h33e[477](https://ci.appveyor.com/project/guolinke/lightgbm/builds/42803460/job/bu9ht0yo0292bxx5#L477)9_2         735 KB  conda-forge
    pcre-8.45                  |       h0e60522_0         518 KB  conda-forge
    pixman-0.40.0              |       h8ffe710_0         482 KB  conda-forge
    python-graphviz-0.19.1     |     pyhaef67bd_0          34 KB  conda-forge
    xorg-kbproto-1.0.7         |    hcd874cb_1002          28 KB  conda-forge
    xorg-libice-1.0.10         |       hcd874cb_0         181 KB  conda-forge
    xorg-libsm-1.2.3           |    hcd874cb_1000          96 KB  conda-forge
    xorg-libx11-1.7.2          |       hcd874cb_0         966 KB  conda-forge
    xorg-libxext-1.3.4         |       hcd874cb_1         282 KB  conda-forge
    xorg-libxpm-3.5.13         |       hcd874cb_0         236 KB  conda-forge
    xorg-libxt-1.2.1           |       hcd874cb_2         811 KB  conda-forge
    xorg-xextproto-7.3.0       |    hcd874cb_1002          28 KB  conda-forge
    xorg-xproto-7.0.31         |    hcd874cb_1007          74 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        27.0 MB

https://ci.appveyor.com/project/guolinke/lightgbm/builds/42803460/job/bu9ht0yo0292bxx5#L431

shiyu1994 · 2022-03-11T13:45:22Z

@StrikerRUS Thanks for this!

jameslamb · 2022-03-11T20:06:08Z

Thank you for doing this!

@shiyu1994 when you merge a PR, could you please also click Delete branch?

By the way, it's now possible to have GitHub automatically delete PR branches once a PR is merged.

On the settings page for the repo:

@shiyu1994 @StrikerRUS what do you think about enabling that? I think it would remove a few clicks for us, and wouldn't have any negative impact on development here.

If you all agree, I think @shiyu1994 would have to make that change...I don't have permission to edit that for LightGBM.

StrikerRUS · 2022-03-12T00:41:00Z

what do you think about enabling that?

Will this trigger branch removal if PR is closed? We often close-reopen PR to retrigger CI jobs.

jameslamb · 2022-03-12T00:46:49Z

nope! only merges. We use this at my current company and I've seen that branches for closed PRs don't get deleted.

shiyu1994 · 2022-03-12T16:08:24Z

Is it possible that in some cases we want to revert some merges? If the branch is deleted, will reverting become unavailable?

shiyu1994 · 2022-03-12T16:09:38Z

Okay, now I see that reverting is still available in this PR. In that case, I think it make sense to enable the automatic deleting.

shiyu1994 · 2022-03-12T16:11:53Z

It is now enabled.

* Added base for minimal variance sampling booster * Implemented MVS booster with support for multioutput targets, deterministic execution on small datasets/ * Updated documentation and fixed some linting errors * fixed python sklearn documentation, tryed to fix R Cran CI * Second attempt to fix R pipeline * Fixed R package build for windows and linting error * Revert "Fixed R package build for windows and linting error" This reverts commit d50769e * Revert "Revert "Fixed R package build for windows and linting error"" This reverts commit f531f3a. * Fixed some documentation * Fixed intendation error in mvs.hpp, fixed some windows build issues, added spinx version upper bound * Fixed intendation error in mvs.hpp, fixed some windows build issues, added spinx version upper bound * Update requirements_base.txt * Update R-package/src/Makevars.in Co-authored-by: James Lamb <jaylamb20@gmail.com> * Update R-package/src/Makevars.win.in Co-authored-by: James Lamb <jaylamb20@gmail.com> * Added MVS booster support for dask tests * Moved CalculateThresholdSequential to array_args.h and renamed it to CalculateThresholdMVS * Added cpp tests for ArrayArgs::CalculateThresholdMVS and ArrayArgs::Partition. * Fix linter errors in test_dask.py * Fixed UB in ArrayArgs::Partition, when it is called with one element. * Fixed linter errors * Added more cpp tests and fixed linting errors * Fixed linting errors * Updated R-package documentation Updated documentation Updated test_mvs_threshold_search.cpp Added parallel computation of regularized absolute value term. Added new mvs parameter from constant. * Updated MVS Lambda algorithm * Updated documentation, MVS::GetLambda, MVS::GetThreshold, updated MVS::ResetConfig * [ci] fix current `master` fails with graphviz-related error (#5068) * Update test_windows.ps1 * Update .appveyor.yml * Update test_windows.ps1 * Update .appveyor.yml Co-authored-by: James Lamb <jaylamb20@gmail.com> Co-authored-by: Nikita Titov <nekit94-08@mail.ru> Co-authored-by: Yu Shi <shiyu_k1994@qq.com>

jameslamb · 2022-03-12T18:35:28Z

thanks very much @shiyu1994 ! I think it will save us a few clicks.

StrikerRUS · 2022-03-12T19:12:30Z

@jameslamb

nope! only merges. We use this at my current company and I've seen that branches for closed PRs don't get deleted.

Great! Thanks for checking that.

github-actions · 2023-08-23T14:09:50Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

StrikerRUS added 4 commits March 11, 2022 04:16

Update test_windows.ps1

72c2506

Update .appveyor.yml

8bb3c34

Update test_windows.ps1

3048798

Update .appveyor.yml

84cfea0

StrikerRUS added blocking maintenance labels Mar 11, 2022

StrikerRUS requested a review from jameslamb as a code owner March 11, 2022 01:53

StrikerRUS mentioned this pull request Mar 11, 2022

[python-package] Better column dtype logging when column has "bad dtype" #5065

Merged

StrikerRUS requested review from guolinke, jmoralez and shiyu1994 March 11, 2022 02:11

shiyu1994 approved these changes Mar 11, 2022

View reviewed changes

shiyu1994 merged commit 31ab4d4 into master Mar 11, 2022

jameslamb deleted the fix_ci branch March 11, 2022 20:06

StrikerRUS removed the blocking label Mar 12, 2022

StrikerRUS mentioned this pull request Apr 5, 2022

[ci] Remove temp workaround for graphviz installation #5126

Merged

StrikerRUS mentioned this pull request May 4, 2022

[ci] Prevent Python downgrading to pypy on Windows #5197

Merged

jameslamb mentioned this pull request Oct 7, 2022

[DO NOT MERGE] Release v3.3.3 #5525

Closed

40 tasks

github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] fix current `master` fails with graphviz-related error #5068

[ci] fix current `master` fails with graphviz-related error #5068

StrikerRUS commented Mar 11, 2022 •

edited

Loading

shiyu1994 commented Mar 11, 2022

jameslamb commented Mar 11, 2022

StrikerRUS commented Mar 12, 2022

jameslamb commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

jameslamb commented Mar 12, 2022

StrikerRUS commented Mar 12, 2022

github-actions bot commented Aug 23, 2023

[ci] fix current master fails with graphviz-related error #5068

[ci] fix current master fails with graphviz-related error #5068

Conversation

StrikerRUS commented Mar 11, 2022 • edited Loading

Why libxml2?

shiyu1994 commented Mar 11, 2022

jameslamb commented Mar 11, 2022

StrikerRUS commented Mar 12, 2022

jameslamb commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

shiyu1994 commented Mar 12, 2022

jameslamb commented Mar 12, 2022

StrikerRUS commented Mar 12, 2022

github-actions bot commented Aug 23, 2023

[ci] fix current `master` fails with graphviz-related error #5068

[ci] fix current `master` fails with graphviz-related error #5068

StrikerRUS commented Mar 11, 2022 •

edited

Loading

Why `libxml2`?