Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] upgrade R to 4.0.5 in CI #4158

Merged
merged 6 commits into from
Apr 9, 2021
Merged

[ci] upgrade R to 4.0.5 in CI #4158

merged 6 commits into from
Apr 9, 2021

Conversation

jameslamb
Copy link
Collaborator

R 4.0.5 ("Shake and Throw") was released two days ago (https://stat.ethz.ch/pipermail/r-announce/2021/000668.html).

This PR updates this project's continuous integration environments to use the new version. Linking #3763.

Release notes: https://cran.r-project.org/bin/windows/base/NEWS.R-4.0.5.html

@jameslamb
Copy link
Collaborator Author

On Mac, this is failing with the error observed in #4147 (comment)

==> Downloading http://mirror.ctan.org/systems/mac/mactex/mactex-basictex-20200407.pkg
==> Downloading from http://ctan.math.utah.edu/ctan/tex-archive/systems/mac/mactex/mactex-basictex-20200407.pkg
curl: (22) The requested URL returned error: 404 Not Found
Error: Download failed on Cask 'basictex' with message: Download failed: http://mirror.ctan.org/systems/mac/mactex/mactex-basictex-20200407.pkg

On ubuntu, looks I'm not sure what the root problem is. I tried removing --no-install-recommends and installing r-base-core explicitly, neither worked. Will have to experiment this weekend.

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 r-base-dev : Depends: r-base-core (>= 4.0.5-1.1804.0) but 4.0.4-1.2004.0 is to be installed
E: Unable to correct problems, you have held broken packages.

@StrikerRUS
Copy link
Collaborator

StrikerRUS commented Apr 3, 2021

@jameslamb

On ubuntu, looks I'm not sure what the root problem is.

This is the same issue I faced during restricting CMake version recently.
#4104 (comment)
You have to (hope temporarily) explicitly specify dependency (r-base-core in this case) with version to resolve the issue.

@jameslamb
Copy link
Collaborator Author

You have to (hope temporarily) explicitly specify dependency (r-base-core in this case) with version to resolve the issue.

Right, I tried this and it did not work. Let me add more details on what I meant by "doesn't work".

Adding r-base-core 4.0.5 explicitly yields this:

The following packages have unmet dependencies:
 r-base-core : Depends: libicu60 (>= 60.1-1~) but it is not installable
               Depends: libreadline7 (>= 6.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

Trying to add libicu60 and libreadline7 explicitly yields this

Package libreadline7 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libicu60 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libicu60' has no installation candidate
E: Package 'libreadline7' has no installation candidate

So maybe we just cannot update to R 4.0.5 on Ubuntu 20.04 yet. libicu60 and libreadline7 are not available on Ubuntu 20.04 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=libicu60, https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libreadline7&searchon=names).

Hopefully if we check back in a few days, it will be possible.

@jameslamb
Copy link
Collaborator Author

Ok I just retried this locally and it seems like just installing r-base-dev should now work without any workarounds! Let's see if CI agrees with me.

To test locally, I replicated LightGBM's CI environment with the commands below. It isn't identical (since R jobs run in the Azure DevOps ubuntu-latest VM image) which is very different from the ubuntu:latest docker image, but I'm hoping that those differences are not meaningful for the purpose of apt resolving dependencies.

Pasting my testing setup here just in case it's useful as a reference in the future.

docker run \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    --env BUILD_DIRECTORY=/opt/LightGBM \
    --env COMPILER=gcc \
    --env GITHUB_ACTIONS="true" \
    --env IN_UBUNTU_LATEST_CONTAINER="true" \
    --env OS_NAME=linux \
    --env R_VERSION=4.0 \
    --env R_BUILD_TYPE=cran \
    --env SETUP_CONDA="true" \
    --env TASK=r-package \
    -it \
    ubuntu:latest \
    /bin/bash

apt update -y
apt install -y sudo
sudo apt-get install -y dirmngr
./.ci/setup.sh
./.ci/test.sh

@jameslamb
Copy link
Collaborator Author

Sad, this is still failing with the same error. Ok, I'll keep investigating locally.

@jameslamb
Copy link
Collaborator Author

Ok, I think I got this working! The trick was to switch the repository for R 4.0.5 to focal-cran40/.


All 3 CUDA jobs failed with no logs after running for more than hour. I just restarted them.

image

@StrikerRUS
Copy link
Collaborator

@jameslamb What do you think about the following trick?
BayesWitnesses/m2cgen@6eabad1
It allows to not change codename of Ubuntu in repo address each time we bump/downgrade/switch it.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Apr 9, 2021

@jameslamb What do you think about the following trick?
BayesWitnesses/m2cgen@6eabad1
It allows to not change codename of Ubuntu in repo address each time we bump/downgrade/switch it.

Nice trick! But that only works if you are not pinning the version of R (further down in that file, you have an unpinned r-base). That's problematic if you want to test on environments other than the most recent R, I think, since the version numbers have distribution information in them.

For example. 3.6.3-1bionic

export R_LINUX_VERSION="3.6.3-1bionic"

So I think it makes sense for mc2gen since there's only a single R job and it always wants to get the newest-released R, but I don't think it would work cleanly here in LightGBM.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Apr 9, 2021

ok merging this one with admin rights, since lack of it is starting to cause others to fail (#4167 (comment)) and since if we rebuild it, it will fail on the check-docs task (fixed in #4168)

@jameslamb jameslamb merged commit 02cccd6 into master Apr 9, 2021
@jameslamb jameslamb deleted the ci/r-4.0.5 branch April 9, 2021 22:12
@jameslamb jameslamb mentioned this pull request Apr 9, 2021
17 tasks
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants