Use conda-env for CI environments #20767

TomAugspurger · 2018-04-20T15:17:18Z

Closes #20720

The old setup where things were installed in multiple places resulted in some strange environments, with multiple copies of the same package in site-packages.

This attempts to reduce the environment definition to a single file per CI job, and the environment installation to 3 steps

create the environment
uninstall pandas if it's present (installing a pandas dependency)
build pandas from source & install

Travis only for now. Will move over the appveyor / circle ones too.

Limit coverage installs

That's pinned to numpy 1.10.4, which forces fastparquet 0.1.3, which is not compatible with thrift & pandas master.

codecov · 2018-04-20T16:12:16Z

Codecov Report

Merging #20767 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #20767      +/-   ##
==========================================
- Coverage   91.85%   91.81%   -0.04%     
==========================================
  Files         153      153              
  Lines       49308    49310       +2     
==========================================
- Hits        45290    45274      -16     
- Misses       4018     4036      +18

Flag	Coverage Δ
#multiple	`90.2% <ø> (-0.04%)`	⬇️
#single	`41.87% <ø> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
pandas/plotting/_converter.py	`65.07% <0%> (-1.74%)`	⬇️
pandas/core/nanops.py	`95.13% <0%> (-1.17%)`	⬇️
pandas/core/indexes/base.py	`96.68% <0%> (ø)`	⬆️
pandas/core/frame.py	`97.17% <0%> (ø)`	⬆️
pandas/core/accessor.py	`98.71% <0%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ae7e90...8b0e277. Read the comment docs.

TomAugspurger · 2018-04-20T16:18:15Z

Just one failure, on the OSX build. That's pinned to numpy 1.10.4, which forces fastparquet 0.1.3, which is not compatible with thrift 0.11. I could pin fastparquet & thrift to those versions, but I think fastparquet is adequately tested elsewhere, so I've removed it.

TomAugspurger · 2018-04-22T13:12:26Z

conda-forge is only used for packages that are not found in the main channel. The 3.5-osx env uses s3fs, moto, and feather which aren't in main.

…

On Sat, Apr 21, 2018 at 4:58 PM, Jeff Reback ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In ci/travis-27.yaml <#20767 (comment)>: > @@ -0,0 +1,49 @@ +name: pandas +channels: can we have one of the main builds NOT use conda-forge (maybe the 3.5 one)? in the original we had *only* the 3.6 build using it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20767 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIm1m0wI5okUh48J5wdN242UHGUIbks5tq6uFgaJpZM4Tdo8r> .

jreback · 2018-04-22T14:26:42Z

conda-forge is only used for packages that are not found in the main
channel. The 3.5-osx env uses s3fs, moto, and feather which aren't in main.

I understand. my point is in the existing setup, we install from defaults ONLY for most packages. I am loath to give that up on at least 1 build. I would either go back to 2 steps on 1 or more builds, or remove these deps (e.g. you can install from pip instead these packages).

jreback · 2018-04-22T14:26:59Z

I think you can completely remove the JOB= env variable as well here (or maybe rename ENV_FILE to JOB)

jreback · 2018-04-22T14:28:03Z

also I don't think the issue number is correct here, #20720 ?

TomAugspurger · 2018-04-23T11:10:29Z

you can install from pip instead these packages

I'd like to avoid installing moto from pip if possible. It brings in a lot of dependencies, which aren't managed by conda.

This closes #20720. It looks like that was caused by the 2 / 3 stage installs, since we ended up with multiple versions of requests, boto, etc., in the same environment. Now we just have the one. I haven't seen any of the network failures on this branch.

jreback · 2018-04-23T12:41:18Z

ci/install_travis.sh

-   pip install cpplint
-fi
-
-if [ "$COVERAGE" ]; then


where did coverage go?

The coverage deps are run on the 3.6 build. travis-36.yaml has the additional coverage deps.

jreback · 2018-04-23T12:42:01Z

ok I recently added some xfails for #20720, can you remove in this PR?

jorisvandenbossche · 2018-04-23T12:55:27Z

Does the conda-forge::packgage syntax work for environment files?
That might be another way to install some packages from conda-forge in an environment without adding conda-forge as a full channel (as then it might be more difficult to control which ones are installed from conda-forge / defaults)

jorisvandenbossche · 2018-04-23T12:57:31Z

Small note: I personally liked the JOB tags to have a quick overview of the different build types. I know the ENV_FILE tag is similarly unique, but a bit less user friendly to read or to refer to ("check travis job ...")

TomAugspurger · 2018-04-23T13:08:48Z

Does the conda-forge::packgage syntax work for environment files?

Not correctly unfortunately :( When I tried that, all the packages came from conda-forge. I suspect conda parses the deps for channels, and if found adds them to the start of the package list. Opened conda/conda#7202 for that.

This reverts commit f70a6dc.

TomAugspurger · 2018-04-23T13:11:18Z

Reverted the JOB removal.

jreback · 2018-04-23T13:30:14Z

Small note: I personally liked the JOB tags to have a quick overview of the different build types. I know the ENV_FILE tag is similarly unique, but a bit less user friendly to read or to refer to ("check travis job ...")

I don't have a problem with the slightly longer ENV_FILE, maybe just rename this to JOB?
having the duplicated ones is a bit annoying

jorisvandenbossche · 2018-04-23T13:42:51Z

Or maybe like before where the JOB tag is the variable part in the file name?

TomAugspurger · 2018-04-23T14:32:45Z

By file name, do you mean the environment.yaml? Because if so, I think that's the case. I don't really care what we do here. In general, I like the idea of a small tag that identifies the relevant bit for each build. Now that we aren't using the `{.build,run,pip,sh}` files, we're free to put whatever we want in `JOB`. I removed the unused 3.5 files.

…

On Mon, Apr 23, 2018 at 8:42 AM, Joris Van den Bossche < ***@***.***> wrote: Or maybe like before where the JOB tag is the variable part in the file name? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20767 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIqGuDliwjnIcLB8pYPZytFQzt3dfks5trdpfgaJpZM4Tdo8r> .

jorisvandenbossche · 2018-04-23T14:36:02Z

By file name, do you mean the environment.yaml? Because if so, I think that's the case.

Yes, that is what I meant. All environment files for travis are like ci/travis-specifics.yml, so JOB could just indicate 'specifics' (which would almost be identical to the current job tag, except for casing) to avoid duplication between JOB and ENV_FILE ?

TomAugspurger · 2018-04-23T18:02:48Z

Re-pushed with more informative job tags. Now they include things like indicating which one is linting, running coverage, etc.

Easier to tell at a glance which ones are special.

Closes pandas-dev#19760

This reverts commit cd50adb.

jreback · 2018-04-24T00:25:23Z

lgtm

merge away

TomAugspurger · 2018-04-24T13:58:34Z

FYI, the channel::package support in environment.yaml was recently added to conda in conda/conda#7178

I tried it out with conda-canary and things seemed to work fine.

TomAugspurger added 15 commits April 20, 2018 09:04

CI: compat

02b2879

CI: 27-locale

586132d

CI: 27

5bcee03

CI: 3.5

8f27bc1

CI: OSX

8d64136

CI: 3.6 slow

5f985ad

CI: 3.6 NumPy Dev

57eef01

DEBUG: show whole environment

5e9b6aa

CI: 3.6 DOC

f040101

CI: 3.6

d08da8f

Revert 2.7-compat

c9b4c68

CI: Revert 3.5

c86eba5

Updated travis install

5d19822

Limit coverage installs

Include conda-forge for 2.7

132e1d8

Removed markers

8071359

TomAugspurger added the CI Continuous Integration label Apr 20, 2018

TomAugspurger added 2 commits April 20, 2018 10:29

Added quiet to the install

828b08b

CI: Removed fastparquet from OSX

65e874a

That's pinned to numpy 1.10.4, which forces fastparquet 0.1.3, which is not compatible with thrift & pandas master.

TomAugspurger added 9 commits April 20, 2018 12:29

CI: Circle 27 compat

dce87b5

CI: circle-36-locale

3116c97

CI: circle-36-locale_slow

0d4ae51

CI: 3.5-ascii

480686b

CI: Remove redundant coverage install

692cc27

CI: updated circle

2ead1be

CI: Appveyor 27

3d29fcc

CI: appveyor-3.6

7dbd646

CI: Updated appveyor

5cf612d

TomAugspurger changed the title ~~[WIP]: Use conda-env for CI environments~~ Use conda-env for CI environments Apr 20, 2018

TomAugspurger added 2 commits April 23, 2018 06:06

CI: Remove conda-forge from 3.5

4161d65

remove JOB

f70a6dc

jreback reviewed Apr 23, 2018

View reviewed changes

TomAugspurger added 2 commits April 23, 2018 07:44

Merge remote-tracking branch 'upstream/master' into ci-envs

ec55f9b

Removed xfails

bf60ba2

Revert "remove JOB"

d8aa12f

This reverts commit f70a6dc.

Removed unused environment

b2e1a5f

Informative JOB tags

be43dd7

TomAugspurger added 2 commits April 23, 2018 13:46

BUG: Fixed NDFrame.transform('abs')

cd50adb

Closes pandas-dev#19760

Revert "BUG: Fixed NDFrame.transform('abs')"

8b0e277

This reverts commit cd50adb.

TomAugspurger merged commit 41db527 into pandas-dev:master Apr 24, 2018

TomAugspurger deleted the ci-envs branch April 24, 2018 01:35

gfyoung mentioned this pull request Oct 27, 2018

CLN: Remove pd.parser, lib, and tslib #23378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use conda-env for CI environments #20767

Use conda-env for CI environments #20767

TomAugspurger commented Apr 20, 2018 •

edited

Loading

codecov bot commented Apr 20, 2018 •

edited

Loading

TomAugspurger commented Apr 20, 2018

TomAugspurger commented Apr 22, 2018 via email

jreback commented Apr 22, 2018

jreback commented Apr 22, 2018 •

edited

Loading

jreback commented Apr 22, 2018

TomAugspurger commented Apr 23, 2018

jreback Apr 23, 2018

TomAugspurger Apr 23, 2018

jreback commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018 via email

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 24, 2018

TomAugspurger commented Apr 24, 2018

Use conda-env for CI environments #20767

Use conda-env for CI environments #20767

Conversation

TomAugspurger commented Apr 20, 2018 • edited Loading

codecov bot commented Apr 20, 2018 • edited Loading

Codecov Report

TomAugspurger commented Apr 20, 2018

TomAugspurger commented Apr 22, 2018 via email

jreback commented Apr 22, 2018

jreback commented Apr 22, 2018 • edited Loading

jreback commented Apr 22, 2018

TomAugspurger commented Apr 23, 2018

jreback Apr 23, 2018

Choose a reason for hiding this comment

TomAugspurger Apr 23, 2018

Choose a reason for hiding this comment

jreback commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 23, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018 via email

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 24, 2018

TomAugspurger commented Apr 24, 2018

TomAugspurger commented Apr 20, 2018 •

edited

Loading

codecov bot commented Apr 20, 2018 •

edited

Loading

jreback commented Apr 22, 2018 •

edited

Loading