CI: Adding build for ARM64 #30641

ossdev07 · 2020-01-03T11:10:46Z

Added arm64 test support in travis-ci .
Modified environment creation by using archiconda instead of miniconda as miniconda is not supported in arm64 currently .

closes Add arm64 builds to CI #28986
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jreback

pls make minimal changes this is changing almost everything

ci/run_tests.sh

.travis.yml

ci/run_tests.sh

ci/setup_env.sh

datapythonista

Not sure if we should be testing ARM, since NumPy is not testing it.

But looks like some old changes got mixed with this PR. And also, I don't think we need most of the if statements that you have. I'd start this PR from zero, you shouldn't need much more than the new build in .travis.yml, the dependencies file, and the downloading the Anaconda installer for ARM.

ci/setup_env.sh

ci/deps/travis-37-aarch64.yaml

datapythonista

Can you please address all the comments and then push? Thanks!

.travis.yml

ci/run_tests.sh

ci/setup_env.sh

WillAyd · 2020-02-12T00:56:59Z

@ossdev07 still active? If so can you fix merge conflicts and address comments?

ossdev07 · 2020-02-13T11:56:26Z

Hi @WillAyd , I am working on it. I will let you know once I am done.

ossdev07 · 2020-03-03T13:25:18Z

Hi @WillAyd and @datapythonista,

I have removed archiconda and updated it to miniforge. It does not require sudo permissions to run conda commands as it is similar to miniconda. By this we can remove all the unwanted code as suggested previously.
For arm64 xvfb need to be installed seperately, so added in apt-addons in travis.
Flags "-n auto --dist=loadfile -s --strict" mentioned in pytest command in run_tests.sh are giving timeout issue in arm64 platform. So, I have removed the same for arm64 platform.

Please review and provide feedback if any more updates are required.

datapythonista

@ossdev07 looks quite good now, added few more comments

ci/run_tests.sh

ci/setup_env.sh

ci/deps/travis-37-aarch64.yaml

.travis.yml

ci/deps/travis-37-arm64.yaml

ci/setup_env.sh

datapythonista

@ossdev07 can you address the pending comments please, this is close now.

.travis.yml

ci/setup_env.sh

ossdev07 · 2020-03-09T13:02:53Z

@datapythonista ,

Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is 887591e.

datapythonista · 2020-03-09T15:50:56Z

Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is 887591e.

You'll have to fix your branch for us to merge this. I wrote some time ago on what I usually do in these cases in this post: https://datapythonista.me/blog/useful-git-commands.html

Hope that helps.

ossdev07 · 2020-03-11T11:35:21Z

Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is 887591e.

You'll have to fix your branch for us to merge this. I wrote some time ago on what I usually do in these cases in this post: https://datapythonista.me/blog/useful-git-commands.html

Hope that helps.

Thanks @datapythonista. I have resolved the conflicts in the commits and added the commit with changes suggested. Please review and provide feedback if anything needs to be updated.

ci/run_tests.sh

.travis.yml

ci/setup_env.sh

rhenwood-arm · 2020-03-11T14:15:19Z

(I work with @ossdev07 ) : @ossdev07 as an independent task, can you make a note of the missing xvfb for Travis-CI in their community forums: https://travis-ci.community/

ossdev07 · 2020-03-11T14:57:56Z

(I work with @ossdev07 ) : @ossdev07 as an independent task, can you make a note of the missing xvfb for Travis-CI in their community forums: https://travis-ci.community/

@rhenwood-arm ,

I will take a note of it and will raise it in the travis community.

datapythonista · 2020-03-13T15:49:55Z

Travis is red, can you have a look and see what's wrong please

ossdev07 · 2020-03-16T05:59:48Z

Travis is red, can you have a look and see what's wrong please

@datapythonista ,

After adding all flags in pytest command, it is failing with the timeout issue at different steps for each build for arm64 platform. That is the reason I have removed those three flags.

Could you please suggest if there is any resolution to run without these flags for a specific platform?

datapythonista · 2020-03-18T19:25:26Z

IIRC the flags you changed were for pytest, and in this case pytest didn't even started, so that's surely not going to solve the problem.

The build was stalled during ci/submit_cython_cache.sh. Can you merge master, so we can see if the problem happens at the same stage. I'm wondering if Travis is reliable for ARM64...

jreback

you need to merge master, lots of things are failing.

jreback · 2020-05-08T12:24:36Z

.travis.yml

@@ -33,6 +35,10 @@ matrix:
    - env:
        - JOB="3.7" ENV_FILE="ci/deps/travis-37.yaml" PATTERN="(not slow and not network and not clipboard)"

+    - arch: arm64
+      env:
+        - JOB="3.7" PYTEST_WORKERS=8 ENV_FILE="ci/deps/travis-37-arm64.yaml" PATTERN="(not slow and not network and not clipboard)"


you are likely to have much better performance with 2 workers here. I would guess that's what we have physically, and what we use on other processors.

I have updated to 2 workers as suggested. Please review.

it still using 8 on the arm build - make this 2

and taking 56 minutes which is too long

put the build in the allowed failures section

I have updated to 2 workers in the previous commit, but the build was getting stalled and failing while running tests. So, I have moved back to 8 workers. Pandas compilation and test execution are taking more time to complete in ARM64 when compared to the AMD64 platform. I have added arm64 build in allowed_failures, but it is not getting triggered in Travis after the commit.

jreback · 2020-05-08T13:58:46Z

these changes are breaking other builds: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=35039&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9&l=17

cphang99 · 2020-05-09T13:52:23Z

Do the workarounds described in #17792 need to be incorporated in order for the test suite to pass?

cphang99 · 2020-05-09T16:17:29Z

It looks like it does. many of the failing tests appear to be DateTime tests as described in #17792

More details in https://gitlab.com/libreml/libreml/-/issues/110#note_339463783

ossdev07 · 2020-05-11T09:15:30Z

these changes are breaking other builds: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=35039&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9&l=17

I have added global variable PYTEST_WORKERS in azure pipelines yml file and issues have been resolved with my azure account. Please review the changes and let me know if any more changes required.

ossdev07 · 2020-05-11T11:12:49Z

@datapythonista / @jreback / @WillAyd ,

All checks are passing as well now. Please review and let me know if anything needs to be updated.

jreback · 2020-05-11T11:52:52Z

this build takes 56 minutes which is way longer than the longest build

jreback · 2020-05-11T19:13:14Z

.travis.yml

-  - dist: bionic
-    python: 3.9-dev
-    env:
+    - arch: arm64


I think you have have to have it in both sections (allow_failures) and the top section; they must be identical.

Thanks for the suggestion. I have added it as suggested and all checks are passing as well. Please review and let me know if any more updates are required.

Signed-off-by: ossdev07 <ossdev@puresoftware.com>

jreback

lgtm. @TomAugspurger ok here

TomAugspurger · 2020-05-12T13:34:05Z

Thanks. So to confirm: we're putting it under allowed failures just because it takes longer? We'd consider any failures on that job to be a blocker to merging / cause for reverting if a failure is noticed?

@ossdev07 will you be available to respond to build / CI issues as they come up?

ossdev07 · 2020-05-12T13:44:29Z

Thanks. So to confirm: we're putting it under allowed failures just because it takes longer? We'd consider any failures on that job to be a blocker to merging / cause for reverting if a failure is noticed?

@ossdev07 will you be available to respond to build / CI issues as they come up?

Yes, I will be available to respond if any issues come up.

jreback · 2020-05-12T14:02:43Z

yeah i think under allowed failures for now until we can get the time to be more reasonable see #34115

@ossdev07 feel free to comment on the issue about building wheels for this (note we would have to actualy run this on the azure side and not here, is this possible?)

can you create an issue about moving this out of allowed failures (we can do this as soon as we see this working reasonbaly for a little)

ossdev07 · 2020-05-12T15:58:02Z

yeah i think under allowed failures for now until we can get the time to be more reasonable see #34115

@ossdev07 feel free to comment on the issue about building wheels for this (note we would have to actualy run this on the azure side and not here, is this possible?)

can you create an issue about moving this out of allowed failures (we can do this as soon as we see this working reasonbaly for a little)

Thanks for the approval. I will raise a ticket about moving this out of allowed failures.

jreback requested changes Jan 3, 2020

View reviewed changes

simonjayhawkins added the CI Continuous Integration label Jan 3, 2020

ossdev07 force-pushed the pandas_ci_arm64 branch from f498d47 to ab43d71 Compare January 3, 2020 12:25

datapythonista requested changes Jan 3, 2020

View reviewed changes

ci/setup_env.sh Outdated Show resolved Hide resolved

ci/setup_env.sh Outdated Show resolved Hide resolved

ci/setup_env.sh Outdated Show resolved Hide resolved

datapythonista changed the title ~~added travis-ci support for arm64~~ CI: Adding build for ARM64 Jan 3, 2020

jbrockmendel reviewed Jan 3, 2020

View reviewed changes

ci/deps/travis-37-aarch64.yaml Outdated Show resolved Hide resolved

datapythonista requested changes Jan 6, 2020

View reviewed changes

ossdev07 force-pushed the pandas_ci_arm64 branch from f85f420 to e258988 Compare March 3, 2020 13:16

datapythonista reviewed Mar 3, 2020

View reviewed changes

ci/run_tests.sh Outdated Show resolved Hide resolved

ci/setup_env.sh Outdated Show resolved Hide resolved

ci/deps/travis-37-aarch64.yaml Outdated Show resolved Hide resolved

jreback requested changes Mar 5, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

ci/deps/travis-37-arm64.yaml Show resolved Hide resolved

ci/setup_env.sh Show resolved Hide resolved

datapythonista reviewed Mar 7, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

ci/setup_env.sh Show resolved Hide resolved

ossdev07 force-pushed the pandas_ci_arm64 branch from 887591e to 06b46fe Compare March 11, 2020 11:30

jreback requested changes Mar 11, 2020

View reviewed changes

ci/run_tests.sh Outdated Show resolved Hide resolved

datapythonista reviewed Mar 11, 2020

View reviewed changes

ci/run_tests.sh Outdated Show resolved Hide resolved

.travis.yml Show resolved Hide resolved

ossdev07 force-pushed the pandas_ci_arm64 branch from 06b46fe to 5f498d5 Compare March 11, 2020 13:30

datapythonista reviewed Mar 11, 2020

View reviewed changes

ci/setup_env.sh Outdated Show resolved Hide resolved

ossdev07 force-pushed the pandas_ci_arm64 branch from 5f498d5 to da2cdad Compare March 13, 2020 07:07

jreback mentioned this pull request Mar 15, 2020

travis: enable bionic & multi-cpu testing #32237

Closed

5 tasks

ossdev07 force-pushed the pandas_ci_arm64 branch from da2cdad to 4a027fe Compare March 19, 2020 06:53

jreback requested changes May 8, 2020

View reviewed changes

ossdev07 force-pushed the pandas_ci_arm64 branch from 269b57b to e251fec Compare May 8, 2020 13:30

ossdev07 force-pushed the pandas_ci_arm64 branch from e251fec to 74194d8 Compare May 11, 2020 09:12

ossdev07 force-pushed the pandas_ci_arm64 branch from 74194d8 to c957ee6 Compare May 11, 2020 10:07

ossdev07 force-pushed the pandas_ci_arm64 branch from c957ee6 to 5169dba Compare May 11, 2020 11:31

jreback reviewed May 11, 2020

View reviewed changes

ossdev07 and others added 4 commits May 12, 2020 04:30

added travis-ci support for arm64

38a6772

Signed-off-by: ossdev07 <ossdev@puresoftware.com>

Update run_tests.sh

854b7f8

Switched from Archiconda to Miniforge

900937d

Signed-off-by: ossdev07 <ossdev@puresoftware.com>

Switched from Archiconda to Miniforge

92d0c12

ossdev07 force-pushed the pandas_ci_arm64 branch from 5169dba to 92d0c12 Compare May 12, 2020 04:33

ossdev07 requested a review from jreback May 12, 2020 05:53

jreback mentioned this pull request May 12, 2020

CI/TST: slow tests #34131

Closed

jreback approved these changes May 12, 2020

View reviewed changes

jreback added this to the 1.1 milestone May 12, 2020

Merge branch 'master' into pandas_ci_arm64

7c85263

jreback merged commit f5ea8ca into pandas-dev:master May 12, 2020

mroeschke mentioned this pull request May 12, 2020

Add arm64 builds to CI #28986

Closed

ossdev07 mentioned this pull request May 13, 2020

CI: Moving ARM64 build from allowed failures #34149

Closed

WillAyd mentioned this pull request Oct 23, 2020

Fix unable to build Pandas with xlc on z/OS #35829

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Adding build for ARM64 #30641

CI: Adding build for ARM64 #30641

ossdev07 commented Jan 3, 2020

jreback left a comment

datapythonista left a comment

datapythonista left a comment

WillAyd commented Feb 12, 2020

ossdev07 commented Feb 13, 2020

ossdev07 commented Mar 3, 2020

datapythonista left a comment

datapythonista left a comment

ossdev07 commented Mar 9, 2020

datapythonista commented Mar 9, 2020

ossdev07 commented Mar 11, 2020

rhenwood-arm commented Mar 11, 2020

ossdev07 commented Mar 11, 2020

datapythonista commented Mar 13, 2020

ossdev07 commented Mar 16, 2020

datapythonista commented Mar 18, 2020

jreback left a comment

jreback May 8, 2020

ossdev07 May 8, 2020

jreback May 11, 2020

ossdev07 May 11, 2020 •

edited

Loading

jreback commented May 8, 2020

cphang99 commented May 9, 2020

cphang99 commented May 9, 2020

ossdev07 commented May 11, 2020

ossdev07 commented May 11, 2020

jreback commented May 11, 2020

jreback May 11, 2020

ossdev07 May 12, 2020

jreback left a comment

TomAugspurger commented May 12, 2020

ossdev07 commented May 12, 2020

jreback commented May 12, 2020

ossdev07 commented May 12, 2020

CI: Adding build for ARM64 #30641

CI: Adding build for ARM64 #30641

Conversation

ossdev07 commented Jan 3, 2020

jreback left a comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

WillAyd commented Feb 12, 2020

ossdev07 commented Feb 13, 2020

ossdev07 commented Mar 3, 2020

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

ossdev07 commented Mar 9, 2020

datapythonista commented Mar 9, 2020

ossdev07 commented Mar 11, 2020

rhenwood-arm commented Mar 11, 2020

ossdev07 commented Mar 11, 2020

datapythonista commented Mar 13, 2020

ossdev07 commented Mar 16, 2020

datapythonista commented Mar 18, 2020

jreback left a comment

Choose a reason for hiding this comment

jreback May 8, 2020

Choose a reason for hiding this comment

ossdev07 May 8, 2020

Choose a reason for hiding this comment

jreback May 11, 2020

Choose a reason for hiding this comment

ossdev07 May 11, 2020 • edited Loading

Choose a reason for hiding this comment

jreback commented May 8, 2020

cphang99 commented May 9, 2020

cphang99 commented May 9, 2020

ossdev07 commented May 11, 2020

ossdev07 commented May 11, 2020

jreback commented May 11, 2020

jreback May 11, 2020

Choose a reason for hiding this comment

ossdev07 May 12, 2020

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

TomAugspurger commented May 12, 2020

ossdev07 commented May 12, 2020

jreback commented May 12, 2020

ossdev07 commented May 12, 2020

ossdev07 May 11, 2020 •

edited

Loading