TST: Make ARM build work (not in the CI) #41739

datapythonista · 2021-05-31T01:04:02Z

closes CI: Set up CI for ARM #41737

This seems to be working, see:

https://app.circleci.com/pipelines/github/datapythonista/pandas/11/workflows/2ca0487f-5a63-4f01-936c-e83827db714f/jobs/11

But tests are failing. I opened #41740 the the failures.

fangchenli · 2021-05-31T04:32:36Z

ci/setup_env.sh

-if [[ "$UNAME_OS" == 'Linux' ]]; then
+DEFAULT_CONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-latest"
+if [[ "$(uname -m)" == 'aarch64' ]]; then
+    CONDA_URL="https://github.com/conda-forge/miniforge/releases/download/4.8.5-1/Miniforge3-4.8.5-1-Linux-aarch64.sh"


Could use the latest version https://github.com/conda-forge/miniforge/releases/download/4.10.1-4/Miniforge3-4.10.1-4-Linux-aarch64.sh

jreback · 2021-05-31T15:42:38Z

any idea why this is not showing up on the checks?

datapythonista · 2021-05-31T16:51:01Z

any idea why this is not showing up on the checks?

The configuration file for CircleCI needs to be in master before it appears. I can add this in two steps. A hello world CI first that activates CircleCI, and this one later, so we can see it running before merging.

jreback · 2021-05-31T18:05:23Z

any idea why this is not showing up on the checks?

The configuration file for CircleCI needs to be in master before it appears. I can add this in two steps. A hello world CI first that activates CircleCI, and this one later, so we can see it running before merging.

sure that would be great

lithomas1 · 2021-06-01T03:49:07Z

@datapythonista Are you sure we'll have enough credits for this? Looking at the plan, we get 2500 credits/week, which would be 250 minutes on Arm? Assuming a build takes 40 min, that would only be ~ 6 builds before we run out.

datapythonista · 2021-06-01T03:58:16Z

@datapythonista Are you sure we'll have enough credits for this? Looking at the plan, we get 2500 credits/week, which would be 250 minutes on Arm? Assuming a build takes 40 min, that would only be ~ 6 builds before we run out.

Thanks for making the numbers. I didn't check in detail, but I assumed with the volume of pandas the plan credits could not be enough. But since CircleCI seems to be the only option, I think it's worth getting started, and then if we need to move to nightly builds or something, it'd be better than not having anything, as we've got now. I hope we have at least enough credits to run the tests once per day. Not as good as running on PRs, but I think that would be good enough.

datapythonista · 2021-06-01T05:09:31Z

/azp run

azure-pipelines · 2021-06-01T05:09:46Z

Azure Pipelines successfully started running 1 pipeline(s).

simonjayhawkins · 2021-06-01T09:33:34Z

I think it's worth getting started, and then if we need to move to nightly builds or something, it'd be better than not having anything, as we've got now.

nightly builds should be happening on python-wheels. but is disabled to save credits for the releases and other projects.

If we do nightly builds, shouldn't this be on the wheels repo.

simonjayhawkins · 2021-06-01T09:35:10Z

nightly builds should be happening on python-wheels. but is disabled to save credits for the releases and other projects.

and because the job was rebuilding the release version and not from master MacPython/pandas-wheels#102

datapythonista · 2021-06-01T11:25:36Z

Ok, since this is now green, and there is no obvious way to move forward with having ARM builds, I think the best is to remove the CircleCI configuration in this PR. We can still merge the rest of changes, and have ARM passing as of today. And if a solution is found to automatically test on ARM, whether it's at each PR, nightly, or whenever, things should be easier to add with the xfails and other changes here.

Does it sound good?

simonjayhawkins · 2021-06-01T11:31:55Z

I've not looked in detail, but maybe the xfails should not be strict. IIUC the latest arm build for python 3.8 and 3.9 on pandas-wheels passed

simonjayhawkins · 2021-06-01T11:43:58Z

there is no obvious way to move forward with having ARM builds,

we have arm builds for conda for both mac (azure) and linux (drone). (IIUC no testing only build package for linux on drone), we have arm builds for PyPI for linux only (travis) (which run tests and fail to upload wheels if tests fail)

from #33971 (comment)

Also note that conda-forge has 1.2.0 packages for aarch64 as well as osx-arm64 packages. They have structurally better support; if you're on ARM64 right now you should strongly consider using conda-forge.

we need some testing on ci and consistency between pandas ci, conda and pip

jreback · 2021-06-01T11:50:18Z

@simonjayhawkins we don't have any arm builds

drone doesn't work for us and azure is emulated and very slow

yes actual wheels are built but w/o CI the only thing u can do us build the wheels and ignore the tests (not great but no option)

simonjayhawkins · 2021-06-01T11:59:22Z

Our basic problem is that the tests are failing on Travis in pandas-wheels on release and no nightly builds in pandas-wheels. The credits issue is part of the problem, but not having visibility of failing tests (on pandas ci and nightly builds) is probably more of an issue. If we cannot solve this issue with the limited credits, then we should also consider moving the pandas-wheels builds to CircleCI for consistency.

jreback · 2021-06-01T12:05:07Z

ignore the arm tests entirely in pandas-wheels

we likely cannot move the wheel builds easily

simonjayhawkins · 2021-06-01T12:09:48Z

ignore the arm tests entirely in pandas-wheels

that's also an option

simonjayhawkins · 2021-06-01T12:12:37Z

I've not looked in detail, but maybe the xfails should not be strict. IIUC the latest arm build for python 3.8 and 3.9 on pandas-wheels passed

if we ignore the tests in pandas-wheels, this won't be an issue.

datapythonista · 2021-06-01T21:09:44Z

I contacted CircleCI to check if they could be interested in sponsoring our ARM builds. Will update if I hear from them. For now looks like we're getting the 2,500 credits that @lithomas1 mentioned (not the 400,000 for open source projects for Linux builds). So, not much we can do with those.

I updated this PR so we don't add the CircleCI build, but still, it has everything else so the build passes when it can be added wherever. Only thing is that we now need to disable Circle CI, so the build doesn't fail after we remove the configuration. But only an admin of the pandas-dev organization can do it. @jreback do you mind going to https://app.circleci.com/settings/project/github/pandas-dev/pandas?return-to=https%3A%2F%2Fapp.circleci.com%2Fpipelines%2Fgithub%2Fpandas-dev%2Fpandas%2F30%2Fworkflows%2Ff9a6db17-510a-4eaf-a10c-b3a800e0a6c4%2Fjobs%2F30872 and clickig on "Stop builds" please?

jreback · 2021-06-01T22:28:37Z

I contacted CircleCI to check if they could be interested in sponsoring our ARM builds. Will update if I hear from them. For now looks like we're getting the 2,500 credits that @lithomas1 mentioned (not the 400,000 for open source projects for Linux builds). So, not much we can do with those.

I updated this PR so we don't add the CircleCI build, but still, it has everything else so the build passes when it can be added wherever. Only thing is that we now need to disable Circle CI, so the build doesn't fail after we remove the configuration. But only an admin of the pandas-dev organization can do it. @jreback do you mind going to https://app.circleci.com/settings/project/github/pandas-dev/pandas?return-to=https%3A%2F%2Fapp.circleci.com%2Fpipelines%2Fgithub%2Fpandas-dev%2Fpandas%2F30%2Fworkflows%2Ff9a6db17-510a-4eaf-a10c-b3a800e0a6c4%2Fjobs%2F30872 and clickig on "Stop builds" please?

ok did that

jreback · 2021-06-01T22:29:54Z

thanks @datapythonista

cc @simonjayhawkins maybe could backport to skip these tests on arm

simonjayhawkins · 2021-06-02T08:00:00Z

cc @simonjayhawkins maybe could backport to skip these tests on arm

sure. IIUC we want to update pandas-wheels, #41739 (comment)

@meeseeksdev backport 1.2.x

simonjayhawkins · 2021-06-02T08:05:50Z

pandas/compat/__init__.py

@@ -92,6 +92,18 @@ def is_platform_mac() -> bool:
    return sys.platform == "darwin"


+def is_platform_arm() -> bool:
+    """
+    Checking if he running platform use ARM architecture.


typo for follow-up

simonjayhawkins · 2021-06-02T08:10:44Z

pandas/tests/tools/test_to_numeric.py

@@ -750,7 +752,7 @@ def test_to_numeric_from_nullable_string(values, nullable_string_dtype, expected
            "UInt64",
            "signed",
            "UInt64",
-            marks=pytest.mark.xfail(reason="GH38798"),
+            marks=pytest.mark.xfail(not is_platform_arm(), reason="GH38798"),


this test added in #38746, not on 1.2.x

simonjayhawkins · 2021-06-02T08:15:40Z

pandas/tests/indexes/interval/test_astype.py

@@ -168,6 +170,7 @@ def test_subtype_integer_with_non_integer_borders(self, subtype):
        )
        tm.assert_index_equal(result, expected)

+    @pytest.mark.xfail(is_platform_arm(), reason="GH 41740")


This one is already xfailed on 1.2.x, xref #38719

Co-authored-by: Marc Garcia <garcia.marc@gmail.com>

fangchenli · 2021-06-04T01:22:15Z

pandas/tests/window/test_rolling.py

@@ -1072,6 +1073,7 @@ def test_rolling_sem(frame_or_series):
    tm.assert_series_equal(result, expected)


+@pytest.mark.xfail(is_platform_arm(), reason="GH 41740")


This only fails on ARM64 Linux. It passes on M1 Mac. #38921

datapythonista added 10 commits May 30, 2021 16:16

CI: Draft of CircleCI for ARM

49a570f

Fixes to circle ci build

7cd7aaf

Fix typo

b3a00f1

Clean up and debug info

b3db6bd

Fixing arm downloading right conda

6e9457b

passing parameters to run tests

32ec532

Improving cirle ci config

78adf49

Restoring workflow

5acb475

Adding debug info

a7c7c79

Fixing PATH, removing debug info

fccf727

datapythonista added CI Continuous Integration ARM aarch64 architecture labels May 31, 2021

datapythonista mentioned this pull request May 31, 2021

TST: Tests broken in ARM #41740

Open

fangchenli requested changes May 31, 2021

View reviewed changes

Updating conda arm url

7df1f52

datapythonista mentioned this pull request Jun 1, 2021

CI: Activating CircleCI #41752

Merged

datapythonista added 2 commits May 31, 2021 20:00

Merge master

0f1257f

Xfailing broken ARM tests

8ebadbf

Fixing xpass

c0fd82d

Removing CircleCI config

e8e2cf9

datapythonista changed the title ~~CI: Add ARM build in CircleCI~~ TST: Make ARM build work (not in the CI) Jun 1, 2021

jreback added this to the 1.3 milestone Jun 1, 2021

jreback merged commit 44237a5 into pandas-dev:master Jun 1, 2021

This comment has been minimized.

Sign in to view

lumberbot-app bot added the Still Needs Manual Backport label Jun 2, 2021

simonjayhawkins reviewed Jun 2, 2021

View reviewed changes

simonjayhawkins mentioned this pull request Jun 2, 2021

Backport PR #41739: TST: Make ARM build work (not in the CI) #41781

Merged

simonjayhawkins removed the Still Needs Manual Backport label Jun 2, 2021

simonjayhawkins added a commit that referenced this pull request Jun 2, 2021

Backport PR #41739: TST: Make ARM build work (not in the CI) (#41781)

29e8da1

Co-authored-by: Marc Garcia <garcia.marc@gmail.com>

fangchenli reviewed Jun 4, 2021

View reviewed changes

lithomas1 mentioned this pull request Jun 5, 2021

CI: Re-enable Circle CI #41829

Closed

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021

TST: Make ARM build work (not in the CI) (pandas-dev#41739)

97faa71

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: Make ARM build work (not in the CI) #41739

TST: Make ARM build work (not in the CI) #41739

datapythonista commented May 31, 2021 •

edited

Loading

fangchenli May 31, 2021

jreback commented May 31, 2021

datapythonista commented May 31, 2021

jreback commented May 31, 2021

lithomas1 commented Jun 1, 2021

datapythonista commented Jun 1, 2021

datapythonista commented Jun 1, 2021

azure-pipelines bot commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

datapythonista commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

datapythonista commented Jun 1, 2021

jreback commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 2, 2021

This comment has been minimized.

simonjayhawkins Jun 2, 2021

simonjayhawkins Jun 2, 2021

simonjayhawkins Jun 2, 2021

fangchenli Jun 4, 2021

		@@ -1072,6 +1073,7 @@ def test_rolling_sem(frame_or_series):
		tm.assert_series_equal(result, expected)


		@pytest.mark.xfail(is_platform_arm(), reason="GH 41740")

TST: Make ARM build work (not in the CI) #41739

TST: Make ARM build work (not in the CI) #41739

Conversation

datapythonista commented May 31, 2021 • edited Loading

fangchenli May 31, 2021

Choose a reason for hiding this comment

jreback commented May 31, 2021

datapythonista commented May 31, 2021

jreback commented May 31, 2021

lithomas1 commented Jun 1, 2021

datapythonista commented Jun 1, 2021

datapythonista commented Jun 1, 2021

azure-pipelines bot commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

datapythonista commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021

datapythonista commented Jun 1, 2021

jreback commented Jun 1, 2021

jreback commented Jun 1, 2021

simonjayhawkins commented Jun 2, 2021

This comment has been minimized.

simonjayhawkins Jun 2, 2021

Choose a reason for hiding this comment

simonjayhawkins Jun 2, 2021

Choose a reason for hiding this comment

simonjayhawkins Jun 2, 2021

Choose a reason for hiding this comment

fangchenli Jun 4, 2021

Choose a reason for hiding this comment

datapythonista commented May 31, 2021 •

edited

Loading