Add test that fails on ppc64le #137

dopplershift · 2023-10-12T18:36:05Z

This tries to reproduce conda-forge/pyproj-feedstock#139 using only PROJ. Unfortunately, this won't directly yield an error, but will produce incorrect values if it fails.

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

This tries to reproduce conda-forge/pyproj-feedstock#139 using only PROJ. Unfortunately, this won't directly yield an error, but will produce incorrect values if it fails.

conda-forge-webservices · 2023-10-12T18:36:12Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

dopplershift · 2023-10-12T19:10:24Z

Hrmph, that gave the expected: 569722.40 4268814.27 0.00

djhoese · 2023-10-13T15:02:45Z

I commented on the pyproj PR referenced above, but the pyproj test has an xfail on a specific PROJ version 9.1.1:

@pytest.mark.xfail(PROJ_911, reason="Patch applied in conda-forge changes behavior")
def test_transform_single_point_nad83_to_nad27():

djhoese · 2023-10-13T17:01:21Z

Sorry, didn't realize conda-forge is well past the version being checked in this xfail. I'd also like to link this PR to conda-forge/pyproj-feedstock#144

djhoese · 2023-10-14T18:20:48Z

recipe/meta.yaml

    - echo -105 40 | proj +proj=utm +zone=13 +ellps=WGS84
    - echo -117 30 | cs2cs +proj=latlong +datum=NAD27 +to +proj=latlong +datum=NAD83
    - echo -105 40 | cs2cs +init=epsg:4326 +to +init=epsg:2975
+    - echo -92.199881 38.56694 | proj EPSG:26915 | cs2cs EPSG:26915 +to EPSG:26715


Based on conda-forge/pyproj-feedstock#144, the first command below should give the wrong answer (second component at about -3.54). I run it first because I noticed in pyproj testing that running cs2cs changes the results of later operations (at least in pyproj).

Suggested change

- echo -105 40 | proj +proj=utm +zone=13 +ellps=WGS84

- echo -117 30 | cs2cs +proj=latlong +datum=NAD27 +to +proj=latlong +datum=NAD83

- echo -105 40 | cs2cs +init=epsg:4326 +to +init=epsg:2975

- echo -92.199881 38.56694 | proj EPSG:26915 | cs2cs EPSG:26915 +to EPSG:26715

- echo 569704.57 4269024.67 | cs2cs EPSG:26915 +to EPSG:26715

- echo -105 40 | proj +proj=utm +zone=13 +ellps=WGS84

- echo -117 30 | cs2cs +proj=latlong +datum=NAD27 +to +proj=latlong +datum=NAD83

- echo -105 40 | cs2cs +init=epsg:4326 +to +init=epsg:2975

- echo -92.199881 38.56694 | proj EPSG:26915 | cs2cs EPSG:26915 +to EPSG:26715

djhoese · 2023-10-15T13:29:30Z

Could someone tell me why this feedstock uses the quay.io/condaforge/linux-anvil-cos7-x86_64 image for building ppc64le packages, but pyproj uses condaforge/linux-anvil-ppc64le?

xylar · 2023-10-15T14:26:12Z

@djhoese, the answer is that this feedstock is emulating ppc64le on linux64:
https://github.com/conda-forge/proj.4-feedstock/blob/main/conda-forge.yml#L3
This lets us run CI on Azure Pipelines, which we have found to be our most reliable option in terms of queue time, performance, etc.

pyproj is not emulating, it's running on ppc6rle CI using Travis CI:
https://github.com/conda-forge/pyproj-feedstock/blob/main/conda-forge.yml
(build platform is not specified for ppc64le so it defaults to native). This also has the upside that we can test the packages, whereas that's not always possible with emulation and cross-compiling.

This might explain differences you're seeing. We could try to build natively here but I believe we switched to emulation because the Travis CI jobs were timing out. We could switch to emulation on pyproj, with the downside that we would potentially lose testng.

@conda-forge/proj-4, please correct me if anything I said is nonsense!

xylar · 2023-10-15T14:28:42Z

It looks like the switch to cross-compiling happened inadvertently via a bot change in #128. Perhaps we want to switch it back.

djhoese · 2023-10-15T14:55:58Z

Thanks @xylar. FYI the test I proposed in my comment fails locally:

export PREFIX=/home/conda/feedstock_root/build_artifacts/proj_1697376485631/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p
export SRC_DIR=/home/conda/feedstock_root/build_artifacts/proj_1697376485631/test_tmp
+ test -f /home/conda/feedstock_root/build_artifacts/proj_1697376485631/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/libproj.so
+ test '!' -f /home/conda/feedstock_root/build_artifacts/proj_1697376485631/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/libproj.a
+ echo -92.199881 38.56694
+ proj EPSG:26915
569704.57       4269024.67
+ echo 569704.57 4269024.67
+ cs2cs EPSG:26915 +to EPSG:26715
589057.58       -3.54 0.00
+ echo 569704.57 4269024.67
+ cs2cs EPSG:26915 +to EPSG:26715
569722.40       4268814.27 0.00
+ echo -92.199881 38.56694
+ proj EPSG:26915
+ cs2cs EPSG:26915 +to EPSG:26715
569722.40       4268814.27 0.00

with these changes:

$ git diff
diff --git a/.ci_support/linux_ppc64le_.yaml b/.ci_support/linux_ppc64le_.yaml
index 2469c00..42bd29f 100644
--- a/.ci_support/linux_ppc64le_.yaml
+++ b/.ci_support/linux_ppc64le_.yaml
@@ -25,3 +25,9 @@ target_platform:
 zip_keys:
 - - c_compiler_version
   - cxx_compiler_version
+build_platform:
+- linux-64
+CMAKE_CROSSCOMPILING_EMULATOR:
+- /usr/bin/qemu-ppc64le-static
+CROSSCOMPILING_EMULATOR:
+- /usr/bin/qemu-ppc64le-static
diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 4ff808f..a84f5d5 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -40,6 +40,10 @@ test:
     - test -f ${PREFIX}/lib/libproj${SHLIB_EXT}  # [unix]
     - test ! -f ${PREFIX}/lib/libproj.a          # [unix]
     - if not exist %LIBRARY_LIB%\\proj.lib exit 1  # [win]
+    - echo -92.199881 38.56694 | proj EPSG:26915
+    - echo 569704.57 4269024.67 | cs2cs EPSG:26915 +to EPSG:26715
+    - echo 569704.57 4269024.67 | cs2cs EPSG:26915 +to EPSG:26715
+    - echo -92.199881 38.56694 | proj EPSG:26915 | cs2cs EPSG:26915 +to EPSG:26715
     - echo -105 40 | proj +proj=utm +zone=13 +ellps=WGS84
     - echo -117 30 | cs2cs +proj=latlong +datum=NAD27 +to +proj=latlong +datum=NAD83
     - echo -105 40 | cs2cs +init=epsg:4326 +to +init=epsg:2975

Not sure where the .ci_support changes came from.

But for the results, note how the same command outputs two different results. First:

589057.58       -3.54 0.00

then:

569722.40       4268814.27 0.00

djhoese · 2023-10-15T14:57:34Z

Perhaps off topic, but since you mentioned the libtiff PR @xylar, I've noticed a libtiff linking error on Python 3.8 (probably not supported by conda-forge anymore?) on MacOS when importing pyproj.

djhoese · 2023-10-15T15:38:29Z

@xylar How do I put it back to match pyproj's build process? I could test it locally so we can ignore travis limits and see if it actually fixes things.

xylar · 2023-10-15T15:44:56Z

@conda-forge-admin, please rerender.

xylar · 2023-10-15T15:45:22Z

@xylar How do I put it back to match pyproj's build process? I could test it locally so we can ignore travis limits and see if it actually fixes things.

What I've just done should hopefully take care of that once the rerender happens.

xylar · 2023-10-15T15:45:49Z

Perhaps off topic, but since you mentioned the libtiff PR @xylar, I've noticed a libtiff linking error on Python 3.8 (probably not supported by conda-forge anymore?) on MacOS when importing pyproj.

I think it would be better to open an issue for that. I'm afraid it will get lost here.

xylar · 2023-10-15T15:46:10Z

Python 3.8 is still supported, at least for now.

…nda-forge-pinning 2023.10.14.15.41.10

djhoese · 2023-10-15T18:16:41Z

I replicated the changes here and re-rendered and I see the same incorrect results with the ppc64le image being used.

djhoese · 2023-10-16T15:41:15Z

FYI I tried other EPSG codes and it seems these ones specifically are initializing something and it gives the wrong results the first time. The other EPSG codes run first in the output below return the same result between executions, but the old EPSG codes I've been using this whole time are wrong the first time, correct the second time:

+ echo 0 0
+ cs2cs EPSG:3034 +to EPSG:3035
363301.33       546152.98 0.00
+ echo 0 0
+ cs2cs EPSG:3034 +to EPSG:3035
363301.33       546152.98 0.00
+ echo 569704.57 4269024.67
+ cs2cs EPSG:26915 +to EPSG:26715
589057.58       -3.54 0.00
+ echo 569704.57 4269024.67
+ cs2cs EPSG:26915 +to EPSG:26715
569722.40       4268814.27 0.00

This may have gotten to the point that I need to try building PROJ without the conda infrastructure and see if I can reproduce it. I guess I could at least file a bug with the information I have so far.

djhoese · 2023-10-16T17:52:29Z

OSGeo/PROJ#3922

djhoese · 2023-10-17T01:16:14Z

@xylar I'll make another PR for the test I've put together since I don't have any permissions to this PR and I can do some more testing with conda-forge's CI. On the referenced PROJ issue above it was determined that the ppc64le emulation is actually the problem and Even was able to setup their Travis CI with a native ppc64le system and their version of my tests pass. Is there a conda-forge compatible way for me to do this without suddenly charging conda-forge a ton of Travis credits for something that isn't in the existing contract/plan?

xylar · 2023-10-17T09:04:07Z

Is there a conda-forge compatible way for me to do this without suddenly charging conda-forge a ton of Travis credits for something that isn't in the existing contract/plan?

You don't need to worry about this. Anything you do is in the noise on the scale of conda-forge and our resources are free. You might just have to sit in the queue.

Add test that fails on ppc64le

d3e17a4

This tries to reproduce conda-forge/pyproj-feedstock#139 using only PROJ. Unfortunately, this won't directly yield an error, but will produce incorrect values if it fails.

djhoese reviewed Oct 14, 2023

View reviewed changes

Stop cross-compiling for linux archs

495f0b7

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

e27c23d

…nda-forge-pinning 2023.10.14.15.41.10

djhoese mentioned this pull request Oct 16, 2023

Inconsistent results on ppc64le on conda-forge with gcc 12 OSGeo/PROJ#3922

Closed

djhoese mentioned this pull request Oct 17, 2023

Force ppc64le builds to use gcc 13 #139

Merged

5 tasks

dopplershift closed this Oct 20, 2023

dopplershift deleted the test-ppc64le branch October 20, 2023 17:02

snowman2 mentioned this pull request Mar 2, 2024

proj.4 v9.4.0 #141

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test that fails on ppc64le #137

Add test that fails on ppc64le #137

dopplershift commented Oct 12, 2023

conda-forge-webservices bot commented Oct 12, 2023

dopplershift commented Oct 12, 2023

djhoese commented Oct 13, 2023

djhoese commented Oct 13, 2023

djhoese Oct 14, 2023

djhoese commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 16, 2023

djhoese commented Oct 16, 2023

djhoese commented Oct 17, 2023

xylar commented Oct 17, 2023

Add test that fails on ppc64le #137

Add test that fails on ppc64le #137

Conversation

dopplershift commented Oct 12, 2023

conda-forge-webservices bot commented Oct 12, 2023

dopplershift commented Oct 12, 2023

djhoese commented Oct 13, 2023

djhoese commented Oct 13, 2023

djhoese Oct 14, 2023

Choose a reason for hiding this comment

djhoese commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

xylar commented Oct 15, 2023

djhoese commented Oct 15, 2023

djhoese commented Oct 16, 2023

djhoese commented Oct 16, 2023

djhoese commented Oct 17, 2023

xylar commented Oct 17, 2023