Fix Windows CI builds #361

duckontheweb · 2021-05-25T16:34:07Z

Related Issues:

Failing tests on Windows #362

Description:

Fix Windows CI jobs
Uses shell: bash to ensure we're running a bash shell, even on Windows.
Fix dependency caching in CI jobs
The cache keys are now specific to Python version and OS and point to the correct cache directories for each OS.
Create temp directories in current working directory
In the GitHub Actions Windows runner the default TMPDIR directory is on a different drive (C:\\) than the code and test data files (D:\\). This was causing failures in os.path.relpath on Windows.
Separates lint and test jobs
The lint and type checking jobs only need to run once per Python version since they are not platform-specific.
Use safe joins and parsing for all paths
In pystac.utils renames _urlparse -> safe_urlparse and _join -> join_path_or_url and uses these functions throughout the codebase and tests to ensure we are joining paths properly. There were a number of places where os.path.join was being used to join URL paths, which caused failures on Windows. Also changes the signature of join_path_or_url to take a JoinType enum as the first argument to make the usage more clear.
Preserves the case of the drive in Windows paths.
When using functions like pystac.utils.make_absolute_href, the drive letter was being converted to lowercase. The safe_urlparse function now preserves the original casing from the start path (see the changes to the tests in 8e9455 for an example)
Uses explicit utf-8 encoding for all file I/O operations
The default encoding on Windows is not necessarily UTF-8, which was causing decoding errors when trying to read test data files. This uses encoding="utf-8" for all file read/write operations to avoid conflicts
Creates separate functions for handling URL v. path cases in make_absolute_href and make_relative_href
When debugging the Windows issues, it was pretty difficult to reason through how a URL would be handled in those functions vs. a local path. This PR creates distinct internal functions to handle each case. The result is some redundancy in the code, but I think the trade-off in clarity is worth it.

UPDATE 2021-06-03

Adds setup.py to linting and auto-formatting
Moves version config from setup.py to setup.cfg. This means we no longer have to use the deprecated imp module.

PR Checklist:

Code is formatted (run scripts/format)
Tests pass (run scripts/test) (see comments above)
This PR maintains or improves overall codebase code coverage.
Changes are added to the CHANGELOG. See the docs for information about adding to the changelog.

codecov-commenter · 2021-05-25T16:36:06Z

Codecov Report

Merging #361 (d23be0a) into main (4658999) will increase coverage by 0.04%.
The diff coverage is 97.77%.

@@            Coverage Diff             @@
##             main     #361      +/-   ##
==========================================
+ Coverage   89.64%   89.69%   +0.04%     
==========================================
  Files          39       39              
  Lines        5169     5209      +40     
==========================================
+ Hits         4634     4672      +38     
- Misses        535      537       +2

Impacted Files	Coverage Δ
pystac/utils.py	`96.12% <96.72%> (-0.91%)`	⬇️
pystac/layout.py	`92.93% <100.00%> (+0.49%)`	⬆️
pystac/stac_io.py	`72.46% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4658999...d23be0a. Read the comment docs.

duckontheweb · 2021-05-28T20:26:28Z

There are still a number of path-related failures on Windows. I opened #362 to track fixes to those rather than try to fix them as part of this PR.

Actually, I think I have some fixes to most of these. I'll update the PR shortly.

- OS-specific dependency caching in GitHub Actions - Separate test requirements for faster builds - Separate lint and test CI jobs

These tests have test cases which are specific to posix paths and will cause failures when run on Windows.

The default encoding on Windows is not necessarily utf-8, which can lead to problems when reading/writing across platforms. This explicitly uses utf-8 for all file encoding & decoding.

The default temp directory in GitHub Actions Windows runners is on the C: drive, but the test code and data files are on the D: drive. This was leading to CI failures due to an error in ntpath when trying to determine relative paths.

This makes the utils for safely parsing and joining URLs "public" and uses them throughout the code base and tests to ensure that we don't use platform-specific separators when it's not appropriate (i.e. for URLs).

Fixes some issues where relative and absolute hrefs were not being constructed correctly for URLs on Windows. Also creates separate internal functions for handling these 2 cases to make the logic more clear.

…clean # Conflicts: # requirements-dev.txt # scripts/test # tests/extensions/test_label.py # tests/test_catalog.py # tests/test_layout.py # tests/test_utils.py # tests/test_writing.py

duckontheweb · 2021-06-01T17:39:35Z

@volaya I would be interested in your review of this since I think it addresses some of the issue you were finding in #271 .

@l0b0 I would also be interested in your review since you've been thinking about pre-commit hooks and CI in #366 .

.github/workflows/continuous-integration.yml

scripts/lint

.github/workflows/continuous-integration.yml

Goes back to using more specific pip- prefix for dependency caches and doesn't needlessly install package for linting

Based heavily on <stac-utils#361>.

pystac/utils.py

setup.cfg

requirements-dev.txt

lossyrob

LGTM. Added a couple of test cases I was curios about in duckontheweb#1, these passed.

tests/data-files/get_examples.py

pystac/utils.py

tests/utils/__init__.py

duckontheweb · 2021-06-04T15:47:46Z

@l0b0 I think I've addressed all of your comments. Let me know what you think.

l0b0 · 2021-06-04T21:12:40Z

@l0b0 I think I've addressed all of your comments. Let me know what you think.

LGTM, cheers!

duckontheweb force-pushed the windows-ci-fixes branch from d8e7cd3 to 79eacfd Compare May 25, 2021 18:50

duckontheweb changed the title ~~[WIP] Fix Windows CI builds~~ Fix Windows CI builds May 25, 2021

duckontheweb requested review from lossyrob and kbgg May 25, 2021 18:53

duckontheweb mentioned this pull request May 25, 2021

Ensure correct url format from local files #271

Closed

duckontheweb force-pushed the windows-ci-fixes branch from 2cfc0ea to 9ddc7a9 Compare May 29, 2021 03:17

duckontheweb removed request for kbgg and lossyrob May 31, 2021 19:40

duckontheweb added 7 commits June 1, 2021 10:33

CI updates

d1cbc57

- OS-specific dependency caching in GitHub Actions - Separate test requirements for faster builds - Separate lint and test CI jobs

Skip posix-specific tests on Windows

b0a8647

These tests have test cases which are specific to posix paths and will cause failures when run on Windows.

Use explicit utf-8 encoding

97653a3

The default encoding on Windows is not necessarily utf-8, which can lead to problems when reading/writing across platforms. This explicitly uses utf-8 for all file encoding & decoding.

Use CWD for temp test directories

e6961b5

The default temp directory in GitHub Actions Windows runners is on the C: drive, but the test code and data files are on the D: drive. This was leading to CI failures due to an error in ntpath when trying to determine relative paths.

Use platform-independent path joins for all URL joins

0af9df8

This makes the utils for safely parsing and joining URLs "public" and uses them throughout the code base and tests to ensure that we don't use platform-specific separators when it's not appropriate (i.e. for URLs).

Fix path handling for relative/absolute hrefs on Windows

f18e90d

Fixes some issues where relative and absolute hrefs were not being constructed correctly for URLs on Windows. Also creates separate internal functions for handling these 2 cases to make the logic more clear.

Some lint fixes

612a737

duckontheweb force-pushed the windows-ci-fixes branch from c92c0df to 612a737 Compare June 1, 2021 15:54

duckontheweb added 2 commits June 1, 2021 12:10

Merge remote-tracking branch 'stac-utils/main' into windows-ci-fixes-…

6e0eef7

…clean # Conflicts: # requirements-dev.txt # scripts/test # tests/extensions/test_label.py # tests/test_catalog.py # tests/test_layout.py # tests/test_utils.py # tests/test_writing.py

Preserve case of drives in Windows paths

8e9455c

duckontheweb requested a review from lossyrob June 1, 2021 17:37

duckontheweb marked this pull request as ready for review June 1, 2021 17:38

duckontheweb mentioned this pull request Jun 1, 2021

raster extension and trouble opening the test data #383

Closed

l0b0 reviewed Jun 1, 2021

View reviewed changes

duckontheweb added 5 commits June 1, 2021 19:47

Updates to workflow config

a858c4c

Goes back to using more specific pip- prefix for dependency caches and doesn't needlessly install package for linting

Merge remote-tracking branch 'stac-utils/main' into windows-ci-fixes

a5dcc7e

Lint setup.py

316c27c

Use importlib instead of deprecated imp

cab1f11

Merge remote-tracking branch 'stac-utils/main' into windows-ci-fixes

e014738

Put version in setup.cfg

b6e778a

duckontheweb mentioned this pull request Jun 3, 2021

Introduce pre-commit #401

Merged

4 tasks

l0b0 added a commit to l0b0/pystac that referenced this pull request Jun 3, 2021

refactor: Split apart linting and testing

f510bc1

Based heavily on <stac-utils#361>.

l0b0 added a commit to l0b0/pystac that referenced this pull request Jun 3, 2021

refactor: Split apart linting and testing

b1bb65e

Based heavily on <stac-utils#361>.

duckontheweb added 2 commits June 3, 2021 20:30

Merge remote-tracking branch 'stac-utils/main' into windows-ci-fixes

5685421

Test preserving trailing slash in relative paths

a56d3e7

l0b0 reviewed Jun 4, 2021

View reviewed changes

duckontheweb commented Jun 4, 2021

View reviewed changes

requirements-dev.txt Show resolved Hide resolved

lossyrob mentioned this pull request Jun 4, 2021

Add additional unit test cases as part of review of #361 duckontheweb/pystac#1

Closed

lossyrob approved these changes Jun 4, 2021

View reviewed changes

tests/data-files/get_examples.py Outdated Show resolved Hide resolved

pystac/utils.py Outdated Show resolved Hide resolved

duckontheweb added 5 commits June 4, 2021 10:43

Newline in setup.cfg and remove unneeded parentheses

c48c15b

Handle relative paths for dotfiles properly

cff9606

Only add trailing slash to derived directory path

b5a0fbd

Simplify dirname logic for URLs

3467b4c

Move custom tempdir logic to test utils

1ed631f

duckontheweb commented Jun 4, 2021

View reviewed changes

tests/utils/__init__.py Outdated Show resolved Hide resolved

l0b0 approved these changes Jun 4, 2021

View reviewed changes

Use function instead of class for temp_dir, fix type issue

d23be0a

duckontheweb merged commit 50c6ee8 into stac-utils:main Jun 5, 2021

duckontheweb mentioned this pull request Jun 8, 2021

Failing tests on Windows #362

Closed

duckontheweb deleted the windows-ci-fixes branch June 9, 2021 14:44

gadomski mentioned this pull request Nov 8, 2022

During CI, run tests in a Windows environment #188

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Windows CI builds #361

Fix Windows CI builds #361

duckontheweb commented May 25, 2021 •

edited

Loading

codecov-commenter commented May 25, 2021 •

edited

Loading

duckontheweb commented May 28, 2021

duckontheweb commented Jun 1, 2021

lossyrob left a comment

duckontheweb commented Jun 4, 2021

l0b0 commented Jun 4, 2021

Fix Windows CI builds #361

Fix Windows CI builds #361

Conversation

duckontheweb commented May 25, 2021 • edited Loading

codecov-commenter commented May 25, 2021 • edited Loading

Codecov Report

duckontheweb commented May 28, 2021

duckontheweb commented Jun 1, 2021

lossyrob left a comment

Choose a reason for hiding this comment

duckontheweb commented Jun 4, 2021

l0b0 commented Jun 4, 2021

duckontheweb commented May 25, 2021 •

edited

Loading

codecov-commenter commented May 25, 2021 •

edited

Loading