Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework libarrow activate.sh with finer-grained checks and logging #1065

Merged
merged 32 commits into from
May 31, 2023

Conversation

maresb
Copy link
Contributor

@maresb maresb commented May 29, 2023

I expect this to fix #1060.

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@maresb maresb mentioned this pull request May 29, 2023
1 task
ln -s "$f" "$WRAPPER_DIR/$(basename "$f")"
symlink="$WRAPPER_DIR/$(basename "$target")"
# Check if symbolic link already exists and points to correct file
if [ -L "$symlink" ] && [ "$(readlink "$symlink")" = "$target" ]; then
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the most important part. Covers every activation after the link is created

# Thus it's safest to delete the symlink in case it already exists.
rm -f "$symlink"
# Check if symlink still exists after trying to delete it
if [ -e "$symlink" ]; then
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of rm + existence check + ln -s you can create a new unique link with random name and mv to the final place to make it atomic. In two parallel processes rm + -e can happen the same time and the two ln -s will still collide.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified it. Is that what you had in mind?

Copy link
Member

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Some comments.

recipe/activate.sh Outdated Show resolved Hide resolved
recipe/activate.sh Show resolved Hide resolved
maresb added 7 commits May 30, 2023 10:05
The effective part of conda-forge#1033 seems to be checking writability
of `WRAPPER_DIR` as opposed to `GDB_PREFIX`. Replacing
`ln -sf` with `rm -f; ln -s` seems to have led to conda-forge#1060.
Set `LIBARROW_ACTIVATE_LOGGING=1` to enable log messages.
@maresb
Copy link
Contributor Author

maresb commented May 30, 2023

The run completed successfully. Now pushing the changes to complete the todo...

Copy link
Member

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (for squash merge)

@maresb
Copy link
Contributor Author

maresb commented May 30, 2023

I can't imagine the last batch of commits changing much. Is the following failure of linux linux_64_cuda_compiler_versionNone a transient error?

__________________________ test_total_bytes_allocated __________________________

    def test_total_bytes_allocated():
>       assert pa.total_allocated_bytes() == 0
E       assert 1344 == 0
E        +  where 1344 = <built-in function total_allocated_bytes>()
E        +    where <built-in function total_allocated_bytes> = pa.total_allocated_bytes

test_array.py:41: AssertionError

@h-vetinari
Copy link
Member

h-vetinari commented May 30, 2023

Is the following failure of linux linux_64_cuda_compiler_versionNone a transient error?

Yes

Edit: sorry, the comment field contained some outdated stuff I didn't mean to send 🙈

@h-vetinari
Copy link
Member

You managed to comprehensively confuse github with two PRs having the exact same HEAD, and now I can't restart the builds. Another reason not to open a new PR unless really necessary. 🙃

@maresb
Copy link
Contributor Author

maresb commented May 30, 2023

Sorry about that, I was trying to save time by starting a new CI run before the old one finished.

Would it help anything if I pushed an empty commit to the other branch and open-close-cycled the other PR, or something like that?

@h-vetinari
Copy link
Member

Would it help anything if I pushed an empty commit to the other branch and open-close-cycled the other PR, or something like that?

No worries, I already fixed it here. That commit will not show up in the squash merge

@maresb
Copy link
Contributor Author

maresb commented May 31, 2023

Looks like one of the runners is a bit messed up.

@h-vetinari
Copy link
Member

Looks like one of the runners is a bit messed up.

Yeah, I don't even get the dialogue for restarting things. Mainly I was waiting if @xhochy still wanted to have some input here. But let's take this for a spin. :)

@h-vetinari h-vetinari merged commit 5f86221 into conda-forge:main May 31, 2023
@maresb maresb deleted the patch-1 branch May 31, 2023 23:01
@plattenschieber
Copy link

We stumbled upon an issue in our ci which seems to be related to this PR, as yesterdays build seems fine and todays doesn't work anymore 😅

Transaction finished
/workspace/repo/.env/etc/conda/activate.d/libarrow_activate.sh: line 11: CF_LIBARROW_ACTIVATE_LOGGING: unbound variable

The only change I could see was in the build number of pyarrow 12.0.0
libarrow 12.0.0 h96638e8_3_cpu worked fine -> libarrow 12.0.0 h96638e8_4_cpu got error from above

@maresb
Copy link
Contributor Author

maresb commented Jun 1, 2023

Hi @plattenschieber, thanks for the report! Indeed, this is would be a natural consequence of your Bash shell having set -u to enable unbound variable checking. We should set a default value to prevent this error in this case. I'll submit a PR for it now.

maresb added a commit to maresb/arrow-cpp-feedstock that referenced this pull request Jun 1, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065>
maresb added a commit to maresb/arrow-cpp-feedstock that referenced this pull request Jun 1, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065 (comment)>.
h-vetinari pushed a commit that referenced this pull request Jun 1, 2023
* Fix unbound variable in activate script

We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<#1065 (comment)>.

* Bump build number
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065 (comment)>.
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065 (comment)>.
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065 (comment)>.
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
h-vetinari pushed a commit to h-vetinari/arrow-cpp-feedstock that referenced this pull request Jun 2, 2023
We were checking `CF_LIBARROW_ACTIVATE_LOGGING` without a default
value. This led to failure under `set -u`. For original report, see
<conda-forge#1065 (comment)>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Activate.sh can't run in parallel
5 participants