Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

semgrep 1.82.0 #176636

Merged
merged 2 commits into from
Aug 1, 2024
Merged

semgrep 1.82.0 #176636

merged 2 commits into from
Aug 1, 2024

Conversation

branchvincent
Copy link
Member

semgrep: migrate to python@3.12

@github-actions github-actions bot added ocaml OCaml use is a significant feature of the PR or issue python Python use is a significant feature of the PR or issue rust Rust use is a significant feature of the PR or issue labels Jul 7, 2024
@branchvincent branchvincent added the CI-no-fail-fast Continue CI tests despite failing GitHub Actions matrix builds. label Jul 7, 2024
@branchvincent
Copy link
Member Author

Unrelated issue on linux:

==> opam install --deps-only -y .
  Warning:  Failed checks on semgrep package definition from source at
            git+file:///tmp/semgrep-20240707-21900-rd915v#HEAD:
             warning 62: License doesn't adhere to the SPDX standard, see
                https://spdx.org/licenses/ : "LGPL-2.1"
  Warning:  Opam packages conf-gmp.4, conf-libcurl.1, conf-libev.4-12,
            conf-libpcre.1 and conf-pkg-config.3 depend on the following system
            packages that are no longer installed: libcurl4-gnutls-dev libev-dev
            libgmp-dev libpcre3-dev pkg-config
  Error:  Package conflict!
    * Missing dependency:
      - conf-libcurl.1: no longer available
  
  No solution found, exiting

@branchvincent branchvincent changed the title semgrep: migrate to python@3.12 semgrep 1.79.0 Jul 11, 2024
@supergibbs
Copy link

Any update on this? VS Code extension is complaining about an outdated CLI version now and 1.81.0 was just released.

image

@branchvincent
Copy link
Member Author

We need to figure out #176636 (comment), @amchiclet do by chance understand what's wrong here?

@amchiclet
Copy link
Contributor

@branchvincent hey, thanks for flagging this. Let me bring it up with the team.

@amchiclet
Copy link
Contributor

@branchvincent it's not obvious to the team what this might be the root cause. Sorry, it may take a while for the team to get to this. They're aware that the brew release has failed for a bit, but everyone right now seems quite occupied with other items.

I'm a little less involved in the release than before, but I'll do my best to help debug during my available time and to make it known more to the team if it doesn't get resolved and causes a lot of issues.

@amchiclet
Copy link
Contributor

Also, the CI failure says that if I use the formula in this PR for homebrew on linux or WSL, it will fail. Is that correct?

@supergibbs
Copy link

supergibbs commented Jul 26, 2024

Also, the CI failure says that if I use the formula in this PR for homebrew on linux or WSL, it will fail. Is that correct?

I've only installed with homebrew on MacOS but semgrep itself definitely runs on linux

Does it have something to do with semgrep requiring v1 here?

Not the same error message but the semgrep build is failing on the same step. I'll open an issue for the semgrep team.

@amchiclet
Copy link
Contributor

Does it have something to do with semgrep requiring v1 here?

It's possible. If I have some time, I'll test it out.

@amchiclet
Copy link
Contributor

I've tried not pinning conf-libcurl.1, but that didn't help for me. I think I got more errors, but I forgot to log them. But more importantly, we need conf-libcurl.1; otherwise we run into issues building for Windows.

My hypothesis is that something happened between opam (one of the build dependencies) version 2.1.5 and 2.2.0. The release date of 2.2.0 seems to be a possible explanation of the errors we are seeing just now and not earlier.

On my linux machine, pinning opam to 2.1.5 allows me to build semgrep successfully.

depends_on "opam" => ["2.1.5", :build]

I don't know if there's anything else going on or whether it will build successfully in homebrew-core's CI though. I assume it is the cause of the failure.

Ideally we would try to make our build work with the newest version of everything, but in this case, I think we're OK sticking to opam 2.1.5 meanwhile to unblock our newer releases. It's a package manager, and not a library, so that makes it a bit less bad. And I'll file a ticket to the team.

@branchvincent do you have strong oppositions against pinning opam to 2.1.5 to try to unblock this release?

The other error that @supergibbs linked seems to be something else. We probably need to try to tackle one error at a time.

@amchiclet
Copy link
Contributor

Actually 2.1.6 may work. I thought 2.1.5 was the latest before 2.2.0 and only tried that.

@amchiclet
Copy link
Contributor

Sadly, no luck with the version thing: #179033

@amchiclet
Copy link
Contributor

I think the brew on linux issue is resolved. Hopefully the PR to fix that gets merged soon.

@branchvincent branchvincent changed the title semgrep 1.79.0 semgrep 1.82.0 Aug 1, 2024
Brew in linux doesn't seem compatible with opam. I believe opam uses
the system's default package manager to check dependencies.

So in this case, brew on linux installs dependencies, but opam thinks
that they're not installed and gives an error.

Worked around this by only checking dependencies on macos. Not ideal
for linux, but if it's missing a dependency, the build should fail,
and if the build doesn't fail, then brew test should fail.

Co-authored-by: Amarin Phaosawasdi <amarin@semgrep.com>
@branchvincent branchvincent added the ready to merge PR can be merged once CI is green label Aug 1, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

🤖 An automated task has requested bottles to be published to this PR.

@github-actions github-actions bot added the CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. label Aug 1, 2024
@BrewTestBot BrewTestBot added this pull request to the merge queue Aug 1, 2024
Merged via the queue into Homebrew:master with commit 4dee31d Aug 1, 2024
15 checks passed
@branchvincent branchvincent deleted the python@3.12-semgrep branch August 1, 2024 14:59
@branchvincent
Copy link
Member Author

@amchiclet thank you again, really appreciate you taking the time to fix this 💯

One last thing, semgrep is currently pinned to glom~=22.1 which fails to build from source with python 3.12 and is fixed in the latest version. It would be great if semgrep could allow that version

@supergibbs
Copy link

@amchiclet thank you again, really appreciate you taking the time to fix this 💯

I second that, you too @branchvincent for keeping semgrep updated in brew. Thanks!

@amchiclet
Copy link
Contributor

@branchvincent We appreciate you helping us move forward to python 3.12! ❤️ Not to mention, you also helped unblock a pip upgrade failure before.

Do you by any chance know if the latest glom would still work with python 3.11? Or do we also need to migrate our makefiles to work with 3.12 in one step?

@branchvincent
Copy link
Member Author

@supergibbs thanks for the kind words 😄

@amchiclet yea latest glom still supports and works with python 3.7+! (3.8+ after mahmoud/glom#283)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-no-fail-fast Continue CI tests despite failing GitHub Actions matrix builds. CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. ocaml OCaml use is a significant feature of the PR or issue python Python use is a significant feature of the PR or issue python-3.12-migration ready to merge PR can be merged once CI is green rust Rust use is a significant feature of the PR or issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants