x.py fails all downloads that use a tempdir with snap curl

[tharunsuresh-code]

Hey everyone,

First time setting up Rust for development in my local system. OS version - Kubuntu 22.10
I installed curl using snap then ran the setup script x as follows:

 $ ./x.py setup

It resulted in the following error message -

downloading https://static.rust-lang.org/dist/2023-01-25/rust-std-beta-x86_64-unknown-linux-gnu.tar.xz
################################################################################################################################################################################# 100.0%
invalid checksum:
    found:    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    expected: 1f95934a853b0ce1f4bc50ac86b1ae74a34acbae8a64903faf6f7ba63e49051b
Traceback (most recent call last):
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/./x.py", line 29, in <module>
    bootstrap.main()
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/src/bootstrap/bootstrap.py", line 939, in main
    bootstrap(args)
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/src/bootstrap/bootstrap.py", line 904, in bootstrap
    build.download_toolchain()
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/src/bootstrap/bootstrap.py", line 433, in download_toolchain
    self._download_component_helper(filename, pattern, tarball_suffix)
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/src/bootstrap/bootstrap.py", line 465, in _download_component_helper
    get(
  File "/media/d_drive/My_files/Visual Studio Code/github/rust/src/bootstrap/bootstrap.py", line 51, in get
    raise RuntimeError("failed verification")
RuntimeError: failed verification

This is primarily due to the "tmp" folder location of curl installed through snap being "/tmp/snap-private-tmp/snap.curl/tmp" which is not the path that x.py is looking for. Alternatively installing curl using "apt install curl", I was able to get x.py setup to run properly.

A message on Zulip discussing this -> https://rust-lang.zulipchat.com/#narrow/stream/122652-new-members/topic/frozendroid/near/285039893
An existing issue #86708

I would like to ask if we can elegantly instruct new users about this issue and guide them in a better way?

[chenyukang]

Is there a proper way to detect current curl is a snap installed one, if so maybe we could make a PR to fix it.

[Teapot4195]

We could call snap to check if it is installed, but I don't think the snap cli is stable

[tharunsuresh-code]

What if we could follow this format of cli:

curl url > filepath

Which can be achieved by below code in python:

with open(path, "wb") as outfile:
  run(["curl", option,
      "-L", # Follow redirect.
      "-y", "30", "-Y", "10",    # timeout if speed is < 10 bytes/sec for > 30 seconds
      "--connect-timeout", "30",  # timeout if cannot connect within 30 seconds
      "--retry", "3", "-Sf", url],  #modified line
      stdout=outfile,  #modified line
      verbose=verbose,
      exception=True, # Will raise RuntimeError on failure
  )

This works for both snap and apt curl commands on Kubuntu.
Is this a good change and work for other distros?

[chenyukang]

I think it's reasonable to store it in a relative path compared to the root dir of rustc source code, do we have such a temporary directory in the bootstrap now？
@jyn514

[jyn514]

I don't see a reason to use the build dir instead of a temporary directory, but using stdout=path instead of -o path on

rust/src/bootstrap/bootstrap.py

Line 94 in dffea43

"--retry", "3", "-Sf", "-o", path, url],

seems fine to me. Weird that snap is messing with the paths like that.

[jyn514]

@tharunsuresh-code are you interested in making a PR with that change? :)

[KittyBorgX]

I'd like to take this up if @tharunsuresh-code does not take it up, if that's alright.

[tharunsuresh-code]

@jyn514 Sure, I will make the necessary changes and raise a PR
@KittyBorgX Sorry for taking away your chance this time, this is one of my first contributions so I would like to take this one. :)

[tharunsuresh-code]

So I guess one of the downloads tries to directly store in the source directory of rust (which can be in any user path)
If the user path consists of spaces, it gives the following error for the above method -

    Finished dev [unoptimized] target(s) in 21.19s
downloading https://static.rust-lang.org/dist/2023-01-30/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
Warning: Failed to open the file /media/d_drive/My_files/Visual Studio                                                               
Warning: Code/github/rust_commit/rust/build/tmp/rustfmt-nightly-x86_64-unknown-
Warning: linux-gnu.tar.xz: Permission denied
curl: (23) Failure writing output to destination

Actual path - /media/d_drive/My_files/Visual Studio Code/github/rust_commit/rust

This is because the -o method of snap curl is being executed as shown in verbose output, however I don't understand why spaces would be an issue (because apt curl seems to work in this filepath) -

command did not execute successfully: "curl" "-#" "-y" "30" "-Y" "10" "--connect-timeout" "30" "--retry" "3" "-Sf" "-o" "/media/d_drive/My_files/Visual Studio Code/github/rust/build/tmp/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz" "https://static.rust-lang.org/dist/2023-01-30/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz"

I am investigating on this

[tharunsuresh-code]

Okay, I guess it is due to the bootstrapping process of using the existing version of bootstrap to compile the current version? So I can ignore this and go ahead to commit our discussed change?

[jyn514]

@tharunsuresh-code there are two languages used in bootstrap, Python and rust. You've fixed a bug in the python code, but it looks like the rust code has a different bug that coincidentally is also related to downloads. I don't have time to look into it right now, but fixing both bugs at once seems reasonable.

https://github.com/rust-lang/rust/blob/master/src/bootstrap/README.md#build-phases

[tharunsuresh-code]

Oh, alright, I will try to check it!

[jyn514]

Fixed in #107790