feat: Add S3 backend support #2825
Conversation
|
Thanks guys, I'm asking @baszalmstra for a review of this work. |
|
this first version should be ready for review as well, only a couple smaller things missing |
|
fyi we temporarily removed the secrets to see if that fixes our release build. |
There was a problem hiding this comment.
from my side the PR is ready for another round of reviews @baszalmstra @ruben-arts @wolfv!
i'm currently still investigating why we get permission errors on windows when compiling but other than that it should be ready
| /// Custom S3 configuration | ||
| #[derive(Debug, Clone, PartialEq, Serialize, Eq, Deserialize)] | ||
| #[serde(rename_all = "kebab-case", deny_unknown_fields)] | ||
| pub struct S3Options { |
There was a problem hiding this comment.
currently we require to set all three values at once. At some point we might want to investigate defaults in the protocol but since it would be backwards compatible, i think this is fine for now
| pub fn build_reqwest_clients(config: Option<&Config>) -> (Client, ClientWithMiddleware) { | ||
| pub fn build_reqwest_clients( | ||
| config: Option<&Config>, | ||
| s3_config_project: Option<HashMap<String, rattler_networking::s3_middleware::S3Config>>, |
There was a problem hiding this comment.
we could pass an empty hashmap instead of None here as well but i thought it's a bit clearer in the API when we actually are using project config and when not like this
| @@ -1,3 +1,4 @@ | |||
| #![feature(once_cell_try)] | |||
There was a problem hiding this comment.
needed because of #2825 (comment), please let me know your thoughts on this, whether a nightly rust version is fine
There was a problem hiding this comment.
the failing windows tests are because for some reason cargo cannot create directories on the Dev drive E: created in #2566.
cargo build --locked --profile $env:CARGO_BUILD_PROFILE --features self_update
shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
env:
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
RUSTUP_MAX_RETRIES: 10
RUST_LOG: info
RUST_BACKTRACE: 1
RUSTFLAGS: -D warnings
CARGO_TERM_COLOR: always
CICD_INTERMEDIATES_DIR: _cicd-intermediates
XDG_CACHE_HOME: C:\a\pixi\pixi/.cache
PYTEST_ADDOPTS: --color=yes
CARGO_BUILD_PROFILE: ci
DEV_DRIVE: E:
TMP: E:/pixi-tmp
TEMP: E:/pixi-tmp
RATTLER_CACHE_DIR: E:/rattler-cache
RUSTUP_HOME: E:/.rustup
CARGO_HOME: E:/.cargo
PIXI_HOME: E:/.pixi
PIXI_WORKSPACE: E:/pixi
CACHE_ON_FAILURE: false
error: failed to create directory `E:\pixi\target`
Caused by:
The request is not supported. (os error 50)
https://github.com/prefix-dev/pixi/actions/runs/13073287838/job/36479432799
my assumption would be that some internals changed with cargo 1.84.0 -> 1.86.0-nightly and that's why it doesn't work anymore. Should i revert the dev drive changes or do you have an other idea?
.github/workflows/ci.yml
Outdated
| @@ -273,6 +285,7 @@ jobs: | |||
| - uses: Swatinem/rust-cache@v2 | |||
| with: | |||
| workspaces: ${{ env.PIXI_WORKSPACE }} | |||
| - uses: mxschmitt/action-tmate@v3 | |||
There was a problem hiding this comment.
| - uses: mxschmitt/action-tmate@v3 |
Motivation
Part of a series of PRs for implementing conda/rattler#960.
Changes
Add S3 backend support for
pixi.tomland global pixi config.Joint work with @moritzwilksch and @pavelzw.