Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add S3 backend support #2825

Open
wants to merge 78 commits into
base: main
Choose a base branch
from
+3,849 −744
Open

feat: Add S3 backend support #2825

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
045b0d8
Add example for s3 usage
delsner Jan 3, 2025
5044bf2
add `S3Options` + `S3Config`
moritzwilksch Jan 3, 2025
9acd6d0
update config structure
moritzwilksch Jan 3, 2025
dc58d10
template global config in `pixi init`
moritzwilksch Jan 3, 2025
19bead5
move config into features, support cli options, init: use in search
moritzwilksch Jan 3, 2025
b4e254a
WIP, todo: move out of feature again?
moritzwilksch Jan 3, 2025
aed25a9
further improvements
pavelzw Jan 3, 2025
52fe381
format
pavelzw Jan 3, 2025
233e6d4
Update docs + minor fix
delsner Jan 5, 2025
b5f7f0d
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 13, 2025
bf9e489
fix merge
pavelzw Jan 13, 2025
f72a184
fix
pavelzw Jan 13, 2025
1f1e2d3
add integration test
pavelzw Jan 13, 2025
e1dcfe7
fix typo
pavelzw Jan 13, 2025
707782f
Merge branch 'main' into s3
delsner Jan 14, 2025
fb0ad2d
fix compilation in ci
pavelzw Jan 14, 2025
4ecb937
adjust schema
pavelzw Jan 14, 2025
b585711
fix lint
pavelzw Jan 14, 2025
2f412bc
fix
pavelzw Jan 14, 2025
15fb412
fix test
pavelzw Jan 14, 2025
cf48809
fix
pavelzw Jan 14, 2025
72853c2
fixes with merge from rattler main
pavelzw Jan 17, 2025
46c8d15
build tests
pavelzw Jan 17, 2025
ba01bb9
update lockfile
pavelzw Jan 17, 2025
2b47aeb
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 17, 2025
1aceb91
.
pavelzw Jan 17, 2025
5098eef
.
pavelzw Jan 17, 2025
99a02ac
.
pavelzw Jan 17, 2025
13eb0c5
Merge branch 'main' into s3
pavelzw Jan 22, 2025
52ecc74
Merge branch 'main' into s3
pavelzw Jan 23, 2025
e844e48
fix typos
pavelzw Jan 24, 2025
5df01a5
fix unwrap
pavelzw Jan 24, 2025
899bd0f
fix unwrap 2
pavelzw Jan 24, 2025
16be383
fix todo
pavelzw Jan 24, 2025
6a3e846
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 24, 2025
14c43c2
fmt
pavelzw Jan 24, 2025
909659c
fix?
pavelzw Jan 24, 2025
2add4db
fix unwraps
pavelzw Jan 24, 2025
edd4d95
clippy
pavelzw Jan 24, 2025
7b76136
fix
pavelzw Jan 24, 2025
c911a73
fix
pavelzw Jan 24, 2025
aebb801
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 29, 2025
9ee40ef
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 30, 2025
2ef96da
wip new hashmap style
pavelzw Jan 30, 2025
428e70d
fix
pavelzw Jan 30, 2025
82854ed
fix
pavelzw Jan 31, 2025
0e796a3
fix pixi init
pavelzw Jan 31, 2025
48259fe
fix
pavelzw Jan 31, 2025
41a34a6
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Jan 31, 2025
004ccce
update lockfile
pavelzw Jan 31, 2025
85c8fe7
fix
pavelzw Jan 31, 2025
8aca6d8
fix
pavelzw Jan 31, 2025
a1fb1a6
bump rattler
pavelzw Jan 31, 2025
5620dad
debug
pavelzw Jan 31, 2025
5725e8f
make clippy happy
pavelzw Jan 31, 2025
4188081
clippy
pavelzw Jan 31, 2025
cabe7d9
clippy
pavelzw Jan 31, 2025
670acd8
fix
pavelzw Jan 31, 2025
0dfe578
update
pavelzw Jan 31, 2025
2ae89e4
update to rattler on main
pavelzw Feb 3, 2025
aaa63d0
Merge branch 'main' into s3
pavelzw Feb 3, 2025
8b9dea8
fix
pavelzw Feb 3, 2025
57d56e6
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 3, 2025
608b133
bump rattler
pavelzw Feb 3, 2025
94b02eb
fix
pavelzw Feb 3, 2025
611fe66
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 5, 2025
d6e96db
fix conflicts
pavelzw Feb 5, 2025
9a84efc
.
pavelzw Feb 5, 2025
528efca
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 9, 2025
48e326e
bump stuff
pavelzw Feb 9, 2025
ee76166
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 10, 2025
c7dd28c
fix
pavelzw Feb 10, 2025
67b9e9c
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 10, 2025
991ef48
Merge remote-tracking branch 'upstream/main' into s3
pavelzw Feb 11, 2025
7529b52
lockfile
pavelzw Feb 11, 2025
90fe2a2
no dev drive for now
pavelzw Feb 11, 2025
74959f0
.
pavelzw Feb 12, 2025
c1e36bd
does this help?
pavelzw Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Copy link
Contributor

@pavelzw pavelzw Jan 31, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the failing windows tests are because for some reason cargo cannot create directories on the Dev drive E: created in #2566.

cargo build --locked --profile $env:CARGO_BUILD_PROFILE --features self_update
  shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
  env:
    CARGO_INCREMENTAL: 0
    CARGO_NET_RETRY: 10
    RUSTUP_MAX_RETRIES: 10
    RUST_LOG: info
    RUST_BACKTRACE: 1
    RUSTFLAGS: -D warnings
    CARGO_TERM_COLOR: always
    CICD_INTERMEDIATES_DIR: _cicd-intermediates
    XDG_CACHE_HOME: C:\a\pixi\pixi/.cache
    PYTEST_ADDOPTS: --color=yes
    CARGO_BUILD_PROFILE: ci
    DEV_DRIVE: E:
    TMP: E:/pixi-tmp
    TEMP: E:/pixi-tmp
    RATTLER_CACHE_DIR: E:/rattler-cache
    RUSTUP_HOME: E:/.rustup
    CARGO_HOME: E:/.cargo
    PIXI_HOME: E:/.pixi
    PIXI_WORKSPACE: E:/pixi
    CACHE_ON_FAILURE: false
  
error: failed to create directory `E:\pixi\target`
Caused by:
  The request is not supported. (os error 50)

https://github.com/prefix-dev/pixi/actions/runs/13073287838/job/36479432799

my assumption would be that some internals changed with cargo 1.84.0 -> 1.86.0-nightly and that's why it doesn't work anymore. Should i revert the dev drive changes or do you have an other idea?

Copy link
Contributor

@pavelzw pavelzw Feb 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deactivated for now 90fe2a2 (#2825)

Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,9 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Test pixi
run: pixi run test-slow
env:
PIXI_TEST_R2_ACCESS_KEY_ID: ${{ secrets.PIXI_TEST_R2_ACCESS_KEY_ID }}
PIXI_TEST_R2_SECRET_ACCESS_KEY: ${{ secrets.PIXI_TEST_R2_SECRET_ACCESS_KEY }}

cargo-test-macos-aarch64:
name: "cargo test | macos aarch64"
Expand All @@ -169,6 +172,9 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Test pixi
run: pixi run test-slow
env:
PIXI_TEST_R2_ACCESS_KEY_ID: ${{ secrets.PIXI_TEST_R2_ACCESS_KEY_ID }}
PIXI_TEST_R2_SECRET_ACCESS_KEY: ${{ secrets.PIXI_TEST_R2_SECRET_ACCESS_KEY }}

cargo-test-macos-x86_64:
name: "cargo test | macos x86_64"
Expand All @@ -188,6 +194,9 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Test pixi
run: pixi run test-slow
env:
PIXI_TEST_R2_ACCESS_KEY_ID: ${{ secrets.PIXI_TEST_R2_ACCESS_KEY_ID }}
PIXI_TEST_R2_SECRET_ACCESS_KEY: ${{ secrets.PIXI_TEST_R2_SECRET_ACCESS_KEY }}

cargo-test-windows:
name: "cargo test | windows"
Expand All @@ -209,6 +218,9 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Test pixi
run: pixi run test-slow
env:
PIXI_TEST_R2_ACCESS_KEY_ID: ${{ secrets.PIXI_TEST_R2_ACCESS_KEY_ID }}
PIXI_TEST_R2_SECRET_ACCESS_KEY: ${{ secrets.PIXI_TEST_R2_SECRET_ACCESS_KEY }}

#
# Builds the binary artifacts on different platforms
Expand Down
Loading
Loading