HTTP status client error (403 Forbidden) for public mirror channel

[chuqingq]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi.toml:

[project]
name = "pixi_helloworld"
version = "0.1.0"
description = "Add a short description here"
authors = ["chuqq <chuqingq@qq.com>"]
#channels = ["conda-forge"]
channels = ["http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/"]
platforms = ["linux-64"]

[tasks]

[dependencies]

$ pixi add go
ERROR rattler_repodata_gateway::fetch: error=HTTP status client error (403 Forbidden) for url (http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch/repodata.json.bz2)
ERROR rattler_repodata_gateway::fetch: error=HTTP status client error (403 Forbidden) for url (http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64/repodata.json.bz2)
  × failed to fetch repodata from channels
  ╰─▶ HTTP status client error (403 Forbidden) for url (http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64/
      repodata.json.bz2)

Issue description

Because of network situation, default "conda-forge" channel is slow for me.
So I tried the near mirrors, like https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/, which is public.
But I don't know why HTTP status client error (403 Forbidden) occured.
I also tried http, same error.

Expected behavior

I use this mirror for conda and micromamba, they are both ok. I think it should be ok for pixi too.

[wolfv]

I think the issue is that the bz2 file is not available, and we should continue trying (until we hit the .json file instead of giving up on a 403 error.

This would need a change in rattler.

[chawyehsu]

that .bz2 file does exist and rattler_repodata_gateway did check its availability - seems to be available, no idea why a 403 http response is returned afterwards.

2024-02-01T16:11:04.877900Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: checking availability of 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.zst'
2024-02-01T16:11:04.878296Z DEBUG fetch_repo_data{cache_path=./cache}: reqwest::connect: starting new connection: http://mirrors.tuna.tsinghua.edu.cn/
2024-02-01T16:11:04.878439Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: checking availability of 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.bz2'
2024-02-01T16:11:04.878462Z DEBUG hyper::client::connect::dns: resolving host="mirrors.tuna.tsinghua.edu.cn"
2024-02-01T16:11:04.878567Z DEBUG fetch_repo_data{cache_path=./cache}: reqwest::connect: starting new connection: http://mirrors.tuna.tsinghua.edu.cn/
2024-02-01T16:11:04.878706Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: checking availability of 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.jlap'
2024-02-01T16:11:04.878794Z DEBUG hyper::client::connect::dns: resolving host="mirrors.tuna.tsinghua.edu.cn"
2024-02-01T16:11:04.878862Z DEBUG fetch_repo_data{cache_path=./cache}: reqwest::connect: starting new connection: http://mirrors.tuna.tsinghua.edu.cn/
2024-02-01T16:11:04.879089Z DEBUG hyper::client::connect::dns: resolving host="mirrors.tuna.tsinghua.edu.cn"
2024-02-01T16:11:04.893928Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connecting to 101.6.15.130:80
2024-02-01T16:11:04.894527Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connecting to 101.6.15.130:80
2024-02-01T16:11:04.894820Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connecting to 101.6.15.130:80
2024-02-01T16:11:06.178976Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connected to 101.6.15.130:80
2024-02-01T16:11:06.179260Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connected to 101.6.15.130:80
2024-02-01T16:11:06.179721Z DEBUG hyper::proto::h1::io: flushed 115 bytes
2024-02-01T16:11:06.179726Z DEBUG hyper::proto::h1::io: flushed 119 bytes
2024-02-01T16:11:06.181447Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::connect::http: connected to 101.6.15.130:80
2024-02-01T16:11:06.181740Z DEBUG hyper::proto::h1::io: flushed 119 bytes
2024-02-01T16:11:06.222914Z DEBUG hyper::proto::h1::io: parsed 10 headers
2024-02-01T16:11:06.223049Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-02-01T16:11:06.223246Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::pool: pooling idle connection for ("http", mirrors.tuna.tsinghua.edu.cn)
2024-02-01T16:11:06.223401Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.bz2' seems to be available
2024-02-01T16:11:06.223890Z DEBUG hyper::proto::h1::io: parsed 11 headers
2024-02-01T16:11:06.223947Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-02-01T16:11:06.224042Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::pool: pooling idle connection for ("http", mirrors.tuna.tsinghua.edu.cn)
2024-02-01T16:11:06.224129Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.jlap' seems to be unavailable
2024-02-01T16:11:06.228865Z DEBUG hyper::proto::h1::io: parsed 11 headers
2024-02-01T16:11:06.228938Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-02-01T16:11:06.229087Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::pool: pooling idle connection for ("http", mirrors.tuna.tsinghua.edu.cn)
2024-02-01T16:11:06.229190Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.zst' seems to be unavailable
2024-02-01T16:11:06.229300Z DEBUG fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: fetching 'http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.bz2'
2024-02-01T16:11:06.229443Z DEBUG fetch_repo_data{cache_path=./cache}: hyper::client::pool: reuse idle connection for ("http", mirrors.tuna.tsinghua.edu.cn)
2024-02-01T16:11:06.229740Z DEBUG hyper::proto::h1::io: flushed 141 bytes
2024-02-01T16:11:06.277112Z DEBUG hyper::proto::h1::io: parsed 9 headers
2024-02-01T16:11:06.277252Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2024-02-01T16:11:06.277393Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x5E0 (1504 bytes)
2024-02-01T16:11:06.277590Z ERROR fetch_repo_data{cache_path=./cache}: rattler_repodata_gateway::fetch: error=Request error: HTTP status client error (403 Forbidden) for url (http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json.bz2)

[chuqingq]

I think the issue is that the bz2 file is not available, and we should continue trying (until we hit the .json file instead of giving up on a 403 error.

This would need a change in rattler.

This bz2 file is available, without any authentication:

$ wget -c http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64/repodata.json.bz2
--2024-02-02 12:16:17--  http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64/repodata.json.bz2
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33480152 (32M) [application/octet-stream]
Saving to: ‘repodata.json.bz2’

repodata.json.bz2                   62%[======================================>                        ]  19.84M  1.14MB/s

[sumanth-manchala]

This has something to do with reqwest.

use reqwest;
let body = reqwest::get("http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch/repodata.json.bz2").await.unwrap()
            .text().await.unwrap();
println!("body = {:?}", body);

The reqwest snippet seem to return a 403 with below content, tried searching for any user-agent header but reqwest doesn't seem to send any

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>清华大学 TUNA 镜像站</title>
    <link href="//cdn.jsdelivr.net/npm/bootstrap@3.3.7/dist/css/bootstrap.min.css" rel="stylesheet">
    <style> body { padding-top: 40px; } </style>
</head>
<body>
<div class="container">
    <div class="well">
        <p lang="zh-cn">尊敬的访问者，</p>
        <p lang="en">Dear visitor, </p>
        <p lang="fr">Chère cliente, </p>
        <p lang="ja">訪問者へ 、</p>
        <p>&nbsp;</p>
        <p lang="zh-cn">您好！</p>
        <p lang="en">Hello! </p>
        <p lang="fr">Bonjour! </p>
        <p lang="ja">こんにちは ！</p>
        <p>&nbsp;</p>
        <p lang="zh-cn">我们检测到您所在的子网和/或所使用的客户端存在大量下载某些较大二进制文件的行为，为保证用户的正常使用，我们阻断了此类请求。</p>
        <p lang="en">We have detected enormous traffic from your network or client and have blocked your requests to ensure the quality of service for normal users.</p>
        <p lang="fr">Nous avons détecté un grand nombre de téléchargements de certains fichiers binaires volumineux à partir de votre sous-réseau et/ou du client que vous utilisez et avons bloqué ces requêtes pour garantir un accès normal aux utilisateurs.</p>
        <p lang="ja">お客様のサブネットおよび/またはクライアントで、特定の大きなバイナリのダウンロードを多数検出しました。お客様の通常の使用のためこれらのリクエストをブロックさせていただきます。</p>
        <p>&nbsp;</p>
        <p lang="zh-cn">您可以尝试更改网络环境或更换客户端；您也可以联系 <a href="mailto:support@tuna.tsinghua.edu.cn">support@tuna.tsinghua.edu.cn</a> 并附上下方的标识符。</p>
        <p lang="en">You could try another network or client, or have your network administrator contact <a href="mailto:support@tuna.tsinghua.edu.cn">support@tuna.tsinghua.edu.cn</a> with the identifier below.</p>
        <p lang="fr">Vous pouvez essayer de modifier votre environnement réseau ou de changer de client ; vous pouvez également contacter <a href="mailto:support@tuna.tsinghua.edu.cn">support@tuna.tsinghua.edu.cn</a> avec l'identifiant ci-dessous.</p>
        <p lang="ja">ネットワーク環境の変更、クライアントの変更などをお試しください。また、下記の識別子を用いて <a href="mailto:support@tuna.tsinghua.edu.cn">support@tuna.tsinghua.edu.cn</a> までご連絡ください。</p>
        <p>&nbsp;</p>
        <p><code>00000070 c9ebf208bde5762193d161ba644843f6</code></p>
    </div>
</div>
</body>
</html>

whereas the same call with Python requests module or curl seems to work fine.

[baszalmstra]

I dont think its a bad idea to add a user-agent to all our requests anyway! Do you know if that fixes the issue @sumanth-manchala ?

[wolfv]

Actually I think we had some similar thing in mamba ...

[wolfv]

mamba-org/mamba#1169

[baszalmstra]

I added a fix in #892, I can verify that sending along a user-agent indeed fixes the issue.

[jezdez]

@baszalmstra @wolfv to verify, there wasn't a user-agent sent at all before the change in #892?

[wolfv]

I think that's right. No user agent was set before #892. We'll release pixi this week or next week with this change.