Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nixpkgs-update-fetch-repology is broken #1192

Open
zowoq opened this issue Apr 4, 2024 · 6 comments
Open

nixpkgs-update-fetch-repology is broken #1192

zowoq opened this issue Apr 4, 2024 · 6 comments

Comments

@zowoq
Copy link
Contributor

zowoq commented Apr 4, 2024

I won't bother disabling it as it doesn't error, it just generates an empty file.

ConnectionError (HttpExceptionRequest Request {
  host                 = "repology.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept","application/json;charset=utf-8,application/json")]
  path                 = "/api/v1/projects"
  queryString          = "?inrepo=nix_unstable&outdated=true"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}
 ConnectionTimeout)

I guess we need to update something in https://github.com/ryantm/nixpkgs-update/blob/main/src/Repology.hs
@zowoq
Copy link
Contributor Author

zowoq commented Apr 4, 2024 •
edited
Loading

Seems we just need to set a user agent.

Not sure what the issue is as nixpkgs-update fetch-repology works on build01 and build02 doesn't seem to have been blocked by repology as the curl command still works.

Works:

[zowoq@build01:~]$ /nix/store/ay4j4690h3izkz1r1bnh6i4dsr8pbriy-nixpkgs-update-0.4.0/bin/nixpkgs-update fetch-repology
starting
Just "blackbox-unclassified"
200
ace 7.0.11 7.1.3 https://repology.org/project/ace-framework/versions
<...>

Fails:

[zowoq@build02:~]$ /nix/store/ay4j4690h3izkz1r1bnh6i4dsr8pbriy-nixpkgs-update-0.4.0/bin/nixpkgs-update fetch-repology
starting
ConnectionError (HttpExceptionRequest Request {
  host                 = "repology.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept","application/json;charset=utf-8,application/json")]
  path                 = "/api/v1/projects"
  queryString          = "?inrepo=nix_unstable&outdated=true"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}
 ConnectionTimeout)

Works:

[zowoq@build02:~]$ curl --user-agent test 'https://repology.org/api/v1/projects?inrepo=nix_unstable&outdated=true'
<!doctype html>
<html lang=en>
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to the target URL: <a href="https://repology.org/api/v1/projects/?inrepo=nix_unstable&amp;outdated=true">https://repology.org/api/v1/projects/?inrepo=nix_unstable&amp;outdated=true</a>. If not, click the link.

@zowoq
Copy link
Contributor Author

zowoq commented May 4, 2024

I've been working around this by pushing the repology updates manually to build02 every few days so I haven't really been in a hurry to fix it.

@ryantm @rhendric I just want to check that I'm not overlooking something on our side before I ask the repology maintainer about it, seems odd to block only the fetch-repology request on build02?

@rhendric
Copy link
Member

rhendric commented May 5, 2024

Nothing jumps out at me.

@zowoq
Copy link
Contributor Author

zowoq commented Jun 30, 2024

Repology fetcher has been working for a couple of days now.

@zowoq zowoq closed this as completed Jun 30, 2024
@zowoq
Copy link
Contributor Author

zowoq commented Aug 5, 2024

Seems we've been getting "Too Many Requests" for the last couple of days.

FailureResponse (Request {requestPath = (BaseUrl {baseUrlScheme = Https, baseUrlHost = "repology.org", baseUrlPort = 443, baseUrlPath = "/api/v1"},"/projects/python:home-assistant-chip-core"), requestQueryString = fromList [("inrepo",Just "nix_unstable"),("outdated",Just "true")], requestBody = Nothing, requestAccept = fromList [application/json;charset=utf-8,application/json], requestHeaders = fromList [], requestHttpVersion = HTTP/1.1, requestMethod = "GET"}) (Response {responseStatusCode = Status {statusCode = 429, statusMessage = "Too Many Requests"}, responseHeaders = fromList [("Server","nginx"),("Date","Mon, 05 Aug 2024 23:34:47 GMT"),("Content-Type","text/html"),("Content-Length","162"),("Connection","keep-alive")], responseHttpVersion = HTTP/1.1, responseBody = "<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"})

@zowoq zowoq reopened this Aug 5, 2024
@zowoq
Copy link
Contributor Author

zowoq commented Aug 7, 2024

https://repology.org/api
Terms of use

Bulk requests to this API are discouraged, consider using a database dump. Bulk clients must identify themselves with a custom user-agent, referring to a description of the client and a way to report misbehavior (such as GitHub repository with an issue tracker). Bulk clients must not do more than one request per second. Miscomplying clients will be blocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhendric @zowoq