Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Flux warmer should handle HTTP 429 responses #1401

Closed
endrec opened this issue Sep 28, 2018 · 8 comments
Closed

Flux warmer should handle HTTP 429 responses #1401

endrec opened this issue Sep 28, 2018 · 8 comments

Comments

@endrec
Copy link
Contributor

endrec commented Sep 28, 2018 •
edited
Loading

When the warmer is fetching images, sometimes it can get an HTTP 429 - Too Many Requests.
Currently it logs that is can't parse this response, and immediately retries, adding more and more load to an already overwhelmed server.

In my opinion it should stop bombarding that server, and come back to it later.

ts=2018-09-28T08:10:37.82216148Z caller=warming.go:209 component=warmer fetching=quay.io/datawire/ambassador total=187 expired=0 missing=187
ts=2018-09-28T08:10:38.332390517Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-c1b3748
ts=2018-09-28T08:10:38.333524948Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-0bad540
ts=2018-09-28T08:10:38.333891205Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-f6821e8
ts=2018-09-28T08:10:38.333974767Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-3d6210b
ts=2018-09-28T08:10:38.334007374Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-d1e3db7
ts=2018-09-28T08:10:38.334052751Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-6f0403a
ts=2018-09-28T08:10:38.334081658Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-add-path-if-exists-2ec69c8
ts=2018-09-28T08:10:38.334129107Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:docs-redirects-e3ceb31
ts=2018-09-28T08:10:38.334160038Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-cf24d38
ts=2018-09-28T08:10:38.334204738Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-feature-l7-redirection-7336d5a
ts=2018-09-28T08:10:38.334234104Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-08ec349
ts=2018-09-28T08:10:38.334261451Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-add-path-if-exists-38001d9
ts=2018-09-28T08:10:38.334310034Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:docs-redirects-5c331da
ts=2018-09-28T08:10:38.334336145Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-7975a10
ts=2018-09-28T08:10:38.33438229Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-7444268
ts=2018-09-28T08:10:38.334409947Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-bd1dbc0
ts=2018-09-28T08:10:38.334456055Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-feature-l7-redirection-98fbd59
ts=2018-09-28T08:10:38.334483185Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0-36-0-unstable-rc0-b0b177d
ts=2018-09-28T08:10:38.334527516Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0-36-0-unstable-rc0-c1b3748
ts=2018-09-28T08:10:38.334556588Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-9f15ce7
ts=2018-09-28T08:10:38.334606246Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-8a06ac8
ts=2018-09-28T08:10:38.334637097Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-0504653
ts=2018-09-28T08:10:38.41521045Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-f0452fa
ts=2018-09-28T08:10:38.420057812Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-aa77cd5
ts=2018-09-28T08:10:38.428076676Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-615d9ba
ts=2018-09-28T08:10:38.428214973Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-forge-link-a1dc612
ts=2018-09-28T08:10:38.430664326Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-envoy-573cf3e
ts=2018-09-28T08:10:38.431945705Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0.36.0-rc1
ts=2018-09-28T08:10:38.433208694Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-c083b04
ts=2018-09-28T08:10:38.434634448Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0-36-0-unstable-rc0-26734a0
ts=2018-09-28T08:10:38.436785656Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-26734a0
ts=2018-09-28T08:10:38.438024367Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-925ef99
ts=2018-09-28T08:10:38.440829819Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-forge-link-04465bd
ts=2018-09-28T08:10:38.443082041Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-envoy-7131b06
ts=2018-09-28T08:10:38.444528419Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-9cff295
ts=2018-09-28T08:10:38.450612896Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-cc0ae34
ts=2018-09-28T08:10:38.451920589Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-d974e64
ts=2018-09-28T08:10:38.453149277Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-master-0-36-0-b4433b6
ts=2018-09-28T08:10:38.454405036Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-f61c3f3
ts=2018-09-28T08:10:38.456878881Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-docs-wrt-master-e448c22
ts=2018-09-28T08:10:38.459272047Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-rebase-master-04c3064
ts=2018-09-28T08:10:38.46387271Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-8677668
ts=2018-09-28T08:10:38.466311179Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-f166784
ts=2018-09-28T08:10:38.467585498Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0-36-0-unstable-rc0-0bd7cfb
ts=2018-09-28T08:10:38.469007264Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0.36.0-unstable-rc0
ts=2018-09-28T08:10:38.503868338Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0.36.1-unstable-rc0
ts=2018-09-28T08:10:38.526825623Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-4e034fb
ts=2018-09-28T08:10:38.527966236Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-baebd16
ts=2018-09-28T08:10:38.529489848Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-changelog-date-328ee3e
ts=2018-09-28T08:10:38.531136199Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:rhs-testing-6843f7d
ts=2018-09-28T08:10:38.533296052Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-587b698
ts=2018-09-28T08:10:38.537004847Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-b11c029
ts=2018-09-28T08:10:38.538805408Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-parallelize-tests-2-41ea21c
ts=2018-09-28T08:10:38.548301036Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-4057273
ts=2018-09-28T08:10:38.548556166Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:rhs-testing-b05c98a
ts=2018-09-28T08:10:38.548653544Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-8a2e6c9
ts=2018-09-28T08:10:38.548779628Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-5badb31
ts=2018-09-28T08:10:38.548904593Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-update-envoy-latest-b8bd583
ts=2018-09-28T08:10:38.54902995Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0-39-0-rc4-ee98074
ts=2018-09-28T08:10:38.549141851Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:0.39.0-rc-latest
ts=2018-09-28T08:10:38.549245659Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:rhs-0-39-0-182a8a5
ts=2018-09-28T08:10:38.549346213Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-fix-add-export-to-helm-7af41af
ts=2018-09-28T08:10:38.54945077Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-8882b38
ts=2018-09-28T08:10:38.549561764Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-eb5e2d0
ts=2018-09-28T08:10:38.549669779Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-fba60ac
ts=2018-09-28T08:10:38.549845358Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:shared-develop-2e6ff89
ts=2018-09-28T08:10:38.549930692Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-b11c029
ts=2018-09-28T08:10:38.581729398Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:concaf-test-add-httpbin-test-b529790
ts=2018-09-28T08:10:38.611964171Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-6aab521
ts=2018-09-28T08:10:38.612290123Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:rhs-kat-2de734a
ts=2018-09-28T08:10:38.612452385Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:master-49bd734
ts=2018-09-28T08:10:38.631818785Z caller=warming.go:234 component=warmer canonical_name=quay.io/datawire/ambassador auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/datawire/ambassador:rhs-kat-c15f6dd
ts=2018-09-28T08:10:43.748833842Z caller=warming.go:262 component=warmer updated=quay.io/datawire/ambassador count=115
@stefanprodan
Copy link
Member

There is a backoff implementation for 429 errors merged in master #1354

@endrec
Copy link
Contributor Author

endrec commented Sep 28, 2018

I missed that I was only searching open issues. :) This can be closed then.

Any ETA on releasing that?

@endrec endrec closed this as completed Sep 28, 2018
@bboreham
Copy link
Contributor

bboreham commented Dec 3, 2018 •
edited
Loading

I have a log sent to me by another user, version 1.8.1, with exactly the same behaviour.

ts=2018-12-03T07:48:28.882994848Z caller=rate_limiter.go:71 component=ratelimiter info="reducing rate limit" host=quay.io limit=100.00
ts=2018-12-03T07:48:28.888272214Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-b75e079
ts=2018-12-03T07:48:28.892728218Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-999b45e
ts=2018-12-03T07:48:28.898906449Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-2d255f8
ts=2018-12-03T07:48:28.899109435Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-6c6caab
ts=2018-12-03T07:48:28.90074269Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-4a9b9cf
ts=2018-12-03T07:48:28.902762452Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-b91acf5
ts=2018-12-03T07:48:28.906052813Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-0272753
ts=2018-12-03T07:48:28.911282146Z caller=warming.go:300 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.14.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" ref=quay.io/weaveworks/flux:master-394784f

this goes on for hundreds of lines about 10ms apart.

@bboreham bboreham reopened this Dec 3, 2018
@stefanprodan
Copy link
Member

@bboreham the rate limit fixes were done in multiple PRs, the latest one didn't made it in 1.8.1 see #1538

@squaremo
Copy link
Member

squaremo commented Dec 3, 2018

In 1.8.1 it does back-off, but it won't cancel outstanding requests, of which there may be hundreds, or suppress the log messages. The recent PR does both of those things.

@squaremo squaremo closed this as completed Dec 3, 2018
@2opremio
Copy link
Contributor

2opremio commented Jan 23, 2020 •
edited
Loading

For the record, I spent quite some time debugging why there is a parsing error error parsing HTTP 429 response body (which is really confusing).

I thought it was a problem with Flux, but it turns it comes from the registry client which expects a json error in the body of the http response.

@bboreham
Copy link
Contributor

bboreham commented Mar 3, 2020

@2opremio should that parsing error be raised as an issue?

@2opremio
Copy link
Contributor

2opremio commented Mar 3, 2020

Maybe, but given how little attention the docker/distribution is getting I am not sure it's worth the effort.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@squaremo @endrec @2opremio @stefanprodan @bboreham