Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Push sporadically fails with huge layer sizes #8

Open
dfunckt opened this issue Feb 14, 2017 · 2 comments
Open

Push sporadically fails with huge layer sizes #8

dfunckt opened this issue Feb 14, 2017 · 2 comments

Comments

@dfunckt
Copy link
Member

dfunckt commented Feb 14, 2017 •
edited
Loading

A user reported that building an image with a layer sized about 2GB failed to be pushed to the registry by the builder.

https://app.frontapp.com/open/cnv_3oo50b

Front logo Front conversations
@dfunckt
Copy link
Member Author

dfunckt commented Mar 22, 2017

Another user reported on gitter failure to upload an image with a huge layer (around 5GB). Logentries reported:

22 Mar 2017 20:58:13.608 9913f31b235e docker-registry[78]: time="2017-03-22T18:58:13.574736698Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/_uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75/data" go.version=go1.6.2 http.request.host="registry2.resin.io:445" http.request.id=452c5cf9-3804-462b-991a-e508a99eb71f http.request.method=PUT http.request.remoteaddr=52.205.226.13 http.request.uri="/v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508" http.request.useragent="docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64" instance.id=7d72c203-c2e9-4d6a-bbb5-e4adc954cce9 vars.name="sensei2/76b52eb4d9f642056be26484c66605ec36eb3272" vars.uuid=648eaf0a-c3d8-4451-80e4-fec70b224a75 version=v2.4.1
22 Mar 2017 20:58:13.674 9913f31b235e docker-registry[78]: time="2017-03-22T18:58:13.574845198Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: Path not found: /docker/registry/v2/repositories/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/_uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75/data" err.message="unknown error" go.version=go1.6.2 http.request.host="registry2.resin.io:445" http.request.id=452c5cf9-3804-462b-991a-e508a99eb71f http.request.method=PUT http.request.remoteaddr=52.205.226.13 http.request.uri="/v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508" http.request.useragent="docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=38.332036ms http.response.status=500 http.response.written=232 instance.id=7d72c203-c2e9-4d6a-bbb5-e4adc954cce9 vars.name="sensei2/76b52eb4d9f642056be26484c66605ec36eb3272" vars.uuid=648eaf0a-c3d8-4451-80e4-fec70b224a75 version=v2.4.1
22 Mar 2017 20:58:13.674 9913f31b235e docker-registry[78]: ::1 - - [22/Mar/2017:18:58:13 +0000] "PUT /v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508 HTTP/1.0" 500 232 "" "docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64"

@dfunckt dfunckt mentioned this issue Dec 11, 2017
Merged
5 tasks
@dfunckt dfunckt mentioned this issue Dec 14, 2017
Merged
dfunckt added a commit that referenced this issue Dec 14, 2017
@dfunckt
Revert "nginx.conf: Use equal values for client_body_buffer_size and …
b5ca550 
…client_max_body_size to avoid writing big files to disk"

Revert commit 4159eb5 due to nginx.conf already specifying a client_max_body_size that conflicts with the newly specified one.

In fact, client_max_body_size is set to `0`, disabling any limits, therefore preventing the registry from responding with 413 when a client tries to upload large layers.

This needs further investigation to determine how Docker behaves with various registry settings, but in the meantime this needs to be reverted.

Change-Type: patch
Connects-To: #8
HQ: balena-io/balena-io#1115
@imrehg
Copy link
Contributor

imrehg commented Feb 5, 2018

It fails for me all the time. Test case is for example:

FROM resin/%%RESIN_MACHINE_NAME%%-alpine

RUN dd if=/dev/urandom of=/goliath.data bs=1M count=2000

CMD ls -lah /goliath.data; while : ; do sleep 200; done

@imrehg imrehg mentioned this issue Feb 5, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@imrehg @dfunckt