Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow images to be created on demand via the imagestreamtag api #14470

Closed
wants to merge 11 commits into from

Conversation

smarterclayton
Copy link
Contributor

A new subresource imagestreamtags/copy is added which accepts an ImageStreamTagCopy. This describes a request to create a new image from the provided docker image metadata (command, labels, env, ports, etc) and provide a single layer via direct submission, dramatically simplifying the act of creating an image. The caller can create a new from scratch image or add a new layer on top of an existing image. The endpoint then talks to the registry and abstracts the necessary details like creating a manifest, uploading the blob contents, and returning the new image.

The API is intended for use with binary uploads and other ad-hoc image composition.

@smarterclayton
Copy link
Contributor Author

Still WIP, tentatively targeted for 3.7

@smarterclayton
Copy link
Contributor Author

I0611 16:45:42.342377   10669 wrap.go:42] POST /apis/image.openshift.io/v1/namespaces/default/imagestreamtags/test:bar/clone: (3.298344ms) 201 [[oc/v1.6.1+5115d708d7 (darwin/amd64) kubernetes/010d313] 192.168.1.105:51585]
I0611 16:45:42.342805   10669 factory.go:185] Image stream "test" updated.
I0611 16:45:42.342840   10669 imagestream_controller.go:145] Queued import of stream default/test...
I0611 16:45:42.342848   10669 image_trigger_controller.go:383] Started syncing image stream "default/test"
I0611 16:45:42.342861   10669 image_trigger_controller.go:385] Finished syncing image stream "default/test" (15.977µs)
I0611 16:45:42.347142   10669 wrap.go:42] GET /api/v1/namespaces/default: (859.859µs) 200 [[openshift/v1.6.1+5115d708d7 (darwin/amd64) kubernetes/010d313] 192.168.1.105:63429]
I0611 16:45:42.347786   10669 wrap.go:42] GET /oapi/v1/users/~: (1.984949ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.348242   10669 wrap.go:42] GET /api/v1/namespaces/default/services/kubernetes: (696.606µs) 200 [[openshift/v1.6.1+5115d708d7 (darwin/amd64) kubernetes/010d313] 192.168.1.105:63429]
I0611 16:45:42.358024   10669 wrap.go:42] GET /oapi/v1/users/~: (8.727905ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.360322   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.716416ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.362271   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.425999ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.366326   10669 wrap.go:42] GET /oapi/v1/users/~: (1.233398ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.368374   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.577408ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.370344   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.478365ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.377686   10669 clone.go:496] Wrote blob of 192 bytes with digests layer=sha256:051652a014c06bb792007f2eefc56909880c60d26813929d261d93fdce5a5fa1 blob=sha256:051652a014c06bb792007f2eefc56909880c60d26813929d261d93fdce5a5fa1
I0611 16:45:42.379753   10669 wrap.go:42] GET /oapi/v1/users/~: (1.118196ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.381727   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.211514ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.383204   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.128543ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.387816   10669 wrap.go:42] GET /oapi/v1/users/~: (1.095776ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.391142   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (2.673396ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.396671   10669 wrap.go:42] GET /oapi/v1/users/~: (1.094566ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.398570   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.375844ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.401629   10669 wrap.go:42] GET /oapi/v1/users/~: (1.037526ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.403139   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.162792ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.404562   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.101572ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.408460   10669 wrap.go:42] GET /oapi/v1/users/~: (1.04311ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.409986   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.148814ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.411404   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.091018ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.415020   10669 wrap.go:42] GET /oapi/v1/users/~: (1.057558ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.416523   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.15025ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.417975   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.118963ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.422262   10669 wrap.go:42] GET /oapi/v1/users/~: (1.000059ms) 200 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.423759   10669 wrap.go:42] POST /oapi/v1/namespaces/default/localsubjectaccessreviews: (1.14641ms) 201 [[dockerregistry/v3.6.0 (darwin/amd64) openshift/4321cf4] 192.168.1.105:63442]
I0611 16:45:42.429215   10669 clone.go:694] Persisted image stream: &api.ImageStream{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test", GenerateName:"", Namespace:"default", SelfLink:"", UID:"4e48e8c9-4ed5-11e7-a403-7831c1b76042", ResourceVersion:"1986", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{sec:63632803170, nsec:649107362, loc:(*time.Location)(0xfe9db60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:api.ImageStreamSpec{LookupPolicy:api.ImageLookupPolicy{Local:false}, DockerImageRepository:"", Tags:map[string]api.TagReference{}}, Status:api.ImageStreamStatus{DockerImageRepository:"127.0.0.1:5000/default/test", Tags:map[string]api.TagEventList{"bar":api.TagEventList{Items:[]api.TagEvent{api.TagEvent{Created:v1.Time{Time:time.Time{sec:63632804584, nsec:433725131, loc:(*time.Location)(0xfe9db60)}}, DockerImageReference:"127.0.0.1:5000/default/test@sha256:56b7f2f33a1d792863c659b4b83f0f7c60cc1530ea1e9a40a85ffcfc5a244991", Image:"sha256:56b7f2f33a1d792863c659b4b83f0f7c60cc1530ea1e9a40a85ffcfc5a244991", Generation:1}, api.TagEvent{Created:v1.Time{Time:time.Time{sec:63632803170, nsec:813666004, loc:(*time.Location)(0xfe9db60)}}, DockerImageReference:"127.0.0.1:5000/default/test@sha256:b9574fbdcd3ae40981970af9f5b8fcda6dddaabc9d7f3fd316d9c54dcf8dad3c", Image:"sha256:b9574fbdcd3ae40981970af9f5b8fcda6dddaabc9d7f3fd316d9c54dcf8dad3c", Generation:1}}, Conditions:[]api.TagEventCondition(nil), PendingCopy:(*api.ImageStreamTagClone)(nil)}}}}
I0611 16:45:42.429633   10669 factory.go:185] Image stream "test" updated.

Registry seems to have huge fan out, this is a local test env but I would have expected ACL caching here @mfojtik @legionus

@smarterclayton
Copy link
Contributor Author

@bparees see the oc push binary ... command.

@mfojtik
Copy link
Contributor

mfojtik commented Jun 12, 2017

@mfojtik
Copy link
Contributor

mfojtik commented Jun 12, 2017

@soltysh FYI

@openshift-bot
Copy link
Contributor

Origin Action Required: Pull request cannot be automatically merged, please rebase your branch from latest HEAD and push again

@openshift-bot openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 12, 2017
@soltysh soltysh self-assigned this Jun 16, 2017
@soltysh soltysh added this to the 3.7.0 milestone Jun 16, 2017
if err != nil {
return fmt.Errorf("--from requires a valid image stream tag or image reference: %v", err)
}
if len(ref.Registry) > 0 || (len(ref.ID) == 0 && len(ref.Tag) == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like parsedockerimagereference will leave the tag empty (vs defaulting it to latest) if unspecified. are you intending to force people to specify :latest?

@smarterclayton
Copy link
Contributor Author

smarterclayton commented Jun 16, 2017 via email

@bparees
Copy link
Contributor

bparees commented Jun 16, 2017

sorry I guess it does, last time i looked into this i thought i found otherwise.

DefaultTransport is the default implementation of Transport and is used by DefaultClient. It establishes network connections as needed and caches them for reuse by subsequent calls. It uses HTTP proxies as directed by the $HTTP_PROXY and $NO_PROXY (or $http_proxy and $no_proxy) environment variables.

@smarterclayton smarterclayton force-pushed the binary_tag branch 3 times, most recently from feb0d39 to 5a3a76d Compare July 24, 2017 00:53
@openshift-merge-robot openshift-merge-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 24, 2017
@openshift-merge-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: smarterclayton

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@smarterclayton
Copy link
Contributor Author

/test all

@openshift-merge-robot openshift-merge-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 27, 2017
This command assists users in creating new images that can be used as part of builds
or as deployed applications. It accepts a zip or tar.gz file that will become a new
image in an image stream. You may also specify image metadata like the entrypoint,
environment variables, or labels to create a runnable image.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's an implicit assumption here that the archive is going to be extracted when it is used to create the image layer. might want to make that explicit. (you get an image containing the contents of the zip, not an image containing the zip)

@bparees
Copy link
Contributor

bparees commented Jul 28, 2017

still seems like a lot of code for a marginal net gain over:

$ oc new-build -D $'FROM scratch\nADD .\n' -n mybuild
$ oc start-build mybuild --from-dir .

@openshift-merge-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: smarterclayton

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2017
@openshift-merge-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: smarterclayton

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 28, 2017
@openshift-merge-robot openshift-merge-robot added the vendor-update Touching vendor dir or related files label Oct 14, 2017
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 16, 2017
@openshift-ci-robot
Copy link

@smarterclayton: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/openshift-jenkins/extended_templates 0eb33cb link /test extended_templates
ci/openshift-jenkins/verify ed60a6b link /test verify
ci/openshift-jenkins/integration ed60a6b link /test integration
ci/openshift-jenkins/unit ed60a6b link /test unit
ci/openshift-jenkins/extended_networking_minimal ed60a6b link /test extended_networking_minimal
ci/openshift-jenkins/extended_conformance_gce ed60a6b link /test extended_conformance_gce
ci/openshift-jenkins/extended_conformance_install_update ed60a6b link /test extended_conformance_install_update

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot
Copy link
Contributor

@smarterclayton PR needs rebase

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 20, 2017
@openshift-bot
Copy link
Contributor

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 25, 2018
@smarterclayton
Copy link
Contributor Author

smarterclayton commented Feb 26, 2018 via email

@openshift-bot
Copy link
Contributor

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 28, 2018
@openshift-bot
Copy link
Contributor

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-api-review needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. vendor-update Touching vendor dir or related files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants