Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ko builder #6041

Closed
halvards opened this issue Jun 18, 2021 · 3 comments
Closed

Add ko builder #6041

halvards opened this issue Jun 18, 2021 · 3 comments
Labels
area/build kind/design discussion kind/feature-request kind/todo implementation task/epic for the skaffold team priority/p2 May take a couple of releases

Comments

@halvards
Copy link
Contributor

halvards commented Jun 18, 2021

Tracking issue for the effort to add the ko builder.

See the design proposal for further details.

Related: #5611, #1958

halvards added a commit to halvards/skaffold that referenced this issue Jun 18, 2021
Small steps, this is just to verify the impact of changes to the
dependency graph.

Tracking issue: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jun 18, 2021
Small steps, this is just to verify the impact of changes to the
dependency graph.

Tracking issue: GoogleContainerTools#6041
tejal29 pushed a commit that referenced this issue Jun 21, 2021
Small steps, this is just to verify the impact of changes to the
dependency graph.

Tracking issue: #6041
halvards added a commit to halvards/skaffold that referenced this issue Jun 22, 2021
This adds the initial implementation for the ko builder.

Read more about the ko builder in the
[design proposal](docs/design_proposals/ko-builder.md).

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jun 22, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (waiting on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
@nkubala nkubala added the priority/p2 May take a couple of releases label Jun 23, 2021
@tejal29
Copy link
Member

tejal29 commented Jul 1, 2021

Thanks for creating the issue!

halvards added a commit to halvards/skaffold that referenced this issue Jul 22, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jul 27, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jul 28, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jul 28, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jul 29, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Aug 5, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Aug 11, 2021
This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: GoogleContainerTools#6041
tejal29 pushed a commit that referenced this issue Aug 11, 2021
* Add core ko builder implementation

This adds the `Build()` method for building artifacts using ko. It
supports both publishing the resulting image to a registry, and
sideloading it to the local Docker daemon.

The Skaffold docker client is used in the ko builder. This ensures that
any Minikube config set by Skaffold is used. The ko builder uses the
docker client when sideloading images to the docker daemon.

The `temporary.go` file contains the structs intended to be added to the
schema.

The additions to the design proposal explain image naming for the ko
builder, specifically how container images are named and how Go import
paths are resolved when using the proposed ko builder.

This commit includes ko builder unit tests for
non-current-working-directory workspace dirs. These verify
that the ko builder works even if the context specified in
`skaffold.yaml` differs from the current working directory.

This implementation is still missing the following features:

- integration test
- dependencies (for file watching)
- insecure registries
- debug mode
- support for `go` flags and environment variables (based on
  ko-build/ko#340)
- actually plumbing the builder into the Skaffold CLI and API :-)

Tracking: #6041

* Improve ko Builder unit tests

Split out the confusing end-to-end unit test with tests for individual
functions.

* Reorder test inputs and expected outputs

Also fix typo in design proposal.

* Add TODO to update import path

Reminders to update the import path once the contents of
`pkg/skaffold/build/ko/schema` have been added to the real schema in
`pkg/skaffold/schema/latest/v1`.

* Fix lint errors for TODO comments in import block

Adding the TODO in the import block resulted in gci linter errors:
https://app.travis-ci.com/github/GoogleContainerTools/skaffold/jobs/530642848#L313
halvards added a commit to halvards/skaffold that referenced this issue Aug 16, 2021
Decide on how ko builder users should specify the location of
`package main` for their images.

Previous discussion: GoogleContainerTools#6054 (comment)

Tracking: GoogleContainerTools#6041
Related: GoogleContainerTools#6054, GoogleContainerTools#6286
halvards added a commit to halvards/skaffold that referenced this issue Aug 16, 2021
Decide on how ko builder users should specify the location of
`package main` for their images.

Previous discussion: GoogleContainerTools#6054 (comment)

Tracking: GoogleContainerTools#6041
Related: GoogleContainerTools#6054, GoogleContainerTools#6286
tejal29 pushed a commit that referenced this issue Aug 16, 2021
* Refine ko builder behavior for main packages

Decide on how ko builder users should specify the location of
`package main` for their images.

Previous discussion: #6054 (comment)

Tracking: #6041
Related: #6054, #6286

* Improve explanation of the Target config field

Clarify how a blank value works, and also how users can use a pattern
with wildcards.
halvards added a commit to halvards/skaffold that referenced this issue Aug 17, 2021
This implements support for the Target config field for the ko builder.
The Target field allows users to specify a target for `go build`. This
is necessary where the main package is not in the context directory.

Tracking: GoogleContainerTools#6041
Related: GoogleContainerTools#6054, GoogleContainerTools#6437
halvards added a commit to halvards/skaffold that referenced this issue Nov 11, 2021
Display the expected tag after successfully building a ko image, for
both sideloaded and published images.

Also handles the `tagPolicy.sha256` scenarios.

Fixes: GoogleContainerTools#6835
Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Nov 11, 2021
Shorten the example values in the config schema. This fixes the issue
on the `skaffold.yaml` reference page of the website, where long
example values push the description to a very narrow space on the page.

Tracking: GoogleContainerTools#6041
tejal29 pushed a commit that referenced this issue Nov 11, 2021
Shorten the example values in the config schema. This fixes the issue
on the `skaffold.yaml` reference page of the website, where long
example values push the description to a very narrow space on the page.

Tracking: #6041
tejal29 pushed a commit that referenced this issue Nov 11, 2021
* fix(log): Output ggcr logs

Previously, Skaffold discarded log messages from
[`go-containerregistry`](https://github.com/google/go-containerregistry)
(a.k.a `ggcr`).

This change enables `ggcr` log messages, with severity tied to
Skaffold's log level.

Related: #6041

* fix(log): Send Go std `log` to `logrus`

Includes `ggcr` logs.
halvards added a commit to halvards/skaffold that referenced this issue Nov 12, 2021
Update explanation of how Go debugging works in Skaffold and add
explanations for ko builder users.

Also update links to the VS Code Go extension.

Related: GoogleContainerTools#6843
Tracking: GoogleContainerTools#6041
tejal29 pushed a commit that referenced this issue Nov 12, 2021
Update explanation of how Go debugging works in Skaffold and add
explanations for ko builder users.

Also update links to the VS Code Go extension.

Related: #6843
Tracking: #6041
halvards added a commit to halvards/skaffold that referenced this issue Nov 17, 2021
`ko` adds the `-trimpath` flag to `go build` by default, as this aids in
achieving reproducible builds.

The `-trimpath` flag removes file system paths from the resulting
binary, and this makes interactive debugging more difficult.

This change instructs `ko` to _not_ add the `-trimpath` flag when users
run `skaffold debug` with the `ko` builder.

Bumps the `ko` version to pick up the `Trimpath` build option.

Also, simplifies the `ko` example config since it's no longer necessary
to add a non-empty `flags` field to `skaffold.yaml` to allow for
interactive debugging.

Fixes: GoogleContainerTools#6843
Tracking: GoogleContainerTools#6041
tejal29 pushed a commit that referenced this issue Nov 18, 2021
`ko` adds the `-trimpath` flag to `go build` by default, as this aids in
achieving reproducible builds.

The `-trimpath` flag removes file system paths from the resulting
binary, and this makes interactive debugging more difficult.

This change instructs `ko` to _not_ add the `-trimpath` flag when users
run `skaffold debug` with the `ko` builder.

Bumps the `ko` version to pick up the `Trimpath` build option.

Also, simplifies the `ko` example config since it's no longer necessary
to add a non-empty `flags` field to `skaffold.yaml` to allow for
interactive debugging.

Fixes: #6843
Tracking: #6041
tejal29 pushed a commit that referenced this issue Nov 18, 2021
`ko` adds the `-trimpath` flag to `go build` by default, as this aids in
achieving reproducible builds.

The `-trimpath` flag removes file system paths from the resulting
binary, and this makes interactive debugging more difficult.

This change instructs `ko` to _not_ add the `-trimpath` flag when users
run `skaffold debug` with the `ko` builder.

Bumps the `ko` version to pick up the `Trimpath` build option.

Also, simplifies the `ko` example config since it's no longer necessary
to add a non-empty `flags` field to `skaffold.yaml` to allow for
interactive debugging.

Fixes: #6843
Tracking: #6041
halvards added a commit to halvards/skaffold that referenced this issue Dec 3, 2021
With this change, the `ko` builder no longer prints the image name to
`stdout` by default.

Originally, this was added to the `ko` builder to mimic the behavior of
the `ko` CLI. Other Skaffold image builders do not print the image name
in this way.

The reason this is useful for existing `ko` CLI users is that some
documented workflows rely on capturing the image name from `stdout`, see
https://github.com/google/ko/blob/v0.9.3/README.md#build-an-image

After some investigation, the better option seems to be to rely on the
existing `--quiet` and `--output` Skaffold flags to format the output of
the image name(s).

This change also updates the `ko` builder documentation to show existing
`ko` CLI users how to capture the image name by providing an appropriate
Go template to `--output`.

Fixes: GoogleContainerTools#6835
Closes: GoogleContainerTools#6836
Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Dec 3, 2021
With this change, the `ko` builder no longer prints the image name to
`stdout` by default.

Originally, this was added to the `ko` builder to mimic the behavior of
the `ko` CLI. Other Skaffold image builders do not print the image name
in this way.

The reason this is useful for existing `ko` CLI users is that some
documented workflows rely on capturing the image name from `stdout`, see
https://github.com/google/ko/blob/v0.9.3/README.md#build-an-image

After some investigation, the better option seems to be to rely on the
existing `--quiet` and `--output` Skaffold flags to format the output of
the image name(s).

This change also updates the `ko` builder documentation to show existing
`ko` CLI users how to capture the image name by providing an appropriate
Go template to `--output`.

Fixes: GoogleContainerTools#6835
Closes: GoogleContainerTools#6836
Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Dec 7, 2021
This change enables `ko` builder users to substitute environment
variable values in the `labels` and `env` config fields.

These fields are used for image labels and build-time environment
variables, respectively.

Envvar templating of image labels can be used to add information such as
the Git commit SHA to the image, see GoogleContainerTools#6916.

Also, environment variable expansion of `flags` and `ldflags` in the
`ko` builder configuration now supports Skaffold's templating syntax,
for consistency.

For backwards compatibility, `ko`'s templating syntax still works with
`flags` and `ldflags`:

Skaffold: `{{.FOO}}`

`ko`: `{{.Env.FOO}}`

Tracking: GoogleContainerTools#6041
Fixes: GoogleContainerTools#6916
halvards added a commit to halvards/skaffold that referenced this issue Dec 8, 2021
This change enables `ko` builder users to substitute environment
variable values in the `labels` and `env` config fields.

These fields are used for image labels and build-time environment
variables, respectively.

Envvar templating of image labels can be used to add information such as
the Git commit SHA to the image, see GoogleContainerTools#6916.

Also, environment variable expansion of `flags` and `ldflags` in the
`ko` builder configuration now supports Skaffold's templating syntax,
for consistency.

For backwards compatibility, `ko`'s templating syntax still works with
`flags` and `ldflags`:

Skaffold: `{{.FOO}}`

`ko`: `{{.Env.FOO}}`

Tracking: GoogleContainerTools#6041
Fixes: GoogleContainerTools#6916
halvards added a commit to halvards/skaffold that referenced this issue Dec 9, 2021
This change enables `ko` builder users to substitute environment
variable values in the `labels` and `env` config fields.

These fields are used for image labels and build-time environment
variables, respectively.

Envvar templating of image labels can be used to add information such as
the Git commit SHA to the image, see GoogleContainerTools#6916.

Also, environment variable expansion of `flags` and `ldflags` in the
`ko` builder configuration now supports Skaffold's templating syntax,
for consistency.

For backwards compatibility, `ko`'s templating syntax still works with
`flags` and `ldflags`:

Skaffold: `{{.FOO}}`

`ko`: `{{.Env.FOO}}`

Tracking: GoogleContainerTools#6041
Fixes: GoogleContainerTools#6916
tejal29 pushed a commit that referenced this issue Dec 13, 2021
This change enables `ko` builder users to substitute environment
variable values in the `labels` and `env` config fields.

These fields are used for image labels and build-time environment
variables, respectively.

Envvar templating of image labels can be used to add information such as
the Git commit SHA to the image, see #6916.

Also, environment variable expansion of `flags` and `ldflags` in the
`ko` builder configuration now supports Skaffold's templating syntax,
for consistency.

For backwards compatibility, `ko`'s templating syntax still works with
`flags` and `ldflags`:

Skaffold: `{{.FOO}}`

`ko`: `{{.Env.FOO}}`

Tracking: #6041
Fixes: #6916
halvards added a commit to halvards/skaffold that referenced this issue Jan 12, 2022
When using the `ko://` scheme prefix for the Skaffold image name, the
rest of the image name, following the prefix, is a Go import path.

This import path maps to the Git repository (typically), and won't match
an image registry name (e.g., `ko://github.com/org/repo` vs
`gcr.io/project_id`).

With this change, if _all_ of the following are true, the build results
in an error that instructs the user to set the default repo:

- the image name in `skaffold.yaml` uses the `ko://` prefix; and
- the Skaffold default repo is _not_ set; and
- the image is being pushed to a registry (rather than sideloaded to a
  Docker daemon).

Fixes: GoogleContainerTools#6933
Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jan 12, 2022
When using the `ko://` scheme prefix for the Skaffold image name, the
rest of the image name, following the prefix, is a Go import path.

This import path maps to the Git repository (typically), and won't match
an image registry name (e.g., `ko://github.com/org/repo` vs
`gcr.io/project_id`).

With this change, if _all_ of the following are true, the build results
in an error that instructs the user to set the default repo:

- the image name in `skaffold.yaml` uses the `ko://` prefix; and
- the Skaffold default repo is _not_ set; and
- the image is being pushed to a registry (rather than sideloaded to a
  Docker daemon).

Fixes: GoogleContainerTools#6933
Tracking: GoogleContainerTools#6041
halvards added a commit to halvards/skaffold that referenced this issue Jan 12, 2022
Document that `ko` builder users who use the `ko://` prefix must set the
default repo if pushing images to a registry.

Related: GoogleContainerTools#6933
Tracking: GoogleContainerTools#6041
@aaron-prindle
Copy link
Contributor

ko builder is added now 🎊! Marking closed(fixed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build kind/design discussion kind/feature-request kind/todo implementation task/epic for the skaffold team priority/p2 May take a couple of releases
Projects
None yet
Development

No branches or pull requests

4 participants