introduce generate command as alpha command #12209

glours · 2024-10-14T14:25:29Z

What I did

Add a command to generate a compose file from a containers list

docker compose alpha generate CONTAINER_NAME or CONTAINER_ID

Missing pieces:

Config support
Container ressource reservations and limitations

Related issue

(not mandatory) A picture of a cute animal, if possible in relation to what you did

pkg/compose/generate.go

pkg/compose/convert.go

lizzthabet

this is awesome to see! i left a few suggestions and pointed out a couple areas where i think the behavior of generate could be non-obvious to users, but those areas aren't blockers here. i'm not an expert on compose and the spec, so you're welcome to take my feedback with a grain of salt. there are definitely areas (like mapping networks and volumes) that i don't know well enough to review precisely.

after this pr, there are two areas of work that i think would improve the value of this command's output:

filtering out redundant/verbose information in the service block that's inherited from images
adding more precise image-handling logic so it can define build blocks where possible

curious what your thoughts are, and i'd be happy to work on those additions next!

lizzthabet · 2024-10-23T14:53:50Z

cmd/compose/generate.go

+}
+
+func runGenerate(ctx context.Context, backend api.Service, opts reverseOptions, containers []string) error {
+	_, _ = fmt.Fprintln(os.Stderr, "reverse command is EXPERIMENTAL")


nit: should this be generate instead of reverse?

cmd/compose/generate.go

lizzthabet · 2024-10-23T16:49:51Z

pkg/compose/generate.go

+	containers, err := s.apiClient().ContainerList(ctx, containerType.ListOptions{
+		Filters: filtersListNames,
+		All:     true,
+	})


til about this filter list approach! nice :)

one side effect of treating the user-provided container references as name filters is that they'll fuzzy-match. it could result in unexpected behavior (or an implicit feature? lol) for users. just something to note!

indeed, filter by name may return more than expected. With an explicit container list better iterate over names and use ContainerInspect

lizzthabet · 2024-10-23T16:54:32Z

pkg/compose/generate.go

+		return nil, err
+	}
+
+	containersByIds, err := s.apiClient().ContainerList(ctx, containerType.ListOptions{


oh neat! it looks like the Container list response is a little simpler than the ContainerJSON response i was parsing from doing an inspect on each container in my hack version

lizzthabet · 2024-10-23T18:42:15Z

pkg/compose/generate.go

+			serviceLabel = getCanonicalContainerName(c)
+		}
+		service, ok := services[serviceLabel]
+		if !ok {


just so i'm understanding this code path properly: if there are two or more containers with the same compose service label name (or parsed container name?), then it'll ignore all but the first.

this could be surprising behavior for users, but also i imagine a rare edge case. maybe another thing to note for future work! i'd lean towards generating a new service name in the case of collisions.

That's a typical ambiguity we should not try to solve by black magic. Create a compose model from live resources should only cover the obvious model, and not try to guess how users will abuse it :P

pkg/compose/generate.go

lizzthabet · 2024-10-23T19:03:43Z

pkg/compose/generate.go

+func cleanDockerPreviousLabels(labels types.Labels) types.Labels {
+	cleanedLabels := types.Labels{}
+	for key, value := range labels {
+		if !strings.HasPrefix(key, "com.docker.compose.") && !strings.HasPrefix(key, "desktop.docker.io") {


suggestion: are there other label paths that make sense to ignore, like org.opencontainers.image?

lizzthabet · 2024-10-23T19:05:11Z

pkg/compose/generate.go

+}
+
+func (s *composeService) extractComposeConfiguration(service *types.ServiceConfig, inspect moby.ContainerJSON, volumes types.Volumes, secrets types.Secrets, networks types.Networks) {
+	service.Environment = types.NewMappingWithEquals(inspect.Config.Env)


do you know how container runs with env files get handled? do those values just get mapped to the Config.Env? i had this as a question / todo in my own work.

Yes the CLI read the file and populate the Config.Env attributes with the content of the file
https://github.com/docker/cli/blob/61baf2a3d984a1e59fcadb38e28d77b571434b88/cli/command/container/opts.go#L496

this would need to exclude environment variables set by image

lizzthabet · 2024-10-23T19:08:06Z

pkg/compose/generate.go

+	return project, nil
+}
+
+func (s *composeService) extractComposeConfiguration(service *types.ServiceConfig, inspect moby.ContainerJSON, volumes types.Volumes, secrets types.Secrets, networks types.Networks) {


wdyt about adding support for cmd, entrypoint, and expose blocks, too? they should be pretty straightforward. i'm also happy to put up a pr for those too!

Sure, that should be part of follow up PRs

would need to compare image CMD and ENTRYPOINT with actual container to detect those have been overridden (same for healthcheck)

ndeloof

LGTM as a first iteration, just some notes

this tries to detect use of compose labels, which I don't expect a user would use for live resources, or .. he already used compose to run those and then, why run this command ? (and/or why would we encourage this practice)
we need to distinguish config set by image vs live config when guessing the compose model, i.e docker run mysql has ENV, CMD and ENTRYPOINT set by image, I don't want those to be reflected in generated compose.yaml until those have been overridden intentionally

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

jhrotko · 2024-10-28T15:05:24Z

pkg/compose/generate.go

+}
+
+func (s *composeService) toComposeVolumes(volumes []moby.MountPoint) (map[string]types.VolumeConfig,
+	[]types.ServiceVolumeConfig, map[string]types.SecretConfig, []types.ServiceSecretConfig) {


Might have missed me, but what about volume labels? Be aware we need to cleanup these labels for volumes

compose/pkg/compose/create.go

Lines 143 to 146 in fafaa9c

volume.Labels = volume.Labels.Add(api.VolumeLabel, k)

volume.Labels = volume.Labels.Add(api.ProjectLabel, project.Name)

volume.Labels = volume.Labels.Add(api.VersionLabel, api.ComposeVersion)

err := s.ensureVolume(ctx, volume, project.Name)

Currently labels are only added to services not to volumes, networks ...

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [docker/compose](https://github.com/docker/compose) | minor | `v2.29.7` -> `v2.30.3` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>docker/compose (docker/compose)</summary> ### [`v2.30.3`](https://github.com/docker/compose/releases/tag/v2.30.3) [Compare Source](docker/compose@v2.30.2...v2.30.3) #### What's Changed ##### 🐛 Fixes - Avoid starting all services on rebuild by [@jhrotko](https://github.com/jhrotko) [(12258)](docker/compose#12258) ##### ⚙️ Dependencies - Bump compose-go v2.4.4 by [@glours](https://github.com/glours) [(12274)](docker/compose#12274) **Full Changelog**: docker/compose@v2.30.2...v2.30.3 ### [`v2.30.2`](https://github.com/docker/compose/releases/tag/v2.30.2) [Compare Source](docker/compose@v2.30.1...v2.30.2) #### What's Changed ##### 🐛 Fixes - Service being declared in a profile must not trigger re-creation by [@ndeloof](https://github.com/ndeloof) in [(12265)](docker/compose#12265) - Remove ArtifactType from Config in OCI v1.1 definition of the artifact by [@glours](https://github.com/glours) [(12266)](docker/compose#12266) ##### 🔧 Internal - Add Joana Hrotko to Maintainers by [@laurazard](https://github.com/laurazard) [(12253)](docker/compose#12253) - Add profile e2e test case to document in compose by [@jhrotko](https://github.com/jhrotko) [(12252)](docker/compose#12252) ##### ⚙️ Dependencies - Bump `compose-go` to version `v2.4.3` by [@glours](https://github.com/glours) in docker/compose#12261 **Full Changelog**: docker/compose@v2.30.1...v2.30.2 ### [`v2.30.1`](https://github.com/docker/compose/releases/tag/v2.30.1) [Compare Source](docker/compose@v2.30.0...v2.30.1) #### What's Changed ##### 🐛 Fixes Fix regression when using stdin as input of `-f` flag [(12248)](docker/compose#12248) Fix regression when using multiple time the same YAML anchor in a Compose file [(12247)](docker/compose#12247) ##### ⚙️ Dependencies - bump compose-go to version v2.4.2 by [@glours](https://github.com/glours) in docker/compose#12249 **Full Changelog**: docker/compose@v2.30.0...v2.30.1 ### [`v2.30.0`](https://github.com/docker/compose/releases/tag/v2.30.0) [Compare Source](docker/compose@v2.29.7...v2.30.0) #### What's Changed ##### ✨ Improvements - Introduce service hooks by [@ndeloof](https://github.com/ndeloof) [(12166)](docker/compose#12166) - Introduce generate command as alpha command by [@glours](https://github.com/glours) [(12209)](docker/compose#12209) - Add export command by [@jarqvi](https://github.com/jarqvi) [(12120)](docker/compose#12120) - Add support for CDI device request using `devices` by [@ndeloof](https://github.com/ndeloof) [(12184)](docker/compose#12184) - Add support for bind recursive by [@ndeloof](https://github.com/ndeloof) [(12210)](docker/compose#12210) - Allow usage of `-f` flag with OCI Compose artifacts by [@glours](https://github.com/glours) [(12220)](docker/compose#12220) ##### 🐛 Fixes - Append unix-style relative path when computing container target path by [@ndeloof](https://github.com/ndeloof) [(12145)](docker/compose#12145) - Wait for dependent service up to delay set by --wait-timeout by [@ndeloof](https://github.com/ndeloof) [(12156)](docker/compose#12156) - Check secret source exists, as bind mount would create target by [@ndeloof](https://github.com/ndeloof) [(12151)](docker/compose#12151) - After container restart register printer consumer by [@jhrotko](https://github.com/jhrotko) [(12158)](docker/compose#12158) - Fix(down): Fix down command if specified services are not running by [@idsulik](https://github.com/idsulik) [(12164)](docker/compose#12164) - Show watch error message and open DD only when w is pressed by [@jhrotko](https://github.com/jhrotko) [(12165)](docker/compose#12165) - Fix(push): Fix unexpected EOF on alpha publish by [@idsulik](https://github.com/idsulik) [(12169)](docker/compose#12169) - Fix(convergence): Serialize access to observed state by [@anantadwi13](https://github.com/anantadwi13) [(12150)](docker/compose#12150) - Remove feature flag integration with Docker Desktop for ComposeUI and ComposeNav by [@jhrotko](https://github.com/jhrotko) [(12192)](docker/compose#12192) - Support Dockerfile-specific ignore-file with watch by [@ndeloof](https://github.com/ndeloof) [(12193)](docker/compose#12193) - Add support for raw env_file format by [@ndeloof](https://github.com/ndeloof) [(12179)](docker/compose#12179) - Convert GPUs to DeviceRequests with implicit "gpu" capability by [@ndeloof](https://github.com/ndeloof) [(12197)](docker/compose#12197) - Improve error message to include expected network label by [@divinity76](https://github.com/divinity76) [(12213)](docker/compose#12213) - Don't use progress to render restart, which hides logs by [@ndeloof](https://github.com/ndeloof) [(12226)](docker/compose#12226) - One-off containers are not indexed, and must be ignored by `exec --index` command by [@ndeloof](https://github.com/ndeloof) [(12224)](docker/compose#12224) - Don't warn about uid/gid not being supported while ... they are by [@ndeloof](https://github.com/ndeloof) [(12232)](docker/compose#12232) - Connect to external networks by name by [@ndeloof](https://github.com/ndeloof) [(12234)](docker/compose#12234) - Fix push error message typo by [@chris-crone](https://github.com/chris-crone) [(12237)](docker/compose#12237) - Fix(dockerignore): Add wildcard support to dockerignore.go by [@idsulik](https://github.com/idsulik) [(12239)](docker/compose#12239) ##### 🔧 Internal - Remove bind options when creating a volume type by [@jhrotko](https://github.com/jhrotko) [(12177)](docker/compose#12177) - pass device.options to engine by [@ndeloof](https://github.com/ndeloof) [(12183)](docker/compose#12183) - Add security policy by [@thaJeztah](https://github.com/thaJeztah) [(12194)](docker/compose#12194) - Gha: set default permissions to "contents: read" by [@thaJeztah](https://github.com/thaJeztah) [(12195)](docker/compose#12195) - Desktop: allow this client to be identified via user-agent by [@djs55](https://github.com/djs55) [(12212)](docker/compose#12212) - Compose-go clean volume target to avoid ambiguous comparisons by [@ndeloof](https://github.com/ndeloof) [(12208)](docker/compose#12208) ##### ⚙️ Dependencies - Bump docker `v27.3.1` by [@ndeloof](https://github.com/ndeloof) [(12178)](docker/compose#12178) - Build(deps): bump `golang.org/x/sys` from `0.25.0` to `0.26.0` by [@dependabot](https://github.com/dependabot) [(12189)](docker/compose#12189) - Bump `compose-go` to `v2.3.0` by [@glours](https://github.com/glours) [(12198)](docker/compose#12198) - Bump `compose-go` to `v2.4.0` by [@glours](https://github.com/glours) [(12231)](docker/compose#12231) - Bump `compose-go` to `v2.4.1` by [@glours](https://github.com/glours) [(12243)](docker/compose#12242) - Build(deps): bump github.com/containerd/containerd from 1.7.22 to 1.7.23 by [@dependabot](https://github.com/dependabot) [(12211)](docker/compose#12211) - Bump golang minimal version to `1.22` in go.mod by [@glours](https://github.com/glours) [(12246)](docker/compose#12246) - Bump `go.uber.org/mock` to `v0.5.0` and `google.golang.org/grpc` to `v1.67.1` by [@glours](https://github.com/glours) [(12245)](docker/compose#12245) #### New Contributors - [@anantadwi13](https://github.com/anantadwi13) made their first contribution in docker/compose#12150 - [@jarqvi](https://github.com/jarqvi) made their first contribution in docker/compose#12120 - [@djs55](https://github.com/djs55) made their first contribution in docker/compose#12212 - [@divinity76](https://github.com/divinity76) made their first contribution in docker/compose#12213 **Full Changelog**: docker/compose@v2.29.7...v2.30.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

glours force-pushed the introduce-generate-cmd branch 3 times, most recently from b80af68 to 2b727f9 Compare October 14, 2024 14:54

ndeloof reviewed Oct 15, 2024

View reviewed changes

pkg/compose/generate.go Outdated Show resolved Hide resolved

ndeloof reviewed Oct 15, 2024

View reviewed changes

pkg/compose/convert.go Outdated Show resolved Hide resolved

glours force-pushed the introduce-generate-cmd branch 3 times, most recently from 2cba772 to 1113d5a Compare October 18, 2024 11:37

glours marked this pull request as ready for review October 18, 2024 14:45

glours requested review from a team, ndeloof, jhrotko and lizzthabet and removed request for a team October 18, 2024 14:45

glours self-assigned this Oct 18, 2024

lizzthabet reviewed Oct 23, 2024

View reviewed changes

glours force-pushed the introduce-generate-cmd branch 2 times, most recently from 754c172 to 6e64691 Compare October 25, 2024 13:41

ndeloof reviewed Oct 25, 2024

View reviewed changes

glours force-pushed the introduce-generate-cmd branch from 6e64691 to d1c6d14 Compare October 25, 2024 14:15

glours enabled auto-merge (rebase) October 25, 2024 14:51

glours force-pushed the introduce-generate-cmd branch 2 times, most recently from 6a1a426 to a14defd Compare October 25, 2024 15:21

glours disabled auto-merge October 25, 2024 15:35

introduce generate command as alpha command

4c071f9

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

glours force-pushed the introduce-generate-cmd branch from a14defd to 4c071f9 Compare October 25, 2024 16:03

glours requested a review from ndeloof October 25, 2024 16:05

jhrotko reviewed Oct 28, 2024

View reviewed changes

jhrotko approved these changes Oct 28, 2024

View reviewed changes

glours merged commit 51ebeb5 into docker:main Oct 28, 2024
30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce generate command as alpha command #12209

introduce generate command as alpha command #12209

glours commented Oct 14, 2024 •

edited

Loading

lizzthabet left a comment

lizzthabet Oct 23, 2024

lizzthabet Oct 23, 2024

ndeloof Oct 25, 2024

lizzthabet Oct 23, 2024

lizzthabet Oct 23, 2024

ndeloof Oct 25, 2024

lizzthabet Oct 23, 2024

lizzthabet Oct 23, 2024

glours Oct 25, 2024

ndeloof Oct 25, 2024

lizzthabet Oct 23, 2024

glours Oct 25, 2024

ndeloof Oct 25, 2024

ndeloof left a comment

jhrotko Oct 28, 2024

glours Oct 28, 2024

	volume.Labels = volume.Labels.Add(api.VolumeLabel, k)
	volume.Labels = volume.Labels.Add(api.ProjectLabel, project.Name)
	volume.Labels = volume.Labels.Add(api.VersionLabel, api.ComposeVersion)
	err := s.ensureVolume(ctx, volume, project.Name)

introduce generate command as alpha command #12209

introduce generate command as alpha command #12209

Conversation

glours commented Oct 14, 2024 • edited Loading

lizzthabet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndeloof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glours commented Oct 14, 2024 •

edited

Loading