Explore simplifying dev container definition with a concept of 'container features'

[chrmarti]

The goal of this feature is to address the complexity of adding simple tools to an existing dev container configuration. Currently this can only be done with sufficient know-how about writing a Dockerfile to get additional tools included.

The proposal to simplify adding tools to an existing dev container configuration is to have a "features" property listing any additional tools (from a given list of available tools) that should be installed. E.g., the following would add Node.js 14 and the GitHub CLI:

{
	"image": "debian",
	"features": {
		"nodejs": "14",
		"github": true
	}
}

A first implementation of this is adding the features by building an additional features image based on the original dev container image. The dev container is then created from this features image. The original image can be from an "image", "dockerfile" or "dockerComposeFile" configuration.

This issue continues a previously internal discussion.

[chrmarti]

Feedback from @Chuxel on the current implementation:

• We really need text for a hint for the choices that are selected. If you pick Node.js and Java right now, you get two sets of version choices without an indication which one is for which.
• Since there are options before you get to the selection of features, I think the hint text for the list of checkboxes should be something like "Select additional features to install"
• Choice/option support is not there for Docker Compose yet which makes the workflow for selecting a version of Python, Node, etc manual still. I assume that's just not implemented yet.
• I already mentioned needing the equivalent of "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ], but I thought I'd repeat it.
• I still do find myself wanting to be able to mark certain choices with the option of allowing someone to select "Other..." and let them enter a value. Right now we need to be vague with version numbers and constantly update. I'd love to let people type in a specific version of node for example.
• Support ${devcontainerName} substitution in the features.json, e.g., scope volume names to a specific dev container.
• Add "init" option to features.
• Add options support for features.

[chrmarti]

Update:

• Added an optional "hint" property for choices that will be shown as the picker's placeholder. E.g., "Select a Node.js version".
• Added "capAdd" and "securityOpt" properties that take a list of capabilities / security options (implemented for single containers and Docker Compose). E.g.:

"capAdd": [
	"SYS_ADMIN",
	"NET_ADMIN"
],
"securityOpt": [
	"label=user:USER",
	"label=role:ROLE"
]

[Chuxel]

//cc: @joshspicer @2percentsilk

[Chuxel]

@chrmarti One thing I realized was missing from this list was the idea of grouping feature options - or more accurately, allowing one feature to have multiple options. One example of this is the kubectl-helm-debian.sh script which has three different versions you can specify. We then have the opposite situation with Java where there's a Gradle and Maven script, but version selection for these should not be presented unless you select Java. //cc: @joshspicer @2percentsilk

[chrmarti]

To keep things simple, I'd prefer to have one version per feature. That also matches what NPM's package.json and the Linux package managers do, they have one version per package. So the kubectl-helm-debian.sh script would install 3 features (the script could be split up by feature if that helps).

If one feature depends on another feature, it might makes sense to simply install a suitable version of the dependency if none is specified. So if Gradle is to be installed, but Java is missing, we could just install the latest stable Java version. This again corresponds to what NPM and Linux package managers do and will result in a working installation. The user can later still refine the choice of the Java version.

To avoid overwhelming the user with a very long list of features, we could also move less relevant features to a 'Show All' list, similar to what we do for definitions. E.g., Gradle and Maven could be shown on the initial list for the Java base image, but move to the 'Show All' list for all other base images.

[Chuxel]

@chrmarti I don't think that works - features have multiple options. Another example is terraform. There's a TFlint version and TFGrunt version neither of which apply if you're not using Terraform. Scripts also have multiple options, like Docker-in-Docker supports using Moby or Docker, but again presenting that without first opting into installing Docker doesn't make sense. Python has options as to whether additional tools should be installed as does Go. Each of these are related to the primary feature.

This seems to be driving complexity into scripting in a way that is not beneficial to customers - the complexity is still there.

[chrmarti]

I'm not familiar with these particular tools, but if their relation to each other is similar to those I know from the JavaScript world: TFlint and TFGrunt are optional tools depending on Terraform and there might be alternatives like TFhint and TFGulp in the future.

With that in mind, it makes more sense to not nest these optional features, but make them top-level and maybe have their dependencies represented in metadata. I think we need to be careful to not let the current granularity of the scripts drive the schema in the devcontainer.json.

E.g.:

{
  "features": {
    "terraform": "1.0.0",
    "tflint": "0.31.0",
    "tfgrunt": "0.31.8"
  }
}

We were also discussing using an array:

{
  "features": [
    {
      "id": "terraform",
      "version": "1.0.0"
    },
    {
      "id": "tflint",
      "version": "0.31.0"
    },
    {
      "id": "tfgrunt",
      "version": "0.31.8"
    }
  ]
}

(I like the object notation better since that makes clear that the order doesn't matter and each feature can be added only once. It is also shorter.)

[stuartleeks]

If the features are split out, might there not be a case where order does matter? In which case, does the array syntax make sense?

[stuartleeks]

Also, the array syntax feels like it could open more options. E.g. for the az and gh CLIs, I'd like to be able to have an argument for whether or not to mount the host config folder (e.g. the .azure folder for az - doing this picks up my auth etc from the host):

{
  "features": [
    {
      "id": "terraform",
      "version": "1.0.0"
    },
    {
      "id": "azure-cli",
      "version": "1.0.0",
      "mountHostConfig": true
    },
   ]
}

Of course, the object syntax could also be expanded:

{
  "features": {
    "terraform": {
      "version": "1.0.0"
    },
    "azure-cli": {
      "version": "1.0.0",
      "mountHostConfig": true
    },
   }
}

[joshspicer]

@stuartleeks - i've been tracking what schema changes i've been proposing, and it's pretty similar to what you're saying here. https://github.com/microsoft/vssaas-planning/issues/5095

@chrmarti and I talked about it this morning. I prefer the array syntax for the main reason that it feels more flexible for the future- and given we're not really too firm on a lot of registry details, having a flexible features schema would be nice. My main hesitation, as Stuart pointed out, is if we would ever want to enforce some sort of ordering. Today I can't confidently say one way or another

[Chuxel]

With that in mind, it makes more sense to not nest these optional features, but make them top-level and maybe have their dependencies represented in metadata. I think we need to be careful to not let the current granularity of the scripts drive the schema in the devcontainer.json.

@chrmarti I think that as we move forward towards Josh's idea of packages, these will be inside the same "feature".

Furthermore, even if they're not, from a UI perspective, I don't want to present options that are not relevant unless they are needed. We should err on the side of user simplification over tech complexity given the goals here - which is inherently to abstract while still leaving full power available for those that want it. So, I'd want this ability in features.json even if it doesn't end up in the devcontainer.json that way. The feature here is "terraform support" or if we take the Java example, "Java support".

As long as we can represent this somewhere in such a way that UI implementations know how to present options, the specifics of how it ends up in devcontainer.json based on selections is less important to the end user. So really my comment here is about the features.json metadata (or whatever that evolves into).

[chrmarti]

(re: @stuartleeks)

If the features are split out, might there not be a case where order does matter? In which case, does the array syntax make sense?

Where the order matters, it should be handled by our implementation. (Similar to how Debian's apt takes care of the order.) Leaving the ordering to the user would ask the user for a decision they don't have the context for.

Of course, the object syntax could also be expanded:

Agreed, also "azure-cli": "1.0.0" can be used as the shorthand for using the defaults for any additional options.

(re: @joshspicer)

@chrmarti and I talked about it this morning. I prefer the array syntax for the main reason that it feels more flexible for the future- and given we're not really too firm on a lot of registry details, having a flexible features schema would be nice. My main hesitation, as Stuart pointed out, is if we would ever want to enforce some sort of ordering. Today I can't confidently say one way or another

Could you give any examples of how the registry might require the array syntax? (Using an array for future use without assigning it any semantics now doesn't seem to make sense since by the time we do assign semantics, users would already have their arrays with some more or less random order.)

(re: @Chuxel)

@chrmarti I think that as we move forward towards Josh's idea of packages, these will be inside the same "feature".

Thinking of features as packages makes me think that we should use a similar granularity as, e.g., Linux packages where each package has a single upstream version.

Furthermore, even if they're not, from a UI perspective, I don't want to present options that are not relevant unless they are needed. We should err on the side of user simplification over tech complexity given the goals here - which is inherently to abstract while still leaving full power available for those that want it. So, I'd want this ability in features.json even if it doesn't end up in the devcontainer.json that way. The feature here is "terraform support" or if we take the Java example, "Java support".

I have the UI in mind and think that we are not limiting ourselves. But we should discuss this here too. I think with the somewhat limiting QuickPick UI we use in VS Code, we might first show a list of features that do not require any other features (e.g., Terraform) and if the user's selection includes any dependencies for additional features, we would then show these additional features in a second list (e.g., TFlint and TFgrunt). This would also manage the length of the first list somewhat.

As long as we can represent this somewhere in such a way that UI implementations know how to present options, the specifics of how it ends up in devcontainer.json based on selections is less important to the end user. So really my comment here is about the features.json metadata (or whatever that evolves into).

We can tune the metadata towards contributors while we tune the devcontainer.json towards users. E.g., multiple features in the metadata could be handled by the same script (if that helps).

[Chuxel]

Thinking of features as packages makes me think that we should use a similar granularity as, e.g., Linux packages where each package has a single upstream version.

I have the UI in mind and think that we are not limiting ourselves. But we should discuss this here too. I think with the somewhat limiting QuickPick UI we use in VS Code, we might first show a list of features that do not require any other features (e.g., Terraform) and if the user's selection includes any dependencies for additional features, we would then show these additional features in a second list (e.g., TFlint and TFgrunt). This would also manage the length of the first list somewhat.

@chrmarti These packages are closer to Chocolatey which provide install options than it is to NuGet which backs Chocolatey. The version of the package here is actually the version of the installer not the thing being installed. It will be super critical that we do not expect people to create new versions of these packages every time there's an update to the underlying tool/sdk/runtime being installed. But at the same time, these scripts will have breaking changes - the analogy to a GitHub Action here isn't a bad one in this regard.

Ultimately, if we have a good UI, what ends up in devcontainer.json is less critical. Fundamentally, we're invoking installer functions here. Installers provide arguments and there will be more than one. Let's stick with some examples.

With the recent updates to Docker, I can see us needing to provide two options to the installer: Whether to use "Moby" rather than Docker and what version to use. Two options, same feature. There's similar things in all the scripts about whether to install supplemental tools. Once again, my concern here is passing through the complexity here in a way that does not provide us the ability to group these options in a logical way. Stuart's mountHostConfig property for the Azure CLI is another example. I think we need to provide enough functionality for feature authors (in this case, me) to craft the experience they want to present to the user.

In each of these cases, we'd want to be able to allow someone to select the feature, and then pick options. If that ends up as flat properties in devcontainer.json, ok - though we need to discuss how features actually identify what applies to them.

[chrmarti]

Thinking of features as packages makes me think that we should use a similar granularity as, e.g., Linux packages where each package has a single upstream version.
I have the UI in mind and think that we are not limiting ourselves. But we should discuss this here too. I think with the somewhat limiting QuickPick UI we use in VS Code, we might first show a list of features that do not require any other features (e.g., Terraform) and if the user's selection includes any dependencies for additional features, we would then show these additional features in a second list (e.g., TFlint and TFgrunt). This would also manage the length of the first list somewhat.

@chrmarti These packages are closer to Chocolatey which provide install options than it is to NuGet which backs Chocolatey. The version of the package here is actually the version of the installer not the thing being installed. It will be super critical that we do not expect people to create new versions of these packages every time there's an update to the underlying tool/sdk/runtime being installed. But at the same time, these scripts will have breaking changes - the analogy to a GitHub Action here isn't a bad one in this regard.

It's important to keep in mind that there are two audiences: The larger audience are the developers writing a devcontainer.json for their project. We want to keep complexity away from them as much as possible. The smaller audience are the contributors writing their own features with metadata that we currently capture in the internal features.json. We will want to give them as much flexibility as they need to implement their features and they will work with the complexity that comes with that.

Ultimately, if we have a good UI, what ends up in devcontainer.json is less critical. Fundamentally, we're invoking installer functions here. Installers provide arguments and there will be more than one. Let's stick with some examples.

I think it's important that we keep the devcontainer.json simple as that will be the file most users will work with at some point. There will be many devcontainer.json files in user repositories and we will support existing ones moving forward, so we need to make sure we pick the right amount of flexiblity. The UI we can improve and rewrite any time and we already know that the existing UI using VS Code's QuickPick comes with its advantages thanks to its simplicity, but also its limitations due to that.

With the recent updates to Docker, I can see us needing to provide two options to the installer: Whether to use "Moby" rather than Docker and what version to use. Two options, same feature. There's similar things in all the scripts about whether to install supplemental tools. Once again, my concern here is passing through the complexity here in a way that does not provide us the ability to group these options in a logical way. Stuart's mountHostConfig property for the Azure CLI is another example. I think we need to provide enough functionality for feature authors (in this case, me) to craft the experience they want to present to the user.

In each of these cases, we'd want to be able to allow someone to select the feature, and then pick options. If that ends up as flat properties in devcontainer.json, ok - though we need to discuss how features actually identify what applies to them.

My proposal is to have:

"azure-cli": "1.0.0"

As a shorthand for:

"azure-cli": {
  "version": "1.0.0"
}

Allowing for additional options like:

"azure-cli": {
  "version": "1.0.0",
  "mountHostConfig": true
}

The version property would be the Azure CLI's version. We can support specifying the installer's version similarly to GitHub Actions, but that should only be needed by CI engineers requiring full control. Most developers will want to use the latest version of the installer.

[Chuxel]

@chrmarti Totally good with that. How do we use features.json to indicate that the version and mountHostConfig prompts should only show if the "Azure CLI" feature is selected? That's my point / question.

[joshspicer]

I would argue for the azurecli case, you generally wouldn't want to specify the version number. I have no idea what version of azure-cli I run, but for the most part that doesn't matter.

I still feel

[
    "azure-cli"
]

is cleaner than

json

{
     "azure-cli": "XXX"    //Where the user doesn't care about the specific version, but would still need to supply _something_ here
}

It's a bit more verbose if you do want to supply extra config to the feature, but sometimes you won't have to, in which case a single string would do.

Other devcontainer fields like appPorts, mounts, runArgs, etc are arrays. While perhaps underneath the order does play some effect, we don't advertise order mattering, and people don't get confused.

Re: flexibility

I still also think theres a possibility this could be a valid collection of features in the future:

[
    {
      "id": "node",
      "version": "14"
    },
    {
      "id": "node",
      "version": "16"
    },
    "azure-cli"
]

In this case, the object notation restricts us.

[joshspicer]

@Chuxel @chrmarti Totally good with that. How do we use features.json to indicate that the version and mountHostConfig prompts should only show if the "Azure CLI" feature is selected? That's my point / question.

I think I address that here

[Chuxel]

@joshspicer Azure CLi is just an example of the general idea. No matter what, each script should include a "latest" version so you don't need to know (and most do already). The real question I had is how we represent scenarios where a feature has multiple options in features.json for our first cut so the UX behaves as expected.

[chrmarti]

(re: @joshspicer )

I still also think theres a possibility this could be a valid collection of features in the future:

[
    {
      "id": "node",
      "version": "14"
    },
    {
      "id": "node",
      "version": "16"
    },
    "azure-cli"
]

This opens questions like: Does the order matter? Which version will be on the PATH? How do you switch between versions? Which features support installing multiple versions?

I prefer solving this with existing solutions like nvm and if multiple versions of Node.js are required, I would make nvm (or any other Node.js version switcher) the feature that optionally takes a list of Node.js versions that should be installed. That would again align features with how Linux packages work, if we go beyond that we need to think about the additional development effort for feature authors.

(re: @Chuxel )

@chrmarti Totally good with that. How do we use features.json to indicate that the version and mountHostConfig prompts should only show if the "Azure CLI" feature is selected? That's my point / question.

We can add options to each feature, but I wouldn't add feature-like options. E.g., TFlint and TFGrunt should be features, not options to the Terraform feature.

I'm not fully convinced we need options in a first implementation and I don't see the current UI scale to options (I can't imagine users hopping through a growing list of QuickPick steps). So if we support options in metadata and the devcontainer.json, we also want to have defaults for those options, so the UI doesn't have to present them.

[Chuxel]

@chrmarti Re: Python, the script builds from source to enable other versions of Python. This is what pyenv does as well. It's not fast, however.

[chrmarti]

@joshspicer I have updated the extension's code to run everything from "options" with a little back compatibility for "type", "values", "default", "customValues" and "hint". You can move these over to a "version" option, like:

"version": {
	"type": "string",
	"proposals": ["latest", "1.0", "2.0"],
	"default": "latest",
	"description": "Version option."
}

(Use "enum" instead of "proposals" when no custom values are allowed.)

@Chuxel For prebuilt images, building from source for flexibility makes sense and the performance impact is in our CI. For features I'd imagine the default should be to install from prebuilt packages to avoid the delay for building from source on the user's side. If features are used as add-ons (as opposed to the primary technology stack used), using the latest prebuilt package might just be what users expect.

[chrmarti]

Previously added "init" to the metadata as discussed. Looked into adding ${devcontainerName} for volume names, the problem there is that volume names cannot use all characters that are allowed in a dev container name and that name can easily change, so a different volume name would result. Maybe we could support using the value from an option for the volume name.

[Chuxel]

@chrmarti Just to check version the only property supported?

I tried updating the docker-in-docker feature as follows and am not seeing it appear when I ran a quick check. I don't see the "moby" option presented in the UX.

"id": "docker-in-docker",
"name": "Docker (Moby) support (Docker-in-Docker)",
"options": {
    "moby": {
        "type": "boolean",
        "default": true,
        "description": "Install OSS Moby build instead of Docker CE"
    }
},
"entrypoint": "/usr/local/share/docker-init.sh",
"privileged": true,
"init": true,
"containerEnv": {
    "DOCKER_BUILDKIT": "1"
},
// ...

I also tried:

//...
"options": {
    "moby": {
        "type": "string",
        "enum": ["true", "false"],
        "default": "true",
        "description": "Install OSS Moby build instead of Docker CE"
    }
},
// ...

Similar things happened when I tried multiple properties. e.g. adding a flag as to whether node-gyp dependencies should be installed for the node feature in addition to version:

"id": "node",
"name": "Node.js (via nvm) and yarn",
"options": {
    "version": {
        "type": "string",
        "proposals": [ "lts", "latest","16", "14", "12" ],
        "default": "latest",
        "description": "Select or enter a Node.js version to install"
    },
    "nodeGypDependencies": {
        "type": "string",
        "enum": [ "true", "false"],
        "default": "true",
        "description": "Install dependencies for compiling native Node.js packages (node-gyp)?"
    }
}

[Chuxel]

I made updates assuming only version is supported in this commit so we can get a build out for testing tomorrow: microsoft/vscode-dev-containers@b99598d

I made a few polish fixes and hit a bug that I resolved. Take a look @joshspicer @2percentsilk and see if there's anything else you want to tweak!

[chrmarti]

The QuickPick UI only offers "version", only IntelliSense has all the options. Since our goal is to have the checkboxes to get users set up and running, I think we should also remove the "version" picker from the UI and just always use "latest".

[Chuxel]

The QuickPick UI only offers "version", only IntelliSense has all the options.

@chrmarti Got it, so currently we can add them and they'll be picked up in json?

Since our goal is to have the checkboxes to get users set up and running, I think we should also remove the "version" picker from the UI and just always use "latest".

@chrmarti I strongly disagree with this part. Our goal is actually to provide a yeoman-like experience in the UI with the ability to do advanced things in json. The problem here again is that people need the option to pin to a version. The way the UI is now with the version picker being available makes that really easy to do which is great - so I don't think we should take this away. This is a major help in discoverability. It has the same advantages that the settings UI does despite the fact that everything is instrumented in the json editor too.

Also, as we add in remote features, will those work in the json editor?

I'd like to expose additional options in the UI, but I'd like to hear @2percentsilk's thoughts here too.

[chrmarti]

@Chuxel Correct, additional options will be available in JSON and suggested in IntelliSense.

I see you've moved all "type": "choice" features over, can we also move the "type": "option" ones over to offer a single "latest" version? (Or more versions where applicable.) That will allow each feature to have the "feature_id": "latest" shorthand and will make it easy to add versions in the future.

[joshspicer]

@Chuxel Correct, additional options will be available in JSON and suggested in IntelliSense.

I see you've moved all "type": "choice" features over, can we also move the "type": "option" ones over to offer a single "latest" version? (Or more versions where applicable.) That will allow each feature to have the "feature_id": "latest" shorthand and will make it easy to add versions in the future.

I'll move these over to make sure I have a good grasp on the new syntax

[joshspicer]

...can we also move the "type": "option" ones over to offer a single "latest" version? (Or more versions where applicable.) That will allow each feature to have the "feature_id": "latest" shorthand and will make it easy to add versions in the future.

Is it required for one of the options to be version? We have no concept of version for some scripts (like docker-in-docker)

I suppose the shorthand then would be broken like you're mentioning. I guess that means authors need to add in a version to their feature's features.json entry? Seems like an unnecessary restriction IMO

[joshspicer]

Please let me know if anything else should be tweaked in the features.json ^

I added in a version option for all of the features that didn't have it, with a description of "Not used" when applicable.

[Chuxel]

@chrmarti lol. As if to accentuate the importance of version selection, we've had two breaks today that could be solved with pinning to a specific version.

1. One was Docker Compose, though we don't expose that version in the script and probably should.
   Here again, people expect this to be there if Docker is installed. curl: (22) The requested URL returned error: 404 in docker-in-docker-debian.sh or docker-debian.sh vscode-dev-containers#1075
2. The other is there have been some issues with the latest terragrunt release. No binaries for v0.32.5 release gruntwork-io/terragrunt#1829, No binaries again for v0.33.0 gruntwork-io/terragrunt#1832

We're finding these in CI, which is great, but if we don't have the ability to recommend people pin to specific releases rather than "latest", this will keep happening. (We also need to recommend pre-building images since that side-steps this entirely, but that's a separate topic.)

[chrmarti]

Is it required for one of the options to be version? We have no concept of version for some scripts (like docker-in-docker)

I suppose the shorthand then would be broken like you're mentioning. I guess that means authors need to add in a version to their feature's features.json entry? Seems like an unnecessary restriction IMO

@joshspicer "version" is optional and "feature_id": true should continue to work for these features.

[chrmarti]

@joshspicer I see a few cases (e.g., PowerShell) where a version makes sense even if we currently don't offer any but the "latest". For these I recommend keeping the version, so we can add versions in the future without changing the schema from boolean to string.

[chrmarti]

Available without experimental setting with Remote-Containers 0.200.0. Closing this issue, let's split out new issues for the remaining tasks.