Kiln bakes tiles
Kiln helps tile developers build products for VMware Tanzu Operations Manager. It provides an opinionated folder structure and templating capabilities. It is designed to be used both in CI environments and in command-line to produce a tile.
More information for those just getting started can be found in the Ops Manager Tile Developer Guide . Looking at an example kiln tile may also be helpful
To install the kiln
CLI
-
install with Homebrew
brew tap pivotal-cf/kiln https://github.com/pivotal-cf/kiln brew install kiln
-
download from the releases page
export KILN_VERSION KILN_VERSION="$(curl -H "Accept: application/vnd.github.v3+json" 'https://api.github.com/repos/pivotal-cf/kiln/releases?per_page=1' | jq -r '.[0].name')" curl -L -o kiln "https://github.com/pivotal-cf/kiln/releases/download/${KILN_VERSION}/kiln-darwin-${KILN_VERSION}" # check the checksum cp kiln "$(go env GOPATH)/bin" kiln version
-
build from source
git clone git@github.com:pivotal-cf/kiln.git cd kiln ./scripts/install.sh
-
copy from a Docker image (to another image)
docker pull pivotalcfreleng/kiln:latest
FROM pivotalcfreleng/kiln:latest as kiln FROM ubuntu COPY --from=kiln /kiln /usr/bin/kiln CMD /usr/bin/bash
Each tile must have a Kilnfile and Kilnfile.lock. Both are YAML files. Kiln won't generate them for you.
The code for parsing the Kilnfile and Kilnfile.lock exists in this package: "github.com/pivotal-cf/kiln/pkg/cargo"
.
Although the package interface is not yet stable, we have found importing it directly to be useful in CI or one-off scripts.
The Specification (source)
This file contains a range of configuration in support of kiln commands.
This field must be a string.
This field should be populated with the TanzuNet product slug where this tile is published.
It is used by kiln publish.
This field must be a list of strings.
This should be populated with the names of TanzuNet User Groups to attach to a pre-release on TanzuNet.
It is used by kiln publish.
This field must be a list of objects with keys from ReleaseSourceConfig
.
All elements must have a type
field.
The values for the type
(string) field are "bosh.io"
, "s3"
, "github"
, or "artifactory"
See fetch
documentation for more details.
Each release you want to add to your tile must have an element in this array. The "name" field with the BOSH Release Name as the value must be set here. The BOSH Release Name must be identical to the name of the release.
You must set a "name" field with the BOSH Release Name as the value must be set here. The BOSH Release Name must be identical to the name of the release.
You may set a "version" field. The value must match the constraints specification for this library.
You may set a "github_repository" field. This should be where the BOSH Release source is maintained. It is used for generating Release Notes for your tile.
You may set a "float_always" field. When you set this, kiln glaze
will not lock this release's version.
You may set a "maintenance_version_bump_policy" field. When you run kiln deglaze
this field specifies how to reset the version constraint after kiln glaze
locked it.
- "LockNone": Given a glazed version value "1.2.3", this setting resets the version constraint to
"*"
- "LockPatch": Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.2.3"
- "LockMinor": Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.2.*"
- "LockMajor": (default) Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.*"
You may set a "slack" field. Kiln does not use this field. It can be useful for product tile Authors to know who to reach out to when something goes wrong.
You may add a list of kiln bake
flags in the Kilnfile to keep a record of how your tile was baked and to keep CI scripts simpler.
If you set add more than one element to the bake_configurations list, you need to select one by adding a kiln bake --variables=tile_name=big-footprint-topology
flag corresponding to a bake configuration with a - tile_name: big-footprint-topology
element
These are the mappings from bake flag to each field in a bake_configurations element:
bake_configurations element field | bake flag | documentation |
---|---|---|
"tile_name" |
--variables=tile_name= |
This field is used when selecting a configuration from a list of bake_configurations. |
"metadata_filepath" |
--metadata= |
This should be the path to the product template entrypoint. Usually called base.yml |
"icon_filepath" |
--icon= |
This may be a path to a png file. |
"forms_directories" |
--forms-directory= |
This may be a list of directories. |
"instance_groups_directories" |
--instance-groups-directory= |
This may be a list of directories. |
"jobs_directories" |
--jobs-directory= |
This may be a list of directories. |
"migrations_directories" |
--migrations-directory= |
This may be a list of directories. |
"properties_directories" |
--properties-directory= |
This may be a list of directories. |
"runtime_configurations_directories" |
--runtime-configs-directory= |
This may be a list of directories. |
"bosh_variables_directories" |
--bosh-variables-directory= |
This may be a list of directories. |
"embed_files" |
--embed= |
This may be a list of filepaths. |
"variable_files" |
--variables-file= |
This may be a list of filepaths. |
The Lock File (source)
This file specifies the exact BOSH Release tarballs to package in a tile.
This is an array of BOSH Release locks.
Elements will be modified by running kiln update-release
.
Each element in the releases array in the Kilnfile will have a corresponding element in the Kilnfile.lock releases array.
The release name, release version, sha1 checksum, remote_source, remote_path are fields on each element.
Usage: kiln [options] <command> [<args>]
--help, -h bool prints this usage information (default: false)
--version, -v bool prints the kiln release version (default: false)
Commands:
bake bakes a tile
cache-compiled-releases Cache compiled releases
fetch fetches releases
find-release-version prints a json string of a remote release satisfying the Kilnfile version and stemcell constraints
find-stemcell-version prints the latest stemcell version from Pivnet using the stemcell type listed in the Kilnfile
generate-osm-manifest Print an OSM-format manifest.
glaze Pin versions in Kilnfile to match lock.
help prints this usage information
publish publish tile on Pivnet
re-bake re-bake constructs a tile from a bake record
release-notes generates release notes from bosh-release release notes
sync-with-local update the Kilnfile.lock based on local releases
test Test manifest for a product
update-release bumps a release to a new version
update-stemcell updates stemcell and release information in Kilnfile.lock
upload-release uploads a BOSH release to an s3 release_source
validate validate Kilnfile and Kilnfile.lock
version prints the kiln release version
It takes release and stemcell tarballs, metadata YAML, and JavaScript migrations as inputs and produces an OpsMan-compatible tile as its output.
The produce a tile, you simply need to be within a tile directory and execute the following command:
$ kiln bake
This will ensure that you have the necessary releases by first calling kiln fetch
.
Refer to the example-tile for a complete example showing the different features kiln supports.
Additional bake options
The --allow-only-publishable-releases
flag should be used for development only
and allows additional releases other than those specified in the kilnfile.lock to
be included in the tile
The --bosh-variables-directory
flag can be used to include CredHub variable
declarations. You should prefer the use of variables rather than Ops Manager
secrets. Each .yml
file in the directory should define a top-level variables
key.
This flag can be specified multiple times if you have organized your variables into subdirectories for development convenience.
Example variables directory.
The --download-threads
flag is for those using S3 as a BOSH release source.
This flag sets the number of parallel threads to download parts from S3
The --embed
flag is for embedding any extra files or directories into the
tile. There are very few reasons a tile developer should want to do this, but if
you do, you can include these extra files here. The flag can be specified
multiple times to embed multiple files or directories.
The --final
flag is to bake a final release tile. When passing the --final flag,
Kiln creates a baked record file with metadata like source revision SHA, tile version, kiln version and
file checksums. This bake record file will be created under bake_records folder. This
bake record file can later be used to re-bake the tile.
The --forms-directory
flag takes a path to a directory that contains one
or more forms. The flag can be specified more than once.
To reference a form file in the directory you can use the form
template helper:
$ cat /path/to/metadata
---
form_types:
- $( form "first" )
Example forms directory.
The --icon
flag takes a path to an icon file.
To include the base64'd representation of the icon you can use the icon
template helper:
$ cat /path/to/metadata
---
icon_image: $( icon )
The --instance-groups-directory
flag takes a path to a directory that contains one
or more instance groups. The flag can be specified more than once.
To reference an instance group in the directory you can use the instance_group
template helper:
$ cat /path/to/metadata
---
job_types:
- $( instance_group "my-instance-group" )
Example instance-groups directory.
The --jobs-directory
flag takes a path to a directory that contains one
or more jobs. The flag can be specified more than once.
To reference a job file in the directory you can use the job
template helper:
$ cat /path/to/instance-group
---
templates:
- $( job "my-job" )
- $( job "my-aliased-job" )
- $( job "my-errand" )
Example jobs directory.
You may find that you want to define different job files for the same BOSH job
with different properties. To do this you add an alias
key to the job which
will be used in preference to the job name when resolving job references:
$ cat /path/to/my-aliased-job
---
name: my-job
alias: my-aliased-job
The --kilnfile
flag is required with kiln version v0.84.0 and later
The flag expects filepath to a Kilnfile (default: Kilnfile). This
file contain links to all the bosh sources used to build a tile
See the Kilnfile section for more information on Kilnfile formatting
Tile authors will also need to include a Kilnfile.lock in the same directory as the Kilnfile.
See the Kilnfile.lock section for more information on Kilnfile.lock formatting
Specify a file path to a tile metadata file for the --metadata
flag. This
metadata file will contain the contents of your tile configuration as specified
in the OpsManager tile development documentation.
The --metadata-only
flag outputs the generated metadata to stdout.
This flag cannot be used with --output-file
.
If your tile has JavaScript migrations, then you will need to include the
--migrations-directory
flag. This flag can be specified multiple times if you
have organized your migrations into subdirectories for development convenience.
The no-confirm
flag will delete extra releases in releases directory without prompting.
This flag defaults to true
The --output-file
flag takes a path to the location on the filesystem where
your tile will be created. The flag expects a full file name like
tiles/my-tile-1.2.3-build.4.pivotal
.
Cannot be used with --metadata-only
.
The --properties-directory
flag takes a path to a directory that contains one
or more blueprint property files. The flag can be specified more than once.
To reference a properties file in the directory you can use the property
template helper:
$ cat /path/to/metadata
---
property_blueprints:
- $( property "rep_password" )
Example properties directory.
The --releases-directory
flag takes a path to a directory that contains one or
many release tarballs. The flag can be specified more than once. This is
useful if you consume bosh releases as Concourse resources. Each release will
likely show up in the task as a separate directory. For example:
$ tree /path/to/releases
|-- first
| |-- cflinuxfs2-release-1.166.0.tgz
| `-- consul-release-190.tgz
`-- second
`-- nats-release-22.tgz
To reference a release you can use the release
template helper:
$ cat /path/to/metadata
---
releases:
- $( release "cflinuxfs2" )
- $( release "consul" )
- $( release "nats" )
Example kiln command line:
$ kiln bake \
--version 2.0.0 \
--metadata /path/to/metadata.yml \
--releases-directory /path/to/releases/first \
--releases-directory /path/to/releases/second \
--stemcells-directory /path/to/stemcells/first \
--stemcells-directory /path/to/stemcells/second \
--output-file /path/to/cf-2.0.0-build.4.pivotal
The --runtime-configs-directory
flag takes a path to a directory that
contains one or more runtime config files. The flag can be specified
more than once.
To reference a runtime config in the directory you can use the runtime_config
template helper:
$ cat /path/to/metadata
---
runtime_configs:
- $( runtime_config "first-runtime-config" )
Example runtime-configs directory.
The --sha256
flag calculates the sha256 checksum of the output file
The --skip-fetch-directories
skips the automatic release fetching of
the specified release directories
Warning: --stemcell-tarball
will be removed in a future version of kiln.
Use --stemcells-directory
in the future.
The --stemcell-tarball
flag takes a path to a stemcell.
To include information about the stemcell in your metadata you can use the
stemcell
template helper:
$ cat /path/to/metadata
---
stemcell_criteria: $( stemcell )
The --stemcells-directory
flag takes a path to a directory containing one
or more stemcells.
To include information about the stemcell in your metadata you can use the
stemcell
template helper. It takes a single argument that specifies which
stemcell os.
The stemcell
helper does not support multiple versions of the same operating
system currently.
$ cat /path/to/metadata
---
stemcell_criteria: $( stemcell "ubuntu-xenial" )
additional_stemcells_criteria:
- $( stemcell "windows" )
For tile developers looking to get some quick feedback about their tile
metadata, the --stub-releases
flag will skip including the release tarballs
into the built tile output. This should result in a much smaller file that
should upload much more quickly to OpsManager.
The --variable
flag takes a key=value
argument that allows you to specify
arbitrary variables for use in your metadata. The flag can be specified
more than once.
To reference a variable you can use the variable
template helper:
$ cat /path/to/metadata
---
$( variable "some-variable" )
The --variables-file
flag takes a path to a YAML file that contains arbitrary
variables for use in your metadata. The flag can be specified more than once.
To reference a variable you can use the variable
template helper:
$ cat /path/to/metadata
---
$( variable "some-variable" )
Example variables file.
The --version
flag takes the version number you want your tile to become.
To reference the version you use the version
template helper:
$ cat /path/to/metadata
---
product_version: $( version )
provides_product_versions:
- name: example
version: $( version )
It constructs a tile from a given bake record file.
To run the command, you simply need to be within a tile directory and execute the following command:
$ kiln re-bake --output-file tile.pivotal bake_records/1.0.0.json
Any variables that Kilnfile needs for the kiln re-bake command should be set in ~/.kiln/credentials.yml file
The test
command exercises to ginkgo tests under the /<tile>/test/manifest
and /<tile>/migrations
paths of the pivotal/tas
repos (where <tile>
is tas, ist, or tasw).
Running these tests require a docker daemon and ssh-agent to be running.
If you run into this docker error could not execute "test": failed to connect to Docker daemon: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running
,
then create a symlink sudo ln -s ~/.docker/run/docker.sock /var/run/docker.sock
If no ssh identity is added (check with ssh-add -l
) , then kiln test
will add a ssh key in the following order, prompting for a passphrase if required:
~/.ssh/id_rs
~/.ssh/id_dsa
~/.ssh/d_ecdsa
~/.ssh/d_ed25519
~/.ssh/dentity
The identity must have access to github.com/pivotal/ops-manager.
Here are command line examples:
$ cd ~/workspace/tas/ist
$ kiln test
cd ~
$ kiln test --verbose -tp ~/workspace/tas/ist --ginkgo-manifest-flags "-p -nodes 8 -v"
Additional test options
The --ginkgo-manifest-flags
flag can be used to pass through Ginkgo test flags. The defaults being passed through are -r -p -slowSpecThreshold 15
. Pass help
as a flag to retrieve the available options for the embeded version of ginkgo.
The --manifest-only
flag can be used to run only Manifest tests. If not passed, kiln test
will run both Manifest and Migration tests by default.
The --migrations-only
flag can be used to run only Migration tests. If not passed, kiln test
will run both Manifest and Migration tests by default.
The --tile-path
(-tp
) flag can be set the path the directory you wish to test. It defaults to the current working directory. For example
$ kiln test -tp ~/workspace/tas/ist
The --verbose
(-v
) flag will log additional debugging info.
The fetch
command downloads bosh release tarballs specified in the Kilnfile and
Kilnfile.lock files to a local directory specified by the --releases-directory
flag.
Kiln verifies that the checksum (SHA1) of the downloaded release matches checksum specified for the release in the Kilnfile.lock file. If the checksums do not match, then the releases that don't match will be deleted from disk. Since BOSH releases from different directors with the same packages result in complied releases with different hashes this may result in some problems where if you download a release that was compiled with a different director those releases will be deleted.
Kiln will not download releases if an existing release exists with the correct release version and checksum.
A Kilnfile contains information about the bosh releases and stemcell used by a particular tile
Example Kilnfile:
---
slug: some-slug #optional but if included should match network.pivotal.io
release_sources:
- type: bosh.io
releases:
- name: bpm
version: '*'
stemcell_criteria:
os: ubuntu-xenial
version: "~621"
release_sources:
- type: bosh.io
release_sources:
- type: s3
id: unique-name
publishable: true # if this bucket contains releases that are suitable to ship to customers
bucket: some-bucket-in-s3
region: us-east-1 # must be the region of the above bucket
access_key_id: $(variable "s3_access_key_id") # Must have at least read permissions to bucket
secret_access_key: $(variable "s3_secret_access_key")
path_template: bosh-releases/compiled/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz # See Templating
- type: github
id: optional-unique-name-defaults-to-github-org-name
org: the-github-org
github_token: $(variable "github_token")
- type: artifactory
id: unique-name
artifactory_host: https://build-artifactory.your-artifactory-url.com
repo: some-artifactory-repo
publishable: true # if this repo contains releases that are suitable to ship to customers
username: $(variable "artifactory_username")
password: $(variable "artifactory_password")
path_template: shared-releases/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz # See Templating
Kilnfile files support the following templating options:
-
{{.Name}}
for release name -
{{.Version}}
for release version -
{{.StemcellOS}}
for stemcell OS -
{{.StemcellVersion}}
for stemcell version -
There's also access to a
trimSuffix
helper (e.g.{{trimSuffix .Name "-release"}}
)
The select
function allows you to pluck values for nested fields from a
template helper.
For instance, this section in our example tile:
my_release_version: $( release "my-release" | select "version" )
Results in:
my_release_version: 1.2.3
release_sources:
- type: s3
compiled: true
bucket: compiled-releases
region: us-west-1
access_key_id: $(variable "aws_access_key_id")
secret_access_key: $(variable "aws_secret_access_key")
path_template: 2.6/{{trimSuffix .Name "-release"}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz
Credentials like S3 keys are not stored in git repos. To support separating that information from non-sensitive configuration, you can reference variables like you do in tile config.
---
aws_access_key_id: SOME_REALLY_SECRET_ID
aws_secret_access_key: SOME_REALLY_SECRET_KEY
Interpolating this file in kiln would look something like this.
- Get your credentials from Lastpass by running:
lpass show --notes 'pas-releng-fetch-releases' > vars.yml
kiln bake --kilnfile random-Kilnfile --variables-file vars.yml
The Kilnfile.lock file name is expected to be a file in the same directory as the
Kilnfile with lock
as as the filename extension.
This file contains the full list of specific versions of all releases, shas, and sources for bosh releases that will go into the tile as well as the target stemcell.
The file has two top level members releases
and stemcell_criteria
.
The releases
member is an array of members with each element having the following members.
name
: bosh release namesha1
: checksum of the tarballversion
: semantic version of the releaseremote_source
: the resource-type for bosh.io or the id for the other typesremote_path
: the path that where the bosh release is stored
The stemcell_criteria
member is defines the stemcell used to generate the tile
os
: the stemcell os used (e.g. ubuntu-xenial)version
: semantic version of the stemcell
Example Kilnfile.lock :
releases:
- name: bpm
sha1: 86675f90d66f7018c57f4ae0312f1b3834dd58c9
version: 1.1.18
remote_source: bosh.io
remote_path: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.1.18
- name: backup-and-restore-sdk
sha1: 0f48faa2f85297043e5201e2200567c2fe5a9f9a
version: 1.18.84
remote_source: unique-name # this could be artifactory or s3
remote_path: bosh-releases/compiled/backup-and-restore-sdk-1.18.84-ubuntu-jammy-1.179.tgz
- name: hello-release
sha1: 06500a2002f6e14f6c258b7ee7044761a28d3d5a
version: 0.1.5
remote_source: the-github-org
remote_path: https://github.com/crhntr/hello-release/releases/download/v0.1.5/hello-release-v0.1.5.tgz
stemcell_criteria:
os: ubuntu-xenial
version: "621.0"