Metadata

[stjohnjohnson]

As a user, I want to pass structured data between my jobs, so that I can make informed decisions about what to deploy or test

Currently, there are almost no built-in ways to pass information between builds. The most common way is to use the value of the current Git tag. But that data cannot be guaranteed to be correct.

Given this simple pipeline:

main -> deploy -> promote

The deploy job would need to know the version of the application that was published in the main job, while the promote job would need to know the version that was tested in the deploy job. This information is not available with just that git tag.

Instead, Screwdriver should provide a simple interface for reading/writing values that can be passed between builds in the same event.

Rules:

• meta is a command-line client (written in go) that is made available to the build container.
• Metadata should be passed from one job to the next in succession (in the same event) (see appendix A).
• When combining the results of matrix builds, they should be merge/replaced in a random order (to ensure users don't depend on a side-effect).
• When the event is done, a record of the metadata should be stored there as well.
• Metadata can contain strings, numbers, boolean, objects, and arrays. The interface should account for this (see appendix B).
• meta set should not output anything, except during failures (STDERR).
• meta get should output in string representation with no newlines (see appendix C).

Stretch (gold bar):

• meta can read in from a file or STDIN.
• meta get --previous can read from the previous successful event.

---

Work

Preparation

• Cucumber feature file
• Meta interface (discussed in this issue)

Implementation

• Screwdriver.yaml changes
• Launcher changes
• API changes
• UI changes
• Store changes
• Schema changes
• Model changes
• Datastore changes
• Executor changes

Outcome

• Cucumber feature passing
• Documentation

---

Appendix

A - Metadata passing

Example pipeline:

main -> package -> deploy

main

• starts with:
  {}
• runs:
  $ meta set coverage 99.52
• results in:
  { "coverage": 99.52 }

package

• starts with:
  { "coverage": 99.52 }
• runs:
  $ meta set version v3.1.2
• results in:
  { "coverage": 99.52, "version": "v3.1.2" }

deploy

• starts with:
  { "coverage": 99.52, "version": "v3.1.2" }
• runs:
  $ meta set url http://example.com
• results in:
  { "coverage": 99.52, "url": "http://example.com", "version": "v3.1.2" }

B - Input Interface Suggestions

note: the json structure below each line is just showing internal representation, not output

$ meta set foo true
{ "foo": true }

$ meta set foo 15.5
{ "foo": 15.5 }

$ meta set foo hello
{ "foo": "hello" }

$ meta set foo[] arrg
{ "foo": ["arrg"] }

$ meta set foo[1] zarp
{ "foo": [null,"zarp"] }

$ meta set foo.bar baz
{ "foo": { "bar": "baz" } }

# Extra cool things, but not required
$ echo "banana" | meta set foo -f -
{ "foo": "banana" }

$ meta set foo -f ./test
{ "foo": "contents of test" }

C - Output Interface Suggestions

note: the json structure above each line is just showing internal representation, not output

{ "foo": true }
$ meta get foo
true

{ "foo": "hello" }
$ meta get foo
hello

{ "foo": ["arrg"] }
$ meta get foo
["arrg"]

{ "foo": { "bar": "baz" } }
$ meta get foo
{"bar":"baz"}

[d2lam]

Hmmm this kinda concerns me: When combining the results of matrix builds, they should be merge/replaced in a random order (to ensure users don't depend on a side-effect).

If main is a matrix job with FOO: [1,2]
FOO=1 sets coverage to 90
FOO=2 sets coverage to 100

main triggers deploy. Then inside deploy, coverage can be either 90 or 100?

[edit] example

[stjohnjohnson]

Yup, it's inspired by GoLang maps. For us to pass metadata consistently between jobs, we need to have an approach that's the same regardless if you're running a matrix build or not. The only option is merging the result. If we don't pick a policy for this, then it will become an undocumented side-effect. But any policy I thought of seemed hard to explain (e.g. merged based on alphabetical serialization of the sorted keys from your matrix builds). So I thought back to the GoLang thing that @jer explained to me and it occurred to me this would be excellent for this problem. If we enforce (and document) randomness, then users must make their own decisions about how to namespace their metadata in matrix builds such that they can retrieve it in later jobs.

[stjohnjohnson]

@catto How would you like to approach working on this? I'm assuming you'll need a new repo for the meta go binary. Would you like:

• Us to create an empty repo first and then you give a PR
• Your team makes the repo first and then we fork it
• Another option

I created #293-metadata in Slack for any open discussions

[catto]

@stjohnjohnson Thanks, I'll contact you on the Slack channel

[tk3fftk]

I have a question about format of meta get .
The rules is defined:

meta get should output in string representation with no newlines (see appendix C).

@minz1027 commented screwdriver-cd/launcher#108 (comment):

Another thing to note: if u do /opt/sd/meta get foo only in a step, u won't see the value of foo in log since there is no newline in the end. And we use newline as separator when parsing logs.

We need newline or not?

[minzcmu]

@tk3fftk I think it should not include the newline in the value since user will probably assign the value to some variables. Just need to document it somewhere so that user won't get confused when they do meta get only.

[stjohnjohnson]

@tk3fftk @minz1027 it sounds like it's a bug if not outputting a newline will result in no log. We should force a newline when we run steps (in the writing of the shell script). But meta should not return a newline, that will allow it to work like a normal unix command.

[tk3fftk]

@minz1027 @stjohnjohnson I understood. You mean that the result is like this image when only run meta get. Thank you!

[minzcmu]

@tk3fftk yes! That's what I mean.

@stjohnjohnson you are right, we don't have a newline for sourcing the step file. The meta get results get printed with the next command ";echo " + guid + " $?\n", which is filtered out for the log... Will test locally to see if that's the problem.

https://github.com/screwdriver-cd/launcher/blob/master/executor/executor.go#L77

[stjohnjohnson]

@minz1027 also related #488

[tk3fftk]

@stjohnjohnson @minz1027
I found a problem😥 meta is not passed in pipeline.

To show log, I changed these lines to fmt and print status of parent build (also add Status to Build in screwdriver.go).
In first line, log shows status of parent build RUNNING. The reason is that build status is not updated until end of a next build at this line. meta and build status are updated in same time. So a triggered build can not get previous meta.

[minzcmu]

@tk3fftk good caught! can u help review the pr?
#494

And there is a bug in the launcher code for meta,
https://github.com/screwdriver-cd/launcher/blob/master/screwdriver/screwdriver.go#L124
the type shouldn't be string, i got error when launcher tried to unmarshal it.
Can u help fix it?

[minzcmu]

@tk3fftk I was writing the docs for meta-cli and found another bug for the meta-cli..

sh-3.2$ ./meta set foo.bar baz
sh-3.2$ ./meta get foo && echo
{"bar":"baz"}
sh-3.2$ ./meta get foo.bar && echo
null

The whole get argument is treated as a string key and the key 'foo.bar' maps to nothing.
https://github.com/screwdriver-cd/meta-cli/blob/master/meta.go#L43

[tk3fftk]

@minz1027 I only implemented meta get to satisfy these output interface suggestions... It seems to need function. I'll fix it.

Should meta get also obtain Array meta?
This means that:

% ./meta set hoge[4] aaa
% ./meta get hoge                                                                                                                                                  
["fuga",null,null,null,"aaa"]%                                                                                                                                                                     % ./meta get hoge[4]                                                                                                                                               
null%

Should meta get hoge[4] show "aaa"? Array pattern is also implemented.

[tk3fftk]

@minz1027 I fixed meta get. Could you review the PR?

[tkyi]

Blog post: http://blog.screwdriver.cd/post/158718077352/build-metadata
Documentation: https://docs.screwdriver.cd/user-guide/metadata