toJson() / fromJson() not symmetric in respect to strain order #71

tripodsan · 2019-03-22T01:07:08Z

Description
the order of properties is not guaranteed to be stable in objects, nor in the JSON serialization. this causes the order of strains to be undeterministic. Since the condition matching relies on the order, it is important to keep it stable.

To Reproduce
not so easy, since the object hash is mostly stable.

Version:

$ hlx --version
0.13.11-pre.9

The text was updated successfully, but these errors were encountered:

tripodsan · 2019-03-22T01:21:13Z

In the following example, the helix-demo is the last strain in the config, but in the received version, it is the first.

config sent:

$ hlx publish --remote
{ configuration:
   { version: 1,
     strains:
      { default: [Object],
        'launch-docs-helixdemo': [Object],
        'xd-docs-helixdemo': [Object],
        'starter-docs-helixdemo': [Object],
        'helix-demo': [Object] } },

config received:

$ wsk activation logs 4e26ade69e834835a6ade69e83f835e8
2019-03-22T01:19:25.903Z       stdout: config { strains:
   { default:
      { code: [Object],
        condition: '',
        content: [Object],
        directoryIndex: 'index.html',
        package:
         'developer-adobe-com/1e21313f7542fd3e00ff0b4601f566375b813bcc',
        perf: [Object],
        static: [Object],
        sticky: false,
        urls: [] },
     'helix-demo':
      { code: [Object],
        condition: '',
        content: [Object],
        directoryIndex: 'index.html',
        package:
         'developer-adobe-com/1e21313f7542fd3e00ff0b4601f566375b813bcc',
        perf: [Object],
        static: [Object],
        sticky: false,
        url: 'https://adobedevsite.helix-demo.xyz/',
        urls: [Array] },
     'launch-docs-helixdemo':
      { code: [Object],
        condition: '',
        content: [Object],
        directoryIndex: 'README.html',
        package:
         'developer-adobe-com/1e21313f7542fd3e00ff0b4601f566375b813bcc',
        perf: [Object],
        static: [Object],
        sticky: false,
        url: 'https://adobedevsite.helix-demo.xyz/launch/docs',
        urls: [Array] },
     'starter-docs-helixdemo':
      { code: [Object],
        condition: '',
        content: [Object],
        directoryIndex: 'Overview.html',
        package:
         'developer-adobe-com/1e21313f7542fd3e00ff0b4601f566375b813bcc',
        perf: [Object],
        static: [Object],
        sticky: false,
        url: 'https://adobedevsite.helix-demo.xyz/starter/docs',
        urls: [Array] },
     'xd-docs-helixdemo':
      { code: [Object],
        condition: '',
        content: [Object],
        directoryIndex: 'README.html',
        package:
         'developer-adobe-com/1e21313f7542fd3e00ff0b4601f566375b813bcc',
        perf: [Object],
        static: [Object],
        sticky: false,
        url: 'https://adobedevsite.helix-demo.xyz/xd/docs',
        urls: [Array] } },
  version: 1 }

tripodsan · 2019-03-22T01:36:42Z

the problem is, that even YAML doesn't mandate an order for mappings:

https://yaml.org/spec/1.2/spec.html#id2764044

Mapping
The content of a mapping node is an unordered set of key: value node pairs, with the restriction that each of the keys is unique. YAML places no further restrictions on the nodes. In particular, keys may be arbitrary nodes, the same node may be used as the value of several key: value pairs, and a mapping could even contain itself as a key or a value (directly or indirectly).

tripodsan · 2019-03-22T01:44:47Z

I see the following solutions:

Use priority property to order the strains. eg:

strains:
   default:
      priority: 0
      url: https://....
   demo:
      priority: 1
      url: https://...

change the config to use a sequence instead (like the original format ;-).

strains: 
  - 
    name: default
    url: "http://...."
  - 
    name: demo
    url: "http://...."

add an extra array property: strainOrder

strainOrder: 
  - default
  - demo
strains: 
  default: 
    url: "http://...."
  demo: 
    url: "http://...."

make the condition evaluation so smart, that the order doesn't matter.

tripodsan · 2019-03-22T01:46:03Z

my preference: 4 > 2 > 3> 1

@trieloff WDYT?

trieloff · 2019-03-22T08:17:45Z

With #20 (conditions language) we might have a good shot at option 4 (which I think is preferred by @kptdobe): sort the strains from most specific condition to least specific condition.

CSS has some kind of specificity rules (https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity) and I think we could construct a partially ordered set on our conditions language, where we apply a set of rules to determine if two conditions are more or less specific, or not comparable or equal.

Rules:

Property Matches between different properties are never comparable

Example:

url_param.foo: bar is not comparable to url_param.bar: foo

For `substring-starts` matching, the longer substring is more specific than the shorter substring.

Examples:

url: https://www.project-helix.io/ is less specific than url: https://www.project-helix.io/docs/
url.path: /docs/README.md is more specific than url.path: /docs/
url.host: www.project-helix.io is not comparable to url.host: project-helix.io
user_agent: Lynx is equal to user_agent: Lynx

Two `strict-match` matches are always equal in specificity

Examples:

client_country_code=: DE is equal to client_country_code=: CH

Two `regex-match` matches are always equal in specificity

Note: We could (in theory) turn the Regular Expressions into two DFA graphs using https://github.com/hokein/automata.js/ and find out if one graph is a subgraph of the other, in which case it would be less specific.

Examples:

accept_language~: en is not comparable to accept_language~: en_US

A `strict-match` is more specific than a `substring-starts` match, if one contains the other

Examples:

url.path=: /docs/README.md is more specific than url.path: /docs/
url.path=: /docs/ is more specific than url.path: /docs/
url.path=: /docs/ is more specific than url.path: /docs/README.md

A `strict-match` is more specific than the `regex-match` it matches

Examples:

url.path=: /docs/ is more specific than url.path~: /docs/
url.path=: /docs/ is more specific than url.path~: ^/docs/$
url.path=: /docs/ is not comparable to url.path~: /frogs/

Property range matches are comparable

Examples:

client_lat>: 100 is more specific than client_lat>: 99

Two `not` expressions are comparable if their children are comparable

Examples:

not:
  url.path: /docs/

is comparable to

not:
  url:path: /docs/foo

but not to

not:
  url.path: /foo/docs

Two `and` or `or` expressions are comparable if children of one expression are a subset of children of the other expression

Note: this condition can potentially be loosened by taking into account the compatibility and specificity of pairs of children.

Example:

and:
  url.hostname: www.adobe.io
  url.path: /docs

and

and:
  url.hostname: www.adobe.io
  url.path: /docs
  client_country_code: DE

are comparable, but both aren't comparable to:

and:
  accept_language~: de
  client_country_code: DE

Of two comparable `or` expressions, the one with more children is less specific

or:
  url.hostname: www.adobe.io
  url.path: /docs

is more specific than

or:
  url.hostname: www.adobe.io
  url.path: /docs
  client_country_code: DE

Of two comparable `and` expressions, the one with more children is more specific

Example:

and:
  url.hostname: www.adobe.io
  url.path: /docs

is less specific than

and:
  url.hostname: www.adobe.io
  url.path: /docs
  client_country_code: DE

Of two comparable `and` and `or` expressions, the `and` expression is more specific

or:
  url.hostname: www.adobe.io
  url.path: /docs

is less specific than

and:
  url.hostname: www.adobe.io
  url.path: /docs

tripodsan · 2019-03-22T08:33:38Z

@trieloff I like. can you add this information to the #20 issue?

As we only have substring match for now, we could add it here to solve the immediate problem. WDYT? (I still have the code that you made me remove :-)

trieloff · 2019-03-22T08:38:14Z

I turned it into a separate issue (#72)

@tripodsan I'd add a straincompare function that can be passed to Array.prototype.sort and we change it when #20 and #72 land.

tripodsan · 2019-03-22T09:01:20Z

I think we should stick to KISS and use a YAML sequence (list) for the strains. the evaluation order is the very simple to explain:

The strains conditions are matched in the order they are configured, the first match wins.

For backward compatibility to those thousands of existing configs, the YAML import will support both, sequence and map. but the YAML and JSON export, will always write the strains as a list.
The schema is applied after the import and will only support the list.

tripodsan · 2019-03-25T07:09:20Z

btw, the yaml parser we use has an option to use javascript maps for maps. I believe this would keep the order stable.

filmaj · 2019-03-25T13:38:31Z

I think we should stick to KISS and use a YAML sequence (list) for the strains.

If I think of examples out there where YAML order matters, travis and circleci configs come to mind. In those configs, the run order of commands matters a lot - and they both use sequences to solve this problem. The sequence solution, to me, follows the principle of least astonishment.

# [0.9.0](v0.8.4...v0.9.0) (2019-03-27) ### Features * **configuration:** Strains can be and should be specified as an ordered list ([38b5e9d](38b5e9d)), closes [#71](#71)

trieloff · 2019-03-27T10:50:50Z

🎉 This issue has been resolved in version 0.9.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

kptdobe · 2019-03-27T12:35:43Z

In the current state, the "map" strains config rewrite generates a config that looks like this: https://github.com/adobe/project-helix.io/blob/test-map-strains/helix-config.yaml

And hlx publish triggers many schema errors!

Invalid configuration:
Invalid Strain .strains should be object: type([{"code":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains"},"content":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-cli"},"static":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains","path":"/htdocs"},"directoryIndex":"index.html","package":"acapt/github-com--adobe--project-helix-io--test-map-strains-dirty","name":"client","urls":["https://alex.helix-demo.xyz/client"],"url":"https://alex.helix-demo.xyz/client"},{"code":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains"},"content":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-pipeline"},"static":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains","path":"/htdocs"},"directoryIndex":"index.html","package":"acapt/github-com--adobe--project-helix-io--test-map-strains-dirty","name":"pipeline","urls":["https://alex.helix-demo.xyz/pipeline"],"url":"https://alex.helix-demo.xyz/pipeline"},{"code":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains"},"content":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-shared"},"static":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains","path":"/htdocs"},"directoryIndex":"index.html","package":"acapt/github-com--adobe--project-helix-io--test-map-strains-dirty","name":"shared","urls":["https://alex.helix-demo.xyz/shared"],"url":"https://alex.helix-demo.xyz/shared"},{"code":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains"},"content":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains"},"static":{"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"project-helix.io","ref":"test-map-strains","path":"/htdocs"},"directoryIndex":"index.html","package":"acapt/github-com--adobe--project-helix-io--test-map-strains-dirty","name":"default","urls":["https://alex.helix-demo.xyz/"],"url":"https://alex.helix-demo.xyz/"},{"code":{"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":""},"content":{"protocol":"https","host":"github.com","port":"","hostname":"github.com","owner":"adobe","repo":"helix-cli","ref":"","path":""},"static":{"magic":false,"allow":[],"deny":[],"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":"/htdocs"},"directoryIndex":"index.html","package":"","name":"clientDev","sticky":false,"condition":"","perf":{"device":"","location":"","connection":""},"urls":["http://localhost:3000/client"],"url":"http://localhost:3000/client"},{"code":{"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":""},"content":{"protocol":"https","host":"github.com","port":"","hostname":"github.com","owner":"adobe","repo":"helix-pipeline","ref":"","path":""},"static":{"magic":false,"allow":[],"deny":[],"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":"/htdocs"},"directoryIndex":"index.html","package":"","name":"pipelineDev","sticky":false,"condition":"","perf":{"device":"","location":"","connection":""},"urls":["http://localhost:3000/pipeline"],"url":"http://localhost:3000/pipeline"},{"code":{"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":""},"content":{"protocol":"https","host":"github.com","port":"","hostname":"github.com","owner":"adobe","repo":"helix-shared","ref":"","path":""},"static":{"magic":false,"allow":[],"deny":[],"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":"/htdocs"},"directoryIndex":"index.html","package":"","name":"sharedDev","sticky":false,"condition":"","perf":{"device":"","location":"","connection":""},"urls":["http://localhost:3000/shared"],"url":"http://localhost:3000/shared"},{"code":{"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":""},"content":{"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":""},"static":{"magic":false,"allow":[],"deny":[],"protocol":"http","host":"localhost","port":"","hostname":"localhost","owner":"local","repo":"default","ref":"","path":"/htdocs"},"directoryIndex":"index.html","package":"","name":"defaultDev","sticky":false,"condition":"","perf":{"device":"","location":"","connection":""},"urls":["http://localhost:3000/"],"url":"http://localhost:3000/"}], {"type":"object"})
Proxy Strain .strains[0] has unknown property 'code'
Proxy Strain .strains[0] has unknown property 'content'
Proxy Strain .strains[0] has unknown property 'static'
Proxy Strain .strains[0] has unknown property 'directoryIndex'
Proxy Strain .strains[0] has unknown property 'package'
Proxy Strain .strains[0] should have required property 'origin'
Invalid Strain .strains[0].content should be string: type({"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-cli"}, {"type":"string"})
Git URL .strains[0].content should have required property 'ref'
Invalid Strain .strains[0].content must be either a Runtime Strain or a Proxy Strain
Invalid Strain .strains[0] must be either a Runtime Strain or a Proxy Strain
Proxy Strain .strains[1] has unknown property 'code'
Proxy Strain .strains[1] has unknown property 'content'
Proxy Strain .strains[1] has unknown property 'static'
Proxy Strain .strains[1] has unknown property 'directoryIndex'
Proxy Strain .strains[1] has unknown property 'package'
Proxy Strain .strains[1] should have required property 'origin'
Invalid Strain .strains[1].content should be string: type({"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-pipeline"}, {"type":"string"})
Git URL .strains[1].content should have required property 'ref'
Invalid Strain .strains[1].content must be either a Runtime Strain or a Proxy Strain
Invalid Strain .strains[1] must be either a Runtime Strain or a Proxy Strain
Proxy Strain .strains[2] has unknown property 'code'
Proxy Strain .strains[2] has unknown property 'content'
Proxy Strain .strains[2] has unknown property 'static'
Proxy Strain .strains[2] has unknown property 'directoryIndex'
Proxy Strain .strains[2] has unknown property 'package'
Proxy Strain .strains[2] should have required property 'origin'
Invalid Strain .strains[2].content should be string: type({"protocol":"https","host":"github.com","hostname":"github.com","owner":"adobe","repo":"helix-shared"}, {"type":"string"})
Git URL .strains[2].content should have required property 'ref'
Invalid Strain .strains[2].content must be either a Runtime Strain or a Proxy Strain
Invalid Strain .strains[2] must be either a Runtime Strain or a Proxy Strain
Strains .strains must be either a Runtime Strain or a Proxy Strain

data.strains should be object, data.strains[0] should NOT have additional properties, data.strains[0] should NOT have additional properties, data.strains[0] should NOT have additional properties, data.strains[0] should NOT have additional properties, data.strains[0] should NOT have additional properties, data.strains[0] should have required property 'origin', data.strains[0].content should be string, data.strains[0].content should have required property 'ref', data.strains[0].content should match exactly one schema in oneOf, data.strains[0] should match exactly one schema in oneOf, data.strains[1] should NOT have additional properties, data.strains[1] should NOT have additional properties, data.strains[1] should NOT have additional properties, data.strains[1] should NOT have additional properties, data.strains[1] should NOT have additional properties, data.strains[1] should have required property 'origin', data.strains[1].content should be string, data.strains[1].content should have required property 'ref', data.strains[1].content should match exactly one schema in oneOf, data.strains[1] should match exactly one schema in oneOf, data.strains[2] should NOT have additional properties, data.strains[2] should NOT have additional properties, data.strains[2] should NOT have additional properties, data.strains[2] should NOT have additional properties, data.strains[2] should NOT have additional properties, data.strains[2] should have required property 'origin', data.strains[2].content should be string, data.strains[2].content should have required property 'ref', data.strains[2].content should match exactly one schema in oneOf, data.strains[2] should match exactly one schema in oneOf, data.strains should match exactly one schema in oneOf

kptdobe · 2019-03-27T12:44:01Z

In the current state, the "sequence" strains seems to be handled properly during deploy (see https://github.com/adobe/project-helix.io/blob/test-sequence-strains/helix-config.yaml).

But publish fails!

Publishing [======================----------------------------] failed to set service config up for Helix 0.0serror: Remote publish service failed StatusCodeError: 400 - {"message":"Error: Invalid configuration:\nStrains .strains should be object: type([{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-cli\"},\"directoryIndex\":\"index.html\",\"name\":\"client\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/client\",\"urls\":[\"https://www.project-helix.io/client\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-pipeline\"},\"directoryIndex\":\"index.html\",\"name\":\"pipeline\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/pipeline\",\"urls\":[\"https://www.project-helix.io/pipeline\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-shared\"},\"directoryIndex\":\"index.html\",\"name\":\"shared\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/shared\",\"urls\":[\"https://www.project-helix.io/shared\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"directoryIndex\":\"index.html\",\"name\":\"default\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/\",\"urls\":[\"https://www.project-helix.io/\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-cli\"},\"directoryIndex\":\"index.html\",\"name\":\"clientDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/client\",\"urls\":[\"http://localhost:3000/client\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-pipeline\"},\"directoryIndex\":\"index.html\",\"name\":\"pipelineDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/pipeline\",\"urls\":[\"http://localhost:3000/pipeline\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-shared\"},\"directoryIndex\":\"index.html\",\"name\":\"sharedDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/shared\",\"urls\":[\"http://localhost:3000/shared\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"directoryIndex\":\"index.html\",\"name\":\"defaultDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/\",\"urls\":[\"http://localhost:3000/\"]}], {\"type\":\"object\"})\n\ndata.strains should be object","stack":["Error: Invalid configuration:","Strains .strains should be object: type([{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-cli\"},\"directoryIndex\":\"index.html\",\"name\":\"client\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/client\",\"urls\":[\"https://www.project-helix.io/client\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-pipeline\"},\"directoryIndex\":\"index.html\",\"name\":\"pipeline\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/pipeline\",\"urls\":[\"https://www.project-helix.io/pipeline\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-shared\"},\"directoryIndex\":\"index.html\",\"name\":\"shared\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/shared\",\"urls\":[\"https://www.project-helix.io/shared\"]},{\"code\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"directoryIndex\":\"index.html\",\"name\":\"default\",\"package\":\"acapt/e179ce0ba371833c8fb3aff82f1e031bea20e2f4\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"github.com\",\"hostname\":\"github.com\",\"magic\":false,\"owner\":\"adobe\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"test-sequence-strains\",\"repo\":\"project-helix.io\"},\"sticky\":false,\"url\":\"https://www.project-helix.io/\",\"urls\":[\"https://www.project-helix.io/\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-cli\"},\"directoryIndex\":\"index.html\",\"name\":\"clientDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/client\",\"urls\":[\"http://localhost:3000/client\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-pipeline\"},\"directoryIndex\":\"index.html\",\"name\":\"pipelineDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/pipeline\",\"urls\":[\"http://localhost:3000/pipeline\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"github.com\",\"hostname\":\"github.com\",\"owner\":\"adobe\",\"path\":\"\",\"port\":\"\",\"protocol\":\"https\",\"ref\":\"\",\"repo\":\"helix-shared\"},\"directoryIndex\":\"index.html\",\"name\":\"sharedDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/shared\",\"urls\":[\"http://localhost:3000/shared\"]},{\"code\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"condition\":\"\",\"content\":{\"host\":\"localhost\",\"hostname\":\"localhost\",\"owner\":\"local\",\"path\":\"\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"directoryIndex\":\"index.html\",\"name\":\"defaultDev\",\"package\":\"\",\"perf\":{\"connection\":\"\",\"device\":\"\",\"location\":\"\"},\"static\":{\"allow\":[],\"deny\":[],\"host\":\"localhost\",\"hostname\":\"localhost\",\"magic\":false,\"owner\":\"local\",\"path\":\"/htdocs\",\"port\":\"\",\"protocol\":\"http\",\"ref\":\"\",\"repo\":\"default\"},\"sticky\":false,\"url\":\"http://localhost:3000/\",\"urls\":[\"http://localhost:3000/\"]}], {\"type\":\"object\"})","","data.strains should be object","    at ConfigValidator.assetValid (/nodejsAction/XXAKkmNn/node_modules/@adobe/helix-shared/src/ConfigValidator.js:84:13)","    at HelixConfig.validate (/nodejsAction/XXAKkmNn/node_modules/@adobe/helix-shared/src/HelixConfig.js:120:27)","    at HelixConfig.init (/nodejsAction/XXAKkmNn/node_modules/@adobe/helix-shared/src/HelixConfig.js:125:16)","    at process._tickCallback (internal/process/next_tick.js:68:7)"],"status":"invalid configuration"}
error: Error while running the Publish command: Error: Unable to setup service config
    at request.post.then.catch (/Users/alex/work/dev/helix/project-helix/modules/helix-cli/src/remotepublish.cmd.js:171:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)
Error while running the Publish command

kptdobe · 2019-03-27T12:45:05Z

In both cases, we cannot publish. The error seems to be the same. Some schema error.

kptdobe · 2019-03-27T12:49:15Z

I assume that if we write some tests for the schemas, we'll find the error quickly.

trieloff · 2019-03-27T12:55:27Z

I think the error is that helix-publish doesn't have the latest helix-shared version (which means, it was a breaking change after all).

kptdobe · 2019-03-27T12:58:03Z

For the 1st, it seems to happen before hitting helix-publish
For the 2 error, correct except it is not necessarily a break change, "my" code have a future feature for helix-publish, I think it is normal that it fails until we patch helix-publish

kptdobe · 2019-03-27T12:59:52Z

We do not have tests on the "map" after the yaml rewrite, I am sure we can easily reproduce the error.

trieloff · 2019-03-27T13:06:13Z

Once this workflow completes, you can try the "sequence" publish again: https://circleci.com/workflow-run/2e744869-fb7e-403a-bee0-ae6a9b75ecf2

kptdobe · 2019-03-27T13:07:13Z

You might be right, I cannot reproduce in helix-shared with both config rewritten.

kptdobe · 2019-03-27T13:09:19Z

Sorry, no, I can reproduce with the "map" rewritten config.
I'll create a test case for both and try to fix the schema.

tripodsan · 2019-03-27T13:15:10Z

i would ignore/delete all map tests. they are not relevant anymore and i would deprecate the format anyways.

trieloff · 2019-03-27T13:18:18Z

What are we doing about adobe/helix-cli#695

tripodsan · 2019-03-27T13:25:19Z

i don’t understand. the map type is still read but saved as sequence style. the json is exported sequence style. and simulator and helix-publish should have the new shared.

kptdobe · 2019-03-27T13:53:42Z

If we ignore the map tests, we ignore backward compatibility. It could be fine (all configs must be updated) but then we must also get rid of the code handling it.

kptdobe · 2019-03-27T13:55:59Z

What are we doing about adobe/helix-cli#695

We need to upgrade the tests. This is the proof on non-backward compatibility ;)

trieloff · 2019-03-27T13:58:58Z

I just updated the tests here: adobe/helix-simulator#172

kptdobe · 2019-03-27T14:03:35Z

Ah... if you look here: https://github.com/adobe/project-helix.io/blob/test-map-strains/helix-config.yaml
The saveConfig for the "map" case is wrong: the strains are listed without the "key". We need to add some logic to use the name as the key.

kptdobe · 2019-03-27T14:23:59Z

I think the strains.schema is wrong.

What do "$ref": "#/definitions/anynamedstrain" and "$ref": "#/definitions/anystrain" refer to ? If I remove those, it validate the schema (well, it is like no validation for the strains basically).

kptdobe · 2019-03-27T14:31:35Z

ah ok, found the definition at the beginning of the file ;)

trieloff · 2019-03-27T14:39:50Z

adobe/helix-cli@ae2769d is the fix (CI pending) for adobe/helix-cli#695

kptdobe · 2019-03-27T15:32:55Z

OK... Found it!!! The ref is missing for certain strains while the GitURL schema requires it: https://github.com/adobe/helix-shared/blob/master/src/schemas/giturl.schema.json#L60

The error message is huge because of the embedded anyOf - none of the combination works and it cannot resolve a working schema. But it is nowhere written "ref is missing". I had to isolate each piece.

Now, next question is how to solve that:

relax the schema (remove ref from the mandatory properties) - tested to publish project-helix.io (config with map), it works: site has been published and seems to be functional (even without the ref in some strains)
force to write ref, would need to set a default value to "master".

I vote for 1.

kptdobe · 2019-03-27T15:38:24Z

Well... especially that the ref defaults to master in the GitURL schema! It should then not be mandatory.

kptdobe · 2019-03-27T15:45:10Z

See #80

kptdobe · 2019-03-28T07:44:29Z

I created:

to follow up on this.

tripodsan self-assigned this Mar 22, 2019

tripodsan added the bug Something isn't working label Mar 22, 2019

trieloff mentioned this issue Mar 22, 2019

Conditions Language: Specificity Rules #72

Closed

trieloff mentioned this issue Mar 22, 2019

Schema for named strains and strains as list #73

Closed

tripodsan mentioned this issue Mar 25, 2019

Sequence strains #75

Merged

trieloff closed this as completed in 38b5e9d Mar 27, 2019

trieloff pushed a commit that referenced this issue Mar 27, 2019

chore(release): 0.9.0 [skip ci]

ca94e93

# [0.9.0](v0.8.4...v0.9.0) (2019-03-27) ### Features * **configuration:** Strains can be and should be specified as an ordered list ([38b5e9d](38b5e9d)), closes [#71](#71)

trieloff added the released label Mar 27, 2019

kptdobe reopened this Mar 27, 2019

kptdobe closed this as completed Mar 27, 2019

kptdobe reopened this Mar 27, 2019

trieloff mentioned this issue Mar 27, 2019

Update dependencies adobe/helix-publish#51

Merged

kptdobe mentioned this issue Mar 27, 2019

Relax GitURL schema: allow no ref - default to master #80

Closed

kptdobe mentioned this issue Mar 28, 2019

Remove strains "map" support #81

Closed

kptdobe closed this as completed Mar 28, 2019

toJson() / fromJson() not symmetric in respect to strain order #71

toJson() / fromJson() not symmetric in respect to strain order #71

Comments

tripodsan commented Mar 22, 2019

tripodsan commented Mar 22, 2019 • edited Loading

tripodsan commented Mar 22, 2019

tripodsan commented Mar 22, 2019 • edited Loading

tripodsan commented Mar 22, 2019

trieloff commented Mar 22, 2019

Rules:

Property Matches between different properties are never comparable

For substring-starts matching, the longer substring is more specific than the shorter substring.

Two strict-match matches are always equal in specificity

Two regex-match matches are always equal in specificity

A strict-match is more specific than a substring-starts match, if one contains the other

A strict-match is more specific than the regex-match it matches

Property range matches are comparable

Two not expressions are comparable if their children are comparable

Two and or or expressions are comparable if children of one expression are a subset of children of the other expression

Of two comparable or expressions, the one with more children is less specific

Of two comparable and expressions, the one with more children is more specific

Of two comparable and and or expressions, the and expression is more specific

tripodsan commented Mar 22, 2019

trieloff commented Mar 22, 2019

tripodsan commented Mar 22, 2019 • edited Loading

tripodsan commented Mar 25, 2019

filmaj commented Mar 25, 2019

trieloff commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019 • edited Loading

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

trieloff commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019 • edited Loading

trieloff commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

tripodsan commented Mar 27, 2019

trieloff commented Mar 27, 2019

tripodsan commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

trieloff commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

trieloff commented Mar 27, 2019

kptdobe commented Mar 27, 2019 • edited Loading

kptdobe commented Mar 27, 2019

kptdobe commented Mar 27, 2019

kptdobe commented Mar 28, 2019

tripodsan commented Mar 22, 2019 •

edited

Loading

tripodsan commented Mar 22, 2019 •

edited

Loading

For `substring-starts` matching, the longer substring is more specific than the shorter substring.

Two `strict-match` matches are always equal in specificity

Two `regex-match` matches are always equal in specificity

A `strict-match` is more specific than a `substring-starts` match, if one contains the other

A `strict-match` is more specific than the `regex-match` it matches

Two `not` expressions are comparable if their children are comparable

Two `and` or `or` expressions are comparable if children of one expression are a subset of children of the other expression

Of two comparable `or` expressions, the one with more children is less specific

Of two comparable `and` expressions, the one with more children is more specific

Of two comparable `and` and `or` expressions, the `and` expression is more specific

tripodsan commented Mar 22, 2019 •

edited

Loading

kptdobe commented Mar 27, 2019 •

edited

Loading

kptdobe commented Mar 27, 2019 •

edited

Loading

kptdobe commented Mar 27, 2019 •

edited

Loading