Configuration Design (super issue)

[malex984]

Initial design issues:

• Global config (Q: Hilbert.yaml?) format: mapping not list (Configuration design: Wrong format: list vs mapping #17). Conclusion: general config structure
• Data fields should not involve non-standard shortcuts (Configuration design: Rename data fields #20). (examples: address, description, services and default_application data fields)
• No use of YAML features like Anchors/Aliases (Configuration design: Use of YAML Anchors/aliases #16). Conclusion: use unique string IDs (see Configuration design: Wrong format: list vs mapping #17)
• Do not use null | void (non-domain words) (Configuration design: Do not use null | void #26).
• Danger of silent overrides due to YAML (Danger of silent overrides due to YAML #27)
• Station's settings (for Hilbert) (Configuration design: Station's settings (for Hilbert) #24). Conclusion: Configuration design: Station's settings (for Hilbert) #24 (comment)
• How to specify resources which are external to the general configuration file (e.g. URI / URL for icons & docker-compose.yml) (Configuration design: Resources specified by URI (e.g. icons) #18, Location of docker-compose.yml is questionable #9)
• Service definition (Configuration design: Service definition #19)
• How does Hilbert-CLI-Server know from the config which ServiceRuntime to use? (How does hilbert know from the config which ServiceRuntime to use? #32). Note: same problem with PowerOnMethod.
• Structure/definition of a profile (Configuration design: Structure/definition of a profile #21). Conclusion: Configuration design: Structure/definition of a profile #21 (comment)
• Matching applications with stations (originally: via profiles) (Configuration design: Matching applications with stations (originally: via profiles) #22) Obsolete due to Configuration design: Reflect n:m relation between stations and applications in the configuration #15.
• n:m relation between stations and applications: Group := "set of stations", Application::compatibleStations: Group (Configuration design: Reflect n:m relation between stations and applications in the configuration #15). Conclusion is here
• Definition of a Station's remote execution/management methods (Host) (Configuration design: Definition of a Station's remote execution/management methods (Host) #23). Detailed discussion is in Configuration design: Separate Access Management? #25. Conclusion is Configuration design: Definition of a Station's remote execution/management methods (Host) #23 (comment).
• Self-contained Access DB in ~/.ssh/config + station addresses should also be SSH aliases (Configuration design: Separate Access Management? #25).
• Rename data fields which are currently in camelCase: intersectWith, compatibleStations. See Configuration design: Rename data fields #20 (comment)

[malex984]

My proposal is described in "Management back-end" google document.
Corresponding .YAML, .JSON (and python) may look as in
https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505

[malex984]

It seems that i used anchors/aliases in a wrong way - currently they refer to strings instead of whole corresponding nodes...

Also I am not sure about a good way to join 3 totally different sections: Services/Applications, Profiles and Stations (with cross-references between them) into one easy to understand and change .YAML file?

Any suggestions are welcome!

[malex984]

@porst17 @elondaits I need your feedback about the proposal...

In the meantime i will try to fit the same structure into .INI file.

[elondaits]

My comments:

General

• 1.1: Yes, anchors seem to be used wrong. I wouldn't use them at all... a simple / plain config is enough and it's not really necessary, nor useful if it makes it more verbose instead of less (which is the intent of anchors).
• 1.2 The format is wrong. If you want to list people you do:

- name: john
  phone: 5411-2222
- name: peter
  phone: 3838-1111

or

john:
  phone: 5411-2222
peter:
  phone: 3838-1111

not

- john:
  phone: 5411-2222
- peter:
  phone: 3838-1111

because that is equivalent to

[
  {john: 
    {phone: 5411-2222}
  },
  {peter:
    {phone: 3838-1111}
  }
]

• 1.3 I see in the Google Doc that some objects have an "icon uri" that does not appear in the YAML file. The icons have to be placed in the "public" directory of the server for them to be accessible and they're kind of UI-dependent, so for now we could keep them out of the config and instead use png files within (public/icons/app) whose name is the app id, for instance.

ServicesAndApplications

• 2.1 I don't understand what all the ref: "docker-compose.yml" are, but if they're all the same they're not needed.
• 2.2 label should be name, since it's the name of the app. A label is something you write a name (or something else) on, not a name.

Profiles.yml

• 3.1 Same thing with labels
• 3.2 The opposite of back(ground) is fore(ground), not "front"... but we should use more expressive names. If I understand correctly under "back" you listed "services" (.
• 3.3 Why do profiles need a default app? If profiles are mostly generic (e.g. "Web kiosk", "Interactive touchscreen", etc.) then they're too generic to have a specific default app, unless that app is something like just a logo, or a test screen to adjust color / brightness / sound volume. I would just keep defaults in the stations. It's not too crazy to require that the admin must configure a default app for each station least he get a black screen.
• 3.4 I don't think profiles should "know" about applications. That's not normally how it's done. Normally a "profile" is a dependency of an application, not the other way around. For instance: an application requires Linux, Linux doesn't have a list of applications it can run. For that reason it seems each application should instead have a list of compatible profiles it can be deployed to. If a new app is added or a new version of an app is created you simply list under what profiles it runs, and that's it.
• 3.5 Use strings instead of anchors for apps and services.

Stations.yml

• 4.1 I don't understand the practical purpose of making a distinction between "stations" and "hosts". It would seem simpler to merge both since as specified the relation is 1:1 anyway.
• 4.2 The first item has a local_config and it's the only one that does. Is that on purpose? If so, what is the local_config?
• 4.3 I assume the "type" key in host is used to classify the host (e.g. Unix, Windows, etc.) but that means there's an extra (hidden) layer that modifies the behavior of the system depending on the type of host. That adds complexity. Are we using that for anything? It would be better if the specific differences were specified in the configuration... for instance, if Windows systems use something other than ssh, then we can use a config key that specifies the remote execution mechanism.
• 4.4 Instead of ip we could have something like address that can be loaded with an IP or a hostname. Usually libraries take either type of address and handle them transparently.
• 4.5 Why do we need key_ref? If you configured passwordless ssh then you only need the username to connect. The correct key is picked by ssh.
• 4.6 Also, once again. I really don't understand why you'd ever publicize a private key and if I can't understand it and you can't explain it to me then you'll have a harder time explaining it to a customer or interested party. We shouldn't "invent" our own security mechanisms but instead use things already in place.

[malex984]

Thanks for your comments. Updated revision:
https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505#file-all_flat_update-yml

1.1: Yes, anchors seem to be used wrong.

Fixed - see update.

I wouldn't use them at all... a simple / plain config is enough and
it's not really necessary, nor useful if it makes it more verbose instead
of less (which is the intent of anchors).

If we choose to use YAML - better use it fully: as far as i understand
anchors and aliases provide safe (automatically verifiable)
cross-references to very often used data.

Say, if we have some (non-trivial) Profile for ~100 Stations - i would not
want to embed its definition directly into all those stations definitions.

1.2 The format is wrong.

Fixed - see update.

2.1 I don't understand what all the ref: "docker-compose.yml" are,
but if they're all the same they're not needed.

All Services (and Applications) have to be defined in some docker-compose
file. Currently:
https://github.com/hilbert/hilbert-cli/blob/devel/config/templates/docker-compose.yml

But this may change.

• 2.2 label should be name, since it's the name of the app. A label
  is something you write a name (or something else) on, not a name.

All label's (and icon's) are meant to be shown by Dashboard only. Field names can be easily changed.

3.1 Same thing with labels

see to 2.2

3.2 The opposite of back(ground) is fore(ground), not "front"... but
we should use more expressive names. If I understand correctly under "back"
you listed "services" (.

Now it is back and apps. Can be easily changed.

• 3.3 Why do profiles need a default app? If profiles are mostly
  generic (e.g. "Web kiosk", "Interactive touchscreen", etc.) then they're
  too generic to have a specific default app, unless that app is something
  like just a logo, or a test screen to adjust color / brightness / sound
  volume. I would just keep defaults in the stations. It's not too crazy to
  require that the admin must configure a default app for each station least
  he get a black screen.

It seems that we lack a common definition of a Profile :-(
Therefore i stick to what i gathered from our meetings with the clients:
my understanding of Profiles is more like "Interactive stations in the same
specified location" (e.g. "InteractivesOnFloor1", "InteractivesInTestRoom",
"InfoScreensAtEntranceHall")

IMO having default application in a profile makes it easier to change the
default GUI application for the whole group of stations (provided they do
not override that default),
e.g. update GUIApp_v1 -> GUIApp_v2 for all InfoScreens in a testing room.

Also it simplifies initial creation of new station definitions.

3.4 I don't think profiles should "know" about applications. That's
not normally how it's done. Normally a "profile" is a dependency of an
application, not the other way around. For instance: an application
requires Linux, Linux doesn't have a list of applications it can run. For
that reason it seems each application should instead have a list of
compatible profiles it can be deployed to. If a new app is added or a new
version of an app is created you simply list under what profiles it runs,
and that's it.

Sorry but I do not believe that there is canonical ("normal") approach which is ALWAYS best with no regard on

• how often one manually add/change Services/Applications/Profiles/Stations and
• how complicated these operations are to perform manually!

I think you consider Profiles as Tags to be attached to Applications, correct?
I would propose to make your "normal" approach also possible (alongside with the current direct approach), e.g. by adding an optional Application filed: set of Profiles.

3.5 Use strings instead of anchors for apps and services.

see 1.1

4.1 I don't understand the practical purpose of making a distinction
between "stations" and "hosts". It would seem simpler to merge both since
as specified the relation is 1:1 anyway.

Stations are top-level logical entities mostly for Dashboard (but they will of
course be used to generate low-level station configurations and for OMD configuration).

Hosts specify the low-level way to communicate and manage with
corresponding host for hilbert-cli-server.
Unfortunately we do not know all possible types of hosts and the means to
communicate and manage them yet. Some known types are visible in the
updated proposal revision.

They both are combined together since currently there is no reuse for
Host objects.

But it is possible to share them between Presets that can be easily
implemented as individual mappings similarly to the current Stations:.
See:
https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505#file-preset_testgui-yml

4.2 The first item has a local_config and it's the only one that
does. Is that on purpose? If so, what is the local_config?

Has been renamed to config_path. It is a location on the remote station
specifying where to deploy generated station configurations for further use
via hilbert-cli-station (which is assumed to be available there).

4.3 I assume the "type" key in host is used to classify the host
(e.g. Unix, Windows, etc.) but that means there's an extra (hidden) layer
that modifies the behavior of the system depending on the type of host.
That adds complexity. Are we using that for anything? It would be better if
the specific differences were specified in the configuration... for
instance, if Windows systems use something other than ssh, then we can use
a config key that specifies the remote execution mechanism.

Corrected. See also 4.1. Is it any better now?

• 4.4 Instead of ip we could have something like address that can be
  loaded with an IP or a hostname. Usually libraries take either type of
  address and handle them transparently.

For some types of hosts already id is a DNS host name for others there
is no DNS name at all. See also 4.1.

4.5 Why do we need key_ref? If you configured passwordless ssh then
you only need the username to connect. The correct key is picked by ssh.

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh
to remote hosts that cannot be accessed via default ssh configuration on
the server.

• 4.6 Also, once again. I really don't understand why you'd ever
  publicize a private key and if I can't understand it and you can't explain
  it to me then you'll have a harder time explaining it to a customer or
  interested party. We shouldn't "invent" our own security mechanisms but
  instead use things already in place.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us
  in advance?

Did you even read the comments regarding ssh keys in the document
"Installation exhibition computers ESO/HITS/Imaginary"?

[malex984]

Proposal for 3.4:
https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505#file-extendingprofiles-yml

[elondaits]

1.1: Yes, anchors seem to be used wrong.

Fixed - see update.

They're a bit of a nightmare now. Resolving references should be done in our code, not in the YAML parser. In your example you described the apps and created an anchor for each and then on the description of the station you used those anchors.

• This means that when you're parsing the description of the stations you'll receive the full config of each app in each station instead of just the ID. This is at best useless repetition...
• ... and at worst dangerous, because YAML doesn't know IDs are IDs (non-repeatable) so it'd be possible to create a config where an app is described twice with the same ID but different values, once through an anchor and the other explicitly, with no way of figuring out they're different except comparing them all.

Anchors can be used by the user to avoid writing the same value many times in the cfg, like a convenient macro... but we shouldn't use it as a structural / organizational element of the config. Also, there's no equivalent in JSON or INIs.

Say, if we have some (non-trivial) Profile for ~100 Stations - i would not
want to embed its definition directly into all those stations definitions.

But this is exactly what you're doing with the anchors. We can reference the profile with a name/id and describe it elsewhere and this is resolved very easily in our program.

2.1 I don't understand what all the ref: "docker-compose.yml" are,
but if they're all the same they're not needed.

All Services (and Applications) have to be defined in some docker-compose
file.

Yes. But if the value will always be the same then we don't need to indicate it in the cfg or it should go elsewhere where it doesn't need to be repeated. If it will be different each time then change it to make that clearer. Also consider using a default filename so it only needs to be configured if we want to override the default. If the cfg is shorter, it's better.

All label's (and icon's) are meant to be shown by Dashboard only. Field names can be easily changed.

I'm suggesting it now so we can easily change it. After we coded it it's more difficult to change and once we released it's VERY hard.

• 3.3 Why do profiles need a default app? If profiles are mostly
  generic (e.g. "Web kiosk", "Interactive touchscreen", etc.) then they're
  too generic to have a specific default app, unless that app is something
  like just a logo, or a test screen to adjust color / brightness / sound
  volume. I would just keep defaults in the stations. It's not too crazy to
  require that the admin must configure a default app for each station least
  he get a black screen.

It seems that we lack a common definition of a Profile :-(
Therefore i stick to what i gathered from our meetings with the clients:
my understanding of Profiles is more like "Interactive stations in the same
specified location" (e.g. "InteractivesOnFloor1", "InteractivesInTestRoom",
"InfoScreensAtEntranceHall")

They physical location and the software running on it are completely different concerns. We can use tags for physical locations, but tags are very lightweight identifiers that can be configured through a single field or even using the frontend.

... So far I understood the profile was the type of station (touch screen interactive, projector, dome projector, non interactive screen, kinect, etc.)

IMO having default application in a profile makes it easier to change the
default GUI application for the whole group of stations (provided they do
not override that default),
e.g. update GUIApp_v1 -> GUIApp_v2 for all InfoScreens in a testing room.
Also it simplifies initial creation of new station definitions.

If Touch screen interactive stations run 5 different apps then having a default one in the cfg doesn't make it any easier to change the assignment because you only affect 1/5th of those... and you'd be configuring 4/5ths explicitly and 1/5th implicitly... which is messy. We should use the front end, and presets, to make it easier / faster to assign apps to stations.

3.4 I don't think profiles should "know" about applications. That's
not normally how it's done. Normally a "profile" is a dependency of an
application, not the other way around. For instance: an application
requires Linux, Linux doesn't have a list of applications it can run. For
that reason it seems each application should instead have a list of
compatible profiles it can be deployed to. If a new app is added or a new
version of an app is created you simply list under what profiles it runs,
and that's it.

Sorry but I do not believe that there is canonical ("normal") approach which is ALWAYS best with no regard on

• how often one manually add/change Services/Applications/Profiles/Stations and
• how complicated these operations are to perform manually!

Consider:

• Applications always "know" what they need to run (dependencies, hardware requirements)
• Systems (soft + hardware) don't "know" what applications might be run on them.

But also:

• Number of Apps > Number of Profiles (and thus apps will be added more often than profiles). Profiles should be redefined if they need to be redefined... not every time an app is added. If you add an app you CAN'T avoid adding an entry to it, but you CAN avoid modifying existing profiles.

I think you consider Profiles as Tags to be attached to Applications, correct?
I would propose to make your "normal" approach also possible (alongside with the current direct approach), e.g. by adding an optional Application filed: set of Profiles.

Not "tags", but "compatible profiles", just like the "compatible hardware / OS" requirements of any application. Making both possible does not make sense, because this configuration is used to restrict what apps can be launched on which profiles... if profiles indicate apps and apps indicate profiles you must maintain both lists manually instead of specifying one and computing the other through the program.

4.1 I don't understand the practical purpose of making a distinction
between "stations" and "hosts". It would seem simpler to merge both since
as specified the relation is 1:1 anyway.

Stations are top-level logical entities mostly for Dashboard (but they will of
course be used to generate low-level station configurations and for OMD configuration).
Hosts specify the low-level way to communicate and manage with
corresponding host for hilbert-cli-server.

I understand the conceptual difference, not the practical purpose. If the relation is necessarily 1:1 (a station doesn't have more than one host, or vice-versa) then separating it makes configuration more complex for no purpose other than expressing a conceptual difference the user might not care about.

Unfortunately we do not know all possible types of hosts and the means to
communicate and manage them yet. Some known types are visible in the
updated proposal revision.

You wrote

 - &GEN GEN: "Generic Host (no remote access). Supporting WOL"
 - &WIN WIN: "Windows Host (with SSH access). Supporting WOL"
 - &UX  UX:  "Unix/Linux (with SSH access). Supporting WOL"
 - &DM  DM:  "Virtualized Station via docker-machine. No WOL!"

then specifying "host type" is just adding an extra layer of opacity, when you could just have an explicit key "wol: yes | no", protocol: ssh | rexec. It only makes sense to "add an extra table" (host type) when there's a need to enforce correlation between several properties... but perhaps there might be a windows system with no WOL support, a non-win / non-unix with ssh, etc. and you'll end up adding ad-hoc types only for specifying every combination.

They both are combined together since currently there is no reuse for
Host objects.

But it is possible to share them between Presets that can be easily
implemented as individual mappings similarly to the current Stations:.
See:
https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505#file-preset_testgui-yml

If this is a way for a logical station to change to a different physical host I don't understand why we'd want this functionality or extra complexity.

• 4.4 Instead of ip we could have something like address that can be
  loaded with an IP or a hostname. Usually libraries take either type of
  address and handle them transparently.

For some types of hosts already id is a DNS host name for others there
is no DNS name at all. See also 4.1.

It doesn't matter if the ID is initially the hostname (although you might want shorter IDs without the full domain)... if there is a single address field then you can use that, be it a hostname, an ip address or something else to connect to the server via ssh. With an ip field instead you can't specify a hostname, just an IP. It's OK if the ID is set to a hostname, but if the IT department has to change all hostnames (I've seen this happen many times in companies, for a number of reasons) it's not necessary to change the IDs, presets, etc.

My point is simply that a single address field could hold either hostname or IP, and that I can't think of a reason where we need the IP and can't resolve it through the DNS if we don't have it.

4.5 Why do we need key_ref? If you configured passwordless ssh then
you only need the username to connect. The correct key is picked by ssh.

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh
to remote hosts that cannot be accessed via default ssh configuration on
the server.

Explain the mechanism. The server has a private key and a public key, and there are no reasons why it wouldn't have them. The stations need the public key in its authorized hosts list, and this is transferred just once in a secure manner because it needs to be done using the station's password... so it makes no sense to have it in the cfg.

Is there some other mechanism? Because anyway nothing but a public key should leave a system... the private key shouldn't be transmitted or known by external systems.

• 4.6 Also, once again. I really don't understand why you'd ever
  publicize a private key and if I can't understand it and you can't explain
  it to me then you'll have a harder time explaining it to a customer or
  interested party. We shouldn't "invent" our own security mechanisms but
  instead use things already in place.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us
  in advance?

The same way they'll install an OS, Docker, setup networking, a user, backups, OMD, etc. on 150 stations... This is well beyond the scope of our tool! There are countless tools for parallel administration / deployment. Setting up passwordless ssh access is a security-sensitive operation that is done just once, so it has no reason to be in the config.

... But, how to set up passwordless ssh to 150 stations?

With a list of 150 hostnames + their passwords + a script that transfers the server's public key to the authorized_keys file in the home of the user that will run docker using ssh with the password. Then you delete the list of hostnames + passwords.

Though, once again, we don't need to take care of this with hilbert.

Did you even read the comments regarding ssh keys in the document
"Installation exhibition computers ESO/HITS/Imaginary"?

I don't have access to that document I think (just searched)... but if the idea completely vulnerates the security of stations and ssh credentials then it doesn't matter.

If I'm failing to understand something and this is indeed reasonable and perfectly secure then consider a prospective customer might have the same concerns I have and go away, so we need to be clear and avoid misunderstandings.

[malex984]

@elondaits sorry but I disagree with you on anchors: i noticed no total embedding according to following sample for Python2 with PyYAML & ruamel (see also http://pyyaml.org/wiki/PyYAMLDocumentation#Aliases):

import yaml
class KV:
    def __init__(self, v):
        self.k = v

a = KV(1)
b = KV(2)
P = { 1: [a, b], 2: [a], 3: [b] }
print yaml.dump(P)
# 1:
# - &id001 !!python/object:__main__.KV {k: 1}
# - &id002 !!python/object:__main__.KV {k: 2}
# 2:
# - *id001
# 3:
# - *id002
d =  """1:
- &id001 !!python/object:__main__.KV {k: 111}
- &id002 !!python/object:__main__.KV {k: 222}
2:
- *id001
3:
- *id002
"""
y = yaml.load(d)
print y
# {1: [<__main__.KV instance at 0x7f60c6925170>, <__main__.KV instance at 0x7f60c6925200>], 2: [<__main__.KV instance at 0x7f60c6925170>], 3: [<__main__.KV instance at 0x7f60c6925200>]}
(y[1][0] == y[2][0]) and (y[1][1] == y[3][0]) and (y[2][0] != y[3][0]) and (yaml.dump(y) == d)
# True

Please also consider the following perfectly valid YAML: &A [ *A ]!

Therefore, for me aliases are exactly cross-references that can serve as unique IDs, provided one would not introduce multiple identical anchors by mistake, which can be detected during loading/parsing... but the situation is identical with using strings as IDs for (loose) references - which looks like re-inventing cross-references where they are already available...

@elondaits could it be that you think about JSON instead of YAML?

[elondaits]

The fact that we have to send each other code samples to show how the parser interprets the file should be enough to make it clear that this feature isn't obvious enough to be used in cfg. The cfg should be obvious to the user.

However...

import yaml

cfg_file = """
people:
  alice: &alice {name: 'Alice Anderson', age: 29, email: 'alice@example.com'}
  john: &john {name: 'John Jameson', age: 30, email: 'john@example.com'}
  peter: &peter {name: 'Peter Parker', age: 25, email: 'peter@example.com'}
teams:
  alpha: [*john, *peter]
  beta: [*peter, {name: 'Green Goblin', age: 60, email: 'hacker@example.com'}]
  gamma: [*john, {name: 'Peter Parker', age: 85, email: 'peter@example.com'}]
"""

cfg = yaml.load(cfg_file)

print cfg.get('teams')['alpha']
# output:
# [{'age': 30, 'name': 'John Jameson', 'email': 'john@example.com'},
# {'age': 25, 'name': 'Peter Parker', 'email': 'peter@example.com'}]

print cfg.get('teams')['beta']
# output:
# [{'age': 25, 'name': 'Peter Parker', 'email': 'peter@example.com'},
# {'age': 60, 'name': 'Green Goblin', 'email': 'hacker@example.com'}]

print cfg.get('teams')['gamma']
# [{'age': 30, 'name': 'John Jameson', 'email': 'john@example.com'}, 
# {'age': 85, 'name': 'Peter Parker', 'email': 'peter@example.com'}]

Notice the Green Goblin was added to team beta and when we do cfg.get('teams')['beta'] it looks JUST like Peter Parker, which is a valid user that exists in the "people" table. So if I validate the teams table it looks like everything is OK, but the user did something illegal.

So how can I validate? Suppose I check that every user in a team is also on the people table... I could do that... but consider team gamma where I added a Peter Parker with the wrong age... so to pick that error in the cfg I would have to compare EVERY FIELD of each person to make sure teams and people are consistent.

Something else: Notice that on the people table I indexed each member with an ID that's non repeatable, because people is a map. If I repeat an ID the parser will overwrite the former occurrence with the latter... just like would happen on an INI file if I set a value twice.

... but now the ID is not accessible on the copies of the people that are on the teams. If I do cfg.get('teams')['gamma'][0] I can't get the ID... unless I repeat the ID within the map, which is error prone, redundant, etc.

[malex984]

I suppose you want to ensure that there are always unique canonical places in a config file for defining ALL possible items of the same type (e.g. Tables for "people" / "teams" etc.), where you additionally introduce (almost) unique IDs (as a primary key for those tables) for referencing from other parts - de-referencing such "ID"s you would have to do manually (i.e. know from context): whether some string is a just some a string or a reference to one of existing table. I hope i am not terribly wrong here?

Also this approach is very generic and JSON-like. I suppose one just cannot do anything else with JSON...

As a developer i am all for that approach!

But after learning YAML i tried to make most use of YAML features to enable config-writers to easily create concise definitions with as small additional overhead as possible. In particular: to eliminate that cross-reference matching (IDs are often only introduced intermediately for correct serialization/deserialization) and necessarily total globalization of all kinds of shared information.

BTW, following my proposal your sample YAML would look as follows:

GenerallyKnownPeople:
 - &alice !!python/object:Person { name: 'Alice Anderson', age: 29, email: 'alice@example.com'}
 - &john !!python/object:Person { name: 'John Jameson', age: 30, email: 'john@example.com'}
 - &peter !!python/object:Person { name: 'Peter Parker', age: 25, email: 'peter@example.com'}
 - &judge !!python/object:Person { name: 'Some Judge', age: 61, email: 'judge@example.com'}
.....
 - &alice !!python/object:Person { name: 'Alice Jameson', age: 29, email: 'alice2@example.com'}

teams:
 - &alpha [*john, *peter]
 - &beta [*peter, &newbie1 !!python/object:Person { name: 'Green Goblin', age: 60, email: 'hacker@example.com'}]
 - &gamma [*john, &newbie2 !!python/object:Person { name: 'Peter Parker', age: 85, email: 'peter@example.com'}]
 - &delta [*newbie1, *newbie2 ]

As far as i understand parser/loader would have to fail on redefining tags (therefore unique IDs!), semantic checker will complain about old team players (if there is such a rule) and warn about direct object definitions within team definition (instead of using a global alias).
I agree that this is not an ideal solution but this is just a possible additional approach, which can be tedious to check but also viable if user sees benefits in using it.

Something else: Notice that on the people table I indexed each member with an ID that's non repeatable, because people is a map. If I repeat an ID the parser will overwrite the former occurrence with the latter...

Wow, sounds really dangerous if that silently goes unmentioned by parser/loader due to language specification... e.g. "alice" immediately visible to a config writer is 'Alice Anderson' instead of later overwriting 'Alice Jameson' in a potentially very long listing of people.

[malex984]

DONE:

• separate initial comments by Eric into smaller issues
• migrate follow-up discussions

[elondaits]

Some ideas on the association of Stations and Applications:

• Our need is for the configuration to indicate which apps can be deployed to which stations. Which software runs on top of which hardware.
• There are "hard" constraints ("this app requires a Windows host", "this app requires a Kinect") and "soft" constraints ("this app has only been tested on these machines"). Soft constraints might change with time (if the app is later tested on another set of computers), which is why I'm not too crazy about putting them in the configuration. Consider:
  • The configuration can break the whole museum.
  • It deals with security and networking, which is the domain of the sys admins (and/or the security department, which is separate for larger orgs).
  • Testing apps is something developers do, not sys admins. Sys admins are involved in deploying apps, but not much more.
• Also, the "hard" constraints can be defined by the developer of the app and have to be retained / remembered across configurations because they don't change. The "soft" constraints are more contextual.

So:

• Hard constraints are important because they're related to basic usability (the app starts without crashing). I'm not sure if we have to take care of soft constraints... or perhaps not at this stage. In the computer world a hard constraint is Windows not launching an ELF binary and a soft constraint is something mentioned in a README.
• So, for conceptual clarity and ease of communication I would use a variation on what I mentioned in the meeting:
  • A profile has a list of capabilities, which are string identifiers (similar to tags). E.g. "windows, windows10, kinect, multitouch"
  • A station can specify its own capabilities to override the ones in the profile. We can do this in one of two ways:
    • Override the list
    • Add / remove individual terms from the base list in the profile
• An app has requirements which is a list of the capabilities it needs in a station (e.g. "windows10, kinect")

This has the following advantages:

• Easy to specify
• Quite expressive
• Can be extended to "full" expressiveness by eventually adding syntax for AND, OR and NOT although I wouldn't start with that.
• The concept behind it is understandable by admins, devs and users. Can be used for filtering.
• Almost the same as the "groups" proposal except that here "groups" are tied conceptually to static/"physical" characteristics of the stations and not logical groupings... BUT it can be used with no changes for logical groupings simply by adding ad-hoc identifiers such as "testing", "type-a", "donation-2016", etc. if the user wishes to do so.

And these problems:

• The addition of an extra capability (e.g. dual screen) could make a station not compatible with an app. This can either be solved in some ways:
  • Adding an extra capability that further identifies compatible apps (e.g. "windows10, kinect, singlescreen")
  • An App specifies capabilities a station SHOULD NOT have (e.g. "windows10, kinect, !dualscreen"
  • Consider that type of incompatibility a "soft constraint" (it will probably work but not look right, not be functional... like running a Windows app in a screen with too little resolution or without a mouse) and hence "not the problem we're attacking".
• Not useful for contextual / temporary / soft constraints (e.g. "not tested"). But I don't think we should concern with those... plus the system should allow running the apps for testing them in those platforms in the first place, while it never makes sense to deploy a UNIX app on a Windows system.

[malex984]

All current design details are in the Management Back-end document (v0.7).
Initial implementation is already available due to PR (#34).
Remaining issues will be managed within (#14)

[malex984]

Note: a few things for this super-issue are part of #44