Extend Maps support.

[blackwinter]

In order to allow lookups to access separately defined maps, I'd like to propose the following changes:

• Let Metafix implement Maps interface to allow adding maps programmatically (including internal Java maps¹).
• Introduce new (custom) Fix functions (names up for discussion):
  • put_map("mapName", "key": "value", ...): Define internal map inside Fix script.
  • put_filemap("fileName"[, "mapName"][, "sep_char": "..."]): Define external file map.
• Update existing lookup function usages:
  • lookup("fieldName", "key": "value", ...): Access local map (unchanged).
  • lookup("fieldName", "fileName"[, "sep_char": "..."]): Access external file map (see above); will be defined on first use (unless already defined via put_filemap) and reused afterwards (ignoring any options).
  • lookup("fieldName", "mapName"): Access internal map (see above); must be defined (via put_map), otherwise interpreted as new file map.

¹ No support for standalone Java maps (put_javamap?) required yet.

[blackwinter]

Functional review: @TobiasNx
Code review: @fsteeg

[TobiasNx]

put_mapseems to work
in one line and with linebreaks:

put_map("map",
"dog":"mammal",
"cat":"mammal",
"parrot":"bird",
"shark":"fish",
"dragon":"ficational animal",
"unicorn":"ficational animal"
	)

put_filemap works too, I was a little bit confused that the path is first an the mapName is second which is "differet" to put_map where the first variable is the first.

Also the reuse of the options in lookup seems to work.

See for tests:
https://gitlab.com/oersi/oersi-etl/-/tree/FunctionalReview/data/experimental/map

+1

[blackwinter]

Thanks for the review!

put_filemap works too, I was a little bit confused that the path is first an the mapName is second which is "differet" to put_map where the first variable is the [mapName].

That's because lookup (in Catmandu) works on the file name, so this is the primary means of addressing external file maps. Naming an external file map is optional, hence the second parameter.

[TobiasNx]

Oh, one side note:

Not all attributes have (at least optional) specifications (what is the specific term idk?) what they are. Only some have as "sep_char":.

I found this a little bit annoying/confusing with catmandu and not user friendly specially with the many attributes/options,
And reviewing your ticket I thought about it again.

Something like this would be nice:
lookup("field":"[fieldName], "in":"[fileName], "sep_char": ";")

Even nice if the options/attributes would not have quotationmarks. This also touches: #74 because if we had consistently
lookup(field:"[fieldName], in:"[fileName], sep_char: ";")

[blackwinter]

Not all attributes have (at least optional) specifications (what is the specific term idk?) what they are.

There are "parameters" and "options", if that's what you mean. Parameters are unnamed, options are named.

The problem with maps in particular is that we (ab)use the options as local map, so we can't have actual options as well.

Even nice if the options/attributes would not have quotationmarks.

You don't need them here, as long as your keys/values only contain word characters¹:

 lookup(field: fieldName, in: fileName, sep_char: ";")

¹ EDIT: And as long as they're not reserved keywords.

[TobiasNx]

The problem with maps in particular is that we (ab)use the options as local map, so we can't have actual options as well.

Even nice if the options/attributes would not have quotationmarks.

You don't need them here, as long as your keys/values only contain word characters¹:

 lookup(field: fieldName, in: fileName, sep_char: ";")

¹ EDIT: And as long as they're not reserved keywords.

But that means that having corresponding parameters/options is not easy/possible?
It makes the fix more readable, it is what i liked about the morph.

[blackwinter]

But that means that having corresponding parameters/options is not easy/possible?

No, I don't think that's possible in general.

[fsteeg]

For the record, if I understand the discussion correctly, the suggestion by @TobiasNx to always use named parameters is basically what I had in mind writing this comment:

metafacture-fix/metafix/src/main/java/org/metafacture/metafix/FixMethod.java

Line 223 in 966e09f

// TODO SPEC: switch to morph-style named params in general?

Right now we have unnamed and named parameters since it's what Catmandu does. In a future attempt to improve the language while working on a common spec (hence TODO SPEC above) it might make sense to consider the approach from Morph, where we have named paramters only.