JSONM

JSON with macros (JSONM) #

JSONM is a JSON extension that allows using macro definitions inside JSON files.

Why? #

The main goals of this format are to make configuration files more readable and to get rid of scripts that generate huge configuration files.

What? #

JSONM is a superset of JSON. Any JSON object may be treated as JSON with macros. JSON with macros is also a valid JSON object – the difference lies in enhancing some JSON properties to allow reuse of similar parts of large JSON objects.

How? #

Using a simple preprocessor, we generate standard JSON objects from user-friendly JSONM by processing all macros and substituting all constants.

Syntax & Definitions #

Comments #

JSONM allows C-style comments, which are removed by the preprocessor. Example:

{
  // some comment here
  "key": /* and one more comment here */ "value/**/"
}

After preprocessing:

{
  "key": "value/**/"
}

Escaping #

All fields in JSONM are preprocessed and characters @, %, (, ) and , have special meaning. You need to add two backslashes (\\) before any such character to avoid it being interpreted as a preprocessor instruction. To escape a backslash, write \\\\. Two backslashes are required because JSON uses a single backslash (\) as an escape character.

value #

Any JSON value (string, object, number, etc.)

paramSubstitution #

A string of form %paramName% that will be replaced by the value of that parameter.

Examples: "%paramName%", "a%paramName%b". %paramName% will be replaced with a value of the corresponding parameter.

If the whole string is a parameter substitution (i.e., "%paramName%") parameter values may be any valueWithParams. Otherwise, it should be a string.

valueWithParams #

A value that may contain paramSubstitutions in it.

valueWithMacros #

A value that may contain macroCalls and paramSubstitutions in it.

macro #

A reusable piece of JSON. One can think about it as a function that takes an arbitrary list of values and returns valueWithMacros.

macroCall #

A call of macro that would replace that call with its result. Most macros have both inline and expanded versions, but several macros have only expanded version.

Inline form: "@macroName(param1,param2,…)". Note: during preprocessing of inline form spaces around special characters are stripped; all arguments that are not macro calls are treated as strings. I.e. a call " @if(@bool(true), foo, bar) " is equivalent to "@if(@bool(true),foo,bar)", also note explicit cast of string "true" to a boolean, without that cast the preprocessing will fail due to wrong argument type.

Expanded form: an object with macro parameters passed as properties of that object plus next special properties

• type string (required): macro name to call
• vars valueWithMacros that get preprocessed to an object (optional): allows to define additional context variables that would be available during preprocessing of parameters for the macro

Example: calling a call to a built-in macro if in extended form

{
  "type": "if",
  "vars": {
    "A": 5
  },
  "condition": true,
  "is_true": "@add(%A%, 1)",
  "is_false": "@add(%A%, 2)",
}

here we have three required parameters (condition, is_true, is_false) for the macro if, we also define a local variable A. We can write similar code in an inline form @if(@bool(true),@add(5,1),@add(5,2)), though we don't have a way to define local vars.

macroDefinition #

Defines a reusable macro. Should be an object that is valueWithMacros and has next properties in it:

• type string (required): should be set to "macroDef"
• params list (optional): defines a list of parameters that this macro accepts, each param is either a string name of the parameter or an object with the following properties:
  • name string (required): name of the parameter
  • optional bool (optional): marks the parameter as optional (i.e. it can be omitted when calling the macro)
  • default valueWithMacros (optional): automatically implies optional = true, defines a default value that should be used for the parameter if it's not specified in macroCall
• result valueWithMacros (required): defines the return value of the macro, contents of the result can reference parameters specified in params list.

Note: if the parameter is defined as optional, all following parameters should be defined as optional too.

constDefinition #

An object that is valueWithMacros. Defines a reusable constant that can be used in paramSubstitution in the same way as params, but on global scope. Should contain next properties:

• type string (required): should be set to "constDef"
• result valueWithMacros (required): the value of the constant

Example:

{
  "type": "constDef",
  "result": {
    "foo": "bar"
  }
}

macroDefinitions #

Set of macroDefinitions and constDefinitions. Can be either an object of type valueWithMacros or a recursive list of such objects. During preprocessing the list is flattened and objects are merged to form a final dictionary of definitions with keys being names of the defined macro or constants.

Example:

[
  [
    {
      "isOdd": {
        "type": "macroDef",
        "params": [ "value" ],
        "result": "@equals(@mod(%value%,2),1)"
      }
    }
  ],
  [
    {
      "dayInSeconds": {
        "type": "constDef",
        "result": "@mul(3600,24)"
      }
    }
  ]
]

JSONM #

JSONM is a JSON object with a special optional key macros:

{
  "macros": macroDefinitions,
  "property1": valueWithMacros,
  "property2": valueWithMacros
}

During the preprocessing all macroDefinitions and constDefinitions from "macros" are imported and afterward all properties are expanded with them.

Macro Library #

JSONM comes with a set of predefined functional macros that extend JSONM to a functional language.

Index #

• @import
• @int
• @bool
• @str
• @not
• @and
• @or
• @less
• @equals
• @if
• @isBool
• @isInt
• @isString
• @isArray
• @isObject
• @add
• @sub
• @mul
• @div
• @mod
• @empty
• @size
• @contains
• @keys
• @values
• @select
• @set
• @shuffle
• @merge
• @slice
• @sort
• @split
• @transform (no-inline)
• @foreach (no-inline)
• @process (no-inline)
• @define
• @range
• @defined
• @fail
• @isLocalIp
• @hash
• @weightedHash

@import other JSON file #

One of the key features of JSONM is that it allows you to make your configs more modular or depend on external configuration sources. @import macro reads some external resource and gets replaced with the contents of that resource. JSONM preprocessor requires a caller to pass in an ImportResolver object that knows how to resolve resource name to its contents. The convention for the resource name is "resource_type:name" (e.g. "file:myFile.json"), this is supported via mcrouter ConfigApi that would see the dependency and monitor changes to it.

Parameters:

• path valueWithMacros that evaluates to string (required): resource identifier (file path)
• default valueWithMacros (optional): default value to use on import failure, if not set import failure would result in preprocessing failure

Return: content of the external resource.

Example: assume we have two files as below

citiesByCountry.json

{
  "ukraine": [ "Kyiv", "Lviv" ],
  "usa": [ "Menlo Park" ]
}

allCities.json

{
  "cities": "@merge(@values(@import(file:citiesByCountry.json)))"
}

after preprocessing, allCities.json would result in

{
  "cities": [ "Kyiv", "Lviv", "Menlo Park" ]
}

Type conversions @int, @bool, @str #

JSONM allows you to do type conversion between integer, boolean and string types. These three macro convert input value parameter to the integer, boolean, string respectively. Conversion happens according to the folly::to<T>(arg) logic.

Parameters:

• value valueWithMacros (required): a value to convert.

Examples: @int(123), @bool(1), @bool(true), @bool(false), @str(12345), @str(true).

Boolean unary @not #

Returns a negation of argument. I.e. converts true to false and false to true.

Parameters:

• A valueWithMacros that evaluates to boolean (required)

Return: not A.

Examples: @not(true) would result in false; @not(@equals(1, 2)) would result in true.

Boolean binary @and, @or #

Macros that represents logical conjunction and disjunction.

Parameters:

• A valueWithMacros that evaluates to boolean (required)
• B valueWithMacros that evaluates to boolean (required)

Return: A and B and A or B respectively.

Examples: @and(@not(true), false) would result in false; @or(false, true) would result in true.

Comparison @less, @equals #

These two macro allow you to compare values and return boolean.

Parameters:

• A valueWithMacros (required): first argument
• B valueWithMacros (required): second argument

Return:

• boolean, result of 'A < B' and 'A == B' respectively.

Examples: @less(5, 3) will return false, @equals(true,true) will return true.

Note: all other comparisons could be expressed by applying boolean operators. I.e. greater can be expressed as @not(@or(@less(A,B),@equals(A,B))).

Conditional macro @if #

Allows doing preprocessing based on some condition.

Parameters:

• condition valueWithMacros that evaluates to boolean (required)
• is_true valueWithMacros (required): value to preprocess and return if condition evaluates to true
• is_false valueWithMacros (required): value to preprocess and return if condition evaluates to false

Return: is_true if condition evaluates to true or is_false otherwise.

Examples: @if(@equals(1,1),same,different) would result in same,

{
  "type": "if",
  "condition": "@equals(1,0)",
  "is_true": [ "foo" ],
  "is_false": [ "bar" ]
}

would result in [ "bar" ].

Testing type of a value @isBool, @isInt, @isString, @isArray, @isObject #

These five macro allow you to check if the parameter is of a certain type (boolean, integer, string, array or an object respectively).

Parameters:

• A valueWithMacros (required).

Return: boolean true if the parameter A is of the tested type, false otherwise.

Examples: @isInt(@int(0)) would result in true, @isInt(false) would result in false.

Arithmetic binary macros @add, @sub, @mul, @div, @mod #

Perform arithmetic operation (summation, subtraction, multiplication, division or modulo respectively) on input parameters.

Parameters:

• A valueWithMacros that evaluates to integer (required).
• B valueWithMacros that evaluates to integer (required).

Return: A+B, A-B, A*B, A/B, A%B respectively.

Example: @add(@mod(5, 2), 1) would result in 2.

Macro for working with compound types (strings, arrays, objects) #

Check if the value is @empty #

Parameters:

• dictionary valueWithMacros that evaluates to string, array or object (required).

Return: true if the value is empty (containing 0 elements), false otherwise.

Example: @empty(“”) would evaluate to true.

Obtain @size of the parameter #

Parameters:

• dictionary valueWithMacros that evaluates to string, array or object (required).

Return: length of a string or a number of elements in array or object.

Examples: @size(abc) would return 3.

Check if the input parameter @contains given substring, value or key #

Parameters:

• dictionary valueWithMacros that evaluates to string, array or object (required)
• key valueWithMacros that evaluates to string if dictionary is a string or an object; any type in case of array (required).

Return: true if

• dictionary is a string and it contains key as a substring
• dictionary is an array and it contains item that is equal to the key
• dictionary is an object and it contains property with a name key,

false otherwise.

Obtain @keys of properties in an object #

Obtain keys of an object and return them as an array.

Parameters:

• dictionary valueWithMacros that evaluates to an object (required).

Return: array of keys without any particular ordering (can be different for the equal input).

Example:

{
  “type”: “keys”,
  “dictionary”: {
    “foo”: “bar”
  }
}

would return [ “foo” ].

Obtain @values of properties in an object #

For a given object returns an array containing values of all properties inside of that object.

Parameters: * dictionary valueWithMacros that evaluates to an object (required).

Return: array of values without any particular ordering (can be different for the equal input).

{
  “type”: “values”,
  “dictionary”: {
    “foo”: “bar”,
    “baz”: {
      “abc”: 1
    }
  }
}

would return

[
  “bar”,
  {
    “abc”: 1
  }
]

@select an item from the input #

Performs a lookup of item in an array (based on index) or in an object (based on key).

Parameters:

• dictionary valueWithMacros that evaluates to an object or an array (required)
• key valueWithMacros that evaluates to a string for an object dictionary and integer for an array dictionary (required)
• default valueWithMacros (optional): a value that would be returned if the key is not found in dictionary.

Return: if the key is found in dictionary returns the value, otherwise if default is set returns default, else throws an error.

Examples:

{
  “type”: “select”,
  “dictionary”: [ 1, 2, 3 ],
  “key”: 0
}

would result in 1,

{
  “type”: “select”,
  “dictionary”: [ 1, 2, 3 ],
  “key”: 3
}

would produce an error,

{
  “type”: “select”,
  “dictionary”: {
    “foo”: “bar”
  },
  “key”: 0
}

would result in ”bar”.

Create a copy of parameter with an item/property @set in it #

Parameters:

• dictionary valueWithMacros that evaluates to an object or an array (required)
• key valueWithMacros that evaluates to a string if the dictionary is object; in case an array dictionary, must be integer an be within the size ranges (i.e. 0 <= key < @size(%dictionary%)); (required)
• value valueWithMacros (required)

Return: for dictionary that evaluates to an object, returns a copy of that dictionary with property key set to value; for a dictionary that evaluates to an array returns array with item at index key set to `value.

Example:

{
  “type”: “set”,
  “dictionary”: {
    “bar”: “baz”
  },
  “key”: “foo”,
  “value”: 1
}

would return

{
  “bar”: “bar”,
  “foo”: 1
}

@shuffle items/properties of the parameter #

Parameters:

• dictionary valueWithMacros that evaluates to an object or an array (required)

Return: for an array would produce a new array with items from dictionary but in random order. Does nothing for objects, because their properties are not ordered.

@merge arrays, objects or strings #

Allows combining multiple values of the same type into a single one.

Parameters:

• params valueWithMacros that evaluates to an array of values of the same type (supported types are arrays, strings, and objects) (required)

Return:

• if the values are strings (s1, s2, s3, ...), produces a string that is a result of concatenation of those strings, i.e. s1 + s2 + s3 + ...
• if the values are arrays (a1, a2, a3, ...), produces an array that contains items of a1 followed by items from a2, and so on. No deduplication or extra ordering is performed
• if the values are objects (o1, o2, o3, ...), produces an object that contains a union of properties from o1, o2, o3, ... with properties of o{N} overriding properties of o{N-1}

Examples:

{
  "type": "merge",
  "params": [ "foo", "bar" ]
}

would result in "foobar",

{
  "type": "merge",
  "params": [ [ "a", "b", "c" ], [ "ab", "c" ] ]
}

would result in [ "a", "b", "c", "ab", "c" ],

{
  "type": "merge",
  "parmas": [
    {
      "foo": 1
    },
    {
      "bar": 2
    },
    {
      "foo": 3
    }
  ]
}

would result in

{
  "foo": 3,
  "bar": 2
}

Obtain subrange of string/array/object (@slice) #

Parameters:

• dictionary valueWithMacros that evaluates to a string, array or an object (required)
• from valueWithMacros that evaluates to string if dictionary is object, int otherwise (required): starting index/key to include (inclusive)
• to valueWithMacros that evaluates to string if dictionary is object, int otherwise (required): ending index/key to include (inclusive)

Return:

• if the dictionary is string, returns substring [from, to]
• if the dictionary is an array, returns an array with items from dictionary with indexes from <= index <= to
• if the dictionary is an object, returns an object with properties from dictionary that have from <= key <= to

Examples:

{
  "type": "slice",
  "dictionary": "test",
  "from": 1,
  "to": 2
}

would result in es,

{
  "type": "slice",
  "dictionary": [ 4, 6, 3, 9 ],
  "from": 0,
  "to": 2
}

would result in [ 4, 6, 3 ],

{
  "type": "slice",
  "dictionary": {
    "a": 1,
    "c": 3,
    "b": 2,
    "z": 4
  }
  "from": "b",
  "to": "d"
}

would result in

{
  "c": 3,
  "b": 2
}

@sort an array of strings/integers #

Parameters:

• dictionary valueWithMacros that evaluates to an array of integers or strings (required)

Return: sorted array of elements

Examples:

{
  "type": "sort",
  "dictionary": [ 3, 2, 5 ]
}

would result in [ 2, 3, 5 ],

{
  "type": "sort",
  "dictionary": [ "c", "a", "b" ]
}

would result in [ "a", "b", "c" ]

@split a string by delimeter #

Parameters:

• dictionary valueWithMacros that evaluates to string (required): string that needs to be split
• delim valueWithMacros that evaluates to string (required): delimiter string

Result: an array of pieces of the string after splitting it, empty strings are not omitted.

Example: @split(foo::bar::baz::) would result in [ "foo", "bar", "baz", "" ].

transform keys and or items of an object or an array #

This macro doesn't provide inline macroCall version.

Parameters:

• dictionary valueWithMacros that evaluates to an array or an object (required): input collection for the transformation
• keyName string (optional, default: "key"): name of parameter to use for keys in extended contexts for itemTransform and keyTransform
• itemName string (optional, default: "item"): name of parameter to use for values in extended contexts for itemTransform and keyTransform
• keyTransform macro with extended context %key% and %item% that returns a string or an array of strings (optional): transformation macro for keys, when an array is returned, the value would be set for each key from the returned array, see example for more details
• itemTransform macro with extended context %key% and %item% that returns any value (required when dictionary is an array and optional when it's an object): macro for transforming values

Return:

• if dictionary is an array, returns a new array with transformed items
• if dictionary is an object, returns a new object with the applied transformation.

Examples:

next example would multiply each item in the list by 2

{
  "type": "transform",
  "dictionary": [ 0, 1, 2, 3, 4 ],
  "itemName": "myCustomItemName",
  "itemTransform": "@mul(2,%myCustomItemName%)"
}

the result would be [ 0, 2, 4, 6, 8 ], next example would keep entries only with positive values and would duplicate them if the values are even

{
  "type": "transform",
  "dictionary": {
    "foo": -1,
    "bar": 1,
    "baz": 2,
  },
  "keyName": "myCustomKeyName",
  "itemName": "myCustomItemName",
  "keyTransform": {
    "type": "if",
    "condition": "@less(%myCustomItemName%,@int(0))",
    "is_true": [],
    "is_false": {
      "type": "if",
      "condition": "@equals(@int(0), @mod(%myCustomItemName%, @int(2)))",
      "is_true": [ "%myCustomKeyName%", "%myCustomKeyName%-clone" ],
      "is_false": "%myCustomKeyName%"
    }
  }
}

this would produce

{
  "bar": 1,
  "baz": 2,
  "baz-clone": 2,
}

foreach item in an array/object #

This macro doesn't provide inline macroCall version.

Provides a more generic way to execute complex query of type "foreach (key, item) from <from> where <where> use <use> and select top <top> items".

Parameters:

• key string (optional, default: "key"): name of parameter to use for keys in extended contexts for where and use
• item string (optional, default: "item"): name of parameter to use for values in extended contexts for where and use
• from valueWithMacros that evaluates to an array or object (required): input collection for the foreach
• where macro with extended context %key% and %item% that returns boolean (optional): select only items for which where macro returns true
• use macro with extended context %key% and %item% that always expands to the same type (either an array or an object) (optional): instead of using origional values, use result of executing use macro on key and item
• top int (optional): select first top items, unlimitted if not set
• noMatchResult any value (optional): return this value if the result is empty.

Return: for top top items from array/object from which satisfy where condition, merge results of expanding use macro into a single collection (array/object), if the result is empty return noMatchResult.

Example:

{
  "type": "foreach",
  "key": "keyRename",
  "item": "itemRename",
  "from": {
    "foo": 1,
    "bar": 2,
    "baz": 3,
    "abc": 5
  }
  "where": "@equals(@mod(@int(%itemRename%), @int(2)), 1)",
  "use": [ "%keyRename%" ],
  "top": 2
}

would return a list of at most two keys of properties whose values are odd, i.e. in this case a list of two items from the set of "foo", "baz", "abc" (remember that properties inside of objects don't have any order).

Iterate over input collection and @process each item with an accumulator #

This macro doesn't provide inline macroCall version.

Provides a generic way to iterate over the collection and get macro called for each item with an accumulator. Both @foreach and @transform can be implemented via @process.

Parameters:

• dictionary valueWithMacros that evaluates to an array or an object (required): input collection for the processing
• initialValue valueWithMacros (required): initial value of the accumulator
• keyName string (optional, default: "key"): name of parameter to use for keys in extended contexts for transform
• itemName string (optional, default: "item"): name of parameter to use for values in extended contexts for transform
• valueName string (optional, default: "value"): name of parameter to use for accumulator value in extended contexts for transform
• transform macro (required): macro to be called for each item, return value of this macro would be passed to the next call of transform

Return: the value of accumulator that was returned from the last call to transform.

Example: next macro would iterate over the input and count how many even items are in the list.

{
  "type": "process",
  "dictionary": [ 0, 1, 2, 5, 9, 22, 24 ],
  "initialValue": 0,
  "transform": {
    "type": "if",
    "condition": "@equals(@mod(%item%, 2), 0)",
    "is_true": "@add(%value%, 1)",
    "is_false": "%value%"
  }
}

The above example would return 4.

Miscellaneous utility macros #

@define a local context for a macro #

No-op macro. Allows you to separate vars section from actual macro code.

Parameters:

• result macro (required): result of this define

Return: result of calling result macro.

Examples: assume we have next macro

{
  "type": "if",
  "vars": {
    "A": 0,
    "B": 1,
    "C": 2
  },
  "condition": "@equals(A, 0)",
  "is_true": "%B%",
  "is_false": "%C"
}

it can be rewritten with @define:

{
  "type": "define",
  "vars": {
    "A": 0,
    "B": 1,
    "C": 2
  },
  "result": {
    "type": "if",
    "condition": "@equals(A, 0)",
    "is_true": "%B%",
    "is_false": "%C"
  }
}

Generate an array with a @range if integers #

Utility macro that generates a range of integers (from <= integer <= to).

Parameters:

• from valueWithMacros that evaluates to integer (required): start of the range
• to valueWithMacros that evaluates to integer (required): end of the range

Return: an array with integers from the range (from <= integer <= to).

Example: @range(3, 5) would produce an array [3, 4, 5]; @range(3, 2) would produce [].

Check if the macro/parameter/constant is @defined #

Parameters:

• name string (required): name of macro/parameter/constant to check for existance

Return: true if the macro/parameter/constant exist in current context, false otherwise

Example:

{
  "type": "define",
  "vars": {
    "A": "B"
  }
  "result": {
    "a-exists": "@defined(A)",
    "non-existing-exist": "@defined(non-existing)"
  }
}

would be preprocessed to

{
  "a-exists": true,
  "non-existing-exist": false
}

unless "non-existing" was defined in some outer context.

@fail preprocessing #

Explicitly direct to fail preprocessor with some message. Useful when some invalid input was provided and you need to signal that things are broken.

Parameters:

• msg valueWithMacros that evaluates to a string (required): message to emit as a failure reason

Return: no return, preprocessing is terminated with the given failure reason.

Example: @fail(failure) will terminate with the message "failure".

Check if an IP address @isLocalIp #

Checks if the parameter is an IP address that belongs to the current host.

Parameters:

• ip valueWithMacros that evaluates to a string (required): IP address to check, but can be any string

Return: true if the supplied ip is a valid IP address and compares true to one of local addresses, false otherwise.

Examples: @isLocalIp(::1) would evaluate to true, while @isLocalIp(blah) would evaluate to false.

Compute @hash of an integer or a string #

Parameters:

• value valueWithMacros that evaluates to an integer or a string (required): value to compute hash for

Return: hash of value.

Examples: @hash(abc), @hash(@int(123)).

Compute @weightedHash of an integer or a string #

Compute Rendezvous Hash for given input and a dictionary of choices.

Parameters:

• dictionary valueWithMacros that evaluates to an object with floating point properties (required): set of choices to choose from
• key valueWithMacros that evaluates to an integer or a string (required): value to hash

Return: some key from dictionary object.

Example:

{
  "type": "weightedHash",
  "dictionary": {
    "a": 0.0,
    "b": 1.0
  },
  "key": 5
}

would return "b".