Dynamic JSON objects with static serialization/deserialization?

[ethindp]

Not sure if this has been asked before, but how does one handle things like the Google discovery API, where you have keys for methods/properties/whatnot that are the name of the method/property/etc.? Or, for another use-case: I have an app I'm wondering about maybe porting over to C++ from Python (and using Glaze for JSON since it seems awesome). In Python I used a dictionary because the JSON reply structure is... Really, really weird, an example being:

{
    "path": "/",
    "data": {
        "properties": {
            "Type": {
                "type": "string",
                "value": "root"
            },
            "SampleRate": {
                "type": "int",
                "values": [
...
                ],
                "value": 48000
            },
...
        },
        "commands": {
            "AddDevice": {
...
            },
            "UpdatePlugInStatus": {}
        },
        "children": {
            "devices": {...},
...
        }
    }
}

So essentially, the entire structure is a recursive tree of the following:

• path key (requests are HTTP-ish)
• parameters key (dynamic object, keys are each parameter as the key and the value of said parameter), and key is optional
• data key:
  • properties key:
    • for each property:
      • type key, required, remaining keys depend on this one
      • enabled key
      • max key (only for numeric types)
      • min key (only for numeric types)
      • default key
      • readonly key
      • plugin key
      • volumes key
      • value key (usually present but not always)
      • values key
  • children key:
    • for each child, a dynamic object with the key of the object being the name, and the subobject being the actual object following the same structure as above

So, yeah... Quite complex, and somehow I don't think this would optimize well. My original idea is to just use glz::json_t for now, but if there's a much better approach, I'd love to know. :D I know I'll probably have to do some postprocessing at least. I'm uncertain if I'll actually go through this but I thought I'd ask since I think dynamic structures like this are fairly common.

[stephenberry]

Dynamic types like this can be handled through maps and variants. Here is a somewhat simplified example without all the fields, but should give you an idea of the approach:

struct string_t
{
  bool enabled{};
  bool readonly{};
  std::string value{};
};

struct int_t
{
  bool enabled{};
  int max{};
  int min{};
  bool readonly{};
  std::vector<int> values{};
  int value{};
};

using properties_t = std::variant<string_t, int_t>;

template <>
struct glz::meta<properties_t>
{
   static constexpr std::string_view tag = "type";
   static constexpr auto ids = std::array{"string", "int"};
};

struct children_t
{
  std::unordered_map<std::string, properties_t> devices{};
};

struct data_t
{
  std::unordered_map<std::string, properties_t> properties{};
  children_t children{};
};

struct root_t
{
  std::string path{};
  data_t data{};
};

See the variant-handling documentation for more details about how variants can work.

glz::json_t is also a solution, but will be slower performance and probably not as clean in the long run.

Note that you can rely on pure reflection for a lot of these structs, but you'll have to write a glz::meta for types with the default key name, because this is a reserved word in C++.

[ethindp]

Wow, I didn't know you could do that. That's honestly really, really cool. Is there a way I can flatten the structure or would that require custom deserialization functions?

[stephenberry]

Glaze has a lot of ways to flatten/remap structures. One way is to use lambdas in a glz::meta, see Object Mapping from the README. You can also use glz::custom wrappers (see documentation), but this is more complicated. Pointers and references also allow you to flatten/remap, because they are treated like the value they point to.

[ethindp]

Wow, I'll definitely look into that -- thank you so much! This library seems really really cool!

[stephenberry]

You're welcome! Feel free to ask more question or post issues as you come across them.

[ethindp]

Reopening this discussion to clarify my question about remapping/flattening (I looked at the docs but I'm still a bit uncertain). So I'll try to explain better what I mean.

The JSON structure I'm working with is an extremely recursive data structure. In my (Python) implementation of this, I have a function that takes paths and traverses the deserialized dict object to try to find the requested item. The problem as you said is that this is slow and not pretty or fun to work with, and is error-prone (even if it works). An example of this "recursive" nature is the following, when the tree has been fully loaded:

{
  "path": "/",
  "parameters": {
    "recursive": 1
  },
  "data": {
    "properties": {
      "Type": {
        "type": "string",
        "value": "root"
      },
      "SampleRate": {
        "type": "int",
        "values": [
          {
            "value": 44100,
            "string": "44.1 kHz",
            "enabled": true
          },
          {
            "value": 48000,
            "string": "48 kHz",
            "enabled": true
          },
          {
            "value": 88200,
            "string": "88.2 kHz",
            "enabled": true
          },
...
        ],
        "value": 48000
      },
...
    "children": {
      "devices": {
        "children": {
          "0": {
            "properties": {
              "Type": {
                "type": "string",
                "value": "device"
              },
              "DeviceHwID": {
                "type": "int64",
                "readonly": true,
                "value": 1210621736
              },
...
            "children": {
              "inputs": {
                "children": {
                  "0": {
                    "properties": {
                      "Type": {
                        "type": "string",
                        "value": "input"
                      },
                      "Name": {
                        "type": "string",
                        "default": "ANALOG 1",
                        "value": "ANALOG 1"
                      },
                    "children": {
                      "effects": {
                        "children": {
                          "0": {
                            "properties": {
                              "Type": {
                                "type": "string",
                                "value": "effect"
                              },
                              "Dirty": {
                                "type": "bool",
                                "value": false
                              },

As you can see, this is not exactly a pleasant format at all, and I don't exactly have control over it :-(. So I want to (maybe) try to take this format and condense it (a lot) into something a lot easier to use (if that's even possible). I did look at discussion #1067, which makes me think that trying to fully define this structure would be good if only for the compiler, but ouch that's gonna be complicated.

[stephenberry]

You'll want to figure out what structs are fully definable: you know all the names. Then, if a single key can refer to multiple types, use a std::variant. If you need a dynamic number of your objects with different keys, use std::map. For recursive structures like trees you can often use std::shared_ptr, which Glaze will treat as a nullable type and allocate if JSON content is read in.

I would recommend trying to make a simplified C++ structure of this data so you can wrap your head around the layout, and if you run into issues post them to this thread and I can try to explain possible solutions.