How to get an ordered list of keys from a json file?

[HLammers]

Let’s say I have a json file with the following structure:

{
"list of objects": {
"key 1": {...},
"key 2": {...},
"key 3": {...},
...
},
...
}

What I need is a list of the keys in ‘list of objects’, in exactly the same order as in the json file.

If I load it this way, I get a randomly ordered list:

import ujson as json
json_file = open('json_file.json')
data = json.load(json_file)
list_of_keys = list(data['list of objects'])

Is there any way to get a list of keys in the original order? I’ve tried using object_pairs_hook=OrderedDict, but the MicroPython implementation of json/ujson doesn’t seem to implement object_pairs_hook.

[andrewleech]

I proposed this by default in #6135

[HLammers]

Thanks @andrewleech , really appreciating your endevours to resolve this!
Since you’ve proposed this about 4 years ago, I’d expect it will take a long time before we see anything like that in the production version of MicroPython. Any idea for a workaround in the meanwhile? Or do I need to rebuild by json file to include an array with ordered keys (assuming arrays are imported as ordered lists - I didn’t test it), plus all the hassle to keep both aligned?

[andrewleech]

I couldn't come up with a workaround myself, hence the PR. I included it in builds for the project that needed json ordering.

Lists are definitely imported in order, so that's possible, but yeah certainly uglier.

[Pharkie]

I think json.loads() is the problem since it jumbles everything up. So only option is to roll your own JSON parser that does it in order?

def json_loads_ordered(raw_json):
    """
    Manually parse JSON and maintain the order of items.
    """

    def parse_json(data):
        data = data.strip()
        if data.startswith("{"):
            items = []
            data = data[1:-1].strip()
            while data:
                key, data = parse_key(data)
                value, data = parse_value(data)
                items.append((key, parse_json(value)))
                data = data.lstrip(",").strip()
            return OrderedDict(items)
        elif data.startswith("["):
            items = []
            data = data[1:-1].strip()
            while data:
                value, data = parse_value(data)
                items.append(parse_json(value))
                data = data.lstrip(",").strip()
            return items
        elif data.startswith('"'):
            return parse_string(data)
        elif data in ("true", "false"):
            return data == "true"
        elif data == "null":
            return None
        else:
            return parse_number(data)

    def parse_key(data):
        if data.startswith('"'):
            end = data.find('"', 1)
            while data[end - 1] == "\\":
                end = data.find('"', end + 1)
            key = data[1:end]
            return key, data[end + 1 :].lstrip(":").strip()
        raise ValueError("Invalid JSON key")

    def parse_value(data):
        if data.startswith('"'):
            end = data.find('"', 1)
            while data[end - 1] == "\\":
                end = data.find('"', end + 1)
            return data[: end + 1], data[end + 1 :].strip()
        elif data.startswith("{"):
            count = 1
            for i in range(1, len(data)):
                if data[i] == "{":
                    count += 1
                elif data[i] == "}":
                    count -= 1
                if count == 0:
                    return data[: i + 1], data[i + 1 :].strip()
        elif data.startswith("["):
            count = 1
            for i in range(1, len(data)):
                if data[i] == "[":
                    count += 1
                elif data[i] == "]":
                    count -= 1
                if count == 0:
                    return data[: i + 1], data[i + 1 :].strip()
        else:
            end = data.find(",")
            if end == -1:
                return data, ""
            return data[:end], data[end + 1 :].strip()

    def parse_string(data):
        end = data.find('"', 1)
        while data[end - 1] == "\\":
            end = data.find('"', end + 1)
        return data[1:end], data[end + 1 :].strip()

    def parse_number(data):
        end = len(data)
        for i, char in enumerate(data):
            if char in " ,]}":
                end = i
                break
        num_str = data[:end]
        if "." in num_str or "e" in num_str or "E" in num_str:
            return float(num_str)
        return int(num_str)

    return parse_json(raw_json)


def load_menu_from_json(file_path):
    try:
        with open(file_path, "r") as file:
            raw_json = file.read()
            return json_loads_ordered(raw_json)
    except OSError as e:
        print(f"Error reading file {file_path}: {e}")
        return OrderedDict()  # Return an empty OrderedDict in case of error

[Josverl]

Would your implementation work with a json array {[]} of things ?
As according to the json spec , arrays are ordered , and object attributes not. So if you roundtrip via another parser, there is no guarantee that the order will be the same.
Keeping order is not problematic, quite a few json parsers do, but any future seems to be a bit of a gamble that nothing else processed the file.

[Pharkie]

I suspect not! Good to be clear about its limitations. Mine is a basic (partial, spec-breaking?) JSON parser that works for what I need. Which is to load up a little Menu from a text file where the order matters. No arrays.

Perhaps YAML would a better format for my menu configured bv a text file. However, there's no decent library to help me parse YAML in Micropython, and my attempt to make my own failed (it couldn't do nested submenus). I'm going to stick with "ordered" JSON for my purposes.