How to deal with data that can be either an array or a single item?

[boydkelly]

I am using the xdxf xml dictionary format used by many dictionaries in the linux world. I convert the file to yaml for easier editing. This mostly works for me but for certain things I run into issues where the conversion of xml to yaml will create inconsistent data. A single entry may have one or more definitions of a word. There may be one or more grammatical parts of speech for a definition, ie (nous verb etc). There may be 0, 1, or more usage examples for a definition.
A sample xml file from the source looks like this.

So when all this gets converted to yaml some entries may be a list/array or not at several levels.

A typical entry might be:

  lexicon:
    ar:
      - k:
          +content: ádamaden
          +@xml:lang: dyu
        def:
          def:
            +@xml:lang: fr
            def:
              +@def-id: 4e3ae9f6-9271-47bd-ab08-9fe01da791f7
              gr:
                abbr:
              def:
                +@def-id: ef8ca352-3dfc-4e93-a672-293b0aa3018b
                deftext: être humain
                ex:
                  +@type: exm
                  +@ex-id: 893361d3-788f-439c-9bfe-f5b3fda11a4b
                  ex_orig: adamaden kɔni tɛ foyi ye.
                  ex_tran: L'homme n'est rien.
          sr:
            kref:
              - +content: adamaden
                +@type: spv
              - +content: hadamaden
                +@type: spv

Where any except the first def could be or not be a list; where sr:kref: could maybe be a single entry, or the ex could be rather an array. I'd like all single entries to be array items in order to be able to avoid array errors using yq on the command line, and also this makes for easy cut and paste between records in the yml file. Chatgpt is suggesting I convert the xml file through a pipe int jq and yq but that is not working for me.

xmlstarlet fo --omit-decl test.xml | jq 'def make_array(x): if x == null then [] elif x | type == "array" then x else [x] end; walk(if type == "object" then with_entries(.value = make_array(.value)) else . end)'
parse error: Invalid numeric literal at line 1, column 10

But I am wondering if there is a way that yq can deal with this directly, or how others handle this kind of problem.

[mikefarah]

You can force elements to be an array, see here: https://mikefarah.gitbook.io/yq/usage/xml#parse-xml-force-as-an-array

The idea is to set the fields to be an empty array concatenated with its current contents.

To convert everything into an array - you could do something like this:

yq '([..] | reverse | .. ) |= [] + .' file.xml -oy 

Explanation

• [] + . will convert a scalar into an array, and have no affect on arrays - as explained in the doc link above.
• [..] | reverse | .. is a little funky. In theory you could just do .. to match all elements and pass those through to be updated. But there's an issue in yq if you update a parent first and then a child in the same expression (bug raised already). So we can reverse the list to ensure we update children first, then the parent.

[boydkelly]

Wow that is awesome! I'm going to go celebrate tonight!

…

On Tue, Sep 26, 2023 at 3:14 AM Mike Farah ***@***.***> wrote: You can force elements to be an array, see here: https://mikefarah.gitbook.io/yq/usage/xml#parse-xml-force-as-an-array The idea is to set the fields to be an empty array concatenated with its current contents. To convert everything into an array - you could do something like this: yq '([..] | reverse | .. ) |= [] + .' file.xml -oy Explanation - [] + . will convert a scalar into an array, and have no affect on arrays - as explained in the doc link above. - [..] | reverse | .. is a little funky. In theory you could just do .. to match all elements and pass those through to be updated. But there's an issue in yq if you update a parent first and then a child in the same expression (bug raised already). So we can reverse the list to ensure we update children first, then the parent. — Reply to this email directly, view it on GitHub <#1795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2BH3HINMWHYBFALVN2GCDX4JCAFANCNFSM6AAAAAA5BKVNHI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Boyd Kelly ***@***.***> www: http://www.coastsystems.net Tel: +1 604 837-0765