split into multiple multidocs files

[benjamin-bergia]

Hello,

I am trying to split documents coming from stdin into multiple files based on a specific field.
Something like

doSomething| yq -s '(.test // "default")`

and I do get one file for each test field value. Unfortunately, I have multiple documents with the same value and I would like them to endup into the same file.

for example this:

---
name: first
test: a
---
name: second
test: b
---
name: third
test: b

Should result in two files:
a.yml

---
name: first
test: a

b.yml

---
name: second
test: b
---
name: third
test: b

I have spent quite some time on this and I didn't really fine anything that could help. Could somebody give me a hint?

[mikefarah]

I can't think of a way to do it in a single yq expression - but you could do it like this:

cat file.yaml | yq ea '[.] | group_by(.test) | .[] | split_doc' -s '.[0].test // "default"'

This will read all the documents from the stream, group it by '.test' and then write out the documents to separate files. Each file is an array of results though, not separate yaml documents. For instance, b.yml:

---
- name: second
  test: b
- name: third
  test: b

Now you need to iterate over all those files (e.g. bash/find) and run this yq command to update each file from an array to a bunch of documents:

yq -i '.[] | split_doc' b.yml

b.yml now looks like:

name: second
test: b
---
name: third
test: b

For example, you can do this using find:

find *.yml -exec yq '.[] | split_doc' -i {} \;

[benjamin-bergia]

Hi,

thank you for your help.
I had an other try at it this morning and I found a second way to achieve this if anybody else encounter this issue. I use:
doSomething| yq -s '(.test // "default") + "_" + $index
then merge all the files with the same prefix with something like:
for f in $(ls *_*\.yml | awk -F'_' '{print $1}' | uniq); do cat ${f}_*.yml |yq ea >${f}.yml; done