Extend "unfold" operation and support it in the compiler plugin

[koperagen]

It covers two interesting use cases:

1. Replace column with multiple, potentially nested. Provides DSL similar to add for that
2. More fine-grained toDataFrame. Instead of converting 20-30 properties to 2-3 level of nesting all at once user can choose to convert toDataFrame(maxDepth = 0) and unfold required properties to whatever level they need

On the compiler plugin side i will continue to support other overloads later in different PR

[github-actions]

Generated sources will be updated after merging this PR.
Please inspect the changes in here.

[Jolanrensen]

I'm wondering, why did you decide to put unfold after replace?
Unfold itself, by definition, already replaces a column with a new column. I think we're making the API more complicated than it needs to be by making our users write df.replace { a }.unfold {} instead of keeping it inside the unfold operation, like df.unfold { a }.by {}. You can even allow df.unfold(maxDepth = 2) { a }. It would keep "replace with" simple and "unfold" more powerful.

[Jolanrensen]

I was very confused, like how can you unfold a primitive? but it's an isPrimitive() which can be a collection too... Can we rename isPrimitive() to something like isPrimitiveOrListLike()? unfold seems to be the only operation using it

[koperagen]

You can't. Have a look unfold primitive test. skipPrimitive = false is needed to make it work, and skipPrimitive = true is needed to avoid unpacking for example a column of String to ColumnGroup, size: Int, the same as we do for toDataFrame with overloads

[Jolanrensen]

Yes, but can you take a look at the isPrimitive() function? That function also returns true when you run in on a collection and an array. My suggestion was only to rename the isPrimitive() function.

[Jolanrensen]

I think the name unfolding would read better, or byUnfolding/withUnfolded. replace {}.unfold {} doesn't read as a sentence anymore.

[zaleslaw]

If it's possible, let's avoid motion or gravity to the native language, I believe, it's not a goal

[koperagen]

Unfold itself, by definition, already replaces a column with a new column

What definition?
Originally i wanted a add replace by function with AddDsl. Then i remembered that toDataFrame DSL is pretty similar to AddDsl and we already have unfold function, it just lacks an overload with a DSL. Such an overload must be a multiplex operation, right?
unfold by is interesting, but it will have exactly one operation probably forever. Sounds good, right, but its semantics is no different from replace.
I'd rather have one entry point (replace) for situations when you want to, like, replace a column with a different one.

[Jolanrensen]

What definition? Originally i wanted a add replace by function with AddDsl. Then i remembered that toDataFrame DSL is pretty similar to AddDsl and we already have unfold function, it just lacks an overload with a DSL. Such an overload must be a multiplex operation, right? unfold by is interesting, but it will have exactly one operation probably forever. Sounds good, right, but its semantics is no different from replace. I'd rather have one entry point (replace) for situations when you want to, like, replace a column with a different one.

I meant the definition of "unfolding", you're unfolding a column with its contents, so its type is bound to change, aka, the column is replaced.

I can see where you're coming from, but it still may be hard for users to have two different ways to unfold, namely .replace {}.unfold {} and .unfold() and both work a little bit different and have different arguments.

So I'd either:

• make df.replace {}.byUnfolding {} (or a name like that) the only version of unfold and deprecate the old one
• or make just df.unfold {} more powerful
• or keep both for discoverability, but both should be equally powerful

wdyt?

[koperagen]

I'd say df.unfold should stay, because use case for simply unfolding a column with objects stays. Worth to add df.unfold(maxDepth = , roots = ) overload too, missed it.

So I'd either:
or make just df.unfold {} more powerful

Please write this API with needed overloads and a few examples of its usage then

[Jolanrensen]

oh, we should also use KCallable instead of KProperty for java classes support :)

[Jolanrensen]

Not specifying maxDepth now breaks, while it worked before.
Try running df1.replace { col }.with { it.unfold() } before the PR and after it.
It works before, but now it gives: java.lang.UnsupportedOperationException: Can not get nested column 'd' from ValueColumn 'bb'

[Jolanrensen]

I'd say df.unfold should stay, because use case for simply unfolding a column with objects stays. Worth to add df.unfold(maxDepth = , roots = ) overload too, missed it.

So I'd either:
or make just df.unfold {} more powerful

Please write this API with needed overloads and a few examples of its usage then

I made a little sample of unfold {}.by {} with the same options as your version. Both df.unfold {} and df.unfold {}.by {} can be used. See the commit here for more details:
20cab7a

[koperagen]

UnfoldingDataFrame looks good, we can use it. Supporting it on the plugin side will require some changes, but it's ok. I'm only worried with return type being different than DataFrame people will be tempted to call df.unfold { col }.by(). It's first time intermediate object in a multiplex operation is also a DataFrame. Or the opposite problem: df.unfold { col }. will print all DataFrame API, with by being somewhere in completion list no different than let's say filter
But if we go this route, i'd also add
df.replace {}.by(CreateDataFrameDsl) (only this, without vararg props: KProperty<*> and maxDepth: Int = 0 overloads)

[Jolanrensen]

@koperagen Yes, I couldn't find another way to have a notation with 2 selectors and the second one being optional while keeping the DataFrame DSL style :/ but indeed, it's something new. A bit like .recursively().

Actually, since by is defined on UnfoldingDataFrame it should appear quite high on the list:

We could also put it inside the class to make it even more discoverable.

Alternatively, we could change the return-type of unfold to something non-dataframe-ish and let people call
df.unfold { a }.byReplacing() or something.

Imagine that XD df.replace { a }.byUnfolding() and df.unfold { a }.byReplacing() and it would do the same.

replace {}.by {} also looks interesting :)

[zaleslaw]

Is this case "More fine-grained toDataFrame. Instead of converting 20-30 properties to 2-3 level of nesting all at once user can choose to convert toDataFrame(maxDepth = 0) and unfold required properties to whatever level they need" covered here, in this test?

[koperagen]

Technically, yes. I intend to have a more representative example as a part of compiler plugin demo. There's a tree of objects with many properties and potentially deep nesting from konsist library. It will be a good illustration. But here it merely unfolds one specific column up to 2 levels.

[zaleslaw]

Could we use replace and unfold independently? If somehow yes, could you please add test for this, of only together, could be combined to one function?

[zaleslaw]

Honetsly, I like the idea from the use-case "More fine-grained toDataFrame. Instead of converting 20-30 properties to 2-3 level of nesting all at once user can choose to convert toDataFrame(maxDepth = 0) and unfold required properties to whatever level they need", defining a level of our unfolding (that is a special situation of convert)

Also I found that we have a lack of docs/examples for this operation https://kotlin.github.io/dataframe/unfold.html

Some crazy ideas

replace {}.by { ::unfold }
replace {}.by { ::unfoldDeeply }

[Jolanrensen]

Since it's a WIP I made it a draft for now

[Jolanrensen]

Just a thought :) We can actually have something like replace by unfolding without changing the replace API. Replace already works like replace with, meaning we can already write something like df.replace { "data"<ColumnGroup<*>>() }.with { it.unfold() } in the current state of the library. Maybe we could expand on DataColumn.unfold to provide the Add-DSL like notation you suggest :)

Something like:

df.replace { data }.with { 
    it.unfold {
         "b" from { it }
         "c" from { DataRow.readJsonStr("""{"prop": 1}""") }
    }
}