Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add onEach scalar function #1138

Closed
norberttech opened this issue Jul 24, 2024 · 1 comment · Fixed by #1168
Closed

Add onEach scalar function #1138

norberttech opened this issue Jul 24, 2024 · 1 comment · Fixed by #1168

Comments

@norberttech
Copy link
Member

norberttech commented Jul 24, 2024

As explained on Discrod we are missing a function that could apply other function on each element of intput array.

Example:

Input List

[
  ["action_type" => "action_1", "value" => 12.123],
  ["action_type" => "action_2", "value" => 10]
]

Function:

ref('cost_per_action_type')->eachElement(ref("value")->cast(type_float())

Inside eval method of OnEach function we would need to get value of provided ref (list/map/struct/array), then iterate through it and convert each element to row, then execute provided scalar function on it and then convert it back to with Row::toArray().
When provided ref does not points to array like structure, it should return null.

Apache spark also provides similar function, however they called it transform:

Image

https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html

@norberttech
Copy link
Member Author

It would be good to first implement #1139 in order to simplify converting array elements into Row so we can execute scalar functions on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant