Compiler plugin #729

koperagen · 2024-06-10T15:43:40Z

There are changes in core module itself to be aware of:

DataRowApi.kt is related to Generate constructor for schema interface #113. Compiler plugin automatically adds DataRowSchema as a supertype to data schemas, enabling those APIs. To be continued later
Changes in insert.kt. An attempt to make it work over a generic tree, in order to share implementation between compiler plugin and core library
Across library added annotations that provide required meta information for compiler plugin
aggregate, toDataFrame functions user to be member functions in Grouped, GroupBy interfaces. Now they are extension functions, in order for plugin to work + it's not clear why they should be member function after all

Compiler plugin module consists of three main parts:

Plugin that generates extension properties, that will replace KSP plugin as it generates declarations in IDE without running a Gradle task. Related to Enhance support for @DataSchema in gradle projects #120
Plugin that adds DataRowSchema supertype to a data schema annotated declarations.
Most importantly, plugin that makes it possible to elevate compile time information from function arguments to type level. Plugin updates types of DataFrame objects whenever supported (=annotated with Refine) function is called with constant arguments. Most of the code here is its implementation

Everything related to compiler plugins aimed at K2 compiler
#704

this type of tests should replace most of the previous API tests that required copy-paste from codegen

… calling it too early

… path

…only for private api Initializing non-private property with transformed DataFrame call leads to "exposes local type" error

# Conflicts: # core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/and.kt # core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/toDataFrame.kt

…llback to an empty list

Jolanrensen · 2024-06-13T10:36:40Z

core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/columns/BaseColumn.kt

@@ -18,7 +19,7 @@ import kotlin.reflect.KType
 *
 * @param T type of values contained in column.
 */
-public interface BaseColumn<out T> : ColumnReference<T> {
+public interface BaseColumn<out T> : ColumnReference<T>, GenericColumn {


@koperagen GenericColumn?
It's fine to have the interface as its supertype to just get the name of a column, but it lives in ./impl/api/insert.kt which breaks our structure. Probably best to move GenericColumn(Group) to ./columns

Ah I see it's mentioned in the comment! thanks

Jolanrensen · 2024-06-13T11:17:34Z

plugins/kotlin-dataframe/src/org/jetbrains/kotlinx/dataframe/plugin/impl/SimpleCol.kt

+
+data class SimpleFrameColumn(
+    override val name: String,
+    private val columns: List<SimpleCol>


If SimpleColumnGroup is a List<SimpleColumn>, then SimpleFrameColumn should be a List<List<SimpleColumn>> or a List<ColumnGroup> right?

Ah no I see, it treats it like that because it tries to generalize the schema's, interesting

Jolanrensen · 2024-06-13T11:28:30Z

...in-dataframe/src/org/jetbrains/kotlinx/dataframe/plugin/impl/data/toPluginDataFrameSchema.kt

+import kotlin.reflect.KTypeProjection
+import kotlin.reflect.KVariance
+
+val KotlinTypeFacade.toPluginDataFrameSchema: DataFrameSchema.() -> PluginDataFrameSchema


Nice! very useful :)

Jolanrensen · 2024-06-13T11:29:38Z

...in-dataframe/src/org/jetbrains/kotlinx/dataframe/plugin/impl/data/toPluginDataFrameSchema.kt

+                    name,
+                    columnSchema.schema.toPluginDataFrameSchema().columns()
+                )
+                else -> TODO()


probably throw a NotImplementedError manually to not clog up the TODO scope

koperagen added 30 commits November 29, 2022 15:26

implement IR lowering for DataSchema generated APIs

8763050

support materialization of schemas with FrameColumns

8515366

Reuse dataframe class ids

db5d59f

WIP

ca0eab4

Support materializing FrameColumn

90210b3

remove duplicated code

17a37a5

reorganize test folder

6dc065f

generate tests for add.kt

616be38

setup df api tests

8d72e51

Support vararg kproperties

c75c208

don't use name "this" because it's a keyword and cannot be used as arg

68c6f87

support member functions

0877e0c

basic support for type arguments

ecc6b5d

choose root marker based in file name to avoid ambiguity in convert.kt

8efa8b0

rearrange convert tests

2ac79b5

test rowValueExpression

0474323

don't throw when optional argument is present in actual

6dd4976

add tests

e396d92

Support columnsSelector atom

59be5fb

support nested schema atom and nested schema select

5c22fb9

Prepare df api tests for new non-serializable TypeApproximation

b582dbe

Use KotlinType instead of fqn approximation

068bfbf

Better column extraction

eeda19d

Support and in columns selector

54f44b1

Support to api

21fbdf6

Setup new df api tests

a680d04

this type of tests should replace most of the previous API tests that required copy-paste from codegen

Tests for help with interpretation errors prototyping

0a772eb

Error reporting

67de729

refactoring

478d2ed

working backend codegen for hardcoded property

997db1b

koperagen added 21 commits May 31, 2024 20:41

[Compiler plugin] Handle star projection argument in columns' types

37da18f

[Compiler plugin] resolvedInitializer has LazyResolve contract. Avoid…

bf81949

… calling it too early

[Compiler plugin] Consider 'this' and 'it' when constructing a column…

0e7b0bb

… path

[Compiler plugin] Transform DataFrame calls in property initializers …

1865e35

…only for private api Initializing non-private property with transformed DataFrame call leads to "exposes local type" error

[Compiler plugin] code cleanup

80eb563

Merge branch 'refs/heads/master' into compiler-plugin

1296fe4

# Conflicts: # core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/and.kt # core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/toDataFrame.kt

[Compiler plugin] update test data

619798f

[Compiler plugin] remove experimental api

acfc91d

[Compiler plugin] update generated sources

e12278a

[Compiler plugin] remove unused code

6f999f2

[Compiler plugin] move ConvertApproximation

1182ac9

[Compiler plugin] move

cb0bd59

[Compiler plugin] move to kotlinx package

f8d0fd3

[Compiler plugin] update test data

3c1811b

[Compiler plugin] remove unused code and outdated overload

1a88d09

Rename classes related to generic insert implementation

72e7506

[Compiler plugin] File with "cached" schemas can become corrupted. Fa…

b73bd04

…llback to an empty list

[Compiler plugin] toDataFrame from support

b23828a

[Compiler plugin] move SimpleCol refactoring

430e2fa

[Compiler plugin] refactor SimpleCol to closed hierarchy

f3e5738

[Compiler plugin] review

6a997f7

koperagen merged commit cf234e7 into master Jun 12, 2024
2 checks passed

Jolanrensen reviewed Jun 13, 2024

View reviewed changes

Jolanrensen mentioned this pull request Jun 13, 2024

Replace Klaxon with kotlinx-serialization #603

Merged

koperagen self-assigned this Jun 17, 2024

Jolanrensen added the Compiler plugin Anything related to the DataFrame Compiler Plugin label Aug 8, 2024

koperagen deleted the compiler-plugin branch August 26, 2024 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compiler plugin #729

Compiler plugin #729

koperagen commented Jun 10, 2024 •

edited

Loading

Jolanrensen Jun 13, 2024 •

edited

Loading

Jolanrensen Jun 13, 2024

Jolanrensen Jun 13, 2024 •

edited

Loading

Jolanrensen Jun 13, 2024

Jolanrensen Jun 13, 2024

Jolanrensen Jun 13, 2024

Compiler plugin #729

Compiler plugin #729

Conversation

koperagen commented Jun 10, 2024 • edited Loading

Jolanrensen Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

Jolanrensen Jun 13, 2024

Choose a reason for hiding this comment

Jolanrensen Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

Jolanrensen Jun 13, 2024

Choose a reason for hiding this comment

Jolanrensen Jun 13, 2024

Choose a reason for hiding this comment

Jolanrensen Jun 13, 2024

Choose a reason for hiding this comment

koperagen commented Jun 10, 2024 •

edited

Loading

Jolanrensen Jun 13, 2024 •

edited

Loading

Jolanrensen Jun 13, 2024 •

edited

Loading