feat: EXPOSED-60 Support json/json(b) column types #1762

bog-walk · 2023-06-15T19:59:15Z

Add 2 new column types: JsonColumnType & JsonBColumnType that can be used 3 ways:

json("column_name", Json, Type.serializer()) // or custom serializer
json<Type>("column_name", Json)
json("column_name", customEncoder(), customDecoder())

Dialects store JSON differently, usually in text or binary format, but may not reflect this in the name of the data type. Some may not even support the JSON data type, but use it as an alias for another type (e.g. MariaDB maps JSON to LONGTEXT and H2 maps to byte[]). To allow for this variability, the user's choice between json() and jsonb() can be flexible depending on dialect, e.g.:

[PostgreSQL] json() -> JSON, jsonb() -> JSONB
[MySQL] Only supports binary format but default type is still JSON, so both column types map to JSON (i.e. using json() and jsonb() is equivalent).
[SQLite] No JSON type but allows JSON functions on TEXT columns, so json() maps to TEXT, while jsonb() throws.

Current caveats:

Oracle and H2 are not yet supported.
Use in DAO not fully supported.
Only 2 functions currently supported: using paths/keys to extract from stored JSON as a scalar value/text or as a JSON object.

Next steps:

Extend support to Oracle and H2 (will likely mean overrides in JsonBColumnType).
Add a few more functions (assess most-used/important/broad functions across dialects).
Test and extend fully to DAO, as well as potential edge cases: e.g. raw Json strings, Json arrays or nested arrays, if field identifiers need escape formatting, indexing +/- misuse as primary/foreign key?
Assess potential use case for extending current cast types, literals, and parameters to include Json type.

Add kotlin serialization plugin to exposed-core and exposed-tests modules, as well as dependency on serialization library. Add build dependency on org.postgresql to allow use of PGobject. Add base JsonColumnType with dialect-specific json sql type and table extension functions.

Add json_extract function for supported dialects and adjust tests. Add apiDump diff. Adjust dependency version to match updated Kotlin version. Caveats: - Oracle and H2 are not yet supported. - Use in DAO also not fully supported.

@param

Add missing @param KDocs. Edit section comment headings for consistency. Remove redundant jsonBType() overrides in dialects.

bog-walk · 2023-06-15T20:04:09Z

exposed-core/build.gradle.kts

@@ -12,5 +13,7 @@ dependencies {
    api(kotlin("stdlib"))
    api(kotlin("reflect"))
    api("org.jetbrains.kotlinx", "kotlinx-coroutines-core", Versions.kotlinCoroutines)
+    api("org.jetbrains.kotlinx", "kotlinx-serialization-json", Versions.kotlinxSerialization)
+    api("org.postgresql", "postgresql", Versions.postgre)


PostgreSQL DB returns a special type PGobject for deserialization. I didn't figure out a way to catch the type that doesn't involve including this specific dependency.

bog-walk · 2023-06-15T20:10:07Z

settings.gradle.kts

@@ -15,5 +15,6 @@ include("exposed-crypt")
 pluginManagement {
    plugins {
        id("org.jetbrains.kotlin.jvm") version "1.8.22"
+        id("org.jetbrains.kotlin.plugin.serialization") version "1.8.22"


I believe that the serialization plugin must always match the kotlin plugin version. If true, does something need to be done with dependabot to ensure they don't diverge?

That's true, I think with different versions build will fail anyway

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/ColumnType.kt

e5l

Well done! LGTM

I would also add tests with a non-serializable class to check if it fails with a good message

e5l · 2023-06-16T07:02:29Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+        Short::class -> ShortColumnType()
+        Byte::class -> ByteColumnType()
+        Double::class -> DoubleColumnType()
+        Float::class -> FloatColumnType()


Should we add ByteArray as well?

What should ByteArray be mapped to? BlobColumnType? Or maybe BasicBinaryColumnType, since the length is unknown?

Move non-member property parameter KDocs to class KDocs. Add ByteArray mapping to jsonExtract(). Add test for error message that results when attempting to use a Json column with a non-serializable class.

xtexChooser · 2023-06-22T01:18:44Z

thanks!

Add kotlin serialization plugin to exposed-core and exposed-tests modules, as well as dependency on serialization library. Add build dependency on org.postgresql to allow use of PGobject. Add base JsonColumnType with dialect-specific json sql type and table extension functions. * Add KDocs and tests. * Add jsonb column type & vendor sqlTypes * Add json_extract function for supported dialects and adjust tests. Add apiDump diff. Adjust dependency version to match updated Kotlin version. Caveats: - Oracle and H2 are not yet supported. - Use in DAO also not fully supported. * Add ByteArray mapping to jsonExtract(). * Add test for error message that results when attempting to use a Json column with a non-serializable class.

bog-walk added 5 commits June 15, 2023 14:42

Add KDocs and tests.

96c1735

Add jsonb column type & vendor sqlTypes

5870932

feat: EXPOSED-60 Support json/json(b) column types

68dae1f

Add json_extract function for supported dialects and adjust tests. Add apiDump diff. Adjust dependency version to match updated Kotlin version. Caveats: - Oracle and H2 are not yet supported. - Use in DAO also not fully supported.

feat: EXPOSED-60 Support json/json(b) column types

954f87e

Add missing @param KDocs. Edit section comment headings for consistency. Remove redundant jsonBType() overrides in dialects.

bog-walk requested review from Tapac, e5l and joc-a June 15, 2023 19:59

bog-walk linked an issue Jun 15, 2023 that may be closed by this pull request

Json #1699

Closed

bog-walk commented Jun 15, 2023

View reviewed changes

This was linked to issues Jun 15, 2023

Support JSON and JSONB types in Postgres #1172

Closed

Json(b) data type support #127

Closed

e5l reviewed Jun 16, 2023

View reviewed changes

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/ColumnType.kt Outdated Show resolved Hide resolved

e5l approved these changes Jun 16, 2023

View reviewed changes

feat: EXPOSED-60 Support json/json(b) column types

3c1ea31

Move non-member property parameter KDocs to class KDocs. Add ByteArray mapping to jsonExtract(). Add test for error message that results when attempting to use a Json column with a non-serializable class.

joc-a approved these changes Jun 16, 2023

View reviewed changes

bog-walk merged commit f9974c5 into main Jun 19, 2023

bog-walk deleted the bog-walk/support-json-column branch June 19, 2023 12:01

sschuberth mentioned this pull request Jul 31, 2023

Switch to Expose's native JSONB support oss-review-toolkit/ort#7349

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: EXPOSED-60 Support json/json(b) column types #1762

feat: EXPOSED-60 Support json/json(b) column types #1762

bog-walk commented Jun 15, 2023

bog-walk Jun 15, 2023

bog-walk Jun 15, 2023

e5l Jun 16, 2023 •

edited

Loading

e5l left a comment

e5l Jun 16, 2023

bog-walk Jun 16, 2023

xtexChooser commented Jun 22, 2023

feat: EXPOSED-60 Support json/json(b) column types #1762

feat: EXPOSED-60 Support json/json(b) column types #1762

Conversation

bog-walk commented Jun 15, 2023

bog-walk Jun 15, 2023

Choose a reason for hiding this comment

bog-walk Jun 15, 2023

Choose a reason for hiding this comment

e5l Jun 16, 2023 • edited Loading

Choose a reason for hiding this comment

e5l left a comment

Choose a reason for hiding this comment

e5l Jun 16, 2023

Choose a reason for hiding this comment

bog-walk Jun 16, 2023

Choose a reason for hiding this comment

xtexChooser commented Jun 22, 2023

e5l Jun 16, 2023 •

edited

Loading