feat: EXPOSED-416 Support adding special database-specific column definitions

[bog-walk]

Column definition in CREATE TABLE DDL currently covers the following:
column_name column_type column_default null_constraint key_constraint

There are some very niched database-specific keywords or syntax that can be appended to a column definition, which are not available to the user unless introduced by feature requests. Some of these would just require hard-coding the additional feature in Column class for the specific database.

This proposal attempts to give user's the ability to customize column definitions that fit the following criteria:

1. It is very database-specific, but low priority.
2. It does not affect any further logic, like column insert/update values, prepared statement, etc.
3. It fits standard ordering rule shared by most databases, i.e. it follows column_name column_type column_default but comes before any constraints.

---

This will not solve all use cases, but gives user's access to the following example features:

MySQL: ON UPDATE CURRENT_TIMESTAMP
Specifies default value when row is updated

object : Table("tester") {
    val created = timestamp("created")
        .defaultExpression(CurrentTimestamp)
        .withDefinition("ON UPDATE", CurrentTimestamp)
}

// generates SQL:
// CREATE TABLE IF NOT EXISTS tester (created DATETIME(6) DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6) NOT NULL)

---

Oracle: DEFAULT ON NULL (also H2)
Specifies default value when row is inserted (or updated) with null value

object : Table("tester") {
    val item = varchar("item", 32).withDefinition("DEFAULT ON NULL", stringLiteral("Item A"))
}

// generates SQL:
// CREATE TABLE TESTER (ITEM VARCHAR2(32 CHAR) DEFAULT ON NULL 'Item A' NOT NULL)

---

SQL Server: MASKED WITH
Masks data returned by SELECT statements for users without proper permissions

object : Table("tester") {
    val email = varchar("email", 128)
        .uniqueIndex()
        .withDefinition("MASKED WITH (FUNCTION = 'email()')")
}

// generates SQL:
// CREATE TABLE tester (email VARCHAR(128) MASKED WITH (FUNCTION = 'email()') NOT NULL)

---

MySQL: COMMENT (also H2, SQLite)
Adds comment to column only

object : Table("tester") {
    val amount = integer("amount").withDefinition("COMMENT", stringLiteral("Amount of testers"))
}

// generates SQL:
// CREATE TABLE IF NOT EXISTS tester (amount INT COMMENT 'Amount of testers' NOT NULL)

---

MySQL: INVISIBLE (also H2, Oracle)
Sets a column's visibility to hidden (so it's not available unless named)

object : Table("tester") {
    val active = bool("active").nullable().withDefinition("INVISIBLE")
}

// generates SQL:
// CREATE TABLE IF NOT EXISTS tester (active BOOLEAN INVISIBLE NULL)

---

PostgreSQL GENERATED ALWAYS AS (...) (supported by all DB actually)
Creates a generated column on database-side based on provided computation

object : Table("tester") {
    val amount = integer("amount").nullable()
    val computedAmount = integer("computed_amount")
        .nullable()
        .databaseGenerated()
        .withDefinition("GENERATED ALWAYS AS (AMOUNT + 1) STORED")
}

// generates SQL:
// CREATE TABLE IF NOT EXISTS tester (
//     amount INT NULL,
//     computed_amount INT GENERATED ALWAYS AS (AMOUNT + 1) STORED NULL
// )

---

The ordering rule works if either a foreign key or check constraint is chained to the column, because these are implemented as generated SQL following the CONSTRAINT keyword, which is separate from the column definition clause itself.

[bog-walk]

Suggestions are welcome both for names and function parameter.

I also considered: withOption(), useDefinition(), addDefinition(), definition().

Parameter alternatives:

Column<T>.withDefinition(vararg definition: Expression<*>)

• Attempting to pass in a string as stringLiteral() would wrap it in single quotes, which is appropriate in query clauses, but not in create DDL.

Column<T>.withDefinition(definition: String)

• Attempting something like .withDefinition("ON UPDATE $CurrentTimestamp"), would cause Please call Database.connect() before using this code because it is trying to resolve the expression to string outside of transaction.

2 overloads, so the 2 options above together

• But I thought it looked more busy: .withDefinition("ON UPDATE").withDefinition(CurrentTimestamp)

[e5l]

withDefinition looks good

[bog-walk]

There are some limitations.
It would be great in this use case to do something like .withDefinition("GENERATED ALWAYS AS", amount.plus(1)).
But this does not work because Exposed generates expression SQL using the column's full identity generated_table.amount, which is valid for queries and other statements, but not valid syntax in a CREATE TABLE ddl.
So available expressions are those that do not use columns of the table being created at the time.

[e5l]

Looks really nice!
It actually can be used to implement a lot of features (being composable)

[e5l]

As a workaround for CREATE TABLE, we can create postprocessing for the statement to replace invalid syntax

[obabichevjb]

Actually, I don't know how it's supposed to work with statementsRequiredToActualizeScheme(), should it return something if new definition was added, but the following test fails:

@Test
fun testUpdateExistingTable() {
    withDb {
        val before = object : IntIdTable("UpdateExistingTable") {
            val field = integer("field")
        }
        SchemaUtils.create(before)

        val after = object : IntIdTable("UpdateExistingTable") {
            val field = integer("field").withDefinition("ON UPDATE", CurrentTimestamp)
        }
        assertTrue(SchemaUtils.statementsRequiredToActualizeScheme(after).isNotEmpty())
    }
}

[bog-walk]

In the ideal world, I think 2 things should happen with statementsRequiredToActualizeScheme():

1. Calling multiple times on an unchanged column should generate no ALTER statements.
2. Calling on a column with added/removed definition should generate an ALTER statement.

But this becomes tricky without knowing exactly what the definition is and because not many special column definitions are returned by metadata (maybe default variants, visibility, comments in this example). Given this I'm not sure how it could be guaranteed using existing table column in database.

Migration script at least can be edited with correctly generated new sql from after.field.modifyStatement().

[obabichevjb]

I would say that in ColumnDefinitionTests correct creation of the table is already checked, and we can trust that the correct table is created. And here we only want to check that the date was updated correctly. But it's minor, I'm also ok, if we make such check for every new type of custom column definition.

[bog-walk]

The intention of line 588 here (and in other file tests) is not to confirm that the correct table is created, but that Exposed considers the created table column as equivalent to the retrieved metadata (by not generating unnecessary ALTER statements for the unchanged column).

It is an extra precaution though, as Exposed currently only retrieves 4 metadata: column name, whether is nullable, its default value, and whether is autoincrement. So it's unlikely, as you've seen above, that a diff will be caught; but not impossible, as with DEFAULT ON NULL.

[obabichevjb]

I was surprised, but it's really not possible to do with replaceColumn(), because varchar() internally tries to register column a second time... probably we could add method removeColumn, or something like item.replaceColumn { varchar("tester")... } for better experience with column manipulation from outside of the library. But it's just thoughts, it's not the problem of PR itself.

[obabichevjb]

Anyway, as an alternative:

withDb(TestDB.ALL_H2_V2 + TestDB.ORACLE) { testDb ->
  val tester = object : Table("tester") {
      val amount = integer("amount")
      var item: Column<String> = if (testDb in TestDB.ALL_H2) {
          varchar("item", 32).default(itemA).withDefinition("DEFAULT ON NULL")
      } else {
          varchar("item", 32).withDefinition("DEFAULT ON NULL", stringLiteral(itemA))
      }
  }
  ...
}

It could be just a bit more readable, in the rest the same.

[bog-walk]

I considered this and hesitated because I wondered if it read weird. Since I'm not the only one, I'll switch to this more simple way.