Not allow to add a not null column, name:timestamp

[zouxiang1993]

Describe this problem

Error message is:

2023-03-27 08:58:03.127 INFO [server/src/proxy/grpc/write.rs:576] Add columns start, request_id:475, table:perflog, columns:[ColumnSchema { id: 2, name: "timestamp", data_type: Timestamp, is_nullable: false, is_tag: false, comment: "", escaped_name: "timestamp", default_value: None }]
2023-03-27 08:58:03.127 ERRO [server/src/proxy/grpc/write.rs:75] Failed to handle write, err:Rpc error, code:500, message:Failed to execute interpreter, err:Failed to execute alter table, err:Not allow to add a not null column, name:timestamp

Server version

CeresDB Server
CeresDB version: 1.0.0
Git branch: main
Git commit: 50471c0
Build time: 2023-03-26T20:55:13.872269184Z
Rustc version: 1.69.0-nightly

Steps to reproduce

1. Create a table, use column "ts" as a TIMESTAMP KEY.

CREATE TABLE xxx (
    ts timestamp not null, 
    ... 
    TIMESTAMP KEY(ts)
)

2. Write points through java SDK:

    Point.PointBuilder builder = Point.newPointBuilder("perflog")
            .setTimestamp(msg.getTime())
            .addTag("tag", msg.getTag())

3. The "timestamp" column from SDK will be treated as a new column and then execute ALTER TABLE.

Expected behavior

No response

Additional Information

No response

[MichaelLeeHZ]

Do you have any interest in fixing this bug? We would be very grateful!

[zouxiang1993]

OK. I will have a try.

[jiacai2050]

It seems your write doesn't match table's schema, so it go to auto add column feature, which is introduced in #749.

Could you reproduce this issue when they are the same?

[zouxiang1993]

My write matches table's schema except the special timestamp column.
The column name in the table's schema can be any string, but the column name from Java SDK is a constant in Schema Config:
https://github.com/CeresDB/ceresdb/blob/50471c0980d4b74a7d3b7491a10d9e10cf87b1b0/sql/src/planner.rs#L469


To fix this, maybe we can add a parameter to 'build_schema_from_write_table_request' and discard the timestamp column when auto add column:
https://github.com/CeresDB/ceresdb/blob/50471c0980d4b74a7d3b7491a10d9e10cf87b1b0/sql/src/planner.rs#L371

[jiacai2050]

My write matches table's schema except the special timestamp column.

That's the problem.

but the column name from Java SDK is a constant in Schema Config:

AFAIK, timestamp is a special field in protobuf, column name for it is not required.
https://github.com/CeresDB/ceresdbproto/blob/37ba6214b131bba1169bbdc6862be4f7408ff803/protos/storage.proto#L70

To fix this, I think we can reuse timestamp name in original schema when invoke find_new_columns, something like this

build_schema_from_write_table_request(
    schema_config: &SchemaConfig,
    write_table_req: &WriteTableRequest,
    timestamp_name: Option<&str> // Some(_) when call from find_new_columns
)

[jiacai2050]

@zouxiang1993 Have you already started working on this issue?

We plan to release 1.1 this Friday, if you haven't started, we will fix this today.

[zouxiang1993]

Sorry, I was going to fix it on the weekend.