SQLite integration

[g-arslan]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add SQLite table engine, table function, database engine.

[kssenii]

Please, remove submodules update, which was added by accident -- looks like all of them were added in the last commit, then just revert it. And sqlite library should be added as submodule - you can delete contrib/sqlite and then add it again by executing in contrib dir git submodule add ...- that's all, then you'll have to commit .gitmodules file and the submodule.

Also, please make integration tests in ClickHouse/tests/integration/test_sqlite_storage and test_sqlite_database_engine (see example in tests/integration/test_postgresql_database_engine and tests/integration/test_postgresql_storage).

Also it will be very easy to add sqlite table function - can be implemented in a few minutes since sqlite storage and method fetchSQLiteTableStructure are already implemented, it also should be added to integration tests.

[alexey-milovidov]

We can add functional tests instead of integration tests - it's easy, because sqlite database is a single file.

[alexey-milovidov]

Tables should be also writable.

[kssenii]

ubsan caught something :)

01889_sqlite_read_write: [ FAIL ] 13.61 sec. - return code 210
2021-06-02 14:14:57 [58e2ccd64857] 2021.06.02 14:14:46.237369 [ 107873 ] BaseDaemon: ########################################
2021-06-02 14:14:57 [58e2ccd64857] 2021.06.02 14:14:46.237496 [ 107873 ] BaseDaemon: (version 21.7.1.7025, build id: 71855C0BDCBDE2FE0A7AE10830E75CAE24E314AC) (from thread 79221) (query_id: d71c8563-4ebe-43f2-9bf5-b951a275e705) Received signal Unknown signal (-3)
2021-06-02 14:14:57 [58e2ccd64857] 2021.06.02 14:14:46.237539 [ 107873 ] BaseDaemon: Sanitizer trap.

[kssenii]

@Mergifyio update

[mergify]

Command update: success

Branch has been successfully updated

[gyuton]

Internal documentation ticket: DOCSUP-11655.

[alexey-milovidov]

@kssenii Need two fixes.

1. After stress test sometimes server failed to start with the message:

2021.07.25 03:55:07.636901 [ 454892 ] {} <Error> Application: DB::Exception: SQLite database path '/var/lib/clickhouse/user_files/db1' is invalid. Error: No such file or directory: Cannot attach table `test_15`.`sqlite_table3` from metadata file /var/lib/clickhouse/store/eef/eef306b5-da98-46ac-aef3-06b5da9826ac/sqlite_table3.sql from query ATTACH TABLE test_15.sqlite_table3 UUID '195e7c83-8d41-489a-995e-7c838d41b89a' (`col1` String, `col2` Int32) ENGINE = SQLite('/var/lib/clickhouse/user_files/db1', 'table3'): while loading database `test_15` from path /var/lib/clickhouse/metadata/test_15

2. SQLite tables should allow efficient key-value requests with WHERE key = xxx or WHERE key IN (...) (assuming table is indexed). PS. You said that it does not work but I wonder, why? Simply sending compatible conditions to SQLite query would suffice (see transformQueryForExternalDatabase).

[kssenii]

@alexey-milovidov

You said that it does not work but I wonder, why?

It seemed too obvious that it just works (because of transformQueryForExternalDatabase), so I thought you mean something more complicated - for example, StorageEmbededRocksDB has additional code for more efficient processing in case there is a where expression which contains primary key (if this key was specified when creating a clickhouse table with StorageEmbededRocksDB engine).

[alexey-milovidov]

Good. So, we can already use pre-indexed sqlite tables for efficient key-value lookups.
Another possibility is to allow PRIMARY KEY in table with SQLite engine in ClickHouse and translate it directly to primary key of created sqlite tables.