Golang SQL database client for ClickHouse.
There are two version of this client, v1 and v2, available as separate branches.
v1 is now in a state of a maintenance - we will only accept PRs for bug and security fixes.
Users should use v2 which is production ready and significantly faster than v1.
v2 has breaking changes for users migrating from v1. These were not properly tracked prior to this client being officially supported. We endeavour to track known differences here and resolve where possible.
The client is tested against the currently supported versions of ClickHouse
Client Version | Golang Versions |
---|---|
=> 2.0 <= 2.2 | 1.17, 1.18 |
>= 2.3 | 1.18.4+, 1.19 |
>= 2.14 | 1.20, 1.21 |
- Uses ClickHouse native format for optimal performance. Utilises low level ch-go client for encoding/decoding and compression (versions >= 2.3.0).
- Supports native ClickHouse TCP client-server protocol
- Compatibility with
database/sql
(slower than native interface!) database/sql
supports http protocol for transport. (Experimental)- Marshal rows into structs (ScanStruct, Select)
- Unmarshal struct to row (AppendStruct)
- Connection pool
- Failover and load balancing
- Bulk write support (for
database/sql
usebegin->prepare->(in loop exec)->commit
) - PrepareBatch options
- AsyncInsert (more details in Async insert section)
- Named and numeric placeholders support
- LZ4/ZSTD compression support
- External data
- Query parameters
Support for the ClickHouse protocol advanced features using Context
:
- Query ID
- Quota Key
- Settings
- Query parameters
- OpenTelemetry
- Execution events:
- Logs
- Progress
- Profile info
- Profile events
https://clickhouse.com/docs/en/integrations/go
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{"127.0.0.1:9000"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
DialContext: func(ctx context.Context, addr string) (net.Conn, error) {
dialCount++
var d net.Dialer
return d.DialContext(ctx, "tcp", addr)
},
Debug: true,
Debugf: func(format string, v ...any) {
fmt.Printf(format+"\n", v...)
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
DialTimeout: time.Second * 30,
MaxOpenConns: 5,
MaxIdleConns: 5,
ConnMaxLifetime: time.Duration(10) * time.Minute,
ConnOpenStrategy: clickhouse.ConnOpenInOrder,
BlockBufferSize: 10,
MaxCompressionBuffer: 10240,
ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
Products: []struct {
Name string
Version string
}{
{Name: "my-app", Version: "0.1"},
},
},
})
if err != nil {
return err
}
return conn.Ping(context.Background())
conn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:9999"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
TLS: &tls.Config{
InsecureSkipVerify: true,
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
DialTimeout: time.Second * 30,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
Debug: true,
BlockBufferSize: 10,
MaxCompressionBuffer: 10240,
ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
Products: []struct {
Name string
Version string
}{
{Name: "my-app", Version: "0.1"},
},
},
})
conn.SetMaxIdleConns(5)
conn.SetMaxOpenConns(10)
conn.SetConnMaxLifetime(time.Hour)
- hosts - comma-separated list of single address hosts for load-balancing and failover
- username/password - auth credentials
- database - select the current default database
- dial_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m". (default 30s)
- connection_open_strategy - random/round_robin/in_order (default in_order).
- random - choose random server from the set
- round_robin - choose a round-robin server from the set
- in_order - first live server is chosen in specified order
- debug - enable debug output (boolean value)
- compress - compress - specify the compression algorithm - “none” (default),
zstd
,lz4
,gzip
,deflate
,br
. If set totrue
,lz4
will be used. - compress_level - Level of compression (default is 0). This is algorithm specific:
gzip
--2
(Best Speed) to9
(Best Compression)deflate
--2
(Best Speed) to9
(Best Compression)br
-0
(Best Speed) to11
(Best Compression)zstd
,lz4
- ignored
- block_buffer_size - size of block buffer (default 2)
- read_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m" (default 5m).
- max_compression_buffer - max size (bytes) of compression buffer during column by column compression (default 10MiB)
- client_info_product - optional list (comma separated) of product name and version pair separated with
/
. This value will be pass a part of client info. e.g.client_info_product=my_app/1.0,my_module/0.1
More details in Client info section.
SSL/TLS parameters:
- secure - establish secure connection (default is false)
- skip_verify - skip certificate verification (default is false)
Example:
clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60
The native format can be used over the HTTP protocol. This is useful in scenarios where users need to proxy traffic e.g. using ChProxy or via load balancers.
This can be achieved by modifying the DSN to specify the HTTP protocol.
http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60
Alternatively, use OpenDB
and specify the interface type.
conn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:8123"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
DialTimeout: 30 * time.Second,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
Protocol: clickhouse.HTTP,
})
Note: using HTTP protocol is possible only with database/sql
interface.
ZSTD/LZ4 compression is supported over native and http protocols. This is performed column by column at a block level and is only used for inserts. Compression buffer size is set as MaxCompressionBuffer
option.
If using Open
via the std interface and specifying a DSN, compression can be enabled via the compress
flag. Currently, this is a boolean flag which enables LZ4
compression.
Other compression methods will be added in future PRs.
At a low level all client connect methods (DSN/OpenDB/Open) will use the Go tls package to establish a secure connection. The client knows to use TLS if the Options struct contains a non-nil tls.Config pointer.
Setting secure in the DSN creates a minimal tls.Config struct with only the InsecureSkipVerify field set (either true or false). It is equivalent to this code:
conn := clickhouse.OpenDB(&clickhouse.Options{
...
TLS: &tls.Config{
InsecureSkipVerify: false
}
...
})
This minimal tls.Config is normally all that is necessary to connect to the secure native port (normally 9440) on a ClickHouse server. If the ClickHouse server does not have a valid certificate (expired, wrong host name, not signed by a publicly recognized root Certificate Authority), InsecureSkipVerify can be to true
, but that is strongly discouraged.
If additional TLS parameters are necessary the application code should set the desired fields in the tls.Config struct. That can include specific cipher suites, forcing a particular TLS version (like 1.2 or 1.3), adding an internal CA certificate chain, adding a client certificate (and private key) if required by the ClickHouse server, and most of the other options that come with a more specialized security setup.
To connect using HTTPS either:
-
Use
https
in your dsn string e.g.https://host1:8443,host2:8443/database?dial_timeout=200ms&max_execution_time=60
-
Specify the interface type as
HttpsInterface
e.g.
conn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:8443"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
Protocol: clickhouse.HTTP,
})
Clickhouse-go implements client info as a part of language client specification. client_name
for native protocol and HTTP User-Agent
header values are provided with the exact client info string.
Users can extend client options with additional product information included in client info. This might be useful for analysis on a server side.
Order is the highest abstraction to the lowest level implementation left to right.
Usage examples for native API and database/sql are provided.
Asynchronous insert is supported via dedicated AsyncInsert
method. This allows to insert data with a non-blocking call.
Effectively, it controls a async_insert
setting for the query.
Using native protocol, asynchronous insert does not support batching. It means, only inline query data is supported. Please see an example here.
HTTP protocol supports batching. It can be enabled by setting async_insert
when using standard Prepare
method.
For more details please see asynchronous inserts documentation.
Available options:
- WithReleaseConnection - after PrepareBatch connection will be returned to the pool. It can help you make a long-lived batch.
V1 (READ) | V2 (READ) std | V2 (READ) clickhouse API |
---|---|---|
1.218s | 924.390ms | 675.721ms |
V1 (WRITE) | V2 (WRITE) std | V2 (WRITE) clickhouse API | V2 (WRITE) by column |
---|---|---|---|
1.899s | 1.177s | 699.203ms | 661.973ms |
go get -u github.com/ClickHouse/clickhouse-go/v2
- batch
- batch with release connection
- async insert
- batch struct
- columnar
- scan struct
- query parameters (deprecated in favour of native query parameters)
- bind params (deprecated in favour of native query parameters)
- client info
- batch
- async insert
- open db
- query parameters
- bind params (deprecated in favour of native query parameters)
- client info
Versions of this client >=2.3.x utilise ch-go for their low level encoding/decoding. This low lever client provides a high performance columnar interface and should be used in performance critical use cases. This client provides more familar row orientated and database/sql
semantics at the cost of some performance.
Both clients are supported by ClickHouse.
-
Database client/clients:
- mailru/go-clickhouse (uses the HTTP protocol)
- uptrace/go-clickhouse (uses the native TCP protocol with
database/sql
-like API) - Drivers with columnar interface:
-
Insert collectors: