batch send err:compress: buf size 607047244 > 134217728 max size

[hanjm]

Issue description

Hi, i am using natvie batch with compress, i found if batch size greater 128MB, will produce a error message when invoke batch.send().
it seems has a hard limit for size of compress data, what should i do.
https://github.com/ClickHouse/ch-go/blob/main/compress/writer.go#L22
https://github.com/ClickHouse/ch-go/blob/main/compress/compress.go#L31

Example code

	batch, err := db.PrepareBatch(ctx, insertSQL)
	if err != nil {
		return fmt.Errorf("prepare batch err:%w", err)
	}
        // vs is 200k length array, total size execeed 128MB 
        for v:=range vs {
            err:=batch.Append(v)
             if err != nil {
			return fmt.Errorf("batch append err:%w", err)
		}
       }
      	if err = batch.Send(); err != nil {
		return fmt.Errorf("batch send err:%w", err)
	}

Error log

batch send err:compress: buf size 607047244 > 134217728 max size

Configuration

OS:
Linux
Interface: E.g. native, database/sql
native
Driver version:
2.23-bata2
Go version: run go version in your console
1.18.2
ClickHouse Server version:
21.8.12

[gingerwizard]

This is a ch-go limitation @ernado can we make this configurable?

[ernado]

Yes, we can do it. It was introduced to prevent user from accidental OOM.

[gingerwizard]

If you expose in ch-go (maybe an env variable or settings passed to the encoder init?) we can document in clickhouse-go.

[ernado]

Lol, now we can close issues with cross-repo references? 🤔

It should be fixed in ch-go@v0.47.3, I've removed this check entirely, we don't need it on data block write.

[gingerwizard]

@hanjm next release. Probably Friday or latest next monday.

[hanjm]

so quickly. thank you. :)