1.2.21 #390
Merged
1.2.21 #390
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. processClickhouseInsertRow => processProtonInsertRow 2. remove an empty blank
…protondb failed in creation.
now incremental supported ``` ./sling run --primary-key id --src-conn PROTON --src-stream 'v' --tgt-conn PROTON --tgt-object test_table --mode incremental ```
…xecution( for example we can do data mutation)
…ry did not work well in clickhouse/protondb, because grammer.
```
./sling conns exec CLICKHOUSE "insert into his_v select * from v"
fatal:
~ SQL Error for:
select * from (
insert into his_v select * from v
) as t limit 100
code: 62, message: Syntax error: failed at position 19 ('insert') (line 2, col 3): insert into his_v select * from v
) as t limit 100. Expected one of: SELECT query, possibly with UNION, list of union elements, SELECT query, subquery, possibly with UNION, SELECT subquery, SELECT query, WITH, FROM, SELECT, EXPLAIN, EXPLAIN
```
Protondb enhance
```
✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test7.csv --src-options '{ delimiter: "|", quote: "'"'"'", escape: "\\" }' --stdout
2024-09-27 02:48:17 DBG Starting pprof webserver @ localhost:6060
2024-09-27 02:48:17 DBG opened "file" connection (conn-file-dUY)
2024-09-27 02:48:17 DBG Sling version: dev (linux amd64)
2024-09-27 02:48:17 DBG type is file-file
2024-09-27 02:48:17 DBG using: {"columns":null,"mode":"full-refresh","transforms":null}
2024-09-27 02:48:17 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"'","max_decimals":-1}
2024-09-27 02:48:17 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2024-09-27 02:48:17 INF reading from source file system (file)
2024-09-27 02:48:17 DBG opened "file" connection (conn-file-L9y)
2024-09-27 02:48:17 DBG reading single datastream from file://cmd/sling/tests/files/test7.csv [format=csv]
2024-09-27 02:48:17 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test7.csv
2024-09-27 02:48:17 DBG processing reader from file://cmd/sling/tests/files/test7.csv
2024-09-27 02:48:17 INF writing to target stream (stdout)
col1,header,col3
data1,data with single quote,data3
data with pipe | inside,data2,data3
data with backslash \\ inside,"data with \""escaped\"" double quotes",data3
2024-09-27 02:48:17 DBG wrote 164 B: 3 rows [483 r/s]
2024-09-27 02:48:17 INF wrote 3 rows to stdout in 0 secs [476 r/s]
2024-09-27 02:48:17 INF execution succeeded
✗ cat cmd/sling/tests/files/test7.csv
col1|header|col3
data1|'data with single quote \' inside'|data3
'data with pipe | inside'|data2|data3
'data with backslash \\ inside'|'data with \"escaped\" double quotes'|data3
✗
```
```
(base) ➜ sling-cli git:(support-quote-customized) ✗ cat cmd/sling/tests/files/test8.csv
col1|header|col3
data1|$data with single quote \' inside$|data3
$data with pipe | inside$|data2|data3
$data with backslash \\ inside$|$data with \"escaped\" double quotes$|data3
(base) ➜ sling-cli git:(support-quote-customized) ✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test8.csv --src-options '{ delimiter: "|", quote: "$", escape: "\\" }' --stdout
2024-09-27 03:08:25 DBG Starting pprof webserver @ localhost:6060
2024-09-27 03:08:25 DBG opened "file" connection (conn-file-PhR)
2024-09-27 03:08:25 DBG Sling version: dev (linux amd64)
2024-09-27 03:08:25 DBG type is file-file
2024-09-27 03:08:25 DBG using: {"columns":null,"mode":"full-refresh","transforms":null}
2024-09-27 03:08:25 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"$","max_decimals":-1}
2024-09-27 03:08:25 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2024-09-27 03:08:25 INF reading from source file system (file)
2024-09-27 03:08:25 DBG opened "file" connection (conn-file-AGU)
2024-09-27 03:08:25 DBG reading single datastream from file://cmd/sling/tests/files/test8.csv [format=csv]
2024-09-27 03:08:25 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test8.csv
2024-09-27 03:08:25 DBG processing reader from file://cmd/sling/tests/files/test8.csv
2024-09-27 03:08:25 INF writing to target stream (stdout)
col1,header,col3
data1,data with single quote \' inside,data3
data with pipe | inside,data2,data3
data with backslash \\ inside,"data with \""escaped\"" double quotes",data3
2024-09-27 03:08:25 DBG wrote 174 B: 3 rows [521 r/s]
2024-09-27 03:08:25 INF wrote 3 rows to stdout in 0 secs [514 r/s]
2024-09-27 03:08:25 INF execution succeeded
```
|
seems some code broke the replication. I locally build with commit 256b075 |
locally done some simple test(including halt fresh insert)
refactor writeto db to smaller function
|
@yokofly no pressure, do you think you'll get to open the other PR this week? Want to release 1.2.21 this weekend. |
correct err usage if err is nil
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1.2.21
GitHub Actions Workflows
flarco/infisical-actionfrom v2 to v3 across multiple workflow filesbuild-dev.ymlConfiguration and Environment Variables
SLING_TASK_CONFIGenvironment variable insling_run.goCode Improvements
sling_run.gosling_conns.goTesting
Documentation
README.mdwith new contributing guidelines and test instructions.goreleaser.notes.mdwith a link to the project repositoryDependencies
Build Scripts