sql: avoid use of txn.Scan API in IndexBackfillMerger #76685
Labels
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Comments
stevendanna
added
the
C-bug
rhu713
pushed a commit
to rhu713/cockroach
that referenced
this issue
Feb 25, 2022
Use Batch API instead of txn.Scan() in order to limit the number of bytes per batch response in the index backfill merger. Fixes cockroachdb#76685. Release note: None
craig bot
pushed a commit
that referenced
this issue
Mar 3, 2022
…77328 #77335 75751: sql: Add DateStyle/IntervalStyle visitor r=e-mbrown a=e-mbrown The DateStyle visitor allows for cast expressions with string to interval and date/interval types to string cast to be rewritten. These stable cast cause issues with DateStyle/IntervalStyle formatting so they need to be wrapped in builtins containing their immutable version. Release note: None Release justification: Low risk update to new functionality 76705: backupccl: add prototype metadata.sst r=rhu713 a=rhu713 This adds writing of an additional file to the completion of BACKUP. This new file is an sstable that contains the same metadata currently stored in the BACKUP_MANIFEST file and statistics files, but organizes that data differently. The current BACKUP_MANIFEST file contains a single binary-encoded protobuf message of type BackupManifest, that in turn has several fields some of which are repeated to contain e.g. the TableDescriptor for every table backed up, or every revision to every table descriptor backed up. This can result in these manifests being quite large in some cases, which is potentially concerning because as a single protobuf message, one has to read and unmarshal the entire struct into memory to read any field(s) of it. Organizing this metadata into an SSTable where repeated fields are instead stored as separate messages under separate keys should instead allow reading it incrementally: one can seek to a particular key or key prefix and then scan, acting on whatever data is found as it is read, without loading the entire file at once (when opened using the same seek-ing remote SST reader we use to read backup data ssts). This initial prototype adds only the writer -- RESTORE does not rely on, or even open, this new file at this time. Release note: none. 77018: release: automate orchestration version update r=celiala a=rail Previously, as a part of the release process we had to bump the orchestration versions using `sed` with some error-prone regexes. This patch adds `set-orchestration-version` subcommand to the release tool. It uses templates in order to generate the orchestration files. Release note: None 77055: sql: change index backfill merger to use batch api r=rhu713 a=rhu713 Use Batch API instead of txn.Scan() in order to limit the number of bytes per batch response in the index backfill merger. Fixes #76685. Release note: None 77065: bazel: use test sharding more liberally r=rail a=rickystewart Closes #76376. Release note: None 77109: ccl/sqlproxyccl: add helpers related to connection migration r=JeffSwenson,andy-kimball a=jaylim-crl #### ccl/sqlproxyccl: add helpers related to connection migration Informs #76000. Extracted from #76805. This commit adds helpers related to connection migration. This includes support for retrieving the transfer state through SHOW TRANSFER STATE, as well as deserializing the session through crdb_internal.deserialize_session. Release note: None Release justification: Helpers added in this commit are needed for the connection migration work. Connection migration is currently not being used in production, and CockroachCloud is the only user of sqlproxy. #### ccl/sqlproxyccl: fix math for defaultBufferSize in interceptors Previously, we incorrectly defined defaultBufferSize as 16K bytes. Note that 2 << 13 is 16K bytes. This commit fixes that behavior to match the original intention of 8K bytes. Release note: None Release justification: This fixes an unintentional buglet within the sqlproxy code that was introduced with the interceptors back then. Not having this in means we're using double the memory for each connection within the sqlproxy. 77307: sql: add cluster setting to limit max size of serialized session r=otan,jaylim-crl a=rafiss fixes #77302 The sql.session_transfer.max_session_size cluster setting can be used to limit the max size of a session that is serialized using crdb_internal.serialize_session. No release note since this is not a public setting. Release justification: high priority fix for new functionality. Release note: None 77318: roachpb: extract keysbase to break some dependencies r=yuzefovich a=yuzefovich This commit extracts a couple of things out of `roachpb` into new `keysbase` package in order to break the dependency of `util/json` and `sql/inverted` on `roachpb` (which is a part of the effort to clean up the dependencies of `execgen`). Addresses: #77234. Release note: None Release justification: low risk change to clean up the dependencies. 77319: sessiondatapb: move one enum definition into lex package r=yuzefovich a=yuzefovich This commit moves the definition of `BytesEncodeFormat` enum from `sessiondatapb` to `lex`. This is done in order to make `lex` not depend on a lot of stuff (eventually on `roachpb`) and is a part of the effort to clean up the dependencies of `execgen`. Note that the proto package name is not changed, so this change is backwards-compatible. Informs: #77234. Release note: None Release justification: low risk change to clean up the dependencies. 77328: roachtest: log stdout and stderr in sstable corruption test r=itsbilal a=nicktrav To aid in debugging #77321, log the contents stdout and stderr if the manifest dump command fails. Release justification: Tests only. Release note: None. 77335: kvserver: fix race that caused truncator to truncate non-alive replica r=tbg,erikgrinaker a=sumeerbhola This was causing truncated state to be written to such a replica, which would then get picked up as the HardState.Commit value when a different replica was later added back for the same range. See #77030 (comment) for the detailed explanation. Also restore the default value of kv.raft_log.loosely_coupled_truncation.enabled to true. Fixes #77030 Release justification: Bug fix. Release note: None Co-authored-by: e-mbrown <ebsonari@gmail.com> Co-authored-by: David Taylor <tinystatemachine@gmail.com> Co-authored-by: Rui Hu <rui@cockroachlabs.com> Co-authored-by: Rail Aliiev <rail@iqchoice.com> Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com> Co-authored-by: Jay <jay@cockroachlabs.com> Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com> Co-authored-by: Nick Travers <travers@cockroachlabs.com> Co-authored-by: sumeerbhola <sumeer@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently we use txn.Scan in the IndexBackfillMerger, but we should probably use the lower level APIs to avoid pulling in more data that we expect.
See: https://github.com/cockroachdb/cockroach/pull/73878/files#r806428509
Jira issue: CRDB-13230
The text was updated successfully, but these errors were encountered: