streaming: track ingested timestamp in ingestion job #57400
Assignees
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-disaster-recovery
Comments
pbardea
added
C-enhancement
craig bot
pushed a commit
that referenced
this issue
Jan 27, 2021
58665: cli: support `cockroach --version`, `cockroach version --build-tag` r=bdarnell a=knz Requested by the CC SRE team: we want a guaranteed machine-readable way to extract the version of a crdb executable. `cockroach version` did not provide this. Release note (cli change): The `cockroach` command now supports the command-line parameter `--version` which reports its version parameters. This makes `cockroach --version` equivalent to `cockroach version`. Release note (ops change); The `cockroach version` command now supports a new parameter `--build-tag`; when specified, it displays the technical build tag, which makes it possible to integrate with automated deployment tools. 58672: streamingccl: add IngestionFrontierProcessor r=pbardea a=adityamaru The IngestionFrontierProcessor (IFP) is fed rows from the IngestionProcessor. Each row represents a partition and the latest resolved ts event received for that partition. The IFP uses the existing util `spanFrontier` (used by changefeeds) to keep track of all the rows returned by the several ingestion procs, and resolves the minimum resolved ts for a particular partition across all procs. This is then written to the ingestion job progress indicating that all processors have ingested a stream up to a particular resolved ts. TODO: Job resumption from the resolved ts reported in its progress. Will do this in a follow up PR, along with a test. Informs: #57400 Release note: None 59051: githooks: new release note category `ops changes` r=jseldess,taroface a=knz Fixes #57898 This new release note category is meant to cover changes that interest primarily the folk responsible for running production clusters and integrating CockroachDB into a larger IT deployment. Note that this new category takes over certain things that would previously be attributed to "cli changes". From now on "cli changes" covers changes that primarily interest app developers, crdb developers and operators that are currently trying out features or setting up experiments in development/staging environments. See this page for more details: https://wiki.crdb.io/wiki/spaces/CRDB/pages/186548364/Release+notes Release note: None Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net> Co-authored-by: Aditya Maru <adityamaru@gmail.com>
|
Going to mark this as closed with the frontier checked in. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The cluster stream ingestion job (#57399) needs to be able to track the ingested timestamp across all of it's partition ingestion processors.
Each partition will need to report back to the job the latest timestamp at which:
a) it has received a resolved timestamp event for
b) it has flushed
The job will then also need to keep track of the earliest of these timestamps across all of its stream ingestion processors.
The text was updated successfully, but these errors were encountered: