Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
…99752 #99774

99433: opt: fixup CTE stats on placeholder queries r=cucaroach a=cucaroach

During optbuilder phase we copy the initial expressions stats into the
fake-rel but this value can change when placeholders are assigned so add
code in AssignPlaceholders to rebuild the cte if the stats change.

Fixes: #99389
Epic: none
Release note: none

99516: metrics: improve ux around _status/vars output r=aadityasondhi a=dhartunian

Previously, the addition of the `tenant` metric label was applied uniformly and could result in confusion for customers who never enable multi-tenancy or c2c. The `tenant="system"` label carries little meaning when there's no tenancy in use.

This change modifies the system tenant label application to only happen when a non- sytem in-process tenant is created.

Additionally, an environment variable:
`COCKROACH_DISABLE_NODE_AND_TENANT_METRIC_LABELS` can be set to `false` to disable the new `tenant` and `node_id` labels. This can be used on single-process tenants to disable the `tenant` label.

Resolves: #94668

Epic: CRDB-18798

Release note (ops change): The
`COCKROACH_DISABLE_NODE_AND_TENANT_METRIC_LABELS` env var can be used to disable the newly introduced metric labels in the `_status/vars` output if they conflict with a customer's scrape configuration.

99522: jobsprofiler: store DistSQL diagram of jobs in job info r=dt a=adityamaru

This change teaches import, cdc, backup and restore
to store their DistSQL plans in the job_info table
under a timestamped info key. The generation and writing
of the plan diagram is done asynchronously so as to not
slow down the execution of the job. A new plan will be
stored everytime the job sets up its DistSQL flow.

Release note: None
Epic: [CRDB-8964](https://cockroachlabs.atlassian.net/browse/CRDB-8964)
Informs: #99729

99574: streamingccl: skip acceptance/c2c on remote cluster setup r=stevendanna a=msbutler

acceptance/c2c currently fails when run on a remote cluster. This patch ensures the test gets skipped when run on a remote cluster. There's no need to run the test on a remote cluster because the other c2c roachtests provide sufficient coverage.

Fixes #99553

Release note: none

99691: codeowners: update sql obs to cluster obs r=maryliag a=maryliag

Update mentions of `sql-observability` to
`cluster-observability`.

Epic: none
Release note: None

99712: ui: connect metrics provider to metrics timescale object r=xinhaoz a=dhartunian

Previously, the `MetricsDataProvider` component queried the redux store for the `TimeScale` object which contained details of the currently active time window. This piece of state was assumed to update to account for the "live" moving window that metrics show when pre-set lookback time windows are selected.

A recent PR: #98331 removed the feature that polled new data from SQL pages, which also disabled polling on metrics pages due to the re-use of `TimeScale`.

This commit modifies the `MetricsDataProvider` to instead read the `metricsTime` field of the `TimeScaleState` object. This object was constructed for use by the `MetricsDataProvider` but was not wired up to the component.

Resolves #99524

Epic: None

Release note: None

99733: telemetry: add FIPS-specific channel r=knz a=rail

Previously, all official builds were reporting using the same telemetry channel.

This PR adds an new telemetry channel for the FIPS build target.

Fixes: CC-24110
Epic: DEVINF-478
Release note: None

99745: spanconfigsqlwatcher: deflake TestSQLWatcherOnEventError r=arulajmani a=arulajmani

Previously, this test was setting the no-op checkpoint duration to be
 every hour to effectively disable checkpoints. Doing so is integral to
what the test is testing. However, this was a lie, given how `util.Every` works -- A call to `ShouldProcess` returns true the very first time.

This patch achieves the original goal by introducing a new testing knob. Previously, the test would fail in < 40 runs locally.  Have this running strong for ~1000 runs.

Fixes #76765

Release note: None

99747: roachtest: use persistent disks for disk-stall tests r=jbowens a=nicktrav

Currently, the `disk-stall` tests use local SSDs. When run on GCE VMs, a higher test flake rate is observed due to known issues with fsync latency for local SSDs.

Switch the test to use persistent disks instead.

Touches: #99372.

Release note: None.

Epic: CRDB-20293

99752: kvserver: bump tolerance more r=ajwerner a=ajwerner

I'm not digging into this more, but the test is flakey.

Epic: none

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests_BazelUnitTests/9161972?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

Release note: None

99774: *: identify remaining uses of TODOSQLCodec r=stevendanna a=knz

The `TODOSQLCodec` was a bug waiting to happen.

The only reasonable remaining purpose is for use in tests. As such, this change moves its definition to a test-only package (we have a linter that verifies that `testutils` is not included in non-test code).

This change then identifies the one non-reasonable remaining purposes and identifies it properly as a bug linked to #48123.

Release note: None
Epic: None

Co-authored-by: Tommy Reilly <treilly@cockroachlabs.com>
Co-authored-by: David Hartunian <davidh@cockroachlabs.com>
Co-authored-by: adityamaru <adityamaru@gmail.com>
Co-authored-by: Michael Butler <butler@cockroachlabs.com>
Co-authored-by: maryliag <marylia@cockroachlabs.com>
Co-authored-by: Rail Aliiev <rail@iqchoice.com>
Co-authored-by: Arul Ajmani <arulajmani@gmail.com>
Co-authored-by: Nick Travers <travers@cockroachlabs.com>
Co-authored-by: ajwerner <awerner32@gmail.com>
Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
  • Loading branch information
11 people committed Mar 28, 2023
12 parents 60b7440 + 2014cf2 + a6a1a4c + da25b21 + a8dd57c + f2c8220 + 9880ad0 + 6100c55 + a7b6432 + 81373fd + cd1299c + 5312840 commit 0745cd4
Show file tree
Hide file tree
Showing 47 changed files with 544 additions and 88 deletions.
20 changes: 10 additions & 10 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@
/pkg/sql/job_exec_context* @cockroachdb/sql-queries @cockroachdb/jobs-prs
/pkg/sql/delegate/*job*.go @cockroachdb/jobs-prs

/pkg/sql/appstatspb @cockroachdb/sql-observability
/pkg/sql/execstats/ @cockroachdb/sql-observability
/pkg/sql/scheduledlogging/ @cockroachdb/sql-observability
/pkg/sql/sqlstats/ @cockroachdb/sql-observability
/pkg/ccl/testccl/sqlstatsccl/ @cockroachdb/sql-observability
/pkg/sql/appstatspb @cockroachdb/cluster-observability
/pkg/sql/execstats/ @cockroachdb/cluster-observability
/pkg/sql/scheduledlogging/ @cockroachdb/cluster-observability
/pkg/sql/sqlstats/ @cockroachdb/cluster-observability
/pkg/ccl/testccl/sqlstatsccl/ @cockroachdb/cluster-observability

/pkg/sql/sem/tree/ @cockroachdb/sql-syntax-prs
/pkg/sql/parser/ @cockroachdb/sql-syntax-prs
Expand Down Expand Up @@ -138,7 +138,7 @@
/pkg/server/authentication*.go @cockroachdb/server-prs @cockroachdb/prodsec
/pkg/server/auto_tls_init*go @cockroachdb/server-prs @cockroachdb/prodsec
/pkg/server/clock_monotonicity.go @cockroachdb/kv-prs
/pkg/server/combined_statement_stats*.go @cockroachdb/sql-observability @cockroachdb/obs-inf-prs
/pkg/server/combined_statement_stats*.go @cockroachdb/cluster-observability @cockroachdb/obs-inf-prs
/pkg/server/decommission*.go @cockroachdb/kv-prs @cockroachdb/server-prs
/pkg/server/drain*.go @cockroachdb/kv-prs @cockroachdb/server-prs
/pkg/server/dumpstore/ @cockroachdb/obs-inf-prs @cockroachdb/server-prs
Expand All @@ -158,10 +158,10 @@
/pkg/server/server_controller_sql.go @cockroachdb/sql-sessions @cockroachdb/server-prs
/pkg/server/serverpb/ @cockroachdb/obs-inf-prs @cockroachdb/server-prs
/pkg/server/serverpb/authentication* @cockroachdb/obs-inf-prs @cockroachdb/prodsec @cockroachdb/server-prs
/pkg/server/serverpb/index_reco* @cockroachdb/sql-observability @cockroachdb/obs-inf-prs
/pkg/server/serverpb/index_reco* @cockroachdb/cluster-observability @cockroachdb/obs-inf-prs
/pkg/server/serverrules/ @cockroachdb/obs-inf-prs @cockroachdb/server-prs
/pkg/server/settingswatcher/ @cockroachdb/multi-tenant @cockroachdb/server-prs
/pkg/server/statements*.go @cockroachdb/sql-observability @cockroachdb/obs-inf-prs
/pkg/server/statements*.go @cockroachdb/cluster-observability @cockroachdb/obs-inf-prs
/pkg/server/status*go @cockroachdb/obs-inf-prs @cockroachdb/server-prs
/pkg/server/status*go @cockroachdb/obs-inf-prs @cockroachdb/server-prs
/pkg/server/status/ @cockroachdb/obs-inf-prs @cockroachdb/server-prs
Expand Down Expand Up @@ -308,7 +308,7 @@
/pkg/ccl/serverccl/ @cockroachdb/server-prs
/pkg/ccl/serverccl/server_sql* @cockroachdb/multi-tenant @cockroachdb/server-prs
/pkg/ccl/serverccl/tenant_* @cockroachdb/multi-tenant @cockroachdb/server-prs
/pkg/ccl/serverccl/statusccl @cockroachdb/sql-observability @cockroachdb/multi-tenant
/pkg/ccl/serverccl/statusccl @cockroachdb/cluster-observability @cockroachdb/multi-tenant
/pkg/ccl/telemetryccl/ @cockroachdb/obs-inf-prs
/pkg/ccl/testccl/authccl/ @cockroachdb/cloud-identity
/pkg/ccl/testccl/sqlccl/ @cockroachdb/sql-queries
Expand Down Expand Up @@ -410,7 +410,7 @@
/pkg/roachpb/.gitattributes @cockroachdb/dev-inf
#!/pkg/roachpb/BUILD.bazel @cockroachdb/kv-prs-noreview
/pkg/roachpb/data* @cockroachdb/kv-prs
/pkg/roachpb/index* @cockroachdb/sql-observability
/pkg/roachpb/index* @cockroachdb/cluster-observability
/pkg/roachpb/internal* @cockroachdb/kv-prs
/pkg/roachpb/io-formats* @cockroachdb/disaster-recovery
#!/pkg/roachpb/main_test.go @cockroachdb/kv-prs-noreview
Expand Down
2 changes: 1 addition & 1 deletion TEAMS.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ cockroachdb/sql-queries:
cockroachdb/sql-optimizer: other
cockroachdb/sql-opt-prs: other
triage_column_id: 13549252
cockroachdb/sql-observability:
cockroachdb/cluster-observability:
triage_column_id: 12618343
cockroachdb/kv:
aliases:
Expand Down
4 changes: 4 additions & 0 deletions pkg/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ ALL_TESTS = [
"//pkg/jobs/joberror:joberror_test",
"//pkg/jobs/jobsauth:jobsauth_test",
"//pkg/jobs/jobspb:jobspb_test",
"//pkg/jobs/jobsprofiler:jobsprofiler_test",
"//pkg/jobs/jobsprotectedts:jobsprotectedts_test",
"//pkg/jobs:jobs_test",
"//pkg/keys:keys_test",
Expand Down Expand Up @@ -1159,6 +1160,8 @@ GO_TARGETS = [
"//pkg/jobs/jobsauth:jobsauth_test",
"//pkg/jobs/jobspb:jobspb",
"//pkg/jobs/jobspb:jobspb_test",
"//pkg/jobs/jobsprofiler:jobsprofiler",
"//pkg/jobs/jobsprofiler:jobsprofiler_test",
"//pkg/jobs/jobsprotectedts:jobsprotectedts",
"//pkg/jobs/jobsprotectedts:jobsprotectedts_test",
"//pkg/jobs/jobstest:jobstest",
Expand Down Expand Up @@ -2633,6 +2636,7 @@ GET_X_DATA_TARGETS = [
"//pkg/jobs/joberror:get_x_data",
"//pkg/jobs/jobsauth:get_x_data",
"//pkg/jobs/jobspb:get_x_data",
"//pkg/jobs/jobsprofiler:get_x_data",
"//pkg/jobs/jobsprotectedts:get_x_data",
"//pkg/jobs/jobstest:get_x_data",
"//pkg/jobs/metricspoller:get_x_data",
Expand Down
7 changes: 6 additions & 1 deletion pkg/build/info.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ var (
binaryVersion = computeBinaryVersion(cockroachVersion, rev)
)

const (
DefaultTelemetryChannel = "official-binary"
FIPSTelemetryChannel = "official-fips-binary"
)

// IsRelease returns true if the binary was produced by a "release" build.
func IsRelease() bool {
return typ == "release"
Expand All @@ -53,7 +58,7 @@ func IsRelease() bool {
// SeemsOfficial reports whether this binary is likely to have come from an
// official release channel.
func SeemsOfficial() bool {
return channel == "official-binary"
return channel == DefaultTelemetryChannel || channel == FIPSTelemetryChannel
}

func computeBinaryVersion(versionTxt, revision string) string {
Expand Down
2 changes: 2 additions & 0 deletions pkg/ccl/backupccl/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ go_library(
"//pkg/jobs",
"//pkg/jobs/joberror",
"//pkg/jobs/jobspb",
"//pkg/jobs/jobsprofiler",
"//pkg/jobs/jobsprotectedts",
"//pkg/keys",
"//pkg/kv",
Expand Down Expand Up @@ -278,6 +279,7 @@ go_test(
"//pkg/testutils/datapathutils",
"//pkg/testutils/distsqlutils",
"//pkg/testutils/jobutils",
"//pkg/testutils/keysutils",
"//pkg/testutils/serverutils",
"//pkg/testutils/skip",
"//pkg/testutils/sqlutils",
Expand Down
8 changes: 8 additions & 0 deletions pkg/ccl/backupccl/backup_processor_planning.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backuppb"
"github.com/cockroachdb/cockroach/pkg/cloud"
"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
"github.com/cockroachdb/cockroach/pkg/jobs/jobsprofiler"
"github.com/cockroachdb/cockroach/pkg/kv"
"github.com/cockroachdb/cockroach/pkg/kv/kvpb"
"github.com/cockroachdb/cockroach/pkg/roachpb"
Expand Down Expand Up @@ -170,7 +171,11 @@ func distBackup(
// Setup a one-stage plan with one proc per input spec.
corePlacement := make([]physicalplan.ProcessorCorePlacement, len(backupSpecs))
i := 0
var jobID jobspb.JobID
for sqlInstanceID, spec := range backupSpecs {
if i == 0 {
jobID = jobspb.JobID(spec.JobID)
}
corePlacement[i].SQLInstanceID = sqlInstanceID
corePlacement[i].Core.BackupData = spec
i++
Expand Down Expand Up @@ -206,6 +211,9 @@ func distBackup(
defer recv.Release()

defer close(progCh)
execCfg := execCtx.ExecCfg()
jobsprofiler.StorePlanDiagram(ctx, execCfg.DistSQLSrv.Stopper, p, execCfg.InternalDB, jobID)

// Copy the evalCtx, as dsp.Run() might change it.
evalCtxCopy := *evalCtx
dsp.Run(ctx, planCtx, noTxn, p, recv, &evalCtxCopy, nil /* finishedSetupFn */)
Expand Down
3 changes: 2 additions & 1 deletion pkg/ccl/backupccl/backup_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/sql/stats"
"github.com/cockroachdb/cockroach/pkg/testutils"
"github.com/cockroachdb/cockroach/pkg/testutils/jobutils"
"github.com/cockroachdb/cockroach/pkg/testutils/keysutils"
"github.com/cockroachdb/cockroach/pkg/testutils/serverutils"
"github.com/cockroachdb/cockroach/pkg/testutils/skip"
"github.com/cockroachdb/cockroach/pkg/testutils/sqlutils"
Expand Down Expand Up @@ -6132,7 +6133,7 @@ func getMockTableDesc(
// methods.
func TestPublicIndexTableSpans(t *testing.T) {
defer leaktest.AfterTest(t)()
codec := keys.TODOSQLCodec
codec := keysutils.TestingSQLCodec
execCfg := &sql.ExecutorConfig{
Codec: codec,
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/ccl/backupccl/restore_job.go
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ func restore(
return distRestore(
ctx,
execCtx,
int64(job.ID()),
job.ID(),
dataToRestore,
endTime,
encryption,
Expand Down
10 changes: 7 additions & 3 deletions pkg/ccl/backupccl/restore_processor_planning.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/base"
"github.com/cockroachdb/cockroach/pkg/cloud"
"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
"github.com/cockroachdb/cockroach/pkg/jobs/jobsprofiler"
"github.com/cockroachdb/cockroach/pkg/kv"
"github.com/cockroachdb/cockroach/pkg/kv/kvpb"
"github.com/cockroachdb/cockroach/pkg/settings"
Expand Down Expand Up @@ -60,7 +61,7 @@ var replanRestoreFrequency = settings.RegisterDurationSetting(
func distRestore(
ctx context.Context,
execCtx sql.JobExecContext,
jobID int64,
jobID jobspb.JobID,
dataToRestore restorationData,
restoreTime hlc.Timestamp,
encryption *jobspb.BackupEncryptionOptions,
Expand Down Expand Up @@ -111,7 +112,7 @@ func distRestore(
p := planCtx.NewPhysicalPlan()

restoreDataSpec := execinfrapb.RestoreDataSpec{
JobID: jobID,
JobID: int64(jobID),
RestoreTime: restoreTime,
Encryption: fileEncryption,
TableRekeys: dataToRestore.getRekeys(),
Expand Down Expand Up @@ -184,7 +185,7 @@ func distRestore(
NumNodes: int64(numNodes),
UseSimpleImportSpans: useSimpleImportSpans,
UseFrontierCheckpointing: spanFilter.useFrontierCheckpointing,
JobID: jobID,
JobID: int64(jobID),
}
if spanFilter.useFrontierCheckpointing {
spec.CheckpointedSpans = persistFrontier(spanFilter.checkpointFrontier, 0)
Expand Down Expand Up @@ -292,6 +293,9 @@ func distRestore(
)
defer recv.Release()

execCfg := execCtx.ExecCfg()
jobsprofiler.StorePlanDiagram(ctx, execCfg.DistSQLSrv.Stopper, p, execCfg.InternalDB, jobID)

// Copy the evalCtx, as dsp.Run() might change it.
evalCtxCopy := *evalCtx
dsp.Run(ctx, planCtx, noTxn, p, recv, &evalCtxCopy, nil /* finishedSetupFn */)
Expand Down
1 change: 1 addition & 0 deletions pkg/ccl/changefeedccl/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ go_library(
"//pkg/jobs",
"//pkg/jobs/jobsauth",
"//pkg/jobs/jobspb",
"//pkg/jobs/jobsprofiler",
"//pkg/jobs/jobsprotectedts",
"//pkg/keys",
"//pkg/kv",
Expand Down
3 changes: 3 additions & 0 deletions pkg/ccl/changefeedccl/changefeed_dist.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeedbase"
"github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed"
"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
"github.com/cockroachdb/cockroach/pkg/jobs/jobsprofiler"
"github.com/cockroachdb/cockroach/pkg/kv"
"github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord"
"github.com/cockroachdb/cockroach/pkg/roachpb"
Expand Down Expand Up @@ -314,6 +315,8 @@ func startDistChangefeed(
finishedSetupFn = func(flowinfra.Flow) { resultsCh <- tree.Datums(nil) }
}

jobsprofiler.StorePlanDiagram(ctx, execCfg.DistSQLSrv.Stopper, p, execCfg.InternalDB, jobID)

// Copy the evalCtx, as dsp.Run() might change it.
evalCtxCopy := *evalCtx
// p is the physical plan, recv is the distsqlreceiver
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -437,18 +437,18 @@ func TestSQLWatcherOnEventError(t *testing.T) {
tdb.Exec(t, `SET CLUSTER SETTING kv.rangefeed.enabled = true`)
tdb.Exec(t, `SET CLUSTER SETTING kv.closed_timestamp.target_duration = '100ms'`)

noopCheckpointDuration := time.Hour // effectively disable no-op checkpoints
sqlWatcher := spanconfigsqlwatcher.New(
keys.SystemSQLCodec,
ts.ClusterSettings(),
ts.RangeFeedFactory().(*rangefeed.Factory),
1<<20, /* 1 MB, bufferMemLimit */
ts.Stopper(),
noopCheckpointDuration,
time.Second, // doesn't matter
&spanconfig.TestingKnobs{
SQLWatcherOnEventInterceptor: func() error {
return errors.New("boom")
},
SQLWatcherSkipNoopCheckpoints: true,
},
)

Expand Down
2 changes: 2 additions & 0 deletions pkg/ccl/streamingccl/streamingest/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ go_library(
"//pkg/cloud/externalconn/connectionpb",
"//pkg/jobs",
"//pkg/jobs/jobspb",
"//pkg/jobs/jobsprofiler",
"//pkg/jobs/jobsprotectedts",
"//pkg/keys",
"//pkg/kv",
Expand Down Expand Up @@ -135,6 +136,7 @@ go_test(
"//pkg/testutils/datapathutils",
"//pkg/testutils/distsqlutils",
"//pkg/testutils/jobutils",
"//pkg/testutils/keysutils",
"//pkg/testutils/serverutils",
"//pkg/testutils/skip",
"//pkg/testutils/sqlutils",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/ccl/streamingccl"
"github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamclient"
"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
"github.com/cockroachdb/cockroach/pkg/jobs/jobsprofiler"
"github.com/cockroachdb/cockroach/pkg/kv"
"github.com/cockroachdb/cockroach/pkg/repstream/streampb"
"github.com/cockroachdb/cockroach/pkg/roachpb"
Expand Down Expand Up @@ -110,7 +111,11 @@ func distStreamIngest(

// Setup a one-stage plan with one proc per input spec.
corePlacement := make([]physicalplan.ProcessorCorePlacement, len(streamIngestionSpecs))
var jobID jobspb.JobID
for i := range streamIngestionSpecs {
if i == 0 {
jobID = jobspb.JobID(streamIngestionSpecs[i].JobID)
}
corePlacement[i].SQLInstanceID = sqlInstanceIDs[i]
corePlacement[i].Core.StreamIngestionData = streamIngestionSpecs[i]
}
Expand Down Expand Up @@ -151,6 +156,8 @@ func distStreamIngest(
)
defer recv.Release()

jobsprofiler.StorePlanDiagram(ctx, execCfg.DistSQLSrv.Stopper, p, execCfg.InternalDB, jobID)

// Copy the evalCtx, as dsp.Run() might change it.
evalCtxCopy := *evalCtx
dsp.Run(ctx, planCtx, noTxn, p, recv, &evalCtxCopy, nil /* finishedSetupFn */)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/storage"
"github.com/cockroachdb/cockroach/pkg/testutils"
"github.com/cockroachdb/cockroach/pkg/testutils/distsqlutils"
"github.com/cockroachdb/cockroach/pkg/testutils/keysutils"
"github.com/cockroachdb/cockroach/pkg/testutils/storageutils"
"github.com/cockroachdb/cockroach/pkg/testutils/testcluster"
"github.com/cockroachdb/cockroach/pkg/util/hlc"
Expand Down Expand Up @@ -363,7 +364,7 @@ func assertEqualKVs(
tableID int,
partitionTimestamp hlc.Timestamp,
) {
key := keys.TODOSQLCodec.TablePrefix(uint32(tableID))
key := keysutils.TestingSQLCodec.TablePrefix(uint32(tableID))

// Iterate over the store.
store := tc.GetFirstStoreFromServer(t, 0)
Expand Down
8 changes: 7 additions & 1 deletion pkg/cmd/publish-artifacts/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ func run(providers []release.ObjectPutGetter, flags runFlags, execFn release.Exe
o.VersionStr = flags.sha
o.AbsolutePath = filepath.Join(flags.pkgDir, "cockroach"+release.SuffixFromPlatform(platform))
o.CockroachSQLAbsolutePath = filepath.Join(flags.pkgDir, "cockroach-sql"+release.SuffixFromPlatform(platform))
o.Channel = release.ChannelFromPlatform(platform)

log.Printf("building %s", pretty.Sprint(o))
buildOneCockroach(providers, o, execFn)
Expand All @@ -111,7 +112,11 @@ func run(providers []release.ObjectPutGetter, flags runFlags, execFn release.Exe

func buildOneCockroach(providers []release.ObjectPutGetter, o opts, execFn release.ExecFn) {
log.Printf("building cockroach %s", pretty.Sprint(o))
if err := release.MakeRelease(o.Platform, release.BuildOptions{ExecFn: execFn}, o.PkgDir); err != nil {
buildOpts := release.BuildOptions{
ExecFn: execFn,
Channel: o.Channel,
}
if err := release.MakeRelease(o.Platform, buildOpts, o.PkgDir); err != nil {
log.Fatal(err)
}
for _, provider := range providers {
Expand Down Expand Up @@ -160,4 +165,5 @@ type opts struct {
AbsolutePath string
CockroachSQLAbsolutePath string
PkgDir string
Channel string
}
10 changes: 5 additions & 5 deletions pkg/cmd/publish-artifacts/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ func TestPublish(t *testing.T) {
"env=[MALLOC_CONF=prof:true] args=./cockroach.linux-2.6.32-gnu-amd64 version",
"env=[] args=ldd ./cockroach.linux-2.6.32-gnu-amd64",
"env=[] args=bazel run @go_sdk//:bin/go -- tool nm ./cockroach.linux-2.6.32-gnu-amd64",
"env=[] args=bazel build //pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos '--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-binary' -c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"env=[] args=bazel build //pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos '--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-fips-binary' -c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"env=[] args=bazel info bazel-bin -c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"env=[MALLOC_CONF=prof:true] args=./cockroach.linux-2.6.32-gnu-amd64-fips version",
"env=[] args=ldd ./cockroach.linux-2.6.32-gnu-amd64-fips",
Expand Down Expand Up @@ -211,22 +211,22 @@ func TestPublish(t *testing.T) {
"gs://edge-binaries-bucket/cockroach/lib/libgeos_c.linux-gnu-amd64.so.LATEST/no-cache REDIRECT /cockroach/lib/libgeos_c.linux-gnu-amd64.1234567890abcdef.so",
"gs://edge-binaries-bucket/cockroach/cockroach.linux-gnu-amd64-fips.1234567890abcdef CONTENTS env=[] args=bazel build " +
"//pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-binary' " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-fips-binary' " +
"-c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"gs://edge-binaries-bucket/cockroach/cockroach.linux-gnu-amd64-fips.LATEST/no-cache REDIRECT /cockroach/cockroach.linux-gnu-amd64-fips.1234567890abcdef",
"gs://edge-binaries-bucket/cockroach/cockroach-sql.linux-gnu-amd64-fips.1234567890abcdef CONTENTS env=[] args=bazel build " +
"//pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-binary' " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-fips-binary' " +
"-c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"gs://edge-binaries-bucket/cockroach/cockroach-sql.linux-gnu-amd64-fips.LATEST/no-cache REDIRECT /cockroach/cockroach-sql.linux-gnu-amd64-fips.1234567890abcdef",
"gs://edge-binaries-bucket/cockroach/lib/libgeos.linux-gnu-amd64-fips.1234567890abcdef.so CONTENTS env=[] args=bazel build " +
"//pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-binary' " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-fips-binary' " +
"-c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"gs://edge-binaries-bucket/cockroach/lib/libgeos.linux-gnu-amd64-fips.so.LATEST/no-cache REDIRECT /cockroach/lib/libgeos.linux-gnu-amd64-fips.1234567890abcdef.so",
"gs://edge-binaries-bucket/cockroach/lib/libgeos_c.linux-gnu-amd64-fips.1234567890abcdef.so CONTENTS env=[] args=bazel build " +
"//pkg/cmd/cockroach //pkg/cmd/cockroach-sql //c-deps:libgeos " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-binary' " +
"'--workspace_status_command=./build/bazelutil/stamp.sh x86_64-pc-linux-gnu official-fips-binary' " +
"-c opt --config=ci --config=force_build_cdeps --config=with_ui --config=crosslinuxfipsbase",
"gs://edge-binaries-bucket/cockroach/lib/libgeos_c.linux-gnu-amd64-fips.so.LATEST/no-cache REDIRECT /cockroach/lib/libgeos_c.linux-gnu-amd64-fips.1234567890abcdef.so",
"gs://edge-binaries-bucket/cockroach/cockroach.linux-gnu-arm64.1234567890abcdef CONTENTS env=[] args=bazel build " +
Expand Down
Loading

0 comments on commit 0745cd4

Please sign in to comment.