backup/restore/ingest 2017Q1

[danhhz]

O: Robust SQL interface to Backup/Restore

• KR: Finalized SQL dialect
• KR: Read-only queries over backups by hooking up our SQL onto a read only kv store (stretch)
• KR: Point in time restore (stretch)
• General Robustness
  • Remove TestDisableTableLeases from the tests (@dt)
    • Error if the table or database exists (@dt)
  • Figure out permissions (@dt)
  • Audit restored TableDescriptors (e.g. FKs need to be marked invalid) (@dt)
    • Don't need to revalidate an fk if both tables involved were restored in the same command
  • Cloud storage credentials (@dt)
  • BACKUP DATABASE foo; DROP DATABASE foo; CREATE DATABASE foo; RESTORE DATABASE foo is broken (no database with ID 50 found). our current tests don't catch this because they restore into new clusters (@dt)
  • Test for tables with secondary indexes
  • Test the MVCCStats behavior of the WriteBatch impl
  • Fix the flaky tests
  • Implement ValidatePreviousBackups by calling restoreTablesRequests
  • Checksumming (sqlccl: figure out import/export checksums (for backup/restore) #13482)
  • Verify kv checksums in WriteBatch
  • Support placeholders in BACKUP's TO and RESTORE's FROM (sqlccl: support placeholders in BACKUP ... TO $1? #13624)
  • Finalize protos
  • Make the logs down the backup/restore calls all use rewritten start/end keys
  • Is TestMVCCIterateIncremental/intents3 correct?
  • [blocked] Test context cancellation
  • [blocked] During restore, wait for the newly created ranges (and leaseholders) to rebalance
• Known-bugs:
  • backup: network errors should be retried (backup: network errors should be retried #14044)
  • restore: TableDescriptors: unexpected value (restore: TableDescriptors: unexpected value #14045)
  • specifying bogus descriptors to BACKUP doesn't fail (backup: specifying bogus descriptors to BACKUP doesn't fail #14118)
  • backup: correctness issues in Import's construction of WriteBatches (backup: correctness issues in Import's construction of WriteBatches #14152)
  • empty SSTs backup: audit all possible zero-entry files #14178
  • RESTORE should return useful info sqlccl: BACKUP and RESTORE should return affected row counts #14296
  • SELECT * FROM system.jobs during a backup on lapis is slowwwww (also system.descriptor)

O: Admin UI for Backup/Restore

• KR: Backup restore state in jobs table (@benesch)

O: Stability and Performance

• KR: Restore 10-node cluster, 2T in 1 hour (@mjibson)
• KR: Integrate with production clusters and run regular backups (@mjibson)
• KR: Regularly (nightly) backup and then restore a production cluster (@mjibson)
• KR: Validate the restored data using a dump (@mjibson)
• KR: Validate restore using SQL fsck tool
• KR: Incremental backups on 10-node cluster of 2T to S3/GS sink (@mjibson)
• KR: Checkpoint restore and recover from a checkpoint (stretch)
• Benchmark incremental backup of a small time window over a large cluster
• Investigate poor restore performance in a geographically distributed cluster (one of the beta customers saw this when using arjun's benchmark data)

O: Enterprise Trailblazing

• KR: Move off of enterprise repo (@danhhz)
• KR: CCL + Apache2 binaries, testing infra (@dt)
• KR: Auditing enterprise feature use (@danhhz)

[danhhz]

Q1 is over. Closing in favor of https://github.com/cockroachdb/cockroach/projects/3 for Q2