Commits on Feb 9, 2021

1. streamingccl: improvements to the random stream test client …

   This change improves on the random stream client to allow for better
   testing of the various components of the stream ingestion job.
   Specifically:
   
   - Adds support for specifying number of partitions. For simplicity,
     a partition generates KVs for a particular table span.
   
   - Generates system KVs (descriptor and namespace) KVs, as the first two
     KVs on the partition stream. I played around with the idea of having a
   separate "system" and "table data" partition, but the code and tests
   became more convoluted, compared to the current approach.
   
   - Hookup the CDC orderValidator to the random stream client's output.
     This gives us some guarantees that the data being generated is
   semantically correct.
   
   - Maintain an in-memory copy of all the streamed events, that can be
     efficiently queried. This allows us to compare the ingested KVs to the
   streamed KVs and gain more confidence in our pipeline.
   
   Release note: None

   

   adityamaru committed Feb 9, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for 8247919
   • Browse repository at this point

   Copy the full SHA

   8247919 View commit details

   Browse the repository at this point in the history