Unexpectedly high memory consumption when using CassandraBatchTimeSeries workload

[bmatican]

Running an in-house test of our a batch workload on YCQL, we are seeing high memory pressure, hitting the soft memory limit.

From the /mem-trackers endpoint in one of the TS:

Id | Current Consumption | Peak consumption | Limit
-- | -- | -- | --
root | 12.57G | 12.57G | 12.35G

Corresponding log messages from peers:

E0326 16:41:54.524134 14680 process_context.cc:180] SQL Error: Execution Error. Write(tablet: 2a0fc6961b134f47a1da85a84d46d0ea, num_ops: 500, num_attempts: 89, txn: 00000000-0000-0000-0000-000000000000) passed its deadline 57676.014s (now: 57689.459s): Remote error (yb/rpc/outbound_call.cc:386): Service unavailable (yb/tserver/tablet_service.cc:239): Soft memory limit exceeded (at 101.64% of capacity)
INSERT INTO batch_ts_metrics_raw (metric_id, ts, value) VALUES (:metric_id, :ts, :value);
       ^^^^
W0326 16:41:54.525184 15093 cql_rpc.cc:271] CQL Call from xx.xx.xx.20:57000 took 73464ms. Details:
W0326 16:41:54.526329 15093 cql_rpc.cc:274] cql_details {
  type: "BATCH"
  call_details {
    sql_id: "a35b1d1d999509e2ab20a7e50d8fb5b3"
    sql_string: "INSERT INTO batch_ts_metrics_raw (metric_id, ts, value) VALUES (:metric_id, :ts, :value);"
  }
  call_details {
    sql_id: "a35b1d1d999509e2ab20a7e50d8fb5b3"
    sql_string: "INSERT INTO batch_ts_metrics_raw (metric_id, ts, value) VALUES (:metric_id, :ts, :value);"
  }
...

cc @kmuthukk