on] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,088][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,088][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,088][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,088][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 239 [2023-11-15T19:32:50,088][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,088][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61614, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 245, request id: 60, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 245B [2023-11-15T19:32:50,088][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 449, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,088][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,089][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,089][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s3] [449][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s2] [60][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,089][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [146b][indices:data/write/bulk[s]] to used bytes [new used: [146b], limit: 4294967296 [4gb], estimate: 292 [292b]] [2023-11-15T19:32:50,089][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61614} [length: 245, request id: 60, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 245B [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [60][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,089][TRACE][o.o.t.TaskManager ] [node_s5] register 660 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,089][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,089][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [450][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [450][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,089][TRACE][o.o.t.TaskManager ] [node_s5] register 661 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,089][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 449, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [449][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,089][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 b0 a1 e1 b1 8c 61 32 37]] (seq no [-2]) [2023-11-15T19:32:50,089][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [103] to [103], current checkpoint is higher [2023-11-15T19:32:50,089][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [102] to [103] [2023-11-15T19:32:50,089][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 659 [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [448][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,089][TRACE][o.o.t.T.tracer ] [node_s5] [448][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,089][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,089][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 658 [2023-11-15T19:32:50,090][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,090][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,090][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,090][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] [2023-11-15T19:32:50,090][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] [2023-11-15T19:32:50,090][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,090][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16586] [2023-11-15T19:32:50,090][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 304, request id: 451, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 304B [2023-11-15T19:32:50,090][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [146] [2023-11-15T19:32:50,090][TRACE][o.o.t.T.tracer ] [node_s5] [451][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,090][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,090][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [202b][indices:data/write/bulk[s][r]] to used bytes [new used: [202b], limit: 4294967296 [4gb], estimate: 404 [404b]] [2023-11-15T19:32:50,090][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 304, request id: 451, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 304B [2023-11-15T19:32:50,090][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [103] to [104] [2023-11-15T19:32:50,090][TRACE][o.o.t.T.tracer ] [node_s3] [451][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,090][TRACE][o.o.t.TaskManager ] [node_s3] register 240 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,090][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 b0 a1 e1 b1 8c 61 32 37]] (seq no [104]) [2023-11-15T19:32:50,091][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,091][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,091][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,091][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,091][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] [2023-11-15T19:32:50,091][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 240 [2023-11-15T19:32:50,091][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16642] [2023-11-15T19:32:50,091][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 451, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,091][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-202] bytes, now [16440] [2023-11-15T19:32:50,091][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,091][TRACE][o.o.t.T.tracer ] [node_s3] [451][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,091][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 451, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,091][TRACE][o.o.t.T.tracer ] [node_s5] [451][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,091][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [103] to [104] [2023-11-15T19:32:50,091][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [104] [2023-11-15T19:32:50,091][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,091][TRACE][o.o.t.TaskManager ] [node_s5] register 662 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,091][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,091][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,091][TRACE][o.o.t.T.tracer ] [node_s5] [452][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,091][TRACE][o.o.t.T.tracer ] [node_s5] [452][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 661 [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s5] [450][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s5] register 663 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s5] [450][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,092][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 660 [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16586] [2023-11-15T19:32:50,092][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61614} [length: 137, request id: 60, type: response, version: 3.0.0, header size: 48B] WRITE: 137B [2023-11-15T19:32:50,092][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-146] bytes, now [16440] [2023-11-15T19:32:50,092][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s5] [60][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,092][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61614, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 137, request id: 60, type: response, version: 3.0.0] READ: 137B [2023-11-15T19:32:50,092][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s2] [60][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,092][TRACE][o.o.a.b.TransportShardBulkAction] [node_s2] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰡ᱌a27]}]] [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 141 [2023-11-15T19:32:50,092][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 453, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 140 [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,092][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [104] to [104], current checkpoint is higher [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s5] [453][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,092][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [103] to [104] [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 664 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,092][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 220, request id: 453, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [node_s3] [453][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s3] register 241 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 665 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,092][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [103] to [104] due to [operation] [2023-11-15T19:32:50,092][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,092][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,092][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [testCommonTermsQuery] [454][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 241 [2023-11-15T19:32:50,092][TRACE][o.o.t.T.tracer ] [testCommonTermsQuery] [454][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,092][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 666 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,092][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 453, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,092][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s3] [453][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,093][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 453, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s5] [453][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,093][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f ea a2 b4 ea a3 87 ea a2 a6 61 33 30]] (seq no [-2]) [2023-11-15T19:32:50,093][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [104] to [104], current checkpoint is higher [2023-11-15T19:32:50,093][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [103] to [104] [2023-11-15T19:32:50,093][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 663 [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s5] [452][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s5] [452][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,093][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,093][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 662 [2023-11-15T19:32:50,093][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,093][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,093][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,093][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}]] [2023-11-15T19:32:50,093][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}]] [2023-11-15T19:32:50,093][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,093][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,093][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 318, request id: 455, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 318B [2023-11-15T19:32:50,093][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s5] [455][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,093][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [216b][indices:data/write/bulk[s][r]] to used bytes [new used: [216b], limit: 4294967296 [4gb], estimate: 432 [432b]] [2023-11-15T19:32:50,093][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,093][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 318, request id: 455, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 318B [2023-11-15T19:32:50,093][TRACE][o.o.t.T.tracer ] [node_s3] [455][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,093][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [104] to [105] [2023-11-15T19:32:50,093][TRACE][o.o.t.TaskManager ] [node_s3] register 242 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,094][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f ea a2 b4 ea a3 87 ea a2 a6 61 33 30]] (seq no [105]) [2023-11-15T19:32:50,094][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,094][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,094][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,094][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,094][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}]] [2023-11-15T19:32:50,094][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 242 [2023-11-15T19:32:50,094][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16656] [2023-11-15T19:32:50,094][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 455, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,094][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-216] bytes, now [16440] [2023-11-15T19:32:50,094][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,094][TRACE][o.o.t.T.tracer ] [node_s3] [455][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,094][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 455, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,094][TRACE][o.o.t.T.tracer ] [node_s5] [455][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,094][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [104] to [105] [2023-11-15T19:32:50,094][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [105] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s5] register 667 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,095][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s5] [456][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s5] [456][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 666 [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s5] [454][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s5] register 668 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s5] [454][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,095][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ꢴ꣇ꢦa30]}]] [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 665 [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 664 [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 120 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ႽჺჹႽႠa16]}] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,095][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,095][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,095][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 457, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 121 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,095][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [105] to [105], current checkpoint is higher [2023-11-15T19:32:50,095][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s5] [457][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,095][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,095][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [104] to [105] [2023-11-15T19:32:50,095][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,095][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,095][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61596, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 263, request id: 63, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 263B [2023-11-15T19:32:50,095][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 220, request id: 457, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,095][TRACE][o.o.t.T.tracer ] [node_s3] [457][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,095][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,095][TRACE][o.o.t.TaskManager ] [node_s3] register 243 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s4] [63][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,096][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [164b][indices:data/write/bulk[s]] to used bytes [new used: [164b], limit: 4294967296 [4gb], estimate: 328 [328b]] [2023-11-15T19:32:50,096][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61596} [length: 263, request id: 63, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 263B [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [63][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,096][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [104] to [105] due to [operation] [2023-11-15T19:32:50,096][TRACE][o.o.t.TaskManager ] [node_s5] register 669 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,096][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,096][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,096][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 243 [2023-11-15T19:32:50,096][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,096][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,096][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 457, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [458][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [458][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,096][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,096][TRACE][o.o.t.TaskManager ] [node_s5] register 670 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,096][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s3] [457][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,096][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 457, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [457][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,096][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 82 bd e1 83 ba e1 83 b9 e1 82 bd e1 82 a0 61 31 36]] (seq no [-2]) [2023-11-15T19:32:50,096][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [105] to [105], current checkpoint is higher [2023-11-15T19:32:50,096][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [104] to [105] [2023-11-15T19:32:50,096][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 668 [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [456][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,096][TRACE][o.o.t.T.tracer ] [node_s5] [456][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,096][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,096][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 667 [2023-11-15T19:32:50,096][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,096][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,096][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,096][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] [2023-11-15T19:32:50,096][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] [2023-11-15T19:32:50,097][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,097][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16604] [2023-11-15T19:32:50,097][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 331, request id: 459, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 331B [2023-11-15T19:32:50,097][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [164] [2023-11-15T19:32:50,097][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,097][TRACE][o.o.t.T.tracer ] [node_s5] [459][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,097][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [229b][indices:data/write/bulk[s][r]] to used bytes [new used: [229b], limit: 4294967296 [4gb], estimate: 458 [458b]] [2023-11-15T19:32:50,097][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 331, request id: 459, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 331B [2023-11-15T19:32:50,097][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [105] to [106] [2023-11-15T19:32:50,097][TRACE][o.o.t.T.tracer ] [node_s3] [459][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,097][TRACE][o.o.t.TaskManager ] [node_s3] register 244 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,097][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 82 bd e1 83 ba e1 83 b9 e1 82 bd e1 82 a0 61 31 36]] (seq no [106]) [2023-11-15T19:32:50,097][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,097][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,098][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,098][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,098][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] [2023-11-15T19:32:50,098][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 244 [2023-11-15T19:32:50,098][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16669] [2023-11-15T19:32:50,098][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 459, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,098][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-229] bytes, now [16440] [2023-11-15T19:32:50,098][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s3] [459][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,098][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 459, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s5] [459][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,098][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [105] to [106] [2023-11-15T19:32:50,098][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [106] [2023-11-15T19:32:50,098][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,098][TRACE][o.o.t.TaskManager ] [node_s5] register 671 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,098][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,098][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s5] [460][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s5] [460][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,098][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 670 [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s5] [458][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,098][TRACE][o.o.t.TaskManager ] [node_s5] register 672 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,098][TRACE][o.o.t.T.tracer ] [node_s5] [458][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,098][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] [2023-11-15T19:32:50,098][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 669 [2023-11-15T19:32:50,098][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16604] [2023-11-15T19:32:50,099][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61596} [length: 146, request id: 63, type: response, version: 3.0.0, header size: 48B] WRITE: 146B [2023-11-15T19:32:50,099][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [164] [2023-11-15T19:32:50,099][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,099][TRACE][o.o.t.T.tracer ] [node_s5] [63][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,099][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61596, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 146, request id: 63, type: response, version: 3.0.0] READ: 146B [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-164] bytes, now [0] [2023-11-15T19:32:50,099][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,099][TRACE][o.o.t.T.tracer ] [node_s4] [63][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,099][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 461, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,099][TRACE][o.o.a.b.TransportShardBulkAction] [node_s4] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ႽჺჹႽႠa16]}]] [2023-11-15T19:32:50,099][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 121 [2023-11-15T19:32:50,099][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 120 [2023-11-15T19:32:50,099][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [106] to [106], current checkpoint is higher [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,099][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [105] to [106] [2023-11-15T19:32:50,099][TRACE][o.o.t.T.tracer ] [node_s5] [461][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,099][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 245 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_𑂸aa9]}] [2023-11-15T19:32:50,099][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 220, request id: 461, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,099][TRACE][o.o.t.T.tracer ] [node_s3] [461][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,099][TRACE][o.o.t.TaskManager ] [node_s3] register 246 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,099][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 247 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,099][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,099][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [105] to [106] due to [operation] [2023-11-15T19:32:50,099][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,099][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,099][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61608, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 245, request id: 65, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 245B [2023-11-15T19:32:50,100][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,100][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 246 [2023-11-15T19:32:50,100][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32981] [2023-11-15T19:32:50,100][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16541] [2023-11-15T19:32:50,100][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 74, request id: 461, type: response, version: 3.0.0, header size: 47B] WRITE: 74B [2023-11-15T19:32:50,100][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s3] [65][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,100][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [146b][indices:data/write/bulk[s]] to used bytes [new used: [146b], limit: 4294967296 [4gb], estimate: 292 [292b]] [2023-11-15T19:32:50,100][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s3] [461][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,100][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61608} [length: 245, request id: 65, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 245B [2023-11-15T19:32:50,100][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 74, request id: 461, type: response, version: 3.0.0] READ: 74B [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [65][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [461][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,100][TRACE][o.o.t.TaskManager ] [node_s5] register 673 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,100][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [106] to [106], current checkpoint is higher [2023-11-15T19:32:50,100][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,100][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [105] to [106] [2023-11-15T19:32:50,100][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 672 [2023-11-15T19:32:50,100][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [460][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [460][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [462][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,100][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,100][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 671 [2023-11-15T19:32:50,100][TRACE][o.o.t.T.tracer ] [node_s5] [462][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,100][TRACE][o.o.t.TaskManager ] [node_s5] register 674 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,100][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 91 82 b8 61 61 39]] (seq no [-2]) [2023-11-15T19:32:50,101][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,101][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,101][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,101][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] [2023-11-15T19:32:50,101][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] [2023-11-15T19:32:50,101][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,101][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16586] [2023-11-15T19:32:50,101][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 304, request id: 463, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 304B [2023-11-15T19:32:50,101][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [146] [2023-11-15T19:32:50,101][TRACE][o.o.t.T.tracer ] [node_s5] [463][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,101][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,101][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [202b][indices:data/write/bulk[s][r]] to used bytes [new used: [202b], limit: 4294967296 [4gb], estimate: 404 [404b]] [2023-11-15T19:32:50,101][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 304, request id: 463, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 304B [2023-11-15T19:32:50,101][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [106] to [107] [2023-11-15T19:32:50,101][TRACE][o.o.t.T.tracer ] [node_s3] [463][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,101][TRACE][o.o.t.TaskManager ] [node_s3] register 248 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,102][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 91 82 b8 61 61 39]] (seq no [107]) [2023-11-15T19:32:50,102][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,102][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,102][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,102][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,102][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] [2023-11-15T19:32:50,102][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 248 [2023-11-15T19:32:50,102][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16642] [2023-11-15T19:32:50,102][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 463, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,102][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-202] bytes, now [16440] [2023-11-15T19:32:50,102][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,102][TRACE][o.o.t.T.tracer ] [node_s3] [463][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,102][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 463, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,102][TRACE][o.o.t.T.tracer ] [node_s5] [463][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,103][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [106] to [107] [2023-11-15T19:32:50,103][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [107] [2023-11-15T19:32:50,103][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s5] register 675 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,103][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,103][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s5] [464][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s5] [464][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 674 [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s5] [462][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s5] [462][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,103][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s5] register 676 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 673 [2023-11-15T19:32:50,103][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16586] [2023-11-15T19:32:50,103][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61608} [length: 137, request id: 65, type: response, version: 3.0.0, header size: 48B] WRITE: 137B [2023-11-15T19:32:50,103][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [146] [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s5] [65][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,103][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,103][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-146] bytes, now [0] [2023-11-15T19:32:50,103][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61608, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 137, request id: 65, type: response, version: 3.0.0] READ: 137B [2023-11-15T19:32:50,103][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,103][TRACE][o.o.t.T.tracer ] [node_s3] [65][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,103][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𑂸aa9]}]] [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 247 [2023-11-15T19:32:50,103][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 245 [2023-11-15T19:32:50,103][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,103][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 465, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,103][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 142 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}] [2023-11-15T19:32:50,103][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [107] to [107], current checkpoint is higher [2023-11-15T19:32:50,103][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,104][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [106] to [107] [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [465][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,104][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 220, request id: 465, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s3] [465][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 143 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,104][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s3] register 249 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,104][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,104][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [106] to [107] due to [operation] [2023-11-15T19:32:50,104][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61612, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 261, request id: 61, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 261B [2023-11-15T19:32:50,104][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 249 [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,104][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 465, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s2] [61][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [162b][indices:data/write/bulk[s]] to used bytes [new used: [162b], limit: 4294967296 [4gb], estimate: 324 [324b]] [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,104][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61612} [length: 261, request id: 61, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 261B [2023-11-15T19:32:50,104][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [61][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s3] [465][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,104][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 465, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s5] register 677 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [465][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,104][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,104][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [107] to [107], current checkpoint is higher [2023-11-15T19:32:50,104][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,104][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [106] to [107] [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 676 [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [466][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [464][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [466][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s5] register 678 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,104][TRACE][o.o.t.T.tracer ] [node_s5] [464][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,104][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,104][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 675 [2023-11-15T19:32:50,105][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f dc 86 dc 81 dc b3 dc 8b dc 9d dc bb dc 84 61 31 30]] (seq no [-2]) [2023-11-15T19:32:50,105][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,105][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,105][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,105][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] [2023-11-15T19:32:50,105][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] [2023-11-15T19:32:50,105][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,105][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16602] [2023-11-15T19:32:50,105][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 328, request id: 467, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 328B [2023-11-15T19:32:50,105][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [162] [2023-11-15T19:32:50,105][TRACE][o.o.t.T.tracer ] [node_s5] [467][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,106][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [226b][indices:data/write/bulk[s][r]] to used bytes [new used: [226b], limit: 4294967296 [4gb], estimate: 452 [452b]] [2023-11-15T19:32:50,106][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 328, request id: 467, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 328B [2023-11-15T19:32:50,106][TRACE][o.o.t.T.tracer ] [node_s3] [467][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,106][TRACE][o.o.t.TaskManager ] [node_s3] register 250 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,106][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f dc 86 dc 81 dc b3 dc 8b dc 9d dc bb dc 84 61 31 30]] (seq no [108]) [2023-11-15T19:32:50,106][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,106][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,106][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,106][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,106][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [107] to [108] [2023-11-15T19:32:50,106][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,107][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] [2023-11-15T19:32:50,107][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 250 [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16666] [2023-11-15T19:32:50,107][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 467, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-226] bytes, now [16440] [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s3] [467][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,107][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 467, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [467][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,107][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [107] to [108] [2023-11-15T19:32:50,107][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [108] [2023-11-15T19:32:50,107][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,107][TRACE][o.o.t.TaskManager ] [node_s5] register 679 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,107][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,107][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [468][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [468][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,107][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 678 [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [466][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,107][TRACE][o.o.t.TaskManager ] [node_s5] register 680 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [466][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,107][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] [2023-11-15T19:32:50,107][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 677 [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16602] [2023-11-15T19:32:50,107][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61612} [length: 145, request id: 61, type: response, version: 3.0.0, header size: 48B] WRITE: 145B [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [162] [2023-11-15T19:32:50,107][TRACE][o.o.t.T.tracer ] [node_s5] [61][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,107][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,107][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-162] bytes, now [0] [2023-11-15T19:32:50,107][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,107][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61612, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 145, request id: 61, type: response, version: 3.0.0] READ: 145B [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s2] [61][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,108][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,108][TRACE][o.o.a.b.TransportShardBulkAction] [node_s2] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_܆܁ܳ܋ܝܻ܄a10]}]] [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 143 [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 469, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 142 [2023-11-15T19:32:50,108][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [108] to [108], current checkpoint is higher [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,108][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [107] to [108] [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s5] [469][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 251 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}] [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 220, request id: 469, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s3] [469][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [node_s3] register 252 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 253 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,108][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,108][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [107] to [108] due to [operation] [2023-11-15T19:32:50,108][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,108][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,108][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 252 [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61604, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 280, request id: 66, type: request, version: 3.0.0, header size: 75B, action: indices:data/write/bulk[s]] WRITE: 280B [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32981] [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 469, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [32880] [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s3] [469][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,108][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [182b][indices:data/write/bulk[s]] to used bytes [new used: [182b], limit: 4294967296 [4gb], estimate: 364 [364b]] [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s3] [66][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61604} [length: 280, request id: 66, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 280B [2023-11-15T19:32:50,108][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 469, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,108][TRACE][o.o.t.T.tracer ] [node_s5] [469][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,109][TRACE][o.o.t.T.tracer ] [node_s5] [66][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,109][TRACE][o.o.t.TaskManager ] [node_s5] register 681 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,109][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [108] to [108], current checkpoint is higher [2023-11-15T19:32:50,109][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [107] to [108] [2023-11-15T19:32:50,109][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,109][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 680 [2023-11-15T19:32:50,109][TRACE][o.o.t.T.tracer ] [node_s5] [468][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,109][TRACE][o.o.t.T.tracer ] [node_s5] [468][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,109][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,109][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,109][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 679 [2023-11-15T19:32:50,109][TRACE][o.o.t.T.tracer ] [node_s5] [470][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,109][TRACE][o.o.t.T.tracer ] [node_s5] [470][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,109][TRACE][o.o.t.TaskManager ] [node_s5] register 682 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,109][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 88 95 e1 88 aa e1 8d 8f e1 8d ba e1 89 a2 e1 88 b4 e1 89 8d e1 8d a4 61 31 31]] (seq no [-2]) [2023-11-15T19:32:50,109][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,109][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,109][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,109][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] [2023-11-15T19:32:50,109][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] [2023-11-15T19:32:50,109][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,109][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,110][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 358, request id: 471, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 358B [2023-11-15T19:32:50,110][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [182] [2023-11-15T19:32:50,110][TRACE][o.o.t.T.tracer ] [node_s5] [471][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,110][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [256b][indices:data/write/bulk[s][r]] to used bytes [new used: [256b], limit: 4294967296 [4gb], estimate: 512 [512b]] [2023-11-15T19:32:50,110][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 358, request id: 471, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 358B [2023-11-15T19:32:50,110][TRACE][o.o.t.T.tracer ] [node_s3] [471][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,110][TRACE][o.o.t.TaskManager ] [node_s3] register 254 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,110][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 88 95 e1 88 aa e1 8d 8f e1 8d ba e1 89 a2 e1 88 b4 e1 89 8d e1 8d a4 61 31 31]] (seq no [109]) [2023-11-15T19:32:50,110][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,110][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,110][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,129][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,129][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,130][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] [2023-11-15T19:32:50,130][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 254 [2023-11-15T19:32:50,130][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16696] [2023-11-15T19:32:50,130][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [108] to [109] [2023-11-15T19:32:50,130][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 471, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,130][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-256] bytes, now [16440] [2023-11-15T19:32:50,130][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s3] [471][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,130][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 471, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s5] [471][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,130][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [108] to [109] [2023-11-15T19:32:50,130][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [109] [2023-11-15T19:32:50,130][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,130][TRACE][o.o.t.TaskManager ] [node_s5] register 683 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,130][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,130][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s5] [472][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s5] [472][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,130][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 682 [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s5] [470][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,130][TRACE][o.o.t.T.tracer ] [node_s5] [470][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,130][TRACE][o.o.t.TaskManager ] [node_s5] register 684 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,130][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] [2023-11-15T19:32:50,130][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 681 [2023-11-15T19:32:50,130][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,130][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61604} [length: 155, request id: 66, type: response, version: 3.0.0, header size: 48B] WRITE: 155B [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-182] bytes, now [16440] [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,131][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,131][TRACE][o.o.t.T.tracer ] [node_s5] [66][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,131][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,131][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61604, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 155, request id: 66, type: response, version: 3.0.0] READ: 155B [2023-11-15T19:32:50,131][TRACE][o.o.t.T.tracer ] [node_s3] [66][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,131][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,131][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ሕሪፏ፺ቢሴቍ፤a11]}]] [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 253 [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 251 [2023-11-15T19:32:50,131][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 473, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,131][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [109] to [109], current checkpoint is higher [2023-11-15T19:32:50,131][TRACE][o.o.t.T.tracer ] [node_s5] [473][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 202 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}] [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,131][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 220, request id: 473, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,131][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [108] to [109] [2023-11-15T19:32:50,131][TRACE][o.o.t.T.tracer ] [node_s3] [473][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [node_s3] register 255 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,131][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [108] to [109] due to [operation] [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 203 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,131][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,131][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,131][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,131][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 255 [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,131][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 281, request id: 343, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 281B [2023-11-15T19:32:50,131][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 473, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,131][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,132][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,132][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s3] [473][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s0] [343][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,132][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [182b][indices:data/write/bulk[s]] to used bytes [new used: [182b], limit: 4294967296 [4gb], estimate: 364 [364b]] [2023-11-15T19:32:50,132][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 473, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,132][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 281, request id: 343, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 281B [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [473][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [343][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,132][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [109] to [109], current checkpoint is higher [2023-11-15T19:32:50,132][TRACE][o.o.t.TaskManager ] [node_s5] register 685 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,132][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [108] to [109] [2023-11-15T19:32:50,132][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,132][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 684 [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [472][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,132][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [472][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [474][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,132][TRACE][o.o.t.T.tracer ] [node_s5] [474][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,132][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,132][TRACE][o.o.t.TaskManager ] [node_s5] register 686 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,132][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 683 [2023-11-15T19:32:50,132][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 9d 99 8e f0 9d 97 b2 f0 9d 99 86 f0 9d 9f 86 61 33 33]] (seq no [-2]) [2023-11-15T19:32:50,132][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,132][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,132][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,133][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] [2023-11-15T19:32:50,133][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] [2023-11-15T19:32:50,133][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,133][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,133][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 358, request id: 475, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 358B [2023-11-15T19:32:50,133][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [182] [2023-11-15T19:32:50,133][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,133][TRACE][o.o.t.T.tracer ] [node_s5] [475][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,133][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [109] to [110] [2023-11-15T19:32:50,133][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [256b][indices:data/write/bulk[s][r]] to used bytes [new used: [256b], limit: 4294967296 [4gb], estimate: 512 [512b]] [2023-11-15T19:32:50,133][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 358, request id: 475, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 358B [2023-11-15T19:32:50,133][TRACE][o.o.t.T.tracer ] [node_s3] [475][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,134][TRACE][o.o.t.TaskManager ] [node_s3] register 256 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,134][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 9d 99 8e f0 9d 97 b2 f0 9d 99 86 f0 9d 9f 86 61 33 33]] (seq no [110]) [2023-11-15T19:32:50,134][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,134][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,134][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,135][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,135][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] [2023-11-15T19:32:50,135][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 256 [2023-11-15T19:32:50,135][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16696] [2023-11-15T19:32:50,135][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 475, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,135][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-256] bytes, now [16440] [2023-11-15T19:32:50,135][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s3] [475][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,135][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 475, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s5] [475][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,135][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [109] to [110] [2023-11-15T19:32:50,135][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [110] [2023-11-15T19:32:50,135][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,135][TRACE][o.o.t.TaskManager ] [node_s5] register 687 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,135][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,135][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s5] [476][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s5] [476][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,135][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 686 [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s5] [474][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,135][TRACE][o.o.t.T.tracer ] [node_s5] [474][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,135][TRACE][o.o.t.TaskManager ] [node_s5] register 688 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,135][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] [2023-11-15T19:32:50,135][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 685 [2023-11-15T19:32:50,135][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 155, request id: 343, type: response, version: 3.0.0, header size: 48B] WRITE: 155B [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [182] [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s5] [343][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-182] bytes, now [0] [2023-11-15T19:32:50,136][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,136][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 155, request id: 343, type: response, version: 3.0.0] READ: 155B [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s0] [343][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,136][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,136][TRACE][o.o.a.b.TransportShardBulkAction] [node_s0] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𝙎𝗲𝙆𝟆a33]}]] [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 203 [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 202 [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 477, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,136][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [110] to [110], current checkpoint is higher [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 149 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}] [2023-11-15T19:32:50,136][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [109] to [110] [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s5] [477][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 220, request id: 477, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s3] [477][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 150 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [node_s3] register 257 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,136][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,136][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [109] to [110] due to [operation] [2023-11-15T19:32:50,136][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61595, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 269, request id: 63, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 269B [2023-11-15T19:32:50,136][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,136][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 257 [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,136][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 477, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s1] [63][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,136][TRACE][o.o.t.T.tracer ] [node_s3] [477][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,136][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [170b][indices:data/write/bulk[s]] to used bytes [new used: [170b], limit: 4294967296 [4gb], estimate: 340 [340b]] [2023-11-15T19:32:50,137][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61595} [length: 269, request id: 63, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 269B [2023-11-15T19:32:50,137][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 477, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [477][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [63][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,137][TRACE][o.o.t.TaskManager ] [node_s5] register 689 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,137][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,137][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [110] to [110], current checkpoint is higher [2023-11-15T19:32:50,137][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,137][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [109] to [110] [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [478][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [478][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,137][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 688 [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [476][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,137][TRACE][o.o.t.TaskManager ] [node_s5] register 690 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,137][TRACE][o.o.t.T.tracer ] [node_s5] [476][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,137][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,137][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 687 [2023-11-15T19:32:50,137][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 b0 a5 e1 b0 b7 e1 b0 87 e1 b0 8d e1 b0 bb e1 b0 80 61 33 39]] (seq no [-2]) [2023-11-15T19:32:50,137][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,137][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,137][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,138][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] [2023-11-15T19:32:50,138][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] [2023-11-15T19:32:50,138][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,138][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16610] [2023-11-15T19:32:50,138][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 340, request id: 479, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 340B [2023-11-15T19:32:50,138][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [170] [2023-11-15T19:32:50,138][TRACE][o.o.t.T.tracer ] [node_s5] [479][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,138][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,138][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [238b][indices:data/write/bulk[s][r]] to used bytes [new used: [238b], limit: 4294967296 [4gb], estimate: 476 [476b]] [2023-11-15T19:32:50,138][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 340, request id: 479, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 340B [2023-11-15T19:32:50,138][TRACE][o.o.t.T.tracer ] [node_s3] [479][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,138][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [110] to [111] [2023-11-15T19:32:50,138][TRACE][o.o.t.TaskManager ] [node_s3] register 258 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,138][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 b0 a5 e1 b0 b7 e1 b0 87 e1 b0 8d e1 b0 bb e1 b0 80 61 33 39]] (seq no [111]) [2023-11-15T19:32:50,139][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,139][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,139][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,139][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,139][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] [2023-11-15T19:32:50,139][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 258 [2023-11-15T19:32:50,139][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16678] [2023-11-15T19:32:50,139][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 479, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,139][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-238] bytes, now [16440] [2023-11-15T19:32:50,139][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,139][TRACE][o.o.t.T.tracer ] [node_s3] [479][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,139][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 479, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,139][TRACE][o.o.t.T.tracer ] [node_s5] [479][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,139][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [110] to [111] [2023-11-15T19:32:50,139][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [111] [2023-11-15T19:32:50,139][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,139][TRACE][o.o.t.TaskManager ] [node_s5] register 691 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,139][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,139][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,139][TRACE][o.o.t.T.tracer ] [node_s5] [480][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,139][TRACE][o.o.t.T.tracer ] [node_s5] [480][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,139][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 690 [2023-11-15T19:32:50,139][TRACE][o.o.t.T.tracer ] [node_s5] [478][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s5] [478][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,140][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s5] register 692 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 689 [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16610] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61595} [length: 149, request id: 63, type: response, version: 3.0.0, header size: 48B] WRITE: 149B [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [170] [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s5] [63][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-170] bytes, now [0] [2023-11-15T19:32:50,140][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61595, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 149, request id: 63, type: response, version: 3.0.0] READ: 149B [2023-11-15T19:32:50,140][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s1] [63][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,140][TRACE][o.o.a.b.TransportShardBulkAction] [node_s1] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᰥ᰷ᰇᰍ᰻ᰀa39]}]] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 150 [2023-11-15T19:32:50,140][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 149 [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 481, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,140][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [111] to [111], current checkpoint is higher [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 122 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}] [2023-11-15T19:32:50,140][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [110] to [111] [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s5] [481][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 220, request id: 481, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s3] [481][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s3] register 259 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 123 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,140][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [110] to [111] due to [operation] [2023-11-15T19:32:50,140][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,140][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,140][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 259 [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,140][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 481, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,140][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61597, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 261, request id: 64, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 261B [2023-11-15T19:32:50,140][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,140][TRACE][o.o.t.T.tracer ] [node_s3] [481][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,141][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,141][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 481, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s4] [64][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [481][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,141][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [162b][indices:data/write/bulk[s]] to used bytes [new used: [162b], limit: 4294967296 [4gb], estimate: 324 [324b]] [2023-11-15T19:32:50,141][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [111] to [111], current checkpoint is higher [2023-11-15T19:32:50,141][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61597} [length: 261, request id: 64, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 261B [2023-11-15T19:32:50,141][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [110] to [111] [2023-11-15T19:32:50,141][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 692 [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [480][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [64][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [480][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,141][TRACE][o.o.t.TaskManager ] [node_s5] register 693 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,141][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,141][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 691 [2023-11-15T19:32:50,141][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,141][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [482][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,141][TRACE][o.o.t.T.tracer ] [node_s5] [482][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,141][TRACE][o.o.t.TaskManager ] [node_s5] register 694 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,141][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f cc b9 cc a6 cc ba cc 9d cc 8e cd a1 cc ab 61 35 31]] (seq no [-2]) [2023-11-15T19:32:50,141][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,141][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,141][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,141][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] [2023-11-15T19:32:50,141][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] [2023-11-15T19:32:50,141][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,142][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16602] [2023-11-15T19:32:50,142][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 328, request id: 483, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 328B [2023-11-15T19:32:50,142][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,142][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [162] [2023-11-15T19:32:50,142][TRACE][o.o.t.T.tracer ] [node_s5] [483][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,142][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [111] to [112] [2023-11-15T19:32:50,142][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [226b][indices:data/write/bulk[s][r]] to used bytes [new used: [226b], limit: 4294967296 [4gb], estimate: 452 [452b]] [2023-11-15T19:32:50,142][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 328, request id: 483, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 328B [2023-11-15T19:32:50,142][TRACE][o.o.t.T.tracer ] [node_s3] [483][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,142][TRACE][o.o.t.TaskManager ] [node_s3] register 260 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,142][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f cc b9 cc a6 cc ba cc 9d cc 8e cd a1 cc ab 61 35 31]] (seq no [112]) [2023-11-15T19:32:50,142][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,142][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,142][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,143][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,143][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] [2023-11-15T19:32:50,143][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 260 [2023-11-15T19:32:50,143][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16666] [2023-11-15T19:32:50,143][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 483, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,143][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-226] bytes, now [16440] [2023-11-15T19:32:50,143][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s3] [483][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,143][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 483, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s5] [483][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [111] to [112] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [112] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,143][TRACE][o.o.t.TaskManager ] [node_s5] register 695 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,143][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s5] [484][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s5] [484][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,143][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 694 [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s5] [482][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,143][TRACE][o.o.t.TaskManager ] [node_s5] register 696 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,143][TRACE][o.o.t.T.tracer ] [node_s5] [482][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,143][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] [2023-11-15T19:32:50,143][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 693 [2023-11-15T19:32:50,143][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16602] [2023-11-15T19:32:50,143][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61597} [length: 145, request id: 64, type: response, version: 3.0.0, header size: 48B] WRITE: 145B [2023-11-15T19:32:50,143][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-162] bytes, now [16440] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,143][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,144][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,144][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,144][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 485, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,144][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [112] to [112], current checkpoint is higher [2023-11-15T19:32:50,144][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [111] to [112] [2023-11-15T19:32:50,144][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,144][TRACE][o.o.t.T.tracer ] [node_s5] [64][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,144][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,144][TRACE][o.o.t.T.tracer ] [node_s5] [485][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,144][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61597, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 145, request id: 64, type: response, version: 3.0.0] READ: 145B [2023-11-15T19:32:50,144][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,144][TRACE][o.o.t.T.tracer ] [node_s4] [64][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,144][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 220, request id: 485, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,144][TRACE][o.o.a.b.TransportShardBulkAction] [node_s4] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_̹̦̺̝̫̎͡a51]}]] [2023-11-15T19:32:50,144][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 123 [2023-11-15T19:32:50,144][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 122 [2023-11-15T19:32:50,144][TRACE][o.o.t.T.tracer ] [node_s3] [485][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s3] register 261 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 204 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}] [2023-11-15T19:32:50,145][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [111] to [112] due to [operation] [2023-11-15T19:32:50,145][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 261 [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,145][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 485, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 205 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s3] [485][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,145][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,145][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 485, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [485][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,145][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,145][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [112] to [112], current checkpoint is higher [2023-11-15T19:32:50,145][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 283, request id: 344, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 283B [2023-11-15T19:32:50,145][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [111] to [112] [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 696 [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [484][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [484][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,145][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 695 [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s0] [344][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,145][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [184b][indices:data/write/bulk[s]] to used bytes [new used: [184b], limit: 4294967296 [4gb], estimate: 368 [368b]] [2023-11-15T19:32:50,145][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 283, request id: 344, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 283B [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [344][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s5] register 697 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,145][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,145][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [486][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,145][TRACE][o.o.t.T.tracer ] [node_s5] [486][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,145][TRACE][o.o.t.TaskManager ] [node_s5] register 698 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,146][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 a7 9c a5 f0 a5 8a 8b f0 a1 82 af f0 a5 ab 81 61 61 35 34]] (seq no [-2]) [2023-11-15T19:32:50,146][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,146][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,146][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,146][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] [2023-11-15T19:32:50,146][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] [2023-11-15T19:32:50,146][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,146][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16624] [2023-11-15T19:32:50,146][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 361, request id: 487, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 361B [2023-11-15T19:32:50,146][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [184] [2023-11-15T19:32:50,146][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,146][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [259b][indices:data/write/bulk[s][r]] to used bytes [new used: [259b], limit: 4294967296 [4gb], estimate: 518 [518b]] [2023-11-15T19:32:50,146][TRACE][o.o.t.T.tracer ] [node_s5] [487][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,146][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 361, request id: 487, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 361B [2023-11-15T19:32:50,146][TRACE][o.o.t.T.tracer ] [node_s3] [487][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,146][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [112] to [113] [2023-11-15T19:32:50,146][TRACE][o.o.t.TaskManager ] [node_s3] register 262 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,147][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 a7 9c a5 f0 a5 8a 8b f0 a1 82 af f0 a5 ab 81 61 61 35 34]] (seq no [113]) [2023-11-15T19:32:50,147][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,147][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,147][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,147][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,147][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] [2023-11-15T19:32:50,147][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 262 [2023-11-15T19:32:50,147][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16699] [2023-11-15T19:32:50,147][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 487, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,147][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-259] bytes, now [16440] [2023-11-15T19:32:50,147][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,147][TRACE][o.o.t.T.tracer ] [node_s3] [487][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,147][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 487, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,147][TRACE][o.o.t.T.tracer ] [node_s5] [487][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,147][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [112] to [113] [2023-11-15T19:32:50,147][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [113] [2023-11-15T19:32:50,147][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,147][TRACE][o.o.t.TaskManager ] [node_s5] register 699 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,147][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,148][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [488][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [488][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 698 [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [486][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s5] register 700 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [486][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,148][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 697 [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16624] [2023-11-15T19:32:50,148][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 156, request id: 344, type: response, version: 3.0.0, header size: 48B] WRITE: 156B [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [184] [2023-11-15T19:32:50,148][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [344][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,148][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-184] bytes, now [0] [2023-11-15T19:32:50,148][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 156, request id: 344, type: response, version: 3.0.0] READ: 156B [2023-11-15T19:32:50,148][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s0] [344][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,148][TRACE][o.o.a.b.TransportShardBulkAction] [node_s0] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𧜥𥊋𡂯𥫁aa54]}]] [2023-11-15T19:32:50,148][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 489, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 205 [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 204 [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,148][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [113] to [113], current checkpoint is higher [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s5] [489][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 263 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ĜĒľŒŶa5]}] [2023-11-15T19:32:50,148][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [112] to [113] [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,148][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 220, request id: 489, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,148][TRACE][o.o.t.T.tracer ] [node_s3] [489][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s3] register 264 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,148][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [112] to [113] due to [operation] [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 265 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,148][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,148][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,148][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 264 [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,148][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 489, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,148][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32981] [2023-11-15T19:32:50,148][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [32880] [2023-11-15T19:32:50,149][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61616, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 251, request id: 67, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 251B [2023-11-15T19:32:50,149][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s3] [489][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,149][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 489, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,149][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s3] [67][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [489][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,149][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [113] to [113], current checkpoint is higher [2023-11-15T19:32:50,149][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [112] to [113] [2023-11-15T19:32:50,149][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 700 [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [488][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [488][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,149][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,149][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 699 [2023-11-15T19:32:50,149][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [152b][indices:data/write/bulk[s]] to used bytes [new used: [152b], limit: 4294967296 [4gb], estimate: 304 [304b]] [2023-11-15T19:32:50,149][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61616} [length: 251, request id: 67, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 251B [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [67][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,149][TRACE][o.o.t.TaskManager ] [node_s5] register 701 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,149][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,149][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [490][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,149][TRACE][o.o.t.T.tracer ] [node_s5] [490][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,149][TRACE][o.o.t.TaskManager ] [node_s5] register 702 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,149][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f c4 9c c4 92 c4 be c5 92 c5 b6 61 35]] (seq no [-2]) [2023-11-15T19:32:50,150][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,150][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,150][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,150][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] [2023-11-15T19:32:50,150][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] [2023-11-15T19:32:50,150][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,150][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16592] [2023-11-15T19:32:50,150][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 313, request id: 491, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 313B [2023-11-15T19:32:50,150][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [152] [2023-11-15T19:32:50,150][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,150][TRACE][o.o.t.T.tracer ] [node_s5] [491][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,150][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [211b][indices:data/write/bulk[s][r]] to used bytes [new used: [211b], limit: 4294967296 [4gb], estimate: 422 [422b]] [2023-11-15T19:32:50,150][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 313, request id: 491, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 313B [2023-11-15T19:32:50,151][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [113] to [114] [2023-11-15T19:32:50,151][TRACE][o.o.t.T.tracer ] [node_s3] [491][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,151][TRACE][o.o.t.TaskManager ] [node_s3] register 266 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,151][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f c4 9c c4 92 c4 be c5 92 c5 b6 61 35]] (seq no [114]) [2023-11-15T19:32:50,151][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,151][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,151][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,151][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,151][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] [2023-11-15T19:32:50,151][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 266 [2023-11-15T19:32:50,151][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16651] [2023-11-15T19:32:50,151][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 491, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,151][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-211] bytes, now [16440] [2023-11-15T19:32:50,151][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,151][TRACE][o.o.t.T.tracer ] [node_s3] [491][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,151][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 491, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [491][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [113] to [114] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [114] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s5] register 703 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,152][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [492][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [492][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 702 [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [490][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s5] register 704 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [490][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,152][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 701 [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16592] [2023-11-15T19:32:50,152][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61616} [length: 140, request id: 67, type: response, version: 3.0.0, header size: 48B] WRITE: 140B [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-152] bytes, now [16440] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [67][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,152][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,152][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61616, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 140, request id: 67, type: response, version: 3.0.0] READ: 140B [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s3] [67][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,152][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 493, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,152][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ĜĒľŒŶa5]}]] [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 265 [2023-11-15T19:32:50,152][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [114] to [114], current checkpoint is higher [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 263 [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,152][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [113] to [114] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s5] [493][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,152][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,152][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 220, request id: 493, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 124 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}] [2023-11-15T19:32:50,152][TRACE][o.o.t.T.tracer ] [node_s3] [493][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,152][TRACE][o.o.t.TaskManager ] [node_s3] register 267 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,153][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [113] to [114] due to [operation] [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 125 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,153][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,153][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 267 [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,153][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,153][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 493, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,153][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61593, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 281, request id: 65, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 281B [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s3] [493][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,153][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 493, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [493][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s4] [65][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,153][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [182b][indices:data/write/bulk[s]] to used bytes [new used: [182b], limit: 4294967296 [4gb], estimate: 364 [364b]] [2023-11-15T19:32:50,153][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [114] to [114], current checkpoint is higher [2023-11-15T19:32:50,153][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61593} [length: 281, request id: 65, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 281B [2023-11-15T19:32:50,153][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [113] to [114] [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 704 [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [492][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [65][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [492][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [node_s5] register 705 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,153][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 703 [2023-11-15T19:32:50,153][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,153][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [494][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,153][TRACE][o.o.t.T.tracer ] [node_s5] [494][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,153][TRACE][o.o.t.TaskManager ] [node_s5] register 706 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,153][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 ab 91 bb f0 ab 89 8c f0 aa ac 96 f0 aa b9 9f 61 31 39]] (seq no [-2]) [2023-11-15T19:32:50,154][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,154][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,154][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,154][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] [2023-11-15T19:32:50,154][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] [2023-11-15T19:32:50,154][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,154][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,154][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 358, request id: 495, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 358B [2023-11-15T19:32:50,154][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [182] [2023-11-15T19:32:50,154][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,154][TRACE][o.o.t.T.tracer ] [node_s5] [495][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,154][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [256b][indices:data/write/bulk[s][r]] to used bytes [new used: [256b], limit: 4294967296 [4gb], estimate: 512 [512b]] [2023-11-15T19:32:50,154][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 358, request id: 495, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 358B [2023-11-15T19:32:50,154][TRACE][o.o.t.T.tracer ] [node_s3] [495][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,154][TRACE][o.o.t.TaskManager ] [node_s3] register 268 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,155][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [114] to [115] [2023-11-15T19:32:50,155][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f f0 ab 91 bb f0 ab 89 8c f0 aa ac 96 f0 aa b9 9f 61 31 39]] (seq no [115]) [2023-11-15T19:32:50,155][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,155][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,155][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,155][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,155][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] [2023-11-15T19:32:50,155][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 268 [2023-11-15T19:32:50,155][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16696] [2023-11-15T19:32:50,155][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 495, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,155][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-256] bytes, now [16440] [2023-11-15T19:32:50,155][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,155][TRACE][o.o.t.T.tracer ] [node_s3] [495][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,155][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 495, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [495][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [114] to [115] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [115] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s5] register 707 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,156][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [496][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [496][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 706 [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [494][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [494][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s5] register 708 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,156][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 705 [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16622] [2023-11-15T19:32:50,156][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61593} [length: 155, request id: 65, type: response, version: 3.0.0, header size: 48B] WRITE: 155B [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-182] bytes, now [16440] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [65][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,156][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,156][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61593, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 155, request id: 65, type: response, version: 3.0.0] READ: 155B [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s4] [65][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,156][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 497, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,156][TRACE][o.o.a.b.TransportShardBulkAction] [node_s4] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_𫑻𫉌𪬖𪹟a19]}]] [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 125 [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,156][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [115] to [115], current checkpoint is higher [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 124 [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s5] [497][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,156][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [114] to [115] [2023-11-15T19:32:50,156][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,156][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 220, request id: 497, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,156][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 269 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}] [2023-11-15T19:32:50,156][TRACE][o.o.t.T.tracer ] [node_s3] [497][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s3] register 270 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,157][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [114] to [115] due to [operation] [2023-11-15T19:32:50,157][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 270 [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 271 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,157][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,157][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 497, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,157][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s3] [497][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,157][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 497, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,157][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61608, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 275, request id: 68, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 275B [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [497][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,157][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [115] to [115], current checkpoint is higher [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s3] [68][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,157][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [114] to [115] [2023-11-15T19:32:50,157][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [176b][indices:data/write/bulk[s]] to used bytes [new used: [176b], limit: 4294967296 [4gb], estimate: 352 [352b]] [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 708 [2023-11-15T19:32:50,157][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61608} [length: 275, request id: 68, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 275B [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [496][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [496][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,157][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [68][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 707 [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s5] register 709 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,157][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,157][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [498][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,157][TRACE][o.o.t.T.tracer ] [node_s5] [498][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,157][TRACE][o.o.t.TaskManager ] [node_s5] register 710 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,157][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 9d a3 e1 9d a4 e1 9d bd e1 9d a1 e1 9d ae e1 9d bf e1 9d b1 61 31 37]] (seq no [-2]) [2023-11-15T19:32:50,158][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,158][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,158][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,158][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] [2023-11-15T19:32:50,158][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] [2023-11-15T19:32:50,158][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,158][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16616] [2023-11-15T19:32:50,158][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 349, request id: 499, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 349B [2023-11-15T19:32:50,158][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [176] [2023-11-15T19:32:50,158][TRACE][o.o.t.T.tracer ] [node_s5] [499][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,158][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [247b][indices:data/write/bulk[s][r]] to used bytes [new used: [247b], limit: 4294967296 [4gb], estimate: 494 [494b]] [2023-11-15T19:32:50,158][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,158][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 349, request id: 499, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 349B [2023-11-15T19:32:50,158][TRACE][o.o.t.T.tracer ] [node_s3] [499][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,158][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [115] to [116] [2023-11-15T19:32:50,158][TRACE][o.o.t.TaskManager ] [node_s3] register 272 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,158][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f e1 9d a3 e1 9d a4 e1 9d bd e1 9d a1 e1 9d ae e1 9d bf e1 9d b1 61 31 37]] (seq no [116]) [2023-11-15T19:32:50,159][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,159][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,159][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,159][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,159][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] [2023-11-15T19:32:50,159][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 272 [2023-11-15T19:32:50,159][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16687] [2023-11-15T19:32:50,159][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 499, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,159][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-247] bytes, now [16440] [2023-11-15T19:32:50,159][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s3] [499][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,159][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 499, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s5] [499][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,159][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [115] to [116] [2023-11-15T19:32:50,159][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [116] [2023-11-15T19:32:50,159][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,159][TRACE][o.o.t.TaskManager ] [node_s5] register 711 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,159][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,159][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s5] [500][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s5] [500][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,159][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 710 [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s5] [498][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,159][TRACE][o.o.t.TaskManager ] [node_s5] register 712 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,159][TRACE][o.o.t.T.tracer ] [node_s5] [498][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,159][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] [2023-11-15T19:32:50,159][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 709 [2023-11-15T19:32:50,159][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16616] [2023-11-15T19:32:50,159][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61608} [length: 152, request id: 68, type: response, version: 3.0.0, header size: 48B] WRITE: 152B [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [176] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s5] [68][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,160][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-176] bytes, now [0] [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61608, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 152, request id: 68, type: response, version: 3.0.0] READ: 152B [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s3] [68][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,160][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,160][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ᝣᝤ᝽ᝡᝮ᝿᝱a17]}]] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 271 [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 501, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 269 [2023-11-15T19:32:50,160][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [116] to [116], current checkpoint is higher [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [115] to [116] [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s5] [501][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 144 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_a25]}] [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 220, request id: 501, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s3] [501][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s3] register 273 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [115] to [116] due to [operation] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 145 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,160][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 273 [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,160][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 501, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61613, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 233, request id: 62, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 233B [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s3] [501][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,160][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,160][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 501, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s2] [62][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s5] [501][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [116] to [116], current checkpoint is higher [2023-11-15T19:32:50,160][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [115] to [116] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 712 [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s5] [500][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,160][TRACE][o.o.t.T.tracer ] [node_s5] [500][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,160][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,160][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 711 [2023-11-15T19:32:50,161][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [134b][indices:data/write/bulk[s]] to used bytes [new used: [134b], limit: 4294967296 [4gb], estimate: 268 [268b]] [2023-11-15T19:32:50,161][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61613} [length: 233, request id: 62, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 233B [2023-11-15T19:32:50,161][TRACE][o.o.t.T.tracer ] [node_s5] [62][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,161][TRACE][o.o.t.TaskManager ] [node_s5] register 713 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,161][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,161][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,161][TRACE][o.o.t.T.tracer ] [node_s5] [502][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,161][TRACE][o.o.t.T.tracer ] [node_s5] [502][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,161][TRACE][o.o.t.TaskManager ] [node_s5] register 714 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,161][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f 61 32 35]] (seq no [-2]) [2023-11-15T19:32:50,161][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,161][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,161][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,161][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] [2023-11-15T19:32:50,161][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] [2023-11-15T19:32:50,161][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,161][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16574] [2023-11-15T19:32:50,161][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 286, request id: 503, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 286B [2023-11-15T19:32:50,162][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [134] [2023-11-15T19:32:50,162][TRACE][o.o.t.T.tracer ] [node_s5] [503][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,162][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [184b][indices:data/write/bulk[s][r]] to used bytes [new used: [184b], limit: 4294967296 [4gb], estimate: 368 [368b]] [2023-11-15T19:32:50,162][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,162][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 286, request id: 503, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 286B [2023-11-15T19:32:50,162][TRACE][o.o.t.T.tracer ] [node_s3] [503][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,162][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [116] to [117] [2023-11-15T19:32:50,162][TRACE][o.o.t.TaskManager ] [node_s3] register 274 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,162][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f 61 32 35]] (seq no [117]) [2023-11-15T19:32:50,162][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,162][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,162][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,162][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,162][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] [2023-11-15T19:32:50,162][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 274 [2023-11-15T19:32:50,162][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16624] [2023-11-15T19:32:50,162][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 503, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,162][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-184] bytes, now [16440] [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s3] [503][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,163][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 503, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [503][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [116] to [117] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [117] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s5] register 715 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,163][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [504][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [504][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 714 [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [502][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s5] register 716 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [502][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,163][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 713 [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16574] [2023-11-15T19:32:50,163][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61613} [length: 131, request id: 62, type: response, version: 3.0.0, header size: 48B] WRITE: 131B [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-134] bytes, now [16440] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [62][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,163][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61613, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 131, request id: 62, type: response, version: 3.0.0] READ: 131B [2023-11-15T19:32:50,163][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s2] [62][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,163][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 505, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,163][TRACE][o.o.a.b.TransportShardBulkAction] [node_s2] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_a25]}]] [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 145 [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 144 [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,163][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [117] to [117], current checkpoint is higher [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s5] [505][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,163][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [116] to [117] [2023-11-15T19:32:50,163][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,163][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 126 [transport] [indices:data/write/delete] [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}] [2023-11-15T19:32:50,163][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 220, request id: 505, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,163][TRACE][o.o.t.T.tracer ] [node_s3] [505][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s3] register 275 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,164][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [116] to [117] due to [operation] [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 127 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,164][TRACE][o.o.a.b.TransportShardBulkAction] [testCommonTermsQuery] send action [indices:data/write/bulk[s]] on primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,164][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 275 [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,164][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 505, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,164][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/write/bulk[s] [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,164][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61596, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 275, request id: 66, type: request, version: 3.0.0, header size: 76B, action: indices:data/write/bulk[s]] WRITE: 275B [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s3] [505][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,164][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 505, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [505][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s4] [66][indices:data/write/bulk[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,164][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [117] to [117], current checkpoint is higher [2023-11-15T19:32:50,164][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [116] to [117] [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 716 [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [504][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [504][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,164][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 715 [2023-11-15T19:32:50,164][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [176b][indices:data/write/bulk[s]] to used bytes [new used: [176b], limit: 4294967296 [4gb], estimate: 352 [352b]] [2023-11-15T19:32:50,164][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61596} [length: 275, request id: 66, type: request, version: 3.0.0, action: indices:data/write/bulk[s]] READ: 275B [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [66][indices:data/write/bulk[s]] received request [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s5] register 717 [transport] [indices:data/write/bulk[s]] [requests[1], index[test][0]] [2023-11-15T19:32:50,164][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] send action [indices:data/write/bulk[s][p]] to local primary [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,164][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][p] [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [506][indices:data/write/bulk[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,164][TRACE][o.o.t.T.tracer ] [node_s5] [506][indices:data/write/bulk[s][p]] received request [2023-11-15T19:32:50,164][TRACE][o.o.t.TaskManager ] [node_s5] register 718 [direct] [indices:data/write/bulk[s][p]] [requests[1], index[test][0]] [2023-11-15T19:32:50,165][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f ed 9f b6 ed 9f 9b ed 9f a1 ed 9f 8d ed 9f 80 ed 9f a4 ed 9f 93 61 33 36]] (seq no [-2]) [2023-11-15T19:32:50,165][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,165][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,165][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,165][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] [2023-11-15T19:32:50,165][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] [[test][0]] sending op [indices:data/write/bulk[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] [2023-11-15T19:32:50,165][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:data/write/bulk[s][r] [2023-11-15T19:32:50,165][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16616] [2023-11-15T19:32:50,165][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 349, request id: 507, type: request, version: 3.0.0, header size: 79B, action: indices:data/write/bulk[s][r]] WRITE: 349B [2023-11-15T19:32:50,165][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [176] [2023-11-15T19:32:50,165][TRACE][o.o.t.T.tracer ] [node_s5] [507][indices:data/write/bulk[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,165][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [247b][indices:data/write/bulk[s][r]] to used bytes [new used: [247b], limit: 4294967296 [4gb], estimate: 494 [494b]] [2023-11-15T19:32:50,165][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,165][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 349, request id: 507, type: request, version: 3.0.0, action: indices:data/write/bulk[s][r]] READ: 349B [2023-11-15T19:32:50,165][TRACE][o.o.t.T.tracer ] [node_s3] [507][indices:data/write/bulk[s][r]] received request [2023-11-15T19:32:50,166][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [117] to [118] [2023-11-15T19:32:50,166][TRACE][o.o.t.TaskManager ] [node_s3] register 276 [transport] [indices:data/write/bulk[s][r]] [requests[1], index[test][0]] [2023-11-15T19:32:50,166][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] delete [[ff 62 6f 67 75 73 5f 64 6f 63 5f ed 9f b6 ed 9f 9b ed 9f a1 ed 9f 8d ed 9f 80 ed 9f a4 ed 9f 93 61 33 36]] (seq no [118]) [2023-11-15T19:32:50,166][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,166][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [32880] [2023-11-15T19:32:50,166][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [16440] [2023-11-15T19:32:50,166][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,166][TRACE][o.o.a.b.TransportShardBulkAction] [node_s3] action [indices:data/write/bulk[s][r]] completed on shard [[test][0]] for request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] [2023-11-15T19:32:50,166][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 276 [2023-11-15T19:32:50,166][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16687] [2023-11-15T19:32:50,166][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61600} [length: 75, request id: 507, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,166][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-247] bytes, now [16440] [2023-11-15T19:32:50,166][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,166][TRACE][o.o.t.T.tracer ] [node_s3] [507][indices:data/write/bulk[s][r]] sent response [2023-11-15T19:32:50,166][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61600, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 507, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,166][TRACE][o.o.t.T.tracer ] [node_s5] [507][indices:data/write/bulk[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,167][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [117] to [118] [2023-11-15T19:32:50,167][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated global checkpoint to [118] [2023-11-15T19:32:50,167][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] syncing global checkpoint for [post-operation] [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s5] register 719 [transport] [indices:admin/seq_no/global_checkpoint_sync] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=DEFAULT}] [2023-11-15T19:32:50,167][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] send action [indices:admin/seq_no/global_checkpoint_sync[p]] to local primary [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,167][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[p] [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [508][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [508][indices:admin/seq_no/global_checkpoint_sync[p]] received request [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 718 [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [506][indices:data/write/bulk[s][p]] sent response [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s5] register 720 [direct] [indices:admin/seq_no/global_checkpoint_sync[p]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [506][indices:data/write/bulk[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,167][TRACE][o.o.a.b.TransportShardBulkAction] [node_s5] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 717 [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16616] [2023-11-15T19:32:50,167][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61596} [length: 152, request id: 66, type: response, version: 3.0.0, header size: 48B] WRITE: 152B [2023-11-15T19:32:50,167][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] op [indices:admin/seq_no/global_checkpoint_sync] completed on primary for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,167][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] [[test][0]] sending op [indices:admin/seq_no/global_checkpoint_sync] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [176] [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [66][indices:data/write/bulk[s]] sent response [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-176] bytes, now [0] [2023-11-15T19:32:50,167][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61596, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 152, request id: 66, type: response, version: 3.0.0] READ: 152B [2023-11-15T19:32:50,167][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/seq_no/global_checkpoint_sync[r] [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s4] [66][indices:data/write/bulk[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,167][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 220, request id: 509, type: request, version: 3.0.0, header size: 96B, action: indices:admin/seq_no/global_checkpoint_sync[r]] WRITE: 220B [2023-11-15T19:32:50,167][TRACE][o.o.a.b.TransportShardBulkAction] [node_s4] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[test][0]] containing [delete {[test][bogus_doc_ퟶퟛퟡퟍퟀퟤퟓa36]}]] [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 127 [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 126 [2023-11-15T19:32:50,167][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [118] to [118], current checkpoint is higher [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,167][TRACE][o.o.t.T.tracer ] [node_s5] [509][indices:admin/seq_no/global_checkpoint_sync[r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,167][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [oIyDaBU5RJCXL98jM8pOHg] on the primary of the global checkpoint from [117] to [118] [2023-11-15T19:32:50,167][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/seq_no/global_checkpoint_sync[r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,167][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 206 [transport] [cluster:monitor/health] [] [2023-11-15T19:32:50,167][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 220, request id: 509, type: request, version: 3.0.0, action: indices:admin/seq_no/global_checkpoint_sync[r]] READ: 220B [2023-11-15T19:32:50,167][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.health.ClusterHealthRequest@770a9554] with cluster state version [24] [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s3] [509][indices:admin/seq_no/global_checkpoint_sync[r]] received request [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [node_s3] register 277 [transport] [indices:admin/seq_no/global_checkpoint_sync[r]] [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,168][TRACE][o.o.c.s.MasterService ] [node_s0] will process [cluster_health (wait_for_events [LANGUID])] [2023-11-15T19:32:50,168][TRACE][o.o.i.s.ReplicationTracker] [node_s3] [test][0] updated global checkpoint from [117] to [118] due to [operation] [2023-11-15T19:32:50,168][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [cluster_health (wait_for_events [LANGUID])] [2023-11-15T19:32:50,168][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to compute cluster state update for [cluster_health (wait_for_events [LANGUID])] [2023-11-15T19:32:50,168][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s3] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[test][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 277 [2023-11-15T19:32:50,168][TRACE][o.o.a.a.c.h.TransportClusterHealthAction] [node_s0] Calculating health based on state version [24] [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,168][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61602} [length: 75, request id: 509, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s3] [509][indices:admin/seq_no/global_checkpoint_sync[r]] sent response [2023-11-15T19:32:50,168][TRACE][o.o.a.a.c.h.TransportClusterHealthAction] [node_s0] Calculating health based on state version [24] [2023-11-15T19:32:50,168][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61602, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 509, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s5] [509][indices:admin/seq_no/global_checkpoint_sync[r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 206 [2023-11-15T19:32:50,168][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [118] to [118], current checkpoint is higher [2023-11-15T19:32:50,168][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on unchanged cluster state for [cluster_health (wait_for_events [LANGUID])] [2023-11-15T19:32:50,168][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] updated local knowledge for [xRjt2CCHQuiHlKD45x631A] on the primary of the global checkpoint from [117] to [118] [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 720 [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s5] [508][indices:admin/seq_no/global_checkpoint_sync[p]] sent response [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 207 [transport] [indices:admin/refresh] [] [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s5] [508][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,168][TRACE][o.o.i.s.GlobalCheckpointSyncAction] [node_s5] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[test][0], timeout=1m, index='test', waitForActiveShards=1}] [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 719 [2023-11-15T19:32:50,168][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 208 [transport] [indices:admin/refresh[s]] [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,168][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [testCommonTermsQuery] send action [indices:admin/refresh[s]] on primary [[test][0]] for request [BasicReplicationRequest{[test][0]}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,168][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:admin/refresh[s] [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,168][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 170, request id: 345, type: request, version: 3.0.0, header size: 74B, action: indices:admin/refresh[s]] WRITE: 170B [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,168][TRACE][o.o.t.T.tracer ] [node_s0] [345][indices:admin/refresh[s]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,168][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adding [73b][indices:admin/refresh[s]] to used bytes [new used: [73b], limit: 4294967296 [4gb], estimate: 146 [146b]] [2023-11-15T19:32:50,168][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 170, request id: 345, type: request, version: 3.0.0, action: indices:admin/refresh[s]] READ: 170B [2023-11-15T19:32:50,169][TRACE][o.o.t.T.tracer ] [node_s5] [345][indices:admin/refresh[s]] received request [2023-11-15T19:32:50,169][TRACE][o.o.t.TaskManager ] [node_s5] register 721 [transport] [indices:admin/refresh[s]] [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,169][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s5] send action [indices:admin/refresh[s][p]] to local primary [[test][0]] for request [BasicReplicationRequest{[test][0]}] with cluster state version [24] to [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,169][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/refresh[s][p] [2023-11-15T19:32:50,169][TRACE][o.o.t.T.tracer ] [node_s5] [510][indices:admin/refresh[s][p]] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,169][TRACE][o.o.t.T.tracer ] [node_s5] [510][indices:admin/refresh[s][p]] received request [2023-11-15T19:32:50,169][TRACE][o.o.t.TaskManager ] [node_s5] register 722 [direct] [indices:admin/refresh[s][p]] [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,169][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] refresh with source [api] [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DW: anyChanges? numDocsInRam=54 deletes=true hasTickets:false pendingChangesInFullFlush: false [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: flush at getReader [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DW: startFullFlush [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DW: anyChanges? numDocsInRam=54 deletes=true hasTickets:false pendingChangesInFullFlush: false [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: compressed 33414 to 33430 bytes (100.05%) for deletes/updates; private segment null [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: flush postings as segment _9 numDocs=54 [2023-11-15T19:32:50,169][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 0 ms to write norms [2023-11-15T19:32:50,172][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 3 ms to write docValues [2023-11-15T19:32:50,174][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 1 ms to write points [2023-11-15T19:32:50,174][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 0 ms to write vectors [2023-11-15T19:32:50,176][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90FieldsIndex-doc_ids_i.tmp [2023-11-15T19:32:50,177][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90FieldsIndexfile_pointers_j.tmp [2023-11-15T19:32:50,177][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 3 ms to finish stored fields [2023-11-15T19:32:50,180][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 2 ms to write postings and finish vectors [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: 0 ms to write fieldInfos [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: new segment has 0 deleted docs [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: new segment has 54 soft-deleted docs [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: new segment has no vectors; no norms; docValues; no prox; freqs [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: flushedFiles=[_9.kdd, _9_Lucene90_1.dvm, _9_Asserting_0.dvd, _9.fnm, _9.kdi, _9.fdm, _9_Lucene90_1.dvd, _9_Asserting_0.tmd, _9.fdx, _9.fdt, _9.kdm, _9_Lucene90_0.dvm, _9_Lucene90_0.dvd, _9_Asserting_0.doc, _9_Asserting_0.tim, _9_Asserting_0.dvm, _9_Asserting_0.tip] [2023-11-15T19:32:50,181][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: flushed codec=Asserting(Lucene95): {_routing=PostingsFormat(name=Asserting), index_uuid=PostingsFormat(name=Asserting), field1=PostingsFormat(name=Asserting), _field_names=Lucene90, field2.keyword=Lucene90, _id=PostingsFormat(name=Asserting), type=Lucene90, field2=Lucene90}, docValues:{__soft_deletes=DocValuesFormat(name=Lucene90), _seq_no=DocValuesFormat(name=Lucene90), _tombstone=DocValuesFormat(name=Asserting), field2.keyword=DocValuesFormat(name=Lucene90), _primary_term=DocValuesFormat(name=Lucene90), _version=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=667, maxMBSortInHeap=6.321628492792106 [2023-11-15T19:32:50,184][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: flushed: segment=_9 ramUsed=0.11 MB newFlushedSize=0.005 MB docs/MB=11,415.948 [2023-11-15T19:32:50,184][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: compressed 33650 to 33666 bytes (100.05%) for deletes/updates; private segment _9(9.8.0):C54:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6 [2023-11-15T19:32:50,184][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: create compound file [2023-11-15T19:32:50,191][TRACE][o.o.i.e.E.DWPT ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DWPT: flush time 22.137156 ms [2023-11-15T19:32:50,191][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: publishFlushedSegment seg-private updates=delGen=-1 fieldUpdates=54 bytesUsed=33666 privateSegment=_9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6 [2023-11-15T19:32:50,191][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: flush: push buffered seg private updates: delGen=-1 fieldUpdates=54 bytesUsed=33666 privateSegment=_9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6 [2023-11-15T19:32:50,191][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: publishFlushedSegment _9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6 [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: push new packet (delGen=11 fieldUpdates=54 bytesUsed=33430), packetCount=1, bytesUsed=0.032 MB [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: push new packet (delGen=12 fieldUpdates=54 bytesUsed=33666 privateSegment=_9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6), packetCount=2, bytesUsed=0.064 MB [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: publish sets newSegment delGen=12 seg=_9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl6 [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now checkpoint "_0(9.8.0):c2:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nj8 _4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nja _5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njh _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njl _8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk6 _7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nka _9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7" [10 segments ; isCommit = false] [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,192][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: 0 ms to checkpoint [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.kdd" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Lucene90_1.dvm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.dvd" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.fnm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.kdi" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.fdm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Lucene90_1.dvd" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.tmd" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.fdx" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.fdt" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9.kdm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Lucene90_0.dvm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Lucene90_0.dvd" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.doc" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.tim" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.dvm" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: will delete new file "_9_Asserting_0.tip" [2023-11-15T19:32:50,193][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 17 files: [_9.kdd, _9_Lucene90_1.dvm, _9_Asserting_0.dvd, _9.fnm, _9.kdi, _9.fdm, _9_Lucene90_1.dvd, _9_Asserting_0.tmd, _9.fdx, _9.fdt, _9.kdm, _9_Lucene90_0.dvm, _9_Lucene90_0.dvd, _9_Asserting_0.doc, _9_Asserting_0.tim, _9_Asserting_0.dvm, _9_Asserting_0.tip] [2023-11-15T19:32:50,194][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.kdd [2023-11-15T19:32:50,194][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90_1.dvm [2023-11-15T19:32:50,195][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.dvd [2023-11-15T19:32:50,195][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.fnm [2023-11-15T19:32:50,196][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.kdi [2023-11-15T19:32:50,196][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.fdm [2023-11-15T19:32:50,197][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90_1.dvd [2023-11-15T19:32:50,198][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.tmd [2023-11-15T19:32:50,198][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.fdx [2023-11-15T19:32:50,199][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.fdt [2023-11-15T19:32:50,199][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.kdm [2023-11-15T19:32:50,200][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90_0.dvm [2023-11-15T19:32:50,201][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Lucene90_0.dvd [2023-11-15T19:32:50,201][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.doc [2023-11-15T19:32:50,202][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.tim [2023-11-15T19:32:50,203][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.dvm [2023-11-15T19:32:50,203][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9_Asserting_0.tip [2023-11-15T19:32:50,204][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 9 segments, mergeGen 0 [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: applyDocValuesUpdates 0.8 msec for 9 segments, 54 field updates; 54 new updates [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: closeSegmentStates: 0 new deleted documents; pool 2 packets; bytesUsed=1212 [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done inner apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 9 segments; 54 new deletes/updates; took 0.001 sec [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: finished packet delGen=11 now completedDelGen=11 [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 9 segments; 54 new deletes/updates; took 0.001 sec; 1 packets remain [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 1 segments, mergeGen 0 [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: applyDocValuesUpdates 0.5 msec for 1 segments, 54 field updates; 0 new updates [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: closeSegmentStates: 0 new deleted documents; pool 1 packets; bytesUsed=1212 [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done inner apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 1 segments; 0 new deletes/updates; took 0.001 sec [2023-11-15T19:32:50,205][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: finished packet delGen=12 now completedDelGen=12 [2023-11-15T19:32:50,206][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@c1dfe60) to 1 segments; 0 new deletes/updates; took 0.001 sec; 0 packets remain [2023-11-15T19:32:50,206][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=1212 [2023-11-15T19:32:50,206][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: waitApply: no deletes to apply [2023-11-15T19:32:50,206][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njl, bytes=0.000 MB [2023-11-15T19:32:50,209][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9; took 0.003s; new files: {8=[_6_1_Lucene90_0.dvm, _6_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,209][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nja, bytes=0.000 MB [2023-11-15T19:32:50,211][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb; took 0.003s; new files: {8=[_4_1_Lucene90_0.dvm, _4_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,212][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk6, bytes=0.000 MB [2023-11-15T19:32:50,215][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld; took 0.003s; new files: {8=[_8_1_Lucene90_0.dvd, _8_1_Lucene90_0.dvm]} [2023-11-15T19:32:50,215][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nj8, bytes=0.000 MB [2023-11-15T19:32:50,218][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf; took 0.003s; new files: {8=[_3_1_Lucene90_0.dvd, _3_1_Lucene90_0.dvm]} [2023-11-15T19:32:50,218][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nka, bytes=0.000 MB [2023-11-15T19:32:50,220][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh; took 0.002s; new files: {8=[_7_1_Lucene90_0.dvm, _7_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,221][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njh, bytes=0.000 MB [2023-11-15T19:32:50,224][TRACE][o.o.i.e.E.BD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] BD: done write field updates for seg=_5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj; took 0.003s; new files: {8=[_5_1_Lucene90_0.dvm, _5_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,224][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now checkpoint "_0(9.8.0):c2:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf _4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb _5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 _8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld _7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh _9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7" [10 segments ; isCommit = false] [2023-11-15T19:32:50,224][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,224][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: 0 ms to checkpoint [2023-11-15T19:32:50,228][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: incRefDeleter for NRT reader version=27 segments=_0(9.8.0):c2:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf _4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb _5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 _8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld _7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh _9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7 [2023-11-15T19:32:50,228][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: return reader version=27 reader=StandardDirectoryReader(segments_2:27:nrt _0(9.8.0):c2/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5/4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf _4(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb _5(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj _6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 _8(9.8.0):c10/10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld _7(9.8.0):c8/8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh _9(9.8.0):c54/54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7) [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_3(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_1(9.8.0):c4/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_8(9.8.0):c10/10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_7(9.8.0):c8/8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_5(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_4(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_2(9.8.0):c5/4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_0(9.8.0):c2/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_9(9.8.0):c54/54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7 size=0.006 MB [floored] [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: findMerges: 10 segments [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: allowedSegmentCount=10 vs count=10 (eligible count=10) [2023-11-15T19:32:50,235][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] DW: opensearch[node_s5][refresh][T#1] finishFullFlush success=true [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_3(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_1(9.8.0):c4/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_8(9.8.0):c10/10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_7(9.8.0):c8/8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.c.c.FollowersChecker ] [node_s0] handleWakeUp: checking {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} with FollowerCheckRequest{term=1, sender={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_5(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_4(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb size=0.006 MB [floored] [2023-11-15T19:32:50,237][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:coordination/fault_detection/follower_check [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_2(9.8.0):c5/4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_0(9.8.0):c2/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: seg=_9(9.8.0):c54/54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7 size=0.006 MB [floored] [2023-11-15T19:32:50,237][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 323, request id: 346, type: request, version: 3.0.0, header size: 101B, action: internal:coordination/fault_detection/follower_check] WRITE: 323B [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: findMerges: 10 segments [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.MP ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] MP: allowedSegmentCount=10 vs count=10 (eligible count=10) [2023-11-15T19:32:50,237][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: getReader took 68 ms [2023-11-15T19:32:50,237][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,237][TRACE][o.o.i.IndexWarmer ] [node_s5] [test][0] top warming [OpenSearchDirectoryReader(FilterLeafReader(_0(9.8.0):c2/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2) FilterLeafReader(_1(9.8.0):c4/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4) FilterLeafReader(_2(9.8.0):c5/4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0) FilterLeafReader(_3(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf) FilterLeafReader(_4(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb) FilterLeafReader(_5(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj) FilterLeafReader(_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9) FilterLeafReader(_8(9.8.0):c10/10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld) FilterLeafReader(_7(9.8.0):c8/8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh) FilterLeafReader(_9(9.8.0):c54/54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7))] [2023-11-15T19:32:50,237][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [199] bytes, now [199] [2023-11-15T19:32:50,237][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 323, request id: 346, type: request, version: 3.0.0, action: internal:coordination/fault_detection/follower_check] READ: 323B [2023-11-15T19:32:50,237][TRACE][o.o.i.w.ShardIndexWarmerService] [node_s5] [test][0] top warming took [41.4micros] [2023-11-15T19:32:50,237][TRACE][o.o.t.TaskManager ] [node_s1] register 151 [transport] [internal:coordination/fault_detection/follower_check] [] [2023-11-15T19:32:50,238][TRACE][o.o.c.c.FollowersChecker ] [node_s1] responding to FollowerCheckRequest{term=1, sender={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} on fast path [2023-11-15T19:32:50,238][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 151 [2023-11-15T19:32:50,238][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16639] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 71, request id: 346, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,238][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][refresh][T#1] IW: decRefDeleter for NRT reader version=24 segments=_0(9.8.0):c2:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nj8 _4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nja _5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njh _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njl _8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk6 _7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nka [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [199] [2023-11-15T19:32:50,238][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s5] [test][0] refresh request executed on primary [2023-11-15T19:32:50,238][TRACE][o.o.c.c.LeaderChecker ] [node_s1] checking {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} with [cluster.fault_detection.leader_check.timeout] = 10s [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-199] bytes, now [0] [2023-11-15T19:32:50,238][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s5] [[test][0]] op [indices:admin/refresh[s]] completed on primary for request [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 71, request id: 346, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,238][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s5] [[test][0]] sending op [indices:admin/refresh[s]] to replica [test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] for request [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,238][DEBUG][o.o.t.TransportService ] [node_s1] Action: internal:coordination/fault_detection/leader_check [2023-11-15T19:32:50,238][TRACE][o.o.c.c.FollowersChecker ] [node_s0] FollowerChecker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} check successful [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,238][DEBUG][o.o.t.TransportService ] [node_s5] Action: indices:admin/refresh[s][r] [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16513] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61568, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 314, request id: 64, type: request, version: 3.0.0, header size: 100B, action: internal:coordination/fault_detection/leader_check] WRITE: 314B [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 201, request id: 511, type: request, version: 3.0.0, header size: 77B, action: indices:admin/refresh[s][r]] WRITE: 201B [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [73] [2023-11-15T19:32:50,238][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [oIyDaBU5RJCXL98jM8pOHg] from [118] to [118], current checkpoint is higher [2023-11-15T19:32:50,238][TRACE][o.o.t.T.tracer ] [node_s5] [511][indices:admin/refresh[s][r]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [191] bytes, now [191] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61568} [length: 314, request id: 64, type: request, version: 3.0.0, action: internal:coordination/fault_detection/leader_check] READ: 314B [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [101b][indices:admin/refresh[s][r]] to used bytes [new used: [101b], limit: 4294967296 [4gb], estimate: 202 [202b]] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 201, request id: 511, type: request, version: 3.0.0, action: indices:admin/refresh[s][r]] READ: 201B [2023-11-15T19:32:50,238][TRACE][o.o.t.T.tracer ] [node_s3] [511][indices:admin/refresh[s][r]] received request [2023-11-15T19:32:50,238][TRACE][o.o.t.TaskManager ] [node_s0] register 209 [transport] [internal:coordination/fault_detection/leader_check] [] [2023-11-15T19:32:50,238][TRACE][o.o.c.c.LeaderChecker ] [node_s0] handling LeaderCheckRequest{sender={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,238][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 209 [2023-11-15T19:32:50,238][TRACE][o.o.t.TaskManager ] [node_s3] register 278 [transport] [indices:admin/refresh[s][r]] [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16631] [2023-11-15T19:32:50,238][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61568} [length: 71, request id: 64, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,238][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [191] [2023-11-15T19:32:50,238][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] refresh with source [api] [2023-11-15T19:32:50,239][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-191] bytes, now [0] [2023-11-15T19:32:50,239][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61568, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 71, request id: 64, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DW: anyChanges? numDocsInRam=53 deletes=true hasTickets:false pendingChangesInFullFlush: false [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DW: anyChanges? numDocsInRam=53 deletes=true hasTickets:false pendingChangesInFullFlush: false [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: nrtIsCurrent: infoVersion matches: true; DW changes: true; BD changes: false [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: flush at getReader [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DW: startFullFlush [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DW: anyChanges? numDocsInRam=53 deletes=true hasTickets:false pendingChangesInFullFlush: false [2023-11-15T19:32:50,239][TRACE][o.o.c.c.LeaderChecker ] [node_s1] scheduling next check of {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} for [cluster.fault_detection.leader_check.interval] = 1s [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: compressed 33410 to 33426 bytes (100.05%) for deletes/updates; private segment null [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: flush postings as segment _8 numDocs=53 [2023-11-15T19:32:50,239][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 0 ms to write norms [2023-11-15T19:32:50,242][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 3 ms to write docValues [2023-11-15T19:32:50,244][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 1 ms to write points [2023-11-15T19:32:50,244][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 0 ms to write vectors [2023-11-15T19:32:50,245][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90FieldsIndex-doc_ids_g.tmp [2023-11-15T19:32:50,246][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90FieldsIndexfile_pointers_h.tmp [2023-11-15T19:32:50,247][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 3 ms to finish stored fields [2023-11-15T19:32:50,251][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 3 ms to write postings and finish vectors [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: 0 ms to write fieldInfos [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: new segment has 0 deleted docs [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: new segment has 53 soft-deleted docs [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: new segment has no vectors; no norms; docValues; no prox; freqs [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: flushedFiles=[_8.fdx, _8_Lucene90_0.dvm, _8_Asserting_0.tmd, _8.fdt, _8_Asserting_0.dvd, _8_Lucene90_0.dvd, _8_Asserting_0.doc, _8_Asserting_0.tim, _8_Lucene90_1.dvd, _8.kdi, _8_Asserting_0.dvm, _8_Asserting_0.tip, _8.kdm, _8.fnm, _8_Lucene90_1.dvm, _8.fdm, _8.kdd] [2023-11-15T19:32:50,252][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: flushed codec=Asserting(Lucene95): {_routing=PostingsFormat(name=Asserting), index_uuid=PostingsFormat(name=Asserting), field1=PostingsFormat(name=Asserting), _field_names=Lucene90, field2.keyword=Lucene90, _id=PostingsFormat(name=Asserting), type=Lucene90, field2=Lucene90}, docValues:{__soft_deletes=DocValuesFormat(name=Lucene90), _seq_no=DocValuesFormat(name=Lucene90), _tombstone=DocValuesFormat(name=Asserting), field2.keyword=DocValuesFormat(name=Lucene90), _primary_term=DocValuesFormat(name=Lucene90), _version=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=667, maxMBSortInHeap=6.321628492792106 [2023-11-15T19:32:50,255][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: flushed: segment=_8 ramUsed=0.11 MB newFlushedSize=0.005 MB docs/MB=11,311.73 [2023-11-15T19:32:50,255][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: compressed 33646 to 33662 bytes (100.05%) for deletes/updates; private segment _8(9.8.0):C53:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk [2023-11-15T19:32:50,255][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: create compound file [2023-11-15T19:32:50,264][TRACE][o.o.i.e.E.DWPT ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DWPT: flush time 25.601964 ms [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: publishFlushedSegment seg-private updates=delGen=-1 fieldUpdates=53 bytesUsed=33662 privateSegment=_8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: flush: push buffered seg private updates: delGen=-1 fieldUpdates=53 bytesUsed=33662 privateSegment=_8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: publishFlushedSegment _8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: push new packet (delGen=11 fieldUpdates=53 bytesUsed=33426), packetCount=1, bytesUsed=0.032 MB [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: push new packet (delGen=12 fieldUpdates=53 bytesUsed=33662 privateSegment=_8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk), packetCount=2, bytesUsed=0.064 MB [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: publish sets newSegment delGen=12 seg=_8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nlk [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now checkpoint "_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njk _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2njr _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njp _4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk2 _3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk3 _5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk4 _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl5 _7(9.8.0):c1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 _8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll" [9 segments ; isCommit = false] [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,265][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: 0 ms to checkpoint [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.fdx" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Lucene90_0.dvm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.tmd" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.fdt" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.dvd" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Lucene90_0.dvd" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.doc" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.tim" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Lucene90_1.dvd" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.kdi" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.dvm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Asserting_0.tip" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.kdm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.fnm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8_Lucene90_1.dvm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.fdm" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: will delete new file "_8.kdd" [2023-11-15T19:32:50,268][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 17 files: [_8.fdx, _8_Lucene90_0.dvm, _8_Asserting_0.tmd, _8.fdt, _8_Asserting_0.dvd, _8_Lucene90_0.dvd, _8_Asserting_0.doc, _8_Asserting_0.tim, _8_Lucene90_1.dvd, _8.kdi, _8_Asserting_0.dvm, _8_Asserting_0.tip, _8.kdm, _8.fnm, _8_Lucene90_1.dvm, _8.fdm, _8.kdd] [2023-11-15T19:32:50,268][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.fdx [2023-11-15T19:32:50,269][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90_0.dvm [2023-11-15T19:32:50,270][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.tmd [2023-11-15T19:32:50,271][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.fdt [2023-11-15T19:32:50,271][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.dvd [2023-11-15T19:32:50,272][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90_0.dvd [2023-11-15T19:32:50,273][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.doc [2023-11-15T19:32:50,273][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.tim [2023-11-15T19:32:50,274][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90_1.dvd [2023-11-15T19:32:50,275][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.kdi [2023-11-15T19:32:50,275][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.dvm [2023-11-15T19:32:50,276][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Asserting_0.tip [2023-11-15T19:32:50,276][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.kdm [2023-11-15T19:32:50,277][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.fnm [2023-11-15T19:32:50,277][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8_Lucene90_1.dvm [2023-11-15T19:32:50,278][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.fdm [2023-11-15T19:32:50,278][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.kdd [2023-11-15T19:32:50,279][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 8 segments, mergeGen 0 [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: applyDocValuesUpdates 0.8 msec for 8 segments, 53 field updates; 53 new updates [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: closeSegmentStates: 0 new deleted documents; pool 2 packets; bytesUsed=1414 [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done inner apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 8 segments; 53 new deletes/updates; took 0.001 sec [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: finished packet delGen=11 now completedDelGen=11 [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 8 segments; 53 new deletes/updates; took 0.001 sec; 1 packets remain [2023-11-15T19:32:50,280][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 1 segments, mergeGen 0 [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: applyDocValuesUpdates 0.3 msec for 1 segments, 53 field updates; 0 new updates [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: closeSegmentStates: 0 new deleted documents; pool 1 packets; bytesUsed=1414 [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done inner apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 1 segments; 0 new deletes/updates; took 0.001 sec [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: finished packet delGen=12 now completedDelGen=12 [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done apply del packet (org.opensearch.index.engine.InternalEngine$AssertingIndexWriter@7d7fb524) to 1 segments; 0 new deletes/updates; took 0.001 sec; 0 packets remain [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=1414 [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: waitApply: no deletes to apply [2023-11-15T19:32:50,281][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl5, bytes=0.000 MB [2023-11-15T19:32:50,285][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln; took 0.004s; new files: {7=[_6_2_Lucene90_0.dvm, _6_2_Lucene90_0.dvd]} [2023-11-15T19:32:50,285][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2njr, bytes=0.000 MB [2023-11-15T19:32:50,288][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp; took 0.003s; new files: {10=[_1_2_Lucene90_1.dvm, _1_2_Lucene90_1.dvd]} [2023-11-15T19:32:50,288][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njk, bytes=0.000 MB [2023-11-15T19:32:50,291][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr; took 0.003s; new files: {7=[_2_1_Lucene90_0.dvm, _2_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,291][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk3, bytes=0.000 MB [2023-11-15T19:32:50,295][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt; took 0.004s; new files: {7=[_3_1_Lucene90_0.dvd, _3_1_Lucene90_0.dvm]} [2023-11-15T19:32:50,295][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njp, bytes=0.000 MB [2023-11-15T19:32:50,298][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv; took 0.002s; new files: {7=[_0_1_Lucene90_0.dvm, _0_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,298][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk2, bytes=0.000 MB [2023-11-15T19:32:50,301][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx; took 0.003s; new files: {7=[_4_1_Lucene90_0.dvm, _4_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,301][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: now write 1 pending numeric DV updates for field=__soft_deletes, seg=_5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk4, bytes=0.000 MB [2023-11-15T19:32:50,304][TRACE][o.o.i.e.E.BD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] BD: done write field updates for seg=_5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz; took 0.003s; new files: {7=[_5_1_Lucene90_0.dvm, _5_1_Lucene90_0.dvd]} [2023-11-15T19:32:50,304][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now checkpoint "_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv _4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx _3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt _5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln _7(9.8.0):c1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 _8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll" [9 segments ; isCommit = false] [2023-11-15T19:32:50,304][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,304][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: 0 ms to checkpoint [2023-11-15T19:32:50,307][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: incRefDeleter for NRT reader version=30 segments=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv _4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx _3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt _5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln _7(9.8.0):c1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 _8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll [2023-11-15T19:32:50,308][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: return reader version=30 reader=StandardDirectoryReader(segments_4:30:nrt _2(9.8.0):c5/5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr _1(9.8.0):c36/33:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp _0(9.8.0):c6/6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv _4(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx _3(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt _5(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz _6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln _7(9.8.0):c1/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 _8(9.8.0):c53/53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll) [2023-11-15T19:32:50,314][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_1(9.8.0):c36/33:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp size=0.010 MB [floored] [2023-11-15T19:32:50,314][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln size=0.006 MB [floored] [2023-11-15T19:32:50,314][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_0(9.8.0):c6/6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv size=0.006 MB [floored] [2023-11-15T19:32:50,314][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_8(9.8.0):c53/53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll size=0.006 MB [floored] [2023-11-15T19:32:50,314][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_4(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx size=0.005 MB [floored] [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_2(9.8.0):c5/5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr size=0.005 MB [floored] [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_5(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz size=0.005 MB [floored] [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_3(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt size=0.005 MB [floored] [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_7(9.8.0):c1/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 size=0.003 MB [floored] [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: findMerges: 9 segments [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: allowedSegmentCount=10 vs count=9 (eligible count=9) [2023-11-15T19:32:50,315][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] DW: opensearch[node_s3][refresh][T#1] finishFullFlush success=true [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_1(9.8.0):c36/33:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp size=0.010 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln size=0.006 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_0(9.8.0):c6/6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv size=0.006 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_8(9.8.0):c53/53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll size=0.006 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_4(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx size=0.005 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_2(9.8.0):c5/5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr size=0.005 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_5(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz size=0.005 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_3(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt size=0.005 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: seg=_7(9.8.0):c1/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 size=0.003 MB [floored] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: findMerges: 9 segments [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.MP ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] MP: allowedSegmentCount=10 vs count=9 (eligible count=9) [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: getReader took 77 ms [2023-11-15T19:32:50,316][TRACE][o.o.i.IndexWarmer ] [node_s3] [test][0] top warming [OpenSearchDirectoryReader(FilterLeafReader(_2(9.8.0):c5/5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr) FilterLeafReader(_1(9.8.0):c36/33:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp) FilterLeafReader(_0(9.8.0):c6/6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv) FilterLeafReader(_4(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx) FilterLeafReader(_3(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt) FilterLeafReader(_5(9.8.0):c3/3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz) FilterLeafReader(_6(9.8.0):c9/9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln) FilterLeafReader(_7(9.8.0):c1/1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3) FilterLeafReader(_8(9.8.0):c53/53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll))] [2023-11-15T19:32:50,316][TRACE][o.o.i.w.ShardIndexWarmerService] [node_s3] [test][0] top warming took [50.2micros] [2023-11-15T19:32:50,316][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IFD: now delete 3 files: [_6_1_Lucene90_0.dvm, _6_1.fnm, _6_1_Lucene90_0.dvd] [2023-11-15T19:32:50,317][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_1_Lucene90_0.dvm [2023-11-15T19:32:50,317][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_1.fnm [2023-11-15T19:32:50,318][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_1_Lucene90_0.dvd [2023-11-15T19:32:50,318][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][refresh][T#1] IW: decRefDeleter for NRT reader version=28 segments=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njk _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2njr _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njp _4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk2 _3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk3 _5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2nk4 _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl5 _7(9.8.0):c1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 [2023-11-15T19:32:50,318][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s3] [test][0] refresh request executed on replica [2023-11-15T19:32:50,318][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s3] action [indices:admin/refresh[s][r]] completed on shard [[test][0]] for request [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,318][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 278 [2023-11-15T19:32:50,318][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16541] [2023-11-15T19:32:50,318][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61603} [length: 75, request id: 511, type: response, version: 3.0.0, header size: 48B] WRITE: 75B [2023-11-15T19:32:50,319][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-101] bytes, now [16440] [2023-11-15T19:32:50,319][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s3] [511][indices:admin/refresh[s][r]] sent response [2023-11-15T19:32:50,319][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61603, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 75, request id: 511, type: response, version: 3.0.0] READ: 75B [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s5] [511][indices:admin/refresh[s][r]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,319][TRACE][o.o.i.s.ReplicationTracker] [node_s5] [test][0] skipped updating local checkpoint of [xRjt2CCHQuiHlKD45x631A] from [118] to [118], current checkpoint is higher [2023-11-15T19:32:50,319][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 722 [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s5] [510][indices:admin/refresh[s][p]] sent response [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s5] [510][indices:admin/refresh[s][p]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,319][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s5] operation succeeded. action [indices:admin/refresh[s]],request [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,319][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 721 [2023-11-15T19:32:50,319][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16513] [2023-11-15T19:32:50,319][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 74, request id: 345, type: response, version: 3.0.0, header size: 48B] WRITE: 74B [2023-11-15T19:32:50,319][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-73] bytes, now [16440] [2023-11-15T19:32:50,319][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s5] [345][indices:admin/refresh[s]] sent response [2023-11-15T19:32:50,319][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 74, request id: 345, type: response, version: 3.0.0] READ: 74B [2023-11-15T19:32:50,319][TRACE][o.o.t.T.tracer ] [node_s0] [345][indices:admin/refresh[s]] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,319][TRACE][o.o.a.a.i.r.TransportShardRefreshAction] [node_s0] operation succeeded. action [indices:admin/refresh[s]],request [BasicReplicationRequest{[test][0]}] [2023-11-15T19:32:50,319][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 208 [2023-11-15T19:32:50,319][TRACE][o.o.a.a.i.r.TransportRefreshAction] [node_s0] indices:admin/refresh: got response from [test][0] [2023-11-15T19:32:50,319][TRACE][o.o.a.a.i.r.TransportRefreshAction] [node_s0] indices:admin/refresh: got all shard responses [2023-11-15T19:32:50,319][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 207 [REFRESH] org.opensearch.action.admin.indices.refresh.RefreshResponse@5e6a1422 [MATCH ALL] [2023-11-15T19:32:50,319][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 152 [transport] [indices:data/read/search] [indices[], search_type[DFS_QUERY_THEN_FETCH], source[{"timeout":"1d","query":{"match_all":{"boost":1.0}}}]] [2023-11-15T19:32:50,319][DEBUG][o.o.t.TaskManager ] [testCommonTermsQuery] Starting resource tracking for task: 152 [2023-11-15T19:32:50,321][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/read/search[phase/query] [2023-11-15T19:32:50,321][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register child node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] task [152] [2023-11-15T19:32:50,321][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,321][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61607, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 265, request id: 65, type: request, version: 3.0.0, header size: 87B, action: indices:data/read/search[phase/query]] WRITE: 265B [2023-11-15T19:32:50,321][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,321][TRACE][o.o.t.T.tracer ] [node_s1] [65][indices:data/read/search[phase/query]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,321][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [155b][indices:data/read/search[phase/query]] to used bytes [new used: [155b], limit: 4294967296 [4gb], estimate: 310 [310b]] [2023-11-15T19:32:50,321][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61607} [length: 265, request id: 65, type: request, version: 3.0.0, action: indices:data/read/search[phase/query]] READ: 265B [2023-11-15T19:32:50,321][TRACE][o.o.t.T.tracer ] [node_s3] [65][indices:data/read/search[phase/query]] received request [2023-11-15T19:32:50,321][TRACE][o.o.t.TaskManager ] [node_s3] register 279 [transport] [indices:data/read/search[phase/query]] [shardId[[test][0]]] [2023-11-15T19:32:50,321][DEBUG][o.o.t.TaskManager ] [node_s3] Starting resource tracking for task: 279 [2023-11-15T19:32:50,322][DEBUG][o.o.t.TaskManager ] [node_s3] Task execution started on thread. Task: 279, Thread: 124 [2023-11-15T19:32:50,325][TRACE][o.o.s.q.QueryPhase ] [node_s3] [test][0] source[{"timeout":"1d","query":{"match_all":{"boost":1.0}}}], id[], [2023-11-15T19:32:50,325][DEBUG][o.o.s.q.QueryPhaseSearcherWrapper] [node_s3] Using non-concurrent aggregation processor over segments for request with context id [-ftBpaDxQ6-s9uvgqbm-1A][1] [2023-11-15T19:32:50,325][DEBUG][o.o.s.q.QueryPhaseSearcherWrapper] [node_s3] Using non-concurrent search over segments for request with context id [-ftBpaDxQ6-s9uvgqbm-1A][1] [2023-11-15T19:32:50,325][TRACE][o.o.s.f.FetchPhase ] [node_s3] [test][0] source[{"timeout":"1d","query":{"match_all":{"boost":1.0}}}], id[], [2023-11-15T19:32:50,326][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 279 [2023-11-15T19:32:50,326][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16595] [2023-11-15T19:32:50,326][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61607} [length: 620, request id: 65, type: response, version: 3.0.0, header size: 48B] WRITE: 620B [2023-11-15T19:32:50,326][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-155] bytes, now [16440] [2023-11-15T19:32:50,326][DEBUG][o.o.t.TaskManager ] [node_s3] Task execution finished on thread. Task: 279, Thread: 124 [2023-11-15T19:32:50,326][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s3] updated resource usage info [resource_stats=[memory_in_bytes], old_end_value=17600, new_end_value=162496] [2023-11-15T19:32:50,326][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s3] updated resource usage info [resource_stats=[cpu_time_in_nanos], old_end_value=322000, new_end_value=4119000] [2023-11-15T19:32:50,326][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,326][DEBUG][o.o.t.TaskManager ] [node_s3] Stopping resource tracking for task: 279 [2023-11-15T19:32:50,326][TRACE][o.o.t.T.tracer ] [node_s3] [65][indices:data/read/search[phase/query]] sent response [2023-11-15T19:32:50,326][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61607, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 620, request id: 65, type: response, version: 3.0.0] READ: 620B [2023-11-15T19:32:50,326][TRACE][o.o.t.T.tracer ] [node_s1] [65][indices:data/read/search[phase/query]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,326][TRACE][o.o.t.TaskManager ] [node_s1] unregister child node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] task [152] [2023-11-15T19:32:50,327][TRACE][o.o.a.s.TransportSearchAction] [node_s1] got first-phase result from [9SWyj4IxQoKQhSS_1-SQNQ][test][0] [2023-11-15T19:32:50,327][TRACE][o.o.a.s.TransportSearchAction] [node_s1] [query] Moving to next phase: [fetch], based on results from: [9SWyj4IxQoKQhSS_1-SQNQ][test][0] (cluster state version: 24) [2023-11-15T19:32:50,327][DEBUG][o.o.t.TaskManager ] [node_s1] Task execution started on thread. Task: 152, Thread: 125 [2023-11-15T19:32:50,327][TRACE][o.o.a.s.TransportSearchAction] [node_s1] [fetch] Moving to next phase: [expand], based on results from: [9SWyj4IxQoKQhSS_1-SQNQ][test][0] (cluster state version: 24) [2023-11-15T19:32:50,327][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 152 [2023-11-15T19:32:50,327][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [0] bytes, now [0] [2023-11-15T19:32:50,327][DEBUG][o.o.t.TaskManager ] [node_s1] Task execution finished on thread. Task: 152, Thread: 125 [2023-11-15T19:32:50,327][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s1] updated resource usage info [resource_stats=[memory_in_bytes], old_end_value=17600, new_end_value=23296] [2023-11-15T19:32:50,327][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s1] updated resource usage info [resource_stats=[cpu_time_in_nanos], old_end_value=281000, new_end_value=458000] [2023-11-15T19:32:50,327][DEBUG][o.o.t.TaskManager ] [node_s1] Stopping resource tracking for task: 152 {"took":7,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":1.0,"hits":[{"_index":"test","_id":"1","_score":1.0,"_source":{"field1":"the quick brown fox"}},{"_index":"test","_id":"2","_score":1.0,"_source":{"field1":"the quick lazy huge brown fox jumps over the tree"}},{"_index":"test","_id":"3","_score":1.0,"_source":{"field1":"quick lazy huge brown pidgin","field2":"the quick lazy huge brown fox jumps over the tree"}}]}} [2023-11-15T19:32:50,328][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 210 [transport] [indices:data/read/search] [indices[], search_type[QUERY_THEN_FETCH], source[{"timeout":"1d","query":{"common":{"field1":{"query":"the huge fox","high_freq_operator":"OR","low_freq_operator":"OR","cutoff_frequency":0.01,"minimum_should_match":{"low_freq":"2"},"boost":1.0}}}}]] [2023-11-15T19:32:50,328][DEBUG][o.o.t.TaskManager ] [testCommonTermsQuery] Starting resource tracking for task: 210 [2023-11-15T19:32:50,328][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:data/read/search[phase/query] [2023-11-15T19:32:50,328][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register child node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] task [210] [2023-11-15T19:32:50,328][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,328][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 294, request id: 347, type: request, version: 3.0.0, header size: 87B, action: indices:data/read/search[phase/query]] WRITE: 294B [2023-11-15T19:32:50,328][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,328][TRACE][o.o.t.T.tracer ] [node_s0] [347][indices:data/read/search[phase/query]] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,328][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adding [184b][indices:data/read/search[phase/query]] to used bytes [new used: [184b], limit: 4294967296 [4gb], estimate: 368 [368b]] [2023-11-15T19:32:50,328][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 294, request id: 347, type: request, version: 3.0.0, action: indices:data/read/search[phase/query]] READ: 294B [2023-11-15T19:32:50,329][TRACE][o.o.t.T.tracer ] [node_s3] [347][indices:data/read/search[phase/query]] received request [2023-11-15T19:32:50,329][TRACE][o.o.t.TaskManager ] [node_s3] register 280 [transport] [indices:data/read/search[phase/query]] [shardId[[test][0]]] [2023-11-15T19:32:50,329][DEBUG][o.o.t.TaskManager ] [node_s3] Starting resource tracking for task: 280 [2023-11-15T19:32:50,329][DEBUG][o.o.t.TaskManager ] [node_s3] Task execution started on thread. Task: 280, Thread: 126 [2023-11-15T19:32:50,331][TRACE][o.o.s.q.QueryPhase ] [node_s3] [test][0] source[{"timeout":"1d","query":{"common":{"field1":{"query":"the huge fox","high_freq_operator":"OR","low_freq_operator":"OR","cutoff_frequency":0.01,"minimum_should_match":{"low_freq":"2"},"boost":1.0}}}}], id[], [2023-11-15T19:32:50,331][DEBUG][o.o.s.q.QueryPhaseSearcherWrapper] [node_s3] Using non-concurrent aggregation processor over segments for request with context id [-ftBpaDxQ6-s9uvgqbm-1A][2] [2023-11-15T19:32:50,331][DEBUG][o.o.s.q.QueryPhaseSearcherWrapper] [node_s3] Using non-concurrent search over segments for request with context id [-ftBpaDxQ6-s9uvgqbm-1A][2] [2023-11-15T19:32:50,336][TRACE][o.o.s.f.FetchPhase ] [node_s3] [test][0] source[{"timeout":"1d","query":{"common":{"field1":{"query":"the huge fox","high_freq_operator":"OR","low_freq_operator":"OR","cutoff_frequency":0.01,"minimum_should_match":{"low_freq":"2"},"boost":1.0}}}}], id[], [2023-11-15T19:32:50,336][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 280 [2023-11-15T19:32:50,336][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16624] [2023-11-15T19:32:50,336][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 516, request id: 347, type: response, version: 3.0.0, header size: 48B] WRITE: 516B [2023-11-15T19:32:50,336][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-184] bytes, now [16440] [2023-11-15T19:32:50,336][DEBUG][o.o.t.TaskManager ] [node_s3] Task execution finished on thread. Task: 280, Thread: 126 [2023-11-15T19:32:50,336][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s3] updated resource usage info [resource_stats=[memory_in_bytes], old_end_value=17600, new_end_value=571144] [2023-11-15T19:32:50,336][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s3] updated resource usage info [resource_stats=[cpu_time_in_nanos], old_end_value=239000, new_end_value=6784000] [2023-11-15T19:32:50,336][DEBUG][o.o.t.TaskManager ] [node_s3] Stopping resource tracking for task: 280 [2023-11-15T19:32:50,336][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,336][TRACE][o.o.t.T.tracer ] [node_s3] [347][indices:data/read/search[phase/query]] sent response [2023-11-15T19:32:50,337][TRACE][o.o.c.c.LeaderChecker ] [node_s2] checking {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} with [cluster.fault_detection.leader_check.timeout] = 10s [2023-11-15T19:32:50,337][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 516, request id: 347, type: response, version: 3.0.0] READ: 516B [2023-11-15T19:32:50,337][TRACE][o.o.t.T.tracer ] [node_s0] [347][indices:data/read/search[phase/query]] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,337][DEBUG][o.o.t.TransportService ] [node_s2] Action: internal:coordination/fault_detection/leader_check [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,337][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61562, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 314, request id: 63, type: request, version: 3.0.0, header size: 100B, action: internal:coordination/fault_detection/leader_check] WRITE: 314B [2023-11-15T19:32:50,337][TRACE][o.o.t.TaskManager ] [node_s0] unregister child node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] task [210] [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,337][TRACE][o.o.a.s.TransportSearchAction] [node_s0] got first-phase result from [9SWyj4IxQoKQhSS_1-SQNQ][test][0] [2023-11-15T19:32:50,337][TRACE][o.o.a.s.TransportSearchAction] [node_s0] [query] Moving to next phase: [fetch], based on results from: [9SWyj4IxQoKQhSS_1-SQNQ][test][0] (cluster state version: 24) [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [191] bytes, now [191] [2023-11-15T19:32:50,337][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61562} [length: 314, request id: 63, type: request, version: 3.0.0, action: internal:coordination/fault_detection/leader_check] READ: 314B [2023-11-15T19:32:50,337][TRACE][o.o.t.TaskManager ] [node_s0] register 211 [transport] [internal:coordination/fault_detection/leader_check] [] [2023-11-15T19:32:50,337][DEBUG][o.o.t.TaskManager ] [node_s0] Task execution started on thread. Task: 210, Thread: 127 [2023-11-15T19:32:50,337][TRACE][o.o.c.c.LeaderChecker ] [node_s0] handling LeaderCheckRequest{sender={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,337][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 211 [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16631] [2023-11-15T19:32:50,337][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61562} [length: 71, request id: 63, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [191] [2023-11-15T19:32:50,337][TRACE][o.o.a.s.TransportSearchAction] [node_s0] [fetch] Moving to next phase: [expand], based on results from: [9SWyj4IxQoKQhSS_1-SQNQ][test][0] (cluster state version: 24) [2023-11-15T19:32:50,337][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-191] bytes, now [0] [2023-11-15T19:32:50,337][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 210 [2023-11-15T19:32:50,337][DEBUG][o.o.t.TaskManager ] [node_s0] Task execution finished on thread. Task: 210, Thread: 127 [COMMON TERMS] [2023-11-15T19:32:50,337][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61562, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 71, request id: 63, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,338][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s0] updated resource usage info [resource_stats=[memory_in_bytes], old_end_value=17600, new_end_value=22184] { "common" : { "field1" : { "query" : "the huge fox", "high_freq_operator" : "OR", "low_freq_operator" : "OR", "cutoff_frequency" : 0.01, "minimum_should_match" : { "low_freq" : "2" }, "boost" : 1.0 } } } [2023-11-15T19:32:50,338][DEBUG][o.o.c.t.r.ResourceUsageInfo] [node_s0] updated resource usage info [resource_stats=[cpu_time_in_nanos], old_end_value=272000, new_end_value=425000] [2023-11-15T19:32:50,338][DEBUG][o.o.t.TaskManager ] [node_s0] Stopping resource tracking for task: 210 [2023-11-15T19:32:50,338][TRACE][o.o.c.c.LeaderChecker ] [node_s2] scheduling next check of {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} for [cluster.fault_detection.leader_check.interval] = 1s {"took":9,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":1.3157668,"hits":[{"_index":"test","_id":"2","_score":1.3157668,"_source":{"field1":"the quick lazy huge brown fox jumps over the tree"}},{"_index":"test","_id":"1","_score":1.1068254,"_source":{"field1":"the quick brown fox"}}]}} [2023-11-15T19:32:50,338][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 128 [transport] [cluster:admin/settings/update] [] [2023-11-15T19:32:50,338][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest@216ad1a4] with cluster state version [24] [2023-11-15T19:32:50,339][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:admin/settings/update [2023-11-15T19:32:50,339][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,339][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61538, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 181, request id: 67, type: request, version: 3.0.0, header size: 79B, action: cluster:admin/settings/update] WRITE: 181B [2023-11-15T19:32:50,339][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,339][TRACE][o.o.t.T.tracer ] [node_s4] [67][cluster:admin/settings/update] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,339][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [79] bytes, now [79] [2023-11-15T19:32:50,339][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61538} [length: 181, request id: 67, type: request, version: 3.0.0, action: cluster:admin/settings/update] READ: 181B [2023-11-15T19:32:50,339][TRACE][o.o.t.T.tracer ] [node_s0] [67][cluster:admin/settings/update] received request [2023-11-15T19:32:50,339][TRACE][o.o.t.TaskManager ] [node_s0] register 212 [transport] [cluster:admin/settings/update] [] [2023-11-15T19:32:50,339][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest@5513b239] with cluster state version [24] [2023-11-15T19:32:50,339][TRACE][o.o.c.s.MasterService ] [node_s0] will process [cluster_update_settings] [2023-11-15T19:32:50,339][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [cluster_update_settings] [2023-11-15T19:32:50,341][DEBUG][o.o.c.s.MasterService ] [node_s0] took [1ms] to compute cluster state update for [cluster_update_settings] [2023-11-15T19:32:50,341][TRACE][o.o.c.s.MasterService ] [node_s0] cluster state updated, source [cluster_update_settings] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: false meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,341][DEBUG][o.o.c.s.MasterService ] [node_s0] publishing cluster state version [25] [2023-11-15T19:32:50,341][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] expecting 6 acknowledgements for cluster_state update (version: 25) [2023-11-15T19:32:50,342][TRACE][o.o.c.c.CompressedStreamUtils] [node_s0] serialized writable object for node version [3.0.0] with size [265] [2023-11-15T19:32:50,342][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] serialized cluster state diff for version [25] in for node version [3.0.0] with size [265] [2023-11-15T19:32:50,342][TRACE][o.o.c.c.CoordinationState] [node_s0] handleClientValue: processing request for version [25] and term [1] [2023-11-15T19:32:50,342][TRACE][o.o.c.c.LeaderChecker ] [node_s0] setCurrentNodes: nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,342][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] publishing PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: false meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned } to [PublicationTarget{discoveryNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}] [2023-11-15T19:32:50,342][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,342][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,342][TRACE][o.o.t.T.tracer ] [node_s0] [348][internal:cluster/coordination/publish_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,342][TRACE][o.o.t.T.tracer ] [node_s0] [348][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,342][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,342][TRACE][o.o.t.TaskManager ] [node_s0] register 213 [direct] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,342][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,342][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,342][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 384, request id: 349, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,342][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,342][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,342][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s0] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32959] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16519] [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 384, request id: 350, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s0] [349][internal:cluster/coordination/publish_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,343][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,343][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 384, request id: 349, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 383, request id: 351, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/publish_state] WRITE: 383B [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s0] [350][internal:cluster/coordination/publish_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s2] [349][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 384, request id: 350, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,343][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s1] [350][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,343][TRACE][o.o.t.TaskManager ] [node_s2] register 146 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s0] [351][internal:cluster/coordination/publish_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 384, request id: 352, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 383, request id: 351, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 383B [2023-11-15T19:32:50,343][TRACE][o.o.t.TaskManager ] [node_s1] register 153 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [25] to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s4] [351][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,343][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32959] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16519] [2023-11-15T19:32:50,343][TRACE][o.o.t.TaskManager ] [node_s4] register 129 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,343][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s2] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 384, request id: 353, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,343][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s1] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s0] [352][internal:cluster/coordination/publish_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 384, request id: 352, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,343][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s4] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s5] [352][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s0] [353][internal:cluster/coordination/publish_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,343][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,343][TRACE][o.o.t.TaskManager ] [node_s5] register 723 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,343][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 384, request id: 353, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,343][TRACE][o.o.c.c.Coordinator ] [node_s0] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: false meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.t.T.tracer ] [node_s3] [353][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,343][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,343][TRACE][o.o.g.PersistedClusterStateService] [node_s0] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,343][TRACE][o.o.t.TaskManager ] [node_s3] register 281 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.CoordinationState] [node_s1] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,343][TRACE][o.o.g.PersistedClusterStateService] [node_s1] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,343][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s5] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.CoordinationState] [node_s2] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,343][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,343][TRACE][o.o.g.PersistedClusterStateService] [node_s2] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,343][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,343][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s3] received diff cluster state version [25] with uuid [MmpCNYRNRHGpyFVnnr2LBw], diff size [265] [2023-11-15T19:32:50,343][TRACE][o.o.c.c.CoordinationState] [node_s4] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,343][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] queuing cluster state update (setting cluster state to 25) [2023-11-15T19:32:50,344][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] resetting newStateQueued [2023-11-15T19:32:50,344][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s4] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s4] not active [2023-11-15T19:32:50,344][TRACE][o.o.g.PersistedClusterStateService] [node_s4] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,344][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,344][TRACE][o.o.c.c.PreVoteCollector ] [node_s4] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.c.c.CoordinationState] [node_s5] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,344][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] queuing cluster state update (setting cluster state to 25) [2023-11-15T19:32:50,344][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 129 [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,344][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] resetting newStateQueued [2023-11-15T19:32:50,344][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s5] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: handling [PublishRequest{term=1, version=25, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.g.PersistedClusterStateService] [node_s5] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,344][TRACE][o.o.c.c.CoordinationState] [node_s3] handlePublishRequest: accepting publish request for version [25] and term [1] [2023-11-15T19:32:50,344][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] queuing cluster state update (setting cluster state to 25) [2023-11-15T19:32:50,344][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s5] not active [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 481, request id: 351, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,344][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] resetting newStateQueued [2023-11-15T19:32:50,344][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,344][TRACE][o.o.c.c.PreVoteCollector ] [node_s5] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s3] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.d.PeerFinder ] [node_s3] not active [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.g.PersistedClusterStateService] [node_s3] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,344][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 723 [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,344][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s4] [351][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,344][TRACE][o.o.c.c.PreVoteCollector ] [node_s3] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,344][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 281 [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 481, request id: 352, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 481, request id: 351, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 481, request id: 353, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s0] [351][internal:cluster/coordination/publish_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s5] [352][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,344][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s3] [353][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 481, request id: 352, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,344][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s0] [352][internal:cluster/coordination/publish_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,344][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 481, request id: 353, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,344][TRACE][o.o.t.T.tracer ] [node_s0] [353][internal:cluster/coordination/publish_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s0] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s4] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s5] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s3] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s2] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.PersistedClusterStateService] [node_s1] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,346][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,348][TRACE][o.o.c.c.FollowersChecker ] [node_s0] handleWakeUp: checking {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} with FollowerCheckRequest{term=1, sender={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,349][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:coordination/fault_detection/follower_check [2023-11-15T19:32:50,349][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,349][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 324, request id: 354, type: request, version: 3.0.0, header size: 102B, action: internal:coordination/fault_detection/follower_check] WRITE: 324B [2023-11-15T19:32:50,350][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,351][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [199] bytes, now [467] [2023-11-15T19:32:50,351][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 324, request id: 354, type: request, version: 3.0.0, action: internal:coordination/fault_detection/follower_check] READ: 324B [2023-11-15T19:32:50,351][TRACE][o.o.t.TaskManager ] [node_s2] register 147 [transport] [internal:coordination/fault_detection/follower_check] [] [2023-11-15T19:32:50,351][TRACE][o.o.c.c.FollowersChecker ] [node_s2] responding to FollowerCheckRequest{term=1, sender={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} on fast path [2023-11-15T19:32:50,352][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 147 [2023-11-15T19:32:50,352][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16907] [2023-11-15T19:32:50,352][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 70, request id: 354, type: response, version: 3.0.0, header size: 47B] WRITE: 70B [2023-11-15T19:32:50,352][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [467] [2023-11-15T19:32:50,352][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-199] bytes, now [268] [2023-11-15T19:32:50,353][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 70, request id: 354, type: response, version: 3.0.0] READ: 70B [2023-11-15T19:32:50,353][TRACE][o.o.c.c.FollowersChecker ] [node_s0] FollowerChecker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} check successful [2023-11-15T19:32:50,372][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,372][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,372][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,372][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,376][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,379][DEBUG][o.o.g.PersistedClusterStateService] [node_s3] writing cluster state took [204ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,379][DEBUG][o.o.g.PersistedClusterStateService] [node_s5] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,379][DEBUG][o.o.g.PersistedClusterStateService] [node_s4] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,379][DEBUG][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,379][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,379][DEBUG][o.o.g.PersistedClusterStateService] [node_s0] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,379][TRACE][o.o.d.PeerFinder ] [node_s2] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,379][TRACE][o.o.d.PeerFinder ] [node_s2] not active [2023-11-15T19:32:50,379][TRACE][o.o.c.c.PreVoteCollector ] [node_s0] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,379][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 213 [2023-11-15T19:32:50,379][TRACE][o.o.c.c.PreVoteCollector ] [node_s2] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,379][TRACE][o.o.t.T.tracer ] [node_s0] [348][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,379][TRACE][o.o.t.T.tracer ] [node_s0] [348][internal:cluster/coordination/publish_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,379][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 146 [2023-11-15T19:32:50,379][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [25] and term [1] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,379][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,379][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 491, request id: 349, type: response, version: 3.0.0, header size: 47B] WRITE: 491B [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,379][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [25] and term [1] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,379][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,379][TRACE][o.o.t.T.tracer ] [node_s2] [349][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,379][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [25] and term [1] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,379][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 491, request id: 349, type: response, version: 3.0.0] READ: 491B [2023-11-15T19:32:50,380][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,380][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [25] and term [1] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s0] [349][internal:cluster/coordination/publish_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,380][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,380][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,380][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [25] and term [1] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,380][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: value committed for version [25] and term [1] [2023-11-15T19:32:50,380][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s0] [355][internal:cluster/coordination/commit_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s0] [355][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,380][TRACE][o.o.t.TaskManager ] [node_s0] register 214 [direct] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,380][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 322, request id: 356, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,380][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s0] [356][internal:cluster/coordination/commit_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 322, request id: 357, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 322, request id: 356, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,380][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s0] [357][internal:cluster/coordination/commit_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 322, request id: 358, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,380][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,380][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 322, request id: 357, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,380][TRACE][o.o.t.T.tracer ] [node_s2] [356][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,381][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,381][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s4] [357][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,381][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,381][TRACE][o.o.t.TaskManager ] [node_s2] register 148 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s0] [358][internal:cluster/coordination/commit_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,381][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,381][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 322, request id: 359, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,381][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 322, request id: 358, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,381][TRACE][o.o.t.TaskManager ] [node_s4] register 130 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,381][TRACE][o.o.c.c.Coordinator ] [node_s2] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,381][TRACE][o.o.c.c.CoordinationState] [node_s2] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,381][TRACE][o.o.c.c.Coordinator ] [node_s4] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,381][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,381][TRACE][o.o.c.c.Coordinator ] [node_s0] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,381][TRACE][o.o.c.c.CoordinationState] [node_s4] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s5] [358][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s0] [359][internal:cluster/coordination/commit_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,381][TRACE][o.o.c.c.CoordinationState] [node_s0] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,381][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,381][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 214 [2023-11-15T19:32:50,381][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 322, request id: 359, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s0] [355][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,381][TRACE][o.o.t.TaskManager ] [node_s5] register 724 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s0] [355][internal:cluster/coordination/commit_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,381][TRACE][o.o.c.c.Coordinator ] [node_s5] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,381][TRACE][o.o.t.T.tracer ] [node_s3] [359][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,381][TRACE][o.o.c.c.CoordinationState] [node_s5] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,381][TRACE][o.o.t.TaskManager ] [node_s3] register 282 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,381][TRACE][o.o.c.c.Coordinator ] [node_s3] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,381][TRACE][o.o.c.c.CoordinationState] [node_s3] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] cluster state updated, version [25], source [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,381][TRACE][o.o.c.s.ClusterApplierService] [node_s2] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] cluster state updated, version [25], source [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,381][TRACE][o.o.c.s.ClusterApplierService] [node_s4] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] cluster state updated, version [25], source [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.s.ClusterApplierService] [node_s5] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] applying settings from cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] applying settings from cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] cluster state updated, version [25], source [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.s.ClusterApplierService] [node_s3] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] applying settings from cluster state with version 25 [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,381][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] applying settings from cluster state with version 25 [2023-11-15T19:32:50,383][DEBUG][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,383][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,383][TRACE][o.o.d.PeerFinder ] [node_s1] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,383][TRACE][o.o.d.PeerFinder ] [node_s1] not active [2023-11-15T19:32:50,383][TRACE][o.o.c.c.PreVoteCollector ] [node_s1] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=25}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,383][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 153 [2023-11-15T19:32:50,383][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] apply cluster state with version 25 [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] apply cluster state with version 25 [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] apply cluster state with version 25 [2023-11-15T19:32:50,383][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 491, request id: 350, type: response, version: 3.0.0, header size: 47B] WRITE: 491B [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.repositories.RepositoriesService@417b8324] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.repositories.RepositoriesService@5daadbc8] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.repositories.RepositoriesService@11ab1a37] with change to version [25] [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] apply cluster state with version 25 [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.indices.cluster.IndicesClusterStateService@45440fd3] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.cluster.IndicesClusterStateService@15948cda] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.cluster.IndicesClusterStateService@599544a5] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.repositories.RepositoriesService@6032f45b] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.script.MockScriptService@46891f] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.cluster.IndicesClusterStateService@640d6c85] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.RestoreService@2cf36cc8] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.script.MockScriptService@1c97d1f3] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.ingest.IngestService@310ca8d1] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.ingest.IngestService@26be5633] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.search.pipeline.SearchPipelineService@545313b4] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.ingest.IngestActionForwarder@60697650] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.search.pipeline.SearchPipelineService@2d41adb7] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.action.ingest.IngestActionForwarder@5e8274d6] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@574bd3b9] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.tasks.TaskManager@3d5b3a1] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.tasks.TaskManager@7cb82ebc] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.SnapshotsService@4f57dedc] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.t.T.tracer ] [node_s1] [350][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,383][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] [test][0] skip writing shard state, has been written before [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] set locally applied cluster state to version 25 [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] set locally applied cluster state to version 25 [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.script.MockScriptService@210ffe60] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResponseCollectorService@28e3e0d6] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.InternalClusterInfoService@6427ea90] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.ingest.IngestService@67038f1b] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.snapshots.SnapshotShardsService@8efdc68] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.search.pipeline.SearchPipelineService@10e43f8c] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResourceUsageCollectorService@45863d67] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.store.IndicesStore@78f636a5] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.action.ingest.IngestActionForwarder@1f021051] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@6df82bb6] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.tasks.TaskManager@4120012f] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 491, request id: 350, type: response, version: 3.0.0] READ: 491B [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.persistent.PersistentTasksNodeService@395e152d] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] [test][0] skip writing shard state, has been written before [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@439ed513] with change to version [25] [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] set locally applied cluster state to version 25 [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@4d476a64] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@324e4282] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResponseCollectorService@3a202547] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.script.MockScriptService@5096100c] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@44d2e77] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResponseCollectorService@69665d31] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.snapshots.SnapshotShardsService@b0965d0] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.t.T.tracer ] [node_s0] [350][internal:cluster/coordination/publish_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,383][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.ingest.IngestService@28bfec03] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResourceUsageCollectorService@4067c78] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.search.pipeline.SearchPipelineService@610c2181] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResourceUsageCollectorService@456f6e30] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 130 [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.store.IndicesStore@456e3cae] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksClusterService@70ec0dc3] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.action.ingest.IngestActionForwarder@50cf8326] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.routing.DelayedAllocationService@30f60b0e] with change to version [25] [2023-11-15T19:32:50,383][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.tasks.TaskManager@30b9ef43] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksNodeService@6b2257f4] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.persistent.PersistentTasksNodeService@7e5f3359] with change to version [25] [2023-11-15T19:32:50,384][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] set locally applied cluster state to version 25 [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.gateway.GatewayService@5582aa4e] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 71, request id: 357, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@3e38eea7] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResponseCollectorService@6731c2a3] with change to version [25] [2023-11-15T19:32:50,384][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@96aa0a4] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 148 [2023-11-15T19:32:50,384][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,384][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.snapshots.SnapshotShardsService@6731e7ed] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=25}] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResourceUsageCollectorService@4bcb8e9e] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 282 [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.store.IndicesStore@34f7b353] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.persistent.PersistentTasksNodeService@5d3a5bb4] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s4] [357][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@35dcd171] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 71, request id: 356, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,384][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@2274062a] with change to version [25] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 71, request id: 357, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,384][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,384][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s0] [357][internal:cluster/coordination/commit_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,384][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 724 [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s2] [356][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 71, request id: 359, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 322, request id: 360, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 70, request id: 358, type: response, version: 3.0.0, header size: 47B] WRITE: 70B [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 71, request id: 356, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s3] [359][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s0] [356][internal:cluster/coordination/commit_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s5] [358][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,384][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25} applied version 25, max now 25 [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 70, request id: 358, type: response, version: 3.0.0] READ: 70B [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [79] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s0] [358][internal:cluster/coordination/commit_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,384][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s0] [360][internal:cluster/coordination/commit_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,384][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=25} applied version 25, max now 25 [2023-11-15T19:32:50,384][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 322, request id: 360, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,384][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,384][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 71, request id: 359, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,384][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25} applied version 25, max now 25 [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s0] [359][internal:cluster/coordination/commit_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,384][TRACE][o.o.t.T.tracer ] [node_s1] [360][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,384][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,384][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25} applied version 25, max now 25 [2023-11-15T19:32:50,384][TRACE][o.o.t.TaskManager ] [node_s1] register 154 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,384][TRACE][o.o.c.c.Coordinator ] [node_s1] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,384][TRACE][o.o.c.c.CoordinationState] [node_s1] handleCommit: applying commit request for term [1] and version [25] [2023-11-15T19:32:50,384][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,385][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] cluster state updated, version [25], source [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: true meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,385][TRACE][o.o.c.s.ClusterApplierService] [node_s1] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,385][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] applying settings from cluster state with version 25 [2023-11-15T19:32:50,386][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] apply cluster state with version 25 [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.repositories.RepositoriesService@2e256902] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.indices.cluster.IndicesClusterStateService@58f1b39a] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.script.MockScriptService@b2181c5] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.RestoreService@57f9c669] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.ingest.IngestService@5211c8ed] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.search.pipeline.SearchPipelineService@729cc56] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.ingest.IngestActionForwarder@814adf] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@19521e28] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.tasks.TaskManager@45d8fb4f] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.SnapshotsService@cdac4de] with change to version [25] [2023-11-15T19:32:50,386][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] set locally applied cluster state to version 25 [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.InternalClusterInfoService@6a31ad84] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@659248ee] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@3abdd4fd] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@23792777] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResponseCollectorService@4443e311] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResourceUsageCollectorService@3361164f] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksClusterService@672f36e2] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.routing.DelayedAllocationService@d07e7ef] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksNodeService@5687d635] with change to version [25] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.gateway.GatewayService@6d6f71c4] with change to version [25] [2023-11-15T19:32:50,386][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=25, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,386][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 154 [2023-11-15T19:32:50,386][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,386][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 70, request id: 360, type: response, version: 3.0.0, header size: 47B] WRITE: 70B [2023-11-15T19:32:50,386][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,386][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,386][TRACE][o.o.t.T.tracer ] [node_s1] [360][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,386][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 70, request id: 360, type: response, version: 3.0.0] READ: 70B [2023-11-15T19:32:50,386][TRACE][o.o.t.T.tracer ] [node_s0] [360][internal:cluster/coordination/commit_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,386][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,386][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=25} applied version 25, max now 25 [2023-11-15T19:32:50,386][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] onPossibleCompletion: [Publication{term=1, version=25}] was successful [2023-11-15T19:32:50,386][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=25}]: execute [2023-11-15T19:32:50,387][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] cluster state updated, version [25], source [Publication{term=1, version=25}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 25 state uuid: MmpCNYRNRHGpyFVnnr2LBw from_diff: false meta data version: 20 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 11): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,387][TRACE][o.o.c.s.ClusterApplierService] [node_s0] connecting to nodes of cluster state with version 25 [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,387][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] applying settings from cluster state with version 25 [2023-11-15T19:32:50,388][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] apply cluster state with version 25 [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.repositories.RepositoriesService@4b52a152] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.indices.cluster.IndicesClusterStateService@4f6d2ce0] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.script.MockScriptService@a6fab69] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.RestoreService@65a9c473] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.ingest.IngestService@3b2fa1a5] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.search.pipeline.SearchPipelineService@1d03900c] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.ingest.IngestActionForwarder@6a405426] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@41e19f41] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.tasks.TaskManager@7052c51] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.SnapshotsService@6075f4d] with change to version [25] [2023-11-15T19:32:50,388][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] set locally applied cluster state to version 25 [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.InternalClusterInfoService@23a71628] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@7c26f627] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@73b53b39] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@170fb472] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResponseCollectorService@5cd3ff7a] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResourceUsageCollectorService@660cd06c] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksClusterService@3cba4367] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.routing.DelayedAllocationService@56fcdca6] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.r.DelayedAllocationService] [node_s0] no need to schedule reroute - no delayed unassigned shards [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksNodeService@1597d12c] with change to version [25] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.gateway.GatewayService@1518ec42] with change to version [25] [2023-11-15T19:32:50,388][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=25}]: took [0s] done applying updated cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) [2023-11-15T19:32:50,388][DEBUG][o.o.c.c.C.CoordinatorPublication] [node_s0] publication ended successfully: Publication{term=1, version=25} [2023-11-15T19:32:50,388][TRACE][o.o.c.c.Reconfigurator ] [node_s0] Reconfigurator{autoShrinkVotingConfiguration=true} reconfiguring VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} based on liveNodes=[{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], retiredNodeIds=[], currentClusterManager={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,388][TRACE][o.o.c.c.LagDetector ] [node_s0] lag detection for version 25 is unnecessary: [NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=25}, NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25}, NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25}, NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=25}, NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=25}] [2023-11-15T19:32:50,388][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 25) [2023-11-15T19:32:50,388][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] all expected nodes acknowledged cluster_state update (version: 25) [2023-11-15T19:32:50,389][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on successful publication of cluster state (version: 25, uuid: MmpCNYRNRHGpyFVnnr2LBw) for [cluster_update_settings] [2023-11-15T19:32:50,389][TRACE][o.o.c.s.MasterService ] [node_s0] will process [reroute_after_cluster_update_settings] [2023-11-15T19:32:50,389][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [reroute_after_cluster_update_settings] [2023-11-15T19:32:50,389][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Assigned shard [[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg]] to node [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,389][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Assigned shard [[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A]] to node [9SWyj4IxQoKQhSS_1-SQNQ] [2023-11-15T19:32:50,389][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start allocating unassigned shards [2023-11-15T19:32:50,389][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] getDiskUsage(subtractLeavingShards=true) returning DiskUsageWithRelocations{diskUsage=[pWGsJ26ZTSmfrCNxbu-KoQ][node_s5][/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0] free: 165gb[35.4%], relocatingShardSize=0} [2023-11-15T19:32:50,389][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] node [pWGsJ26ZTSmfrCNxbu-KoQ] has 35.45555978821858% free disk (177264742400 bytes) [2023-11-15T19:32:50,390][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] getDiskUsage(subtractLeavingShards=true) returning DiskUsageWithRelocations{diskUsage=[9SWyj4IxQoKQhSS_1-SQNQ][node_s3][/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0] free: 165gb[35.4%], relocatingShardSize=0} [2023-11-15T19:32:50,390][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] node [9SWyj4IxQoKQhSS_1-SQNQ] has 35.45555978821858% free disk (177264742400 bytes) [2023-11-15T19:32:50,390][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start balancing cluster [2023-11-15T19:32:50,390][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Stop balancing index [test] min_node [9SWyj4IxQoKQhSS_1-SQNQ] weight: [0.3333333] max_node [OhN8Eeh3SuiVD7m3P0ly7Q] weight: [-0.6666667] delta: [1.0] [2023-11-15T19:32:50,390][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No unassigned remote shards found. [2023-11-15T19:32:50,390][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No excluded nodes found. Returning... [2023-11-15T19:32:50,390][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Performing balancing for remote shards. [2023-11-15T19:32:50,390][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No eligible remote nodes found to perform balancing [2023-11-15T19:32:50,390][DEBUG][o.o.c.s.MasterService ] [node_s0] took [1ms] to compute cluster state update for [reroute_after_cluster_update_settings] [2023-11-15T19:32:50,390][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 212 [2023-11-15T19:32:50,390][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16519] [2023-11-15T19:32:50,390][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61538} [length: 74, request id: 67, type: response, version: 3.0.0, header size: 48B] WRITE: 74B [2023-11-15T19:32:50,390][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-79] bytes, now [16440] [2023-11-15T19:32:50,390][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on unchanged cluster state for [reroute_after_cluster_update_settings] [2023-11-15T19:32:50,390][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,390][TRACE][o.o.t.T.tracer ] [node_s0] [67][cluster:admin/settings/update] sent response [2023-11-15T19:32:50,391][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61538, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 74, request id: 67, type: response, version: 3.0.0] READ: 74B [2023-11-15T19:32:50,391][TRACE][o.o.t.T.tracer ] [node_s4] [67][cluster:admin/settings/update] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,391][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 128 [2023-11-15T19:32:50,391][INFO ][o.o.s.q.SearchQueryIT ] [testCommonTermsQuery] [SearchQueryIT#testCommonTermsQuery [p0={"search.concurrent_segment_search.enabled":"false"} seed=[D490856B1361B346:5B09F6EE7D3C6926]]]: cleaning up after test [2023-11-15T19:32:50,391][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 155 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,391][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@2f2f1aaf] with cluster state version [25] [2023-11-15T19:32:50,391][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:monitor/state [2023-11-15T19:32:50,391][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,391][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61566, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 142, request id: 66, type: request, version: 3.0.0, header size: 71B, action: cluster:monitor/state] WRITE: 142B [2023-11-15T19:32:50,391][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,392][TRACE][o.o.t.T.tracer ] [node_s1] [66][cluster:monitor/state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,392][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [48] bytes, now [48] [2023-11-15T19:32:50,392][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61566} [length: 142, request id: 66, type: request, version: 3.0.0, action: cluster:monitor/state] READ: 142B [2023-11-15T19:32:50,392][TRACE][o.o.t.T.tracer ] [node_s0] [66][cluster:monitor/state] received request [2023-11-15T19:32:50,392][TRACE][o.o.t.TaskManager ] [node_s0] register 215 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,392][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@577f8e91] with cluster state version [25] [2023-11-15T19:32:50,392][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [node_s0] Serving cluster state request using version 25 [2023-11-15T19:32:50,392][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 215 [2023-11-15T19:32:50,392][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16488] [2023-11-15T19:32:50,392][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61566} [length: 2979, request id: 66, type: response, version: 3.0.0, header size: 48B] WRITE: 2979B [2023-11-15T19:32:50,392][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [48] [2023-11-15T19:32:50,392][TRACE][o.o.t.T.tracer ] [node_s0] [66][cluster:monitor/state] sent response [2023-11-15T19:32:50,392][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-48] bytes, now [0] [2023-11-15T19:32:50,392][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61566, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 2979, request id: 66, type: response, version: 3.0.0] READ: 2979B [2023-11-15T19:32:50,392][TRACE][o.o.t.T.tracer ] [node_s1] [66][cluster:monitor/state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,393][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 155 [2023-11-15T19:32:50,393][TRACE][o.o.s.q.SearchQueryIT ] [testCommonTermsQuery] Check consistency for [6] nodes [2023-11-15T19:32:50,393][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 156 [transport] [cluster:monitor/health] [] [2023-11-15T19:32:50,393][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.health.ClusterHealthRequest@1ea3d82e] with cluster state version [25] [2023-11-15T19:32:50,393][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:monitor/health [2023-11-15T19:32:50,393][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,393][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61561, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 152, request id: 67, type: request, version: 3.0.0, header size: 72B, action: cluster:monitor/health] WRITE: 152B [2023-11-15T19:32:50,393][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,393][TRACE][o.o.t.T.tracer ] [node_s1] [67][cluster:monitor/health] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,393][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [57] bytes, now [57] [2023-11-15T19:32:50,393][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61561} [length: 152, request id: 67, type: request, version: 3.0.0, action: cluster:monitor/health] READ: 152B [2023-11-15T19:32:50,393][TRACE][o.o.t.T.tracer ] [node_s0] [67][cluster:monitor/health] received request [2023-11-15T19:32:50,393][TRACE][o.o.t.TaskManager ] [node_s0] register 216 [transport] [cluster:monitor/health] [] [2023-11-15T19:32:50,393][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.health.ClusterHealthRequest@1fb56c3d] with cluster state version [25] [2023-11-15T19:32:50,393][TRACE][o.o.a.a.c.h.TransportClusterHealthAction] [node_s0] Calculating health based on state version [25] [2023-11-15T19:32:50,393][TRACE][o.o.a.a.c.h.TransportClusterHealthAction] [node_s0] Calculating health based on state version [25] [2023-11-15T19:32:50,394][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 216 [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16497] [2023-11-15T19:32:50,394][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61561} [length: 229, request id: 67, type: response, version: 3.0.0, header size: 48B] WRITE: 229B [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [57] [2023-11-15T19:32:50,394][TRACE][o.o.t.T.tracer ] [node_s0] [67][cluster:monitor/health] sent response [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-57] bytes, now [0] [2023-11-15T19:32:50,394][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61561, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 229, request id: 67, type: response, version: 3.0.0] READ: 229B [2023-11-15T19:32:50,394][TRACE][o.o.t.T.tracer ] [node_s1] [67][cluster:monitor/health] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,394][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 156 [2023-11-15T19:32:50,394][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 157 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,394][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@50424314] with cluster state version [25] [2023-11-15T19:32:50,394][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:monitor/state [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,394][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61568, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 142, request id: 68, type: request, version: 3.0.0, header size: 71B, action: cluster:monitor/state] WRITE: 142B [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,394][TRACE][o.o.t.T.tracer ] [node_s1] [68][cluster:monitor/state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,394][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [48] bytes, now [48] [2023-11-15T19:32:50,394][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61568} [length: 142, request id: 68, type: request, version: 3.0.0, action: cluster:monitor/state] READ: 142B [2023-11-15T19:32:50,395][TRACE][o.o.t.T.tracer ] [node_s0] [68][cluster:monitor/state] received request [2023-11-15T19:32:50,395][TRACE][o.o.t.TaskManager ] [node_s0] register 217 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,395][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@2adb973e] with cluster state version [25] [2023-11-15T19:32:50,395][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [node_s0] Serving cluster state request using version 25 [2023-11-15T19:32:50,395][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 217 [2023-11-15T19:32:50,395][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16488] [2023-11-15T19:32:50,395][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61568} [length: 2979, request id: 68, type: response, version: 3.0.0, header size: 48B] WRITE: 2979B [2023-11-15T19:32:50,395][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [48] [2023-11-15T19:32:50,395][TRACE][o.o.t.T.tracer ] [node_s0] [68][cluster:monitor/state] sent response [2023-11-15T19:32:50,395][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-48] bytes, now [0] [2023-11-15T19:32:50,395][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61568, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 2979, request id: 68, type: response, version: 3.0.0] READ: 2979B [2023-11-15T19:32:50,395][TRACE][o.o.t.T.tracer ] [node_s1] [68][cluster:monitor/state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,396][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 157 [2023-11-15T19:32:50,397][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 218 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,397][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@7aa33a10] with cluster state version [25] [2023-11-15T19:32:50,397][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,397][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 218 [2023-11-15T19:32:50,399][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 158 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,399][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@500a23db] with cluster state version [25] [2023-11-15T19:32:50,399][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,399][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 158 [2023-11-15T19:32:50,401][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 149 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,401][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@5bbfe07e] with cluster state version [25] [2023-11-15T19:32:50,401][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,402][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 149 [2023-11-15T19:32:50,403][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 283 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,403][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@1cc165e4] with cluster state version [25] [2023-11-15T19:32:50,403][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,404][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 283 [2023-11-15T19:32:50,406][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 131 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,406][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@5708881c] with cluster state version [25] [2023-11-15T19:32:50,406][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,406][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 131 [2023-11-15T19:32:50,408][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 725 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,408][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@719c667d] with cluster state version [25] [2023-11-15T19:32:50,408][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [testCommonTermsQuery] Serving cluster state request using version 25 [2023-11-15T19:32:50,408][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] unregister task for id: 725 [2023-11-15T19:32:50,410][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 159 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,410][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@5138969e] with cluster state version [25] [2023-11-15T19:32:50,411][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:monitor/state [2023-11-15T19:32:50,411][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,411][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61566, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 142, request id: 69, type: request, version: 3.0.0, header size: 71B, action: cluster:monitor/state] WRITE: 142B [2023-11-15T19:32:50,411][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,411][TRACE][o.o.t.T.tracer ] [node_s1] [69][cluster:monitor/state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,411][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [48] bytes, now [48] [2023-11-15T19:32:50,411][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61566} [length: 142, request id: 69, type: request, version: 3.0.0, action: cluster:monitor/state] READ: 142B [2023-11-15T19:32:50,411][TRACE][o.o.t.T.tracer ] [node_s0] [69][cluster:monitor/state] received request [2023-11-15T19:32:50,411][TRACE][o.o.t.TaskManager ] [node_s0] register 219 [transport] [cluster:monitor/state] [] [2023-11-15T19:32:50,411][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.state.ClusterStateRequest@4b7b60f1] with cluster state version [25] [2023-11-15T19:32:50,411][TRACE][o.o.a.a.c.s.TransportClusterStateAction] [node_s0] Serving cluster state request using version 25 [2023-11-15T19:32:50,412][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 219 [2023-11-15T19:32:50,412][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16488] [2023-11-15T19:32:50,412][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61566} [length: 2979, request id: 69, type: response, version: 3.0.0, header size: 48B] WRITE: 2979B [2023-11-15T19:32:50,412][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [48] [2023-11-15T19:32:50,412][TRACE][o.o.t.T.tracer ] [node_s0] [69][cluster:monitor/state] sent response [2023-11-15T19:32:50,412][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-48] bytes, now [0] [2023-11-15T19:32:50,412][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61566, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 2979, request id: 69, type: response, version: 3.0.0] READ: 2979B [2023-11-15T19:32:50,412][TRACE][o.o.t.T.tracer ] [node_s1] [69][cluster:monitor/state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,412][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 159 [2023-11-15T19:32:50,416][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 726 [transport] [indices:admin/data_stream/delete] [] [2023-11-15T19:32:50,416][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.indices.datastream.DeleteDataStreamAction$Request@49] with cluster state version [25] [2023-11-15T19:32:50,416][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:admin/data_stream/delete [2023-11-15T19:32:50,416][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,416][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61550, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 141, request id: 512, type: request, version: 3.0.0, header size: 82B, action: indices:admin/data_stream/delete] WRITE: 141B [2023-11-15T19:32:50,416][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,416][TRACE][o.o.t.T.tracer ] [node_s5] [512][indices:admin/data_stream/delete] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,416][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adding [36b][indices:admin/data_stream/delete] to used bytes [new used: [36b], limit: 4294967296 [4gb], estimate: 72 [72b]] [2023-11-15T19:32:50,416][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61550} [length: 141, request id: 512, type: request, version: 3.0.0, action: indices:admin/data_stream/delete] READ: 141B [2023-11-15T19:32:50,417][TRACE][o.o.t.T.tracer ] [node_s0] [512][indices:admin/data_stream/delete] received request [2023-11-15T19:32:50,417][TRACE][o.o.t.TaskManager ] [node_s0] register 220 [transport] [indices:admin/data_stream/delete] [] [2023-11-15T19:32:50,417][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.indices.datastream.DeleteDataStreamAction$Request@49] with cluster state version [25] [2023-11-15T19:32:50,417][TRACE][o.o.c.s.MasterService ] [node_s0] will process [remove-data-stream [*]] [2023-11-15T19:32:50,417][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [remove-data-stream [*]] [2023-11-15T19:32:50,417][TRACE][o.o.c.m.MetadataDeleteIndexService] [node_s0] 0 tombstones purged from the cluster state. Previous tombstone size: 1. Current tombstone size: 1. [2023-11-15T19:32:50,417][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Assigned shard [[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg]] to node [pWGsJ26ZTSmfrCNxbu-KoQ] [2023-11-15T19:32:50,417][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Assigned shard [[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A]] to node [9SWyj4IxQoKQhSS_1-SQNQ] [2023-11-15T19:32:50,417][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start allocating unassigned shards [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] getDiskUsage(subtractLeavingShards=true) returning DiskUsageWithRelocations{diskUsage=[pWGsJ26ZTSmfrCNxbu-KoQ][node_s5][/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0] free: 165gb[35.4%], relocatingShardSize=0} [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] node [pWGsJ26ZTSmfrCNxbu-KoQ] has 35.45555978821858% free disk (177264742400 bytes) [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] getDiskUsage(subtractLeavingShards=true) returning DiskUsageWithRelocations{diskUsage=[9SWyj4IxQoKQhSS_1-SQNQ][node_s3][/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0] free: 165gb[35.4%], relocatingShardSize=0} [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.d.DiskThresholdDecider] [node_s0] node [9SWyj4IxQoKQhSS_1-SQNQ] has 35.45555978821858% free disk (177264742400 bytes) [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start balancing cluster [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Stop balancing index [test] min_node [9SWyj4IxQoKQhSS_1-SQNQ] weight: [0.3333333] max_node [OhN8Eeh3SuiVD7m3P0ly7Q] weight: [-0.6666667] delta: [1.0] [2023-11-15T19:32:50,418][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No unassigned remote shards found. [2023-11-15T19:32:50,418][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No excluded nodes found. Returning... [2023-11-15T19:32:50,418][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Performing balancing for remote shards. [2023-11-15T19:32:50,418][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No eligible remote nodes found to perform balancing [2023-11-15T19:32:50,418][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to compute cluster state update for [remove-data-stream [*]] [2023-11-15T19:32:50,418][TRACE][o.o.c.s.MasterService ] [node_s0] cluster state updated, source [remove-data-stream [*]] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: false meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,419][DEBUG][o.o.c.s.MasterService ] [node_s0] publishing cluster state version [26] [2023-11-15T19:32:50,419][TRACE][o.o.c.c.CompressedStreamUtils] [node_s0] serialized writable object for node version [3.0.0] with size [265] [2023-11-15T19:32:50,419][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] serialized cluster state diff for version [26] in for node version [3.0.0] with size [265] [2023-11-15T19:32:50,419][TRACE][o.o.c.c.CoordinationState] [node_s0] handleClientValue: processing request for version [26] and term [1] [2023-11-15T19:32:50,419][TRACE][o.o.c.c.LeaderChecker ] [node_s0] setCurrentNodes: nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,419][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] publishing PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: false meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned } to [PublicationTarget{discoveryNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}] [2023-11-15T19:32:50,419][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,419][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,419][TRACE][o.o.t.T.tracer ] [node_s0] [361][internal:cluster/coordination/publish_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,419][TRACE][o.o.t.T.tracer ] [node_s0] [361][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,420][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,420][TRACE][o.o.t.TaskManager ] [node_s0] register 221 [direct] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,420][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 384, request id: 362, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,420][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,420][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s0] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,420][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 384, request id: 363, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s0] [362][internal:cluster/coordination/publish_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,420][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,420][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s0] [363][internal:cluster/coordination/publish_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 384, request id: 362, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61583, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 384, request id: 364, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s2] [362][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 384, request id: 363, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,420][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s0] [364][internal:cluster/coordination/publish_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,420][TRACE][o.o.t.TaskManager ] [node_s2] register 150 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61583} [length: 384, request id: 364, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,420][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s1] [363][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s4] [364][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,420][TRACE][o.o.t.TaskManager ] [node_s1] register 160 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,420][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 384, request id: 365, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,420][TRACE][o.o.t.TaskManager ] [node_s4] register 132 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,420][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s2] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,420][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [26] to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,420][TRACE][o.o.t.T.tracer ] [node_s0] [365][internal:cluster/coordination/publish_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,420][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s4] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,420][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,420][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s1] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,420][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,421][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,421][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 384, request id: 365, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,421][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 384, request id: 366, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 384B [2023-11-15T19:32:50,421][TRACE][o.o.t.T.tracer ] [node_s5] [365][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,421][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,421][TRACE][o.o.t.TaskManager ] [node_s5] register 727 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,421][TRACE][o.o.t.T.tracer ] [node_s0] [366][internal:cluster/coordination/publish_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,421][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [268] bytes, now [268] [2023-11-15T19:32:50,421][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 384, request id: 366, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 384B [2023-11-15T19:32:50,421][TRACE][o.o.t.T.tracer ] [node_s3] [366][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s0] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: false meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,421][TRACE][o.o.t.TaskManager ] [node_s3] register 284 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,421][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s5] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.CoordinationState] [node_s2] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.CoordinationState] [node_s1] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.CoordinationState] [node_s4] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s0] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,421][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] queuing cluster state update (setting cluster state to 26) [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s2] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s1] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,421][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] resetting newStateQueued [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s2] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s1] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,421][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,421][TRACE][o.o.d.PeerFinder ] [node_s4] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s4] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,421][TRACE][o.o.d.PeerFinder ] [node_s4] not active [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s0] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.PreVoteCollector ] [node_s4] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,421][TRACE][o.o.g.PersistedClusterStateService] [node_s4] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,421][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,421][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 132 [2023-11-15T19:32:50,421][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,421][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,421][TRACE][o.o.c.c.CoordinationState] [node_s5] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,421][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] queuing cluster state update (setting cluster state to 26) [2023-11-15T19:32:50,421][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61583} [length: 481, request id: 364, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,421][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,421][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,421][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,421][TRACE][o.o.d.PeerFinder ] [node_s5] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,421][TRACE][o.o.d.PeerFinder ] [node_s5] not active [2023-11-15T19:32:50,421][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s3] received diff cluster state version [26] with uuid [cIp1K_y6SQyeYbeamk1ymg], diff size [265] [2023-11-15T19:32:50,422][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] resetting newStateQueued [2023-11-15T19:32:50,422][TRACE][o.o.c.c.PreVoteCollector ] [node_s5] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,422][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 727 [2023-11-15T19:32:50,422][TRACE][o.o.g.PersistedClusterStateService] [node_s5] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,422][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,422][TRACE][o.o.g.PersistedClusterStateService] [node_s5] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,422][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,422][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 481, request id: 365, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,422][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,421][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,422][TRACE][o.o.t.T.tracer ] [node_s4] [364][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,422][TRACE][o.o.t.T.tracer ] [node_s5] [365][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,422][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: handling [PublishRequest{term=1, version=26, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,422][TRACE][o.o.c.c.CoordinationState] [node_s3] handlePublishRequest: accepting publish request for version [26] and term [1] [2023-11-15T19:32:50,422][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] queuing cluster state update (setting cluster state to 26) [2023-11-15T19:32:50,422][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,422][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61583, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 481, request id: 364, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,422][TRACE][o.o.d.PeerFinder ] [node_s3] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,422][TRACE][o.o.d.PeerFinder ] [node_s3] not active [2023-11-15T19:32:50,422][TRACE][o.o.t.T.tracer ] [node_s0] [364][internal:cluster/coordination/publish_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,422][TRACE][o.o.c.c.PreVoteCollector ] [node_s3] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,422][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 284 [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,422][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 481, request id: 366, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,422][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,422][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] resetting newStateQueued [2023-11-15T19:32:50,422][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 481, request id: 365, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,422][TRACE][o.o.t.T.tracer ] [node_s0] [365][internal:cluster/coordination/publish_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,422][TRACE][o.o.g.PersistedClusterStateService] [node_s3] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,422][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,423][TRACE][o.o.g.PersistedClusterStateService] [node_s3] no action required for [[test/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,423][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,423][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,423][TRACE][o.o.t.T.tracer ] [node_s3] [366][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,423][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,423][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 481, request id: 366, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,423][TRACE][o.o.t.T.tracer ] [node_s0] [366][internal:cluster/coordination/publish_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s4] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s3] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,429][TRACE][o.o.d.PeerFinder ] [node_s2] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,429][TRACE][o.o.d.PeerFinder ] [node_s2] not active [2023-11-15T19:32:50,429][TRACE][o.o.c.c.PreVoteCollector ] [node_s2] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 150 [2023-11-15T19:32:50,429][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,429][TRACE][o.o.d.PeerFinder ] [node_s1] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,429][TRACE][o.o.d.PeerFinder ] [node_s1] not active [2023-11-15T19:32:50,429][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s0] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][TRACE][o.o.c.c.PreVoteCollector ] [node_s1] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,429][DEBUG][o.o.g.PersistedClusterStateService] [node_s5] writing cluster state took [0ms]; wrote global metadata [false] and metadata for [0] indices and skipped [1] unchanged indices [2023-11-15T19:32:50,429][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 160 [2023-11-15T19:32:50,429][TRACE][o.o.c.c.PreVoteCollector ] [node_s0] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=26}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,429][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16708] [2023-11-15T19:32:50,429][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 492, request id: 362, type: response, version: 3.0.0, header size: 48B] WRITE: 492B [2023-11-15T19:32:50,430][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 221 [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s0] [361][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,430][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,430][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 491, request id: 363, type: response, version: 3.0.0, header size: 47B] WRITE: 491B [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s0] [361][internal:cluster/coordination/publish_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,430][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,430][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-268] bytes, now [16440] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [26] and term [1] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s2] [362][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,430][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s1] [363][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,430][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 492, request id: 362, type: response, version: 3.0.0] READ: 492B [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s0] [362][internal:cluster/coordination/publish_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,430][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [26] and term [1] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 491, request id: 363, type: response, version: 3.0.0] READ: 491B [2023-11-15T19:32:50,430][TRACE][o.o.t.T.tracer ] [node_s0] [363][internal:cluster/coordination/publish_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,430][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [26] and term [1] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,430][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [26] and term [1] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,430][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,431][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [26] and term [1] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,431][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: value committed for version [26] and term [1] [2023-11-15T19:32:50,431][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 321, request id: 367, type: request, version: 3.0.0, header size: 91B, action: internal:cluster/coordination/commit_state] WRITE: 321B [2023-11-15T19:32:50,431][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 322, request id: 368, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s0] [367][internal:cluster/coordination/commit_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 321, request id: 367, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 321B [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,431][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s0] [368][internal:cluster/coordination/commit_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s2] [367][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 322, request id: 368, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 322, request id: 369, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,431][TRACE][o.o.t.TaskManager ] [node_s2] register 151 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s1] [368][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,431][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s0] [369][internal:cluster/coordination/commit_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,431][TRACE][o.o.c.c.Coordinator ] [node_s2] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 322, request id: 369, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,431][TRACE][o.o.t.TaskManager ] [node_s1] register 161 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,431][TRACE][o.o.c.c.CoordinationState] [node_s2] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,431][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 322, request id: 370, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s4] [369][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,431][TRACE][o.o.c.c.Coordinator ] [node_s1] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,431][TRACE][o.o.c.c.CoordinationState] [node_s1] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,431][TRACE][o.o.t.TaskManager ] [node_s4] register 133 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,431][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,431][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,431][TRACE][o.o.t.T.tracer ] [node_s0] [370][internal:cluster/coordination/commit_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,431][TRACE][o.o.c.c.Coordinator ] [node_s4] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,432][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 322, request id: 371, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,432][TRACE][o.o.c.c.CoordinationState] [node_s4] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,431][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,432][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 322, request id: 370, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,432][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [36] [2023-11-15T19:32:50,432][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s5] [370][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,432][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,432][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=26}] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,432][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 322, request id: 371, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s0] [371][internal:cluster/coordination/commit_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,432][TRACE][o.o.t.TaskManager ] [node_s5] register 728 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,432][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s3] [371][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,432][TRACE][o.o.c.c.Coordinator ] [node_s5] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,432][TRACE][o.o.c.c.CoordinationState] [node_s5] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s0] [372][internal:cluster/coordination/commit_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s0] [372][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] cluster state updated, version [26], source [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] cluster state updated, version [26], source [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.t.TaskManager ] [node_s3] register 285 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.t.TaskManager ] [node_s0] register 222 [direct] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.c.Coordinator ] [node_s3] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,432][TRACE][o.o.c.c.Coordinator ] [node_s0] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.c.CoordinationState] [node_s3] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] cluster state updated, version [26], source [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.c.CoordinationState] [node_s0] handleCommit: applying commit request for term [1] and version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,432][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 222 [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s0] [372][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.t.T.tracer ] [node_s0] [372][internal:cluster/coordination/commit_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] applying settings from cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] apply cluster state with version 26 [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] applying settings from cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.repositories.RepositoriesService@2e256902] with change to version [26] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] apply cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.indices.cluster.IndicesClusterStateService@58f1b39a] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.repositories.RepositoriesService@417b8324] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.indices.cluster.IndicesClusterStateService@45440fd3] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.script.MockScriptService@b2181c5] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.RestoreService@57f9c669] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.script.MockScriptService@46891f] with change to version [26] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] applying settings from cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.ingest.IngestService@5211c8ed] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.RestoreService@2cf36cc8] with change to version [26] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] apply cluster state with version 26 [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] cluster state updated, version [26], source [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.repositories.RepositoriesService@5daadbc8] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.search.pipeline.SearchPipelineService@729cc56] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.cluster.IndicesClusterStateService@599544a5] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s3] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.ingest.IngestService@310ca8d1] with change to version [26] [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] cluster state updated, version [26], source [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: true meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.ingest.IngestActionForwarder@814adf] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.search.pipeline.SearchPipelineService@545313b4] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.script.MockScriptService@1c97d1f3] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.ingest.IngestActionForwarder@60697650] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.ingest.IngestService@26be5633] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s5] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@19521e28] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.tasks.TaskManager@45d8fb4f] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@574bd3b9] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.SnapshotsService@cdac4de] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.search.pipeline.SearchPipelineService@2d41adb7] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.tasks.TaskManager@3d5b3a1] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] set locally applied cluster state to version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.InternalClusterInfoService@6a31ad84] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.SnapshotsService@4f57dedc] with change to version [26] [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.action.ingest.IngestActionForwarder@5e8274d6] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,433][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] set locally applied cluster state to version 26 [2023-11-15T19:32:50,433][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.InternalClusterInfoService@6427ea90] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@6df82bb6] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] applying settings from cluster state with version 26 [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] applying settings from cluster state with version 26 [2023-11-15T19:32:50,432][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@659248ee] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] apply cluster state with version 26 [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@439ed513] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.repositories.RepositoriesService@11ab1a37] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] apply cluster state with version 26 [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@3abdd4fd] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.cluster.IndicesClusterStateService@15948cda] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.tasks.TaskManager@7cb82ebc] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@23792777] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResponseCollectorService@4443e311] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.repositories.RepositoriesService@6032f45b] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResourceUsageCollectorService@3361164f] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] set locally applied cluster state to version 26 [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.cluster.IndicesClusterStateService@640d6c85] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResponseCollectorService@28e3e0d6] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.snapshots.SnapshotShardsService@8efdc68] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResourceUsageCollectorService@45863d67] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@324e4282] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.store.IndicesStore@78f636a5] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksClusterService@672f36e2] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResponseCollectorService@69665d31] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.s.IndexShard ] [node_s3] [test][0] [test][0] skip writing shard state, has been written before [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.routing.DelayedAllocationService@d07e7ef] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.persistent.PersistentTasksNodeService@395e152d] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.script.MockScriptService@210ffe60] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@4d476a64] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksNodeService@5687d635] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@44d2e77] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResourceUsageCollectorService@456f6e30] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksClusterService@70ec0dc3] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.ingest.IngestService@67038f1b] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 133 [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.routing.DelayedAllocationService@30f60b0e] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.gateway.GatewayService@6d6f71c4] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksNodeService@6b2257f4] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.search.pipeline.SearchPipelineService@10e43f8c] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,433][TRACE][o.o.i.s.IndexShard ] [node_s5] [test][0] [test][0] skip writing shard state, has been written before [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.action.ingest.IngestActionForwarder@1f021051] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.tasks.TaskManager@4120012f] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.gateway.GatewayService@5582aa4e] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] set locally applied cluster state to version 26 [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 70, request id: 369, type: response, version: 3.0.0, header size: 47B] WRITE: 70B [2023-11-15T19:32:50,433][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 161 [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.script.MockScriptService@5096100c] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResponseCollectorService@3a202547] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.ingest.IngestService@28bfec03] with change to version [26] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.snapshots.SnapshotShardsService@b0965d0] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.search.pipeline.SearchPipelineService@610c2181] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.action.ingest.IngestActionForwarder@50cf8326] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResourceUsageCollectorService@4067c78] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.tasks.TaskManager@30b9ef43] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 70, request id: 368, type: response, version: 3.0.0, header size: 47B] WRITE: 70B [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] set locally applied cluster state to version 26 [2023-11-15T19:32:50,433][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 151 [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResponseCollectorService@6731c2a3] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.T.tracer ] [node_s4] [369][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.snapshots.SnapshotShardsService@6731e7ed] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.store.IndicesStore@456e3cae] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResourceUsageCollectorService@4bcb8e9e] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.persistent.PersistentTasksNodeService@7e5f3359] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.store.IndicesStore@34f7b353] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 70, request id: 369, type: response, version: 3.0.0] READ: 70B [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.persistent.PersistentTasksNodeService@5d3a5bb4] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@3e38eea7] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 71, request id: 367, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@35dcd171] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@96aa0a4] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.T.tracer ] [node_s1] [368][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,433][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@2274062a] with change to version [26] [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 70, request id: 368, type: response, version: 3.0.0] READ: 70B [2023-11-15T19:32:50,433][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 285 [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,433][TRACE][o.o.t.T.tracer ] [node_s0] [369][internal:cluster/coordination/commit_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,433][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=26, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 71, request id: 371, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,433][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 728 [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,433][TRACE][o.o.t.T.tracer ] [node_s0] [368][internal:cluster/coordination/commit_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,433][TRACE][o.o.t.T.tracer ] [node_s2] [367][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,433][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,433][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26} applied version 26, max now 26 [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 71, request id: 367, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,433][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 71, request id: 370, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,434][TRACE][o.o.t.T.tracer ] [node_s0] [367][internal:cluster/coordination/commit_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,434][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,434][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=26} applied version 26, max now 26 [2023-11-15T19:32:50,434][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,434][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 71, request id: 371, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,434][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,434][TRACE][o.o.t.T.tracer ] [node_s3] [371][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,434][TRACE][o.o.t.T.tracer ] [node_s5] [370][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,434][TRACE][o.o.t.T.tracer ] [node_s0] [371][internal:cluster/coordination/commit_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,434][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=26} applied version 26, max now 26 [2023-11-15T19:32:50,434][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26} applied version 26, max now 26 [2023-11-15T19:32:50,434][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 71, request id: 370, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,434][TRACE][o.o.t.T.tracer ] [node_s0] [370][internal:cluster/coordination/commit_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,434][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26} applied version 26, max now 26 [2023-11-15T19:32:50,434][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] onPossibleCompletion: [Publication{term=1, version=26}] was successful [2023-11-15T19:32:50,434][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=26}]: execute [2023-11-15T19:32:50,434][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] cluster state updated, version [26], source [Publication{term=1, version=26}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 26 state uuid: cIp1K_y6SQyeYbeamk1ymg from_diff: false meta data version: 21 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] [test/VYTqcztNQ8mZP5spA2-HmQ]: v[5], mv[2], sv[1], av[1] 0: p_term [1], isa_ids [oIyDaBU5RJCXL98jM8pOHg, xRjt2CCHQuiHlKD45x631A] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 12): -- index [[test/VYTqcztNQ8mZP5spA2-HmQ]] ----shard_id [test][0] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] --------[test][0], node[pWGsJ26ZTSmfrCNxbu-KoQ], [P], s[STARTED], a[id=oIyDaBU5RJCXL98jM8pOHg] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] --------[test][0], node[9SWyj4IxQoKQhSS_1-SQNQ], [R], s[STARTED], a[id=xRjt2CCHQuiHlKD45x631A] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,434][TRACE][o.o.c.s.ClusterApplierService] [node_s0] connecting to nodes of cluster state with version 26 [2023-11-15T19:32:50,434][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,434][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] applying settings from cluster state with version 26 [2023-11-15T19:32:50,435][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] apply cluster state with version 26 [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.repositories.RepositoriesService@4b52a152] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.indices.cluster.IndicesClusterStateService@4f6d2ce0] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.script.MockScriptService@a6fab69] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.RestoreService@65a9c473] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.ingest.IngestService@3b2fa1a5] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.search.pipeline.SearchPipelineService@1d03900c] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.ingest.IngestActionForwarder@6a405426] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@41e19f41] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.tasks.TaskManager@7052c51] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.SnapshotsService@6075f4d] with change to version [26] [2023-11-15T19:32:50,435][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] set locally applied cluster state to version 26 [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.InternalClusterInfoService@23a71628] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@7c26f627] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@73b53b39] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@170fb472] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResponseCollectorService@5cd3ff7a] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResourceUsageCollectorService@660cd06c] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksClusterService@3cba4367] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.routing.DelayedAllocationService@56fcdca6] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.r.DelayedAllocationService] [node_s0] no need to schedule reroute - no delayed unassigned shards [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksNodeService@1597d12c] with change to version [26] [2023-11-15T19:32:50,435][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.gateway.GatewayService@1518ec42] with change to version [26] [2023-11-15T19:32:50,435][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=26}]: took [0s] done applying updated cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) [2023-11-15T19:32:50,435][DEBUG][o.o.c.c.C.CoordinatorPublication] [node_s0] publication ended successfully: Publication{term=1, version=26} [2023-11-15T19:32:50,435][TRACE][o.o.c.c.Reconfigurator ] [node_s0] Reconfigurator{autoShrinkVotingConfiguration=true} reconfiguring VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} based on liveNodes=[{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], retiredNodeIds=[], currentClusterManager={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,435][TRACE][o.o.c.c.LagDetector ] [node_s0] lag detection for version 26 is unnecessary: [NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=26}, NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26}, NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26}, NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=26}, NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=26}] [2023-11-15T19:32:50,435][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 220 [2023-11-15T19:32:50,435][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16476] [2023-11-15T19:32:50,436][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61550} [length: 72, request id: 512, type: response, version: 3.0.0, header size: 48B] WRITE: 72B [2023-11-15T19:32:50,436][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-36] bytes, now [16440] [2023-11-15T19:32:50,436][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on successful publication of cluster state (version: 26, uuid: cIp1K_y6SQyeYbeamk1ymg) for [remove-data-stream [*]] [2023-11-15T19:32:50,436][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,436][TRACE][o.o.t.T.tracer ] [node_s0] [512][indices:admin/data_stream/delete] sent response [2023-11-15T19:32:50,436][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61550, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 72, request id: 512, type: response, version: 3.0.0] READ: 72B [2023-11-15T19:32:50,436][TRACE][o.o.t.T.tracer ] [node_s5] [512][indices:admin/data_stream/delete] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,436][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 726 [2023-11-15T19:32:50,436][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 134 [transport] [indices:admin/delete] [] [2023-11-15T19:32:50,436][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.indices.delete.DeleteIndexRequest@744456c6] with cluster state version [26] [2023-11-15T19:32:50,436][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:admin/delete [2023-11-15T19:32:50,436][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,436][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61556, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 140, request id: 68, type: request, version: 3.0.0, header size: 68B, action: indices:admin/delete] WRITE: 140B [2023-11-15T19:32:50,436][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,436][TRACE][o.o.t.T.tracer ] [node_s4] [68][indices:admin/delete] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,436][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adding [49b][indices:admin/delete] to used bytes [new used: [49b], limit: 4294967296 [4gb], estimate: 98 [98b]] [2023-11-15T19:32:50,437][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61556} [length: 140, request id: 68, type: request, version: 3.0.0, action: indices:admin/delete] READ: 140B [2023-11-15T19:32:50,437][TRACE][o.o.t.T.tracer ] [node_s0] [68][indices:admin/delete] received request [2023-11-15T19:32:50,437][TRACE][o.o.t.TaskManager ] [node_s0] register 223 [transport] [indices:admin/delete] [] [2023-11-15T19:32:50,437][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.indices.delete.DeleteIndexRequest@12d68ea1] with cluster state version [26] [2023-11-15T19:32:50,437][TRACE][o.o.c.s.MasterService ] [node_s0] will process [delete-index [[test/VYTqcztNQ8mZP5spA2-HmQ]]] [2023-11-15T19:32:50,437][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [delete-index [[test/VYTqcztNQ8mZP5spA2-HmQ]]] [2023-11-15T19:32:50,437][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_s0] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index [2023-11-15T19:32:50,437][TRACE][o.o.c.m.MetadataDeleteIndexService] [node_s0] 0 tombstones purged from the cluster state. Previous tombstone size: 1. Current tombstone size: 2. [2023-11-15T19:32:50,437][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start allocating unassigned shards [2023-11-15T19:32:50,437][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Start balancing cluster [2023-11-15T19:32:50,437][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No unassigned remote shards found. [2023-11-15T19:32:50,437][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No excluded nodes found. Returning... [2023-11-15T19:32:50,437][TRACE][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] Performing balancing for remote shards. [2023-11-15T19:32:50,437][DEBUG][o.o.c.r.a.a.BalancedShardsAllocator] [node_s0] No eligible remote nodes found to perform balancing [2023-11-15T19:32:50,438][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to compute cluster state update for [delete-index [[test/VYTqcztNQ8mZP5spA2-HmQ]]] [2023-11-15T19:32:50,438][TRACE][o.o.c.s.MasterService ] [node_s0] cluster state updated, source [delete-index [[test/VYTqcztNQ8mZP5spA2-HmQ]]] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: false meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,438][DEBUG][o.o.c.s.MasterService ] [node_s0] publishing cluster state version [27] [2023-11-15T19:32:50,438][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] expecting 6 acknowledgements for cluster_state update (version: 27) [2023-11-15T19:32:50,438][TRACE][o.o.c.c.CompressedStreamUtils] [node_s0] serialized writable object for node version [3.0.0] with size [320] [2023-11-15T19:32:50,438][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] serialized cluster state diff for version [27] in for node version [3.0.0] with size [320] [2023-11-15T19:32:50,439][TRACE][o.o.c.c.CoordinationState] [node_s0] handleClientValue: processing request for version [27] and term [1] [2023-11-15T19:32:50,439][TRACE][o.o.c.c.LeaderChecker ] [node_s0] setCurrentNodes: nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,439][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] publishing PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: false meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned } to [PublicationTarget{discoveryNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}] [2023-11-15T19:32:50,439][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,439][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,439][TRACE][o.o.t.T.tracer ] [node_s0] [373][internal:cluster/coordination/publish_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,439][TRACE][o.o.t.T.tracer ] [node_s0] [373][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,439][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,439][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,439][TRACE][o.o.t.TaskManager ] [node_s0] register 224 [direct] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,439][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,439][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 439, request id: 374, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 439B [2023-11-15T19:32:50,439][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,439][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,439][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32929] [2023-11-15T19:32:50,439][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16489] [2023-11-15T19:32:50,439][TRACE][o.o.t.T.tracer ] [node_s0] [374][internal:cluster/coordination/publish_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,439][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s0] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [323] bytes, now [323] [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 439, request id: 375, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 439B [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 439, request id: 374, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 439B [2023-11-15T19:32:50,440][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s2] [374][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,440][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s0] [375][internal:cluster/coordination/publish_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [323] bytes, now [323] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,440][TRACE][o.o.t.TaskManager ] [node_s2] register 152 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 439, request id: 375, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 439B [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 439, request id: 376, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 439B [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s1] [375][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,440][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,440][TRACE][o.o.t.TaskManager ] [node_s1] register 162 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,440][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,440][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s2] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s0] [376][internal:cluster/coordination/publish_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [323] bytes, now [323] [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 439, request id: 376, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 439B [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 439, request id: 377, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 439B [2023-11-15T19:32:50,440][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [27] to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,440][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s1] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s4] [376][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,440][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,440][TRACE][o.o.t.T.tracer ] [node_s0] [377][internal:cluster/coordination/publish_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [323] bytes, now [323] [2023-11-15T19:32:50,440][TRACE][o.o.t.TaskManager ] [node_s4] register 135 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,440][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,440][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 439, request id: 377, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 439B [2023-11-15T19:32:50,446][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 439, request id: 378, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 439B [2023-11-15T19:32:50,446][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,446][TRACE][o.o.c.c.CoordinationState] [node_s2] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,446][TRACE][o.o.t.T.tracer ] [node_s5] [377][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,446][TRACE][o.o.g.PersistedClusterStateService] [node_s2] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,446][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,446][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,446][TRACE][o.o.t.TaskManager ] [node_s5] register 729 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,446][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,446][TRACE][o.o.c.c.CoordinationState] [node_s1] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,446][TRACE][o.o.g.PersistedClusterStateService] [node_s1] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,446][TRACE][o.o.t.T.tracer ] [node_s0] [378][internal:cluster/coordination/publish_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,446][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,446][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s4] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,446][TRACE][o.o.c.c.Coordinator ] [node_s0] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: false meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,446][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,446][TRACE][o.o.g.PersistedClusterStateService] [node_s0] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,447][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [323] bytes, now [323] [2023-11-15T19:32:50,447][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 439, request id: 378, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 439B [2023-11-15T19:32:50,447][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s5] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,447][TRACE][o.o.t.T.tracer ] [node_s3] [378][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,447][TRACE][o.o.c.c.CoordinationState] [node_s4] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,447][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] queuing cluster state update (setting cluster state to 27) [2023-11-15T19:32:50,447][TRACE][o.o.t.TaskManager ] [node_s3] register 286 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,447][TRACE][o.o.d.PeerFinder ] [node_s4] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,447][TRACE][o.o.d.PeerFinder ] [node_s4] not active [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,447][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,447][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s3] received diff cluster state version [27] with uuid [mZPa6luYTgq1hARtxYz38w], diff size [320] [2023-11-15T19:32:50,447][TRACE][o.o.c.c.CoordinationState] [node_s5] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,447][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] resetting newStateQueued [2023-11-15T19:32:50,447][TRACE][o.o.c.c.PreVoteCollector ] [node_s4] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,447][TRACE][o.o.g.PersistedClusterStateService] [node_s4] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,447][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 135 [2023-11-15T19:32:50,447][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,447][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16763] [2023-11-15T19:32:50,447][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 481, request id: 376, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: handling [PublishRequest{term=1, version=27, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,447][TRACE][o.o.c.c.CoordinationState] [node_s3] handlePublishRequest: accepting publish request for version [27] and term [1] [2023-11-15T19:32:50,447][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] queuing cluster state update (setting cluster state to 27) [2023-11-15T19:32:50,447][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-323] bytes, now [16440] [2023-11-15T19:32:50,447][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] queuing cluster state update (setting cluster state to 27) [2023-11-15T19:32:50,447][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,448][TRACE][o.o.d.PeerFinder ] [node_s3] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,448][TRACE][o.o.t.T.tracer ] [node_s4] [376][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,448][TRACE][o.o.d.PeerFinder ] [node_s3] not active [2023-11-15T19:32:50,447][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,448][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] resetting newStateQueued [2023-11-15T19:32:50,448][TRACE][o.o.c.c.PreVoteCollector ] [node_s3] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,447][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,448][TRACE][o.o.g.PersistedClusterStateService] [node_s3] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,448][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 286 [2023-11-15T19:32:50,448][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16763] [2023-11-15T19:32:50,448][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 481, request id: 376, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,448][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] resetting newStateQueued [2023-11-15T19:32:50,448][TRACE][o.o.t.T.tracer ] [node_s0] [376][internal:cluster/coordination/publish_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,448][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 481, request id: 378, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,448][TRACE][o.o.g.PersistedClusterStateService] [node_s5] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-323] bytes, now [16440] [2023-11-15T19:32:50,448][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,447][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,448][TRACE][o.o.d.PeerFinder ] [node_s5] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,448][TRACE][o.o.d.PeerFinder ] [node_s5] not active [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,448][TRACE][o.o.t.T.tracer ] [node_s3] [378][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,448][TRACE][o.o.c.c.PreVoteCollector ] [node_s5] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,448][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 729 [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16763] [2023-11-15T19:32:50,448][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,448][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,448][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 481, request id: 378, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,448][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 481, request id: 377, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,448][TRACE][o.o.t.T.tracer ] [node_s0] [378][internal:cluster/coordination/publish_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-323] bytes, now [16440] [2023-11-15T19:32:50,448][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,448][TRACE][o.o.t.T.tracer ] [node_s5] [377][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,449][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 481, request id: 377, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,449][TRACE][o.o.t.T.tracer ] [node_s0] [377][internal:cluster/coordination/publish_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] removing metadata for [VYTqcztNQ8mZP5spA2-HmQ] [2023-11-15T19:32:50,450][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,474][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s5] Recording memory usage: 2% [2023-11-15T19:32:50,474][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s5] Recording cpu usage: 40% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s1] Recording memory usage: 2% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s3] Recording memory usage: 2% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s1] Recording cpu usage: 33% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s3] Recording cpu usage: 75% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s2] Recording memory usage: 2% [2023-11-15T19:32:50,476][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s2] Recording cpu usage: 31% [2023-11-15T19:32:50,479][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s0] Recording memory usage: 2% [2023-11-15T19:32:50,482][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,482][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,482][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,482][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,483][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s0] Recording cpu usage: 43% [2023-11-15T19:32:50,485][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [node_s4] Recording memory usage: 2% [2023-11-15T19:32:50,485][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [node_s4] Recording cpu usage: 49% [2023-11-15T19:32:50,489][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,493][DEBUG][o.o.g.PersistedClusterStateService] [node_s3] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,494][DEBUG][o.o.g.PersistedClusterStateService] [node_s4] writing cluster state took [203ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,494][DEBUG][o.o.g.PersistedClusterStateService] [node_s5] writing cluster state took [201ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,494][DEBUG][o.o.g.PersistedClusterStateService] [node_s0] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,495][TRACE][o.o.c.c.PreVoteCollector ] [node_s0] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,495][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 224 [2023-11-15T19:32:50,495][TRACE][o.o.t.T.tracer ] [node_s0] [373][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,495][TRACE][o.o.t.T.tracer ] [node_s0] [373][internal:cluster/coordination/publish_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,495][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [27] and term [1] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,495][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [27] and term [1] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,495][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,495][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [27] and term [1] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,496][DEBUG][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [201ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,496][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,496][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,496][TRACE][o.o.d.PeerFinder ] [node_s2] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,496][TRACE][o.o.d.PeerFinder ] [node_s2] not active [2023-11-15T19:32:50,496][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,496][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [27] and term [1] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,496][TRACE][o.o.c.c.PreVoteCollector ] [node_s2] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,496][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 152 [2023-11-15T19:32:50,496][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16763] [2023-11-15T19:32:50,496][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 492, request id: 374, type: response, version: 3.0.0, header size: 48B] WRITE: 492B [2023-11-15T19:32:50,496][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-323] bytes, now [16440] [2023-11-15T19:32:50,496][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,496][TRACE][o.o.t.T.tracer ] [node_s2] [374][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,496][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 492, request id: 374, type: response, version: 3.0.0] READ: 492B [2023-11-15T19:32:50,496][TRACE][o.o.t.T.tracer ] [node_s0] [374][internal:cluster/coordination/publish_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,497][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,497][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,497][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [27] and term [1] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,497][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: value committed for version [27] and term [1] [2023-11-15T19:32:50,497][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,497][TRACE][o.o.t.T.tracer ] [node_s0] [379][internal:cluster/coordination/commit_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,497][TRACE][o.o.t.T.tracer ] [node_s0] [379][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,497][TRACE][o.o.t.TaskManager ] [node_s0] register 225 [direct] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,497][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,497][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,497][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 322, request id: 380, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,497][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,497][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [16489] [2023-11-15T19:32:50,497][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [32929] [2023-11-15T19:32:50,497][TRACE][o.o.t.T.tracer ] [node_s0] [380][internal:cluster/coordination/commit_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,497][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61583, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 322, request id: 381, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,497][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,497][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 322, request id: 380, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,498][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s2] [380][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s0] [381][internal:cluster/coordination/commit_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,498][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61583} [length: 322, request id: 381, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,498][TRACE][o.o.t.TaskManager ] [node_s2] register 153 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,498][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 322, request id: 382, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,498][TRACE][o.o.c.c.Coordinator ] [node_s2] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,498][TRACE][o.o.c.c.CoordinationState] [node_s2] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s4] [381][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,498][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s0] [382][internal:cluster/coordination/commit_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,498][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 322, request id: 383, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,498][TRACE][o.o.t.TaskManager ] [node_s4] register 136 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,498][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,498][TRACE][o.o.c.c.Coordinator ] [node_s4] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,498][TRACE][o.o.c.c.Coordinator ] [node_s0] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,498][TRACE][o.o.c.c.CoordinationState] [node_s4] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s0] [383][internal:cluster/coordination/commit_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,498][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 322, request id: 382, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,498][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,498][TRACE][o.o.c.c.CoordinationState] [node_s0] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,498][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 322, request id: 383, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,498][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 225 [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s0] [379][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s5] [382][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s3] [383][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,498][TRACE][o.o.t.T.tracer ] [node_s0] [379][internal:cluster/coordination/commit_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,498][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,498][TRACE][o.o.t.TaskManager ] [node_s3] register 287 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,498][TRACE][o.o.t.TaskManager ] [node_s5] register 730 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,498][TRACE][o.o.c.c.Coordinator ] [node_s3] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,498][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] cluster state updated, version [27], source [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,498][TRACE][o.o.c.c.CoordinationState] [node_s3] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,498][TRACE][o.o.c.s.ClusterApplierService] [node_s2] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,498][TRACE][o.o.c.c.Coordinator ] [node_s5] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,498][TRACE][o.o.c.c.CoordinationState] [node_s5] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,498][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,498][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] applying settings from cluster state with version 27 [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] cluster state updated, version [27], source [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] apply cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.repositories.RepositoriesService@417b8324] with change to version [27] [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s4] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] cluster state updated, version [27], source [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s3] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,499][DEBUG][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] cluster state updated, version [27], source [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s5] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.d.PeerFinder ] [node_s1] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.d.PeerFinder ] [node_s1] not active [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.c.PreVoteCollector ] [node_s1] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=27}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 162 [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] applying settings from cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16763] [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] applying settings from cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] apply cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.repositories.RepositoriesService@11ab1a37] with change to version [27] [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] applying settings from cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.cluster.IndicesClusterStateService@15948cda] with change to version [27] [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] apply cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.repositories.RepositoriesService@5daadbc8] with change to version [27] [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.indices.cluster.IndicesClusterStateService@45440fd3] with change to version [27] [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.cluster.IndicesClusterStateService@599544a5] with change to version [27] [2023-11-15T19:32:50,499][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s5] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,499][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s4] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,499][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s2] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,499][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] apply cluster state with version 27 [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.repositories.RepositoriesService@6032f45b] with change to version [27] [2023-11-15T19:32:50,499][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.cluster.IndicesClusterStateService@640d6c85] with change to version [27] [2023-11-15T19:32:50,499][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s3] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndicesService ] [node_s5] [test] closing ... (reason [DELETED]) [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndicesService ] [node_s3] [test] closing ... (reason [DELETED]) [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndicesService ] [node_s5] [test/VYTqcztNQ8mZP5spA2-HmQ] closing index service (reason [DELETED][index no longer part of the metadata]) [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndicesService ] [node_s3] [test/VYTqcztNQ8mZP5spA2-HmQ] closing index service (reason [DELETED][index no longer part of the metadata]) [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndexService ] [node_s5] [test] [0] closing... (reason: [index no longer part of the metadata]) [2023-11-15T19:32:50,499][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 492, request id: 375, type: response, version: 3.0.0, header size: 48B] WRITE: 492B [2023-11-15T19:32:50,499][DEBUG][o.o.i.IndexService ] [node_s3] [test] [0] closing... (reason: [index no longer part of the metadata]) [2023-11-15T19:32:50,499][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-323] bytes, now [16440] [2023-11-15T19:32:50,499][DEBUG][o.o.i.s.IndexShard ] [node_s5] [test][0] state: [STARTED]->[CLOSED], reason [index no longer part of the metadata] [2023-11-15T19:32:50,499][DEBUG][o.o.i.s.IndexShard ] [node_s3] [test][0] state: [STARTED]->[CLOSED], reason [index no longer part of the metadata] [2023-11-15T19:32:50,499][DEBUG][o.o.i.e.Engine ] [node_s3] [test][0] close now acquiring writeLock [2023-11-15T19:32:50,500][DEBUG][o.o.i.e.Engine ] [node_s3] [test][0] close acquired writeLock [2023-11-15T19:32:50,500][DEBUG][o.o.i.e.Engine ] [node_s5] [test][0] close now acquiring writeLock [2023-11-15T19:32:50,500][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,500][DEBUG][o.o.i.e.Engine ] [node_s5] [test][0] close acquired writeLock [2023-11-15T19:32:50,500][TRACE][o.o.t.T.tracer ] [node_s1] [375][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,500][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 492, request id: 375, type: response, version: 3.0.0] READ: 492B [2023-11-15T19:32:50,500][TRACE][o.o.t.T.tracer ] [node_s0] [375][internal:cluster/coordination/publish_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,500][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,500][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,500][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,500][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=27}] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,500][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,500][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IW: decRefDeleter for NRT reader version=27 segments=_0(9.8.0):c2:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098368982}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=1 :id=805sznvlx1pzvtosjx1bs2nj2 _1(9.8.0):c4:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369009}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nj4 _2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369108}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=4 :id=805sznvlx1pzvtosjx1bs2nj0 _3(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369223}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlf _4(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369257}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlb _5(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369367}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nlj _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369423}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nl9 _8(9.8.0):c10:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369742}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=10 :id=805sznvlx1pzvtosjx1bs2nld _7(9.8.0):c8:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369776}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2nlh _9(9.8.0):c54:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370184}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=54 :id=805sznvlx1pzvtosjx1bs2nl7 [2023-11-15T19:32:50,500][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,500][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IW: decRefDeleter for NRT reader version=30 segments=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=5 :id=805sznvlx1pzvtosjx1bs2nlr _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=33 :id=805sznvlx1pzvtosjx1bs2nlp _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=6 :id=805sznvlx1pzvtosjx1bs2nlv _4(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlx _3(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369655}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlt _5(9.8.0):c3:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369654}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=3 :id=805sznvlx1pzvtosjx1bs2nlz _6(9.8.0):c9:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369768}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=2:dvGen=2 :softDel=9 :id=805sznvlx1pzvtosjx1bs2nln _7(9.8.0):c1:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369913}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=1 :id=805sznvlx1pzvtosjx1bs2nl3 _8(9.8.0):c53:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098370255}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :softDel=53 :id=805sznvlx1pzvtosjx1bs2nll [2023-11-15T19:32:50,500][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 322, request id: 384, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,500][TRACE][o.o.i.t.LocalTranslog ] [node_s5] [test][0] closing files. translog is closed and there are no pending retention locks [2023-11-15T19:32:50,500][DEBUG][o.o.i.IndicesService ] [node_s4] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [deleted index was not assigned to local node] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s4] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s4] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,500][DEBUG][o.o.i.IndicesService ] [node_s2] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [deleted index was not assigned to local node] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s4] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s2] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s2] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,500][TRACE][o.o.e.NodeEnvironment ] [node_s2] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,501][TRACE][o.o.i.t.LocalTranslog ] [node_s3] [test][0] closing files. translog is closed and there are no pending retention locks [2023-11-15T19:32:50,501][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [49] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s4] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s2] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,501][TRACE][o.o.t.T.tracer ] [node_s0] [384][internal:cluster/coordination/commit_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,501][DEBUG][o.o.i.t.LocalTranslog ] [node_s5] [test][0] translog closed [2023-11-15T19:32:50,501][TRACE][o.o.i.e.Engine ] [node_s5] [test][0] rollback indexWriter [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IW: rollback [2023-11-15T19:32:50,501][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,501][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 322, request id: 384, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IW: all running merges have aborted [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IW: rollback: done finish merges [2023-11-15T19:32:50,501][DEBUG][o.o.i.t.LocalTranslog ] [node_s3] [test][0] translog closed [2023-11-15T19:32:50,501][TRACE][o.o.i.e.Engine ] [node_s3] [test][0] rollback indexWriter [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] DW: abort [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IW: rollback [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IW: all running merges have aborted [2023-11-15T19:32:50,501][TRACE][o.o.t.T.tracer ] [node_s1] [384][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.DW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] DW: done abort success=true [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IW: rollback: done finish merges [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s4] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s4] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s4] released shard lock for [[test][0]] [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IW ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IW: rollback: infos= [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IFD: now checkpoint "" [0 segments ; isCommit = false] [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IFD: now delete 54 files: [_5_1.fnm, _6.cfe, _3.si, _7.cfs, _5_1_Lucene90_0.dvm, _5_1_Lucene90_0.dvd, _6.cfs, _7.cfe, _4_1_Lucene90_0.dvm, _7.si, _1_1.fnm, _4_1_Lucene90_0.dvd, _7_1_Lucene90_0.dvm, _2.si, _5.cfs, _4.cfe, _3.cfe, _7_1_Lucene90_0.dvd, _8_1.fnm, _3.cfs, _5.cfe, _4.cfs, _6.si, _0_1_Lucene90_0.dvm, _1.cfe, _1.si, _2.cfs, _7_1.fnm, _1_1_Lucene90_0.dvd, _3_1.fnm, _1_1_Lucene90_0.dvm, _1.cfs, _2.cfe, _0_1_Lucene90_0.dvd, _5.si, _9.si, _3_1_Lucene90_0.dvd, _0.si, _8_1_Lucene90_0.dvd, _0.cfs, _6_1.fnm, _4_1.fnm, _8.cfe, _9.cfs, _8_1_Lucene90_0.dvm, _6_1_Lucene90_0.dvm, _0.cfe, _3_1_Lucene90_0.dvm, _6_1_Lucene90_0.dvd, _9.cfe, _0_1.fnm, _4.si, _8.cfs, _8.si] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s2] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s2] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,501][TRACE][o.o.e.NodeEnvironment ] [node_s2] released shard lock for [[test][0]] [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] DW: abort [2023-11-15T19:32:50,501][TRACE][o.o.i.e.E.DW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] DW: done abort success=true [2023-11-15T19:32:50,501][TRACE][o.o.t.TaskManager ] [node_s1] register 163 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,502][TRACE][o.o.c.c.Coordinator ] [node_s1] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,502][TRACE][o.o.c.c.CoordinationState] [node_s1] handleCommit: applying commit request for term [1] and version [27] [2023-11-15T19:32:50,502][DEBUG][o.o.i.IndicesService ] [node_s4] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,502][TRACE][o.o.i.e.E.IW ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IW: rollback: infos=_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njk _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2njr _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njp [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.script.MockScriptService@46891f] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.script.MockScriptService@1c97d1f3] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.RestoreService@2cf36cc8] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.ingest.IngestService@310ca8d1] with change to version [27] [2023-11-15T19:32:50,502][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5_1.fnm [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.search.pipeline.SearchPipelineService@545313b4] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.ingest.IngestActionForwarder@60697650] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@574bd3b9] with change to version [27] [2023-11-15T19:32:50,502][DEBUG][o.o.i.IndicesService ] [node_s2] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.tasks.TaskManager@3d5b3a1] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.SnapshotsService@4f57dedc] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,502][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: now checkpoint "_2(9.8.0):c5:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369401}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njk _1(9.8.0):c36:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369469}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=8 :id=805sznvlx1pzvtosjx1bs2njr _0(9.8.0):c6:[diagnostics={source=flush, lucene.version=9.8.0, os.version=13.5.2, os.arch=x86_64, os=Mac OS X, java.vendor=Amazon.com Inc., java.runtime.version=17.0.9+8-LTS, timestamp=1700098369516}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=805sznvlx1pzvtosjx1bs2njp" [3 segments ; isCommit = false] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] set locally applied cluster state to version 27 [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.ingest.IngestService@26be5633] with change to version [27] [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,502][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: now delete 39 files: [_5_1.fnm, _6.cfe, _3.si, _0_1_Lucene90_0.dvm, _7.cfs, _5_1_Lucene90_0.dvm, _1_2.fnm, _1_2_Lucene90_1.dvd, _3_1.fnm, _5_1_Lucene90_0.dvd, _1_2_Lucene90_1.dvm, _6.cfs, _7.cfe, _4_1_Lucene90_0.dvm, _7.si, _0_1_Lucene90_0.dvd, _5.si, _4_1_Lucene90_0.dvd, _3_1_Lucene90_0.dvd, _6_2_Lucene90_0.dvm, _2_1_Lucene90_0.dvm, _5.cfs, _4.cfe, _4_1.fnm, _3.cfe, _8.cfe, _6_2_Lucene90_0.dvd, _2_1.fnm, _6_2.fnm, _3_1_Lucene90_0.dvm, _3.cfs, _5.cfe, _4.cfs, _0_1.fnm, _2_1_Lucene90_0.dvd, _4.si, _8.cfs, _6.si, _8.si] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.search.pipeline.SearchPipelineService@2d41adb7] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.action.ingest.IngestActionForwarder@5e8274d6] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.tasks.TaskManager@7cb82ebc] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.InternalClusterInfoService@6427ea90] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] set locally applied cluster state to version 27 [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s4] released shard lock for [[test][0]] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResponseCollectorService@28e3e0d6] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.snapshots.SnapshotShardsService@8efdc68] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResourceUsageCollectorService@45863d67] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@6df82bb6] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.store.IndicesStore@78f636a5] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@439ed513] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@324e4282] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResponseCollectorService@69665d31] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.e.NodeEnvironment ] [node_s2] released shard lock for [[test][0]] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.persistent.PersistentTasksNodeService@395e152d] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResourceUsageCollectorService@456f6e30] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksClusterService@70ec0dc3] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@4d476a64] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.routing.DelayedAllocationService@30f60b0e] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@44d2e77] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksNodeService@6b2257f4] with change to version [27] [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.gateway.GatewayService@5582aa4e] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 136 [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] cluster state updated, version [27], source [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: true meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s1] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,502][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 153 [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61583} [length: 71, request id: 381, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,502][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5_1.fnm [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61577} [length: 71, request id: 380, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,502][TRACE][o.o.t.T.tracer ] [node_s4] [381][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.t.T.tracer ] [node_s2] [380][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,502][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,502][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61583, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 71, request id: 381, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] applying settings from cluster state with version 27 [2023-11-15T19:32:50,502][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6.cfe [2023-11-15T19:32:50,502][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] apply cluster state with version 27 [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.repositories.RepositoriesService@2e256902] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.indices.cluster.IndicesClusterStateService@58f1b39a] with change to version [27] [2023-11-15T19:32:50,502][TRACE][o.o.t.T.tracer ] [node_s0] [381][internal:cluster/coordination/commit_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,503][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61577, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 71, request id: 380, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,503][TRACE][o.o.t.T.tracer ] [node_s0] [380][internal:cluster/coordination/commit_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,503][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s1] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,503][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,503][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27} applied version 27, max now 27 [2023-11-15T19:32:50,503][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,503][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=27} applied version 27, max now 27 [2023-11-15T19:32:50,503][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6.cfe [2023-11-15T19:32:50,503][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3.si [2023-11-15T19:32:50,504][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3.si [2023-11-15T19:32:50,504][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7.cfs [2023-11-15T19:32:50,504][DEBUG][o.o.i.IndicesService ] [node_s1] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [deleted index was not assigned to local node] [2023-11-15T19:32:50,504][TRACE][o.o.e.NodeEnvironment ] [node_s1] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,504][TRACE][o.o.e.NodeEnvironment ] [node_s1] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,504][TRACE][o.o.e.NodeEnvironment ] [node_s1] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,504][TRACE][o.o.e.NodeEnvironment ] [node_s1] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,504][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5_1_Lucene90_0.dvm [2023-11-15T19:32:50,504][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _0_1_Lucene90_0.dvm [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] released shard lock for [[test][0]] [2023-11-15T19:32:50,505][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5_1_Lucene90_0.dvd [2023-11-15T19:32:50,505][DEBUG][o.o.i.IndicesService ] [node_s1] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.script.MockScriptService@b2181c5] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.RestoreService@57f9c669] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.ingest.IngestService@5211c8ed] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.search.pipeline.SearchPipelineService@729cc56] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,505][TRACE][o.o.e.NodeEnvironment ] [node_s1] released shard lock for [[test][0]] [2023-11-15T19:32:50,505][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _7.cfs [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.ingest.IngestActionForwarder@814adf] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@19521e28] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.tasks.TaskManager@45d8fb4f] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.SnapshotsService@cdac4de] with change to version [27] [2023-11-15T19:32:50,505][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] set locally applied cluster state to version 27 [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.InternalClusterInfoService@6a31ad84] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@659248ee] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@3abdd4fd] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@23792777] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResponseCollectorService@4443e311] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResourceUsageCollectorService@3361164f] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksClusterService@672f36e2] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.routing.DelayedAllocationService@d07e7ef] with change to version [27] [2023-11-15T19:32:50,505][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksNodeService@5687d635] with change to version [27] [2023-11-15T19:32:50,506][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.gateway.GatewayService@6d6f71c4] with change to version [27] [2023-11-15T19:32:50,506][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,506][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 163 [2023-11-15T19:32:50,506][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,506][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61560} [length: 71, request id: 384, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,506][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6.cfs [2023-11-15T19:32:50,506][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,506][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,506][TRACE][o.o.t.T.tracer ] [node_s1] [384][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,506][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5_1_Lucene90_0.dvm [2023-11-15T19:32:50,506][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61560, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 71, request id: 384, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,506][TRACE][o.o.t.T.tracer ] [node_s0] [384][internal:cluster/coordination/commit_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,506][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,506][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=27} applied version 27, max now 27 [2023-11-15T19:32:50,506][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7.cfe [2023-11-15T19:32:50,506][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _1_2.fnm [2023-11-15T19:32:50,507][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4_1_Lucene90_0.dvm [2023-11-15T19:32:50,507][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _1_2_Lucene90_1.dvd [2023-11-15T19:32:50,507][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7.si [2023-11-15T19:32:50,508][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3_1.fnm [2023-11-15T19:32:50,508][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1_1.fnm [2023-11-15T19:32:50,508][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5_1_Lucene90_0.dvd [2023-11-15T19:32:50,509][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _1_2_Lucene90_1.dvm [2023-11-15T19:32:50,509][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4_1_Lucene90_0.dvd [2023-11-15T19:32:50,509][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6.cfs [2023-11-15T19:32:50,509][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7_1_Lucene90_0.dvm [2023-11-15T19:32:50,510][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _7.cfe [2023-11-15T19:32:50,510][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _2.si [2023-11-15T19:32:50,510][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5.cfs [2023-11-15T19:32:50,510][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4_1_Lucene90_0.dvm [2023-11-15T19:32:50,511][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4.cfe [2023-11-15T19:32:50,511][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _7.si [2023-11-15T19:32:50,511][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3.cfe [2023-11-15T19:32:50,512][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _0_1_Lucene90_0.dvd [2023-11-15T19:32:50,512][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7_1_Lucene90_0.dvd [2023-11-15T19:32:50,512][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5.si [2023-11-15T19:32:50,512][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8_1.fnm [2023-11-15T19:32:50,512][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4_1_Lucene90_0.dvd [2023-11-15T19:32:50,513][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3.cfs [2023-11-15T19:32:50,513][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3_1_Lucene90_0.dvd [2023-11-15T19:32:50,513][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5.cfe [2023-11-15T19:32:50,514][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_2_Lucene90_0.dvm [2023-11-15T19:32:50,514][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4.cfs [2023-11-15T19:32:50,514][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _2_1_Lucene90_0.dvm [2023-11-15T19:32:50,515][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6.si [2023-11-15T19:32:50,515][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5.cfs [2023-11-15T19:32:50,515][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0_1_Lucene90_0.dvm [2023-11-15T19:32:50,515][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4.cfe [2023-11-15T19:32:50,516][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1.cfe [2023-11-15T19:32:50,516][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4_1.fnm [2023-11-15T19:32:50,516][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1.si [2023-11-15T19:32:50,516][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3.cfe [2023-11-15T19:32:50,517][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _2.cfs [2023-11-15T19:32:50,517][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.cfe [2023-11-15T19:32:50,517][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _7_1.fnm [2023-11-15T19:32:50,517][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_2_Lucene90_0.dvd [2023-11-15T19:32:50,518][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1_1_Lucene90_0.dvd [2023-11-15T19:32:50,518][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _2_1.fnm [2023-11-15T19:32:50,518][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3_1.fnm [2023-11-15T19:32:50,518][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6_2.fnm [2023-11-15T19:32:50,519][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3_1_Lucene90_0.dvm [2023-11-15T19:32:50,519][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1_1_Lucene90_0.dvm [2023-11-15T19:32:50,519][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _3.cfs [2023-11-15T19:32:50,519][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _1.cfs [2023-11-15T19:32:50,519][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _5.cfe [2023-11-15T19:32:50,520][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _2.cfe [2023-11-15T19:32:50,520][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4.cfs [2023-11-15T19:32:50,520][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0_1_Lucene90_0.dvd [2023-11-15T19:32:50,520][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _0_1.fnm [2023-11-15T19:32:50,520][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _5.si [2023-11-15T19:32:50,521][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _2_1_Lucene90_0.dvd [2023-11-15T19:32:50,521][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.si [2023-11-15T19:32:50,521][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _4.si [2023-11-15T19:32:50,521][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3_1_Lucene90_0.dvd [2023-11-15T19:32:50,522][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.cfs [2023-11-15T19:32:50,522][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0.si [2023-11-15T19:32:50,522][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _6.si [2023-11-15T19:32:50,522][TRACE][i.s.deletes ] [node_s3][test][0] StoreDirectory.deleteFile: delete file _8.si [2023-11-15T19:32:50,522][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8_1_Lucene90_0.dvd [2023-11-15T19:32:50,523][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: 21 ms to checkpoint [2023-11-15T19:32:50,523][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0.cfs [2023-11-15T19:32:50,523][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,523][TRACE][o.o.i.e.E.IFD ] [node_s3] [test][0] opensearch[node_s3][clusterApplierService#updateTask][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,523][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6_1.fnm [2023-11-15T19:32:50,524][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4_1.fnm [2023-11-15T19:32:50,524][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8.cfe [2023-11-15T19:32:50,524][TRACE][o.o.i.e.Engine ] [node_s3] [test][0] rollback indexWriter done [2023-11-15T19:32:50,524][DEBUG][o.o.i.e.Engine ] [node_s3] [test][0] engine closed [api] [2023-11-15T19:32:50,524][TRACE][o.o.i.IndicesService ] [node_s3] [test][0] deleting shard reason [delete index] [2023-11-15T19:32:50,524][TRACE][o.o.e.NodeEnvironment ] [node_s3] acquiring node shardlock on [[test][0]], timeout [0], details [checking if shard is locked] [2023-11-15T19:32:50,524][TRACE][o.o.e.NodeEnvironment ] [node_s3] shard lock wait count for [test][0] is now [1] [2023-11-15T19:32:50,525][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.cfs [2023-11-15T19:32:50,525][TRACE][o.o.e.NodeEnvironment ] [node_s3] acquiring locks for [test][0], paths: [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ/0]] [2023-11-15T19:32:50,525][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8_1_Lucene90_0.dvm [2023-11-15T19:32:50,525][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6_1_Lucene90_0.dvm [2023-11-15T19:32:50,526][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0.cfe [2023-11-15T19:32:50,526][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _3_1_Lucene90_0.dvm [2023-11-15T19:32:50,527][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _6_1_Lucene90_0.dvd [2023-11-15T19:32:50,527][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _9.cfe [2023-11-15T19:32:50,528][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _0_1.fnm [2023-11-15T19:32:50,528][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _4.si [2023-11-15T19:32:50,529][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8.cfs [2023-11-15T19:32:50,529][TRACE][i.s.deletes ] [node_s5][test][0] StoreDirectory.deleteFile: delete file _8.si [2023-11-15T19:32:50,530][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IFD: 28 ms to checkpoint [2023-11-15T19:32:50,530][TRACE][o.o.i.e.E.IFD ] [node_s5] [test][0] opensearch[node_s5][clusterApplierService#updateTask][T#1] IFD: now delete 0 files: [] [2023-11-15T19:32:50,531][TRACE][o.o.i.e.Engine ] [node_s5] [test][0] rollback indexWriter done [2023-11-15T19:32:50,531][DEBUG][o.o.i.e.Engine ] [node_s5] [test][0] engine closed [api] [2023-11-15T19:32:50,531][TRACE][o.o.i.IndicesService ] [node_s5] [test][0] deleting shard reason [delete index] [2023-11-15T19:32:50,531][TRACE][o.o.e.NodeEnvironment ] [node_s5] acquiring node shardlock on [[test][0]], timeout [0], details [checking if shard is locked] [2023-11-15T19:32:50,531][TRACE][o.o.e.NodeEnvironment ] [node_s5] shard lock wait count for [test][0] is now [1] [2023-11-15T19:32:50,531][TRACE][o.o.e.NodeEnvironment ] [node_s5] acquiring locks for [test][0], paths: [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ/0]] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] deleted shard [test][0] directory, paths: [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ/0]] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] released shard lock for [[test][0]] [2023-11-15T19:32:50,537][DEBUG][o.o.i.s.Store ] [node_s3] [test][0] store reference count on close: 0 [2023-11-15T19:32:50,537][DEBUG][o.o.i.IndexService ] [node_s3] [test] [0] closed (reason: [index no longer part of the metadata]) [2023-11-15T19:32:50,537][DEBUG][o.o.i.c.b.BitsetFilterCache] [node_s3] [test] clearing all bitsets because [close] [2023-11-15T19:32:50,537][DEBUG][o.o.i.c.q.IndexQueryCache] [node_s3] [test] full cache clear, reason [close] [2023-11-15T19:32:50,537][DEBUG][o.o.i.c.b.BitsetFilterCache] [node_s3] [test] clearing all bitsets because [close] [2023-11-15T19:32:50,537][DEBUG][o.o.i.IndicesService ] [node_s3] [test/VYTqcztNQ8mZP5spA2-HmQ] closed... (reason [DELETED][index no longer part of the metadata]) [2023-11-15T19:32:50,537][DEBUG][o.o.i.IndicesService ] [node_s3] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [index no longer part of the metadata] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,537][TRACE][o.o.e.NodeEnvironment ] [node_s3] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,538][TRACE][o.o.e.NodeEnvironment ] [node_s5] deleted shard [test][0] directory, paths: [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ/0]] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] released shard lock for [[test][0]] [2023-11-15T19:32:50,539][DEBUG][o.o.i.s.Store ] [node_s5] [test][0] store reference count on close: 0 [2023-11-15T19:32:50,539][DEBUG][o.o.i.IndexService ] [node_s5] [test] [0] closed (reason: [index no longer part of the metadata]) [2023-11-15T19:32:50,539][DEBUG][o.o.i.c.b.BitsetFilterCache] [node_s5] [test] clearing all bitsets because [close] [2023-11-15T19:32:50,539][DEBUG][o.o.i.c.q.IndexQueryCache] [node_s5] [test] full cache clear, reason [close] [2023-11-15T19:32:50,539][DEBUG][o.o.i.c.b.BitsetFilterCache] [node_s5] [test] clearing all bitsets because [close] [2023-11-15T19:32:50,539][DEBUG][o.o.i.IndicesService ] [node_s5] [test/VYTqcztNQ8mZP5spA2-HmQ] closed... (reason [DELETED][index no longer part of the metadata]) [2023-11-15T19:32:50,539][DEBUG][o.o.i.IndicesService ] [node_s5] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [index no longer part of the metadata] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s5] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s3] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s3] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,539][TRACE][o.o.e.NodeEnvironment ] [node_s3] released shard lock for [[test][0]] [2023-11-15T19:32:50,540][DEBUG][o.o.i.IndicesService ] [node_s3] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.script.MockScriptService@210ffe60] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.ingest.IngestService@67038f1b] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.search.pipeline.SearchPipelineService@10e43f8c] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.action.ingest.IngestActionForwarder@1f021051] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.tasks.TaskManager@4120012f] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.e.NodeEnvironment ] [node_s3] released shard lock for [[test][0]] [2023-11-15T19:32:50,540][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] set locally applied cluster state to version 27 [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResponseCollectorService@3a202547] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.snapshots.SnapshotShardsService@b0965d0] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResourceUsageCollectorService@4067c78] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.store.IndicesStore@456e3cae] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.persistent.PersistentTasksNodeService@7e5f3359] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@3e38eea7] with change to version [27] [2023-11-15T19:32:50,540][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@96aa0a4] with change to version [27] [2023-11-15T19:32:50,540][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,540][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 287 [2023-11-15T19:32:50,540][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,540][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61581} [length: 71, request id: 383, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,540][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,540][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,540][TRACE][o.o.t.T.tracer ] [node_s3] [383][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,540][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61581, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 71, request id: 383, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,540][TRACE][o.o.t.T.tracer ] [node_s0] [383][internal:cluster/coordination/commit_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,541][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27} applied version 27, max now 27 [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] released shard lock for [[test][0]] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.script.MockScriptService@5096100c] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.ingest.IngestService@28bfec03] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.search.pipeline.SearchPipelineService@610c2181] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.action.ingest.IngestActionForwarder@50cf8326] with change to version [27] [2023-11-15T19:32:50,541][DEBUG][o.o.i.IndicesService ] [node_s5] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.tasks.TaskManager@30b9ef43] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,541][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] set locally applied cluster state to version 27 [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResponseCollectorService@6731c2a3] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.snapshots.SnapshotShardsService@6731e7ed] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResourceUsageCollectorService@4bcb8e9e] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.store.IndicesStore@34f7b353] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,541][TRACE][o.o.e.NodeEnvironment ] [node_s5] released shard lock for [[test][0]] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.persistent.PersistentTasksNodeService@5d3a5bb4] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@35dcd171] with change to version [27] [2023-11-15T19:32:50,541][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@2274062a] with change to version [27] [2023-11-15T19:32:50,542][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=27, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,542][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 730 [2023-11-15T19:32:50,542][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,542][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61582} [length: 71, request id: 382, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,542][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,542][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,542][TRACE][o.o.t.T.tracer ] [node_s5] [382][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,542][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61582, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 71, request id: 382, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,542][TRACE][o.o.t.T.tracer ] [node_s0] [382][internal:cluster/coordination/commit_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,542][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,542][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27} applied version 27, max now 27 [2023-11-15T19:32:50,542][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] onPossibleCompletion: [Publication{term=1, version=27}] was successful [2023-11-15T19:32:50,542][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=27}]: execute [2023-11-15T19:32:50,542][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] cluster state updated, version [27], source [Publication{term=1, version=27}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 27 state uuid: mZPa6luYTgq1hARtxYz38w from_diff: false meta data version: 22 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,542][TRACE][o.o.c.s.ClusterApplierService] [node_s0] connecting to nodes of cluster state with version 27 [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,542][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] applying settings from cluster state with version 27 [2023-11-15T19:32:50,542][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] apply cluster state with version 27 [2023-11-15T19:32:50,542][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.repositories.RepositoriesService@4b52a152] with change to version [27] [2023-11-15T19:32:50,542][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.indices.cluster.IndicesClusterStateService@4f6d2ce0] with change to version [27] [2023-11-15T19:32:50,542][DEBUG][o.o.i.c.IndicesClusterStateService] [node_s0] [[test/VYTqcztNQ8mZP5spA2-HmQ]] cleaning index, no longer part of the metadata [2023-11-15T19:32:50,543][DEBUG][o.o.i.IndicesService ] [node_s0] [test/VYTqcztNQ8mZP5spA2-HmQ] deleting index store reason [deleted index was not assigned to local node] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] acquiring node shardlock on [[test][0]], timeout [0], details [deleting index directory] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] deleting index [test/VYTqcztNQ8mZP5spA2-HmQ] directory, paths(1): [[/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/indices/VYTqcztNQ8mZP5spA2-HmQ]] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,543][TRACE][o.o.e.NodeEnvironment ] [node_s0] released shard lock for [[test][0]] [2023-11-15T19:32:50,543][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.script.MockScriptService@a6fab69] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.RestoreService@65a9c473] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.ingest.IngestService@3b2fa1a5] with change to version [27] [2023-11-15T19:32:50,544][DEBUG][o.o.i.IndicesService ] [node_s0] [test/VYTqcztNQ8mZP5spA2-HmQ] processing pending deletes [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.search.pipeline.SearchPipelineService@1d03900c] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] locking all shards for index [test/VYTqcztNQ8mZP5spA2-HmQ] - [1] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.ingest.IngestActionForwarder@6a405426] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] acquiring node shardlock on [[test][0]], timeout [1800000], details [process pending deletes] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@41e19f41] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] successfully acquired shardlock for [[test][0]] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.tasks.TaskManager@7052c51] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.SnapshotsService@6075f4d] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] shard lock wait count for [test][0] is now [0] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] last shard lock wait decremented, removing lock for [test][0] [2023-11-15T19:32:50,544][TRACE][o.o.e.NodeEnvironment ] [node_s0] released shard lock for [[test][0]] [2023-11-15T19:32:50,544][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] set locally applied cluster state to version 27 [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.InternalClusterInfoService@23a71628] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@7c26f627] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@73b53b39] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@170fb472] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResponseCollectorService@5cd3ff7a] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResourceUsageCollectorService@660cd06c] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksClusterService@3cba4367] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.routing.DelayedAllocationService@56fcdca6] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.r.DelayedAllocationService] [node_s0] no need to schedule reroute - no delayed unassigned shards [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksNodeService@1597d12c] with change to version [27] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.gateway.GatewayService@1518ec42] with change to version [27] [2023-11-15T19:32:50,544][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=27}]: took [0s] done applying updated cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) [2023-11-15T19:32:50,544][DEBUG][o.o.c.c.C.CoordinatorPublication] [node_s0] publication ended successfully: Publication{term=1, version=27} [2023-11-15T19:32:50,544][TRACE][o.o.c.c.Reconfigurator ] [node_s0] Reconfigurator{autoShrinkVotingConfiguration=true} reconfiguring VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} based on liveNodes=[{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], retiredNodeIds=[], currentClusterManager={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,544][TRACE][o.o.c.c.LagDetector ] [node_s0] lag detection for version 27 is unnecessary: [NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=27}, NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27}, NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27}, NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=27}, NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=27}] [2023-11-15T19:32:50,544][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] ack received from node [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}], cluster_state update (version: 27) [2023-11-15T19:32:50,544][TRACE][o.o.c.s.M.AckCountDownListener] [node_s0] all expected nodes acknowledged cluster_state update (version: 27) [2023-11-15T19:32:50,544][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 223 [2023-11-15T19:32:50,544][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16489] [2023-11-15T19:32:50,544][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61556} [length: 72, request id: 68, type: response, version: 3.0.0, header size: 48B] WRITE: 72B [2023-11-15T19:32:50,544][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-49] bytes, now [16440] [2023-11-15T19:32:50,544][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on successful publication of cluster state (version: 27, uuid: mZPa6luYTgq1hARtxYz38w) for [delete-index [[test/VYTqcztNQ8mZP5spA2-HmQ]]] [2023-11-15T19:32:50,544][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,544][TRACE][o.o.t.T.tracer ] [node_s0] [68][indices:admin/delete] sent response [2023-11-15T19:32:50,544][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61556, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 72, request id: 68, type: response, version: 3.0.0] READ: 72B [2023-11-15T19:32:50,544][TRACE][o.o.t.T.tracer ] [node_s4] [68][indices:admin/delete] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,544][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 134 [2023-11-15T19:32:50,544][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 164 [transport] [indices:admin/template/get] [] [2023-11-15T19:32:50,544][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.indices.template.get.GetIndexTemplatesRequest@4a4b5ed1] with cluster state version [27] [2023-11-15T19:32:50,544][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:admin/template/get [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61561, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 134, request id: 70, type: request, version: 3.0.0, header size: 76B, action: indices:admin/template/get] WRITE: 134B [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s1] [70][indices:admin/template/get] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adding [35b][indices:admin/template/get] to used bytes [new used: [35b], limit: 4294967296 [4gb], estimate: 70 [70b]] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61561} [length: 134, request id: 70, type: request, version: 3.0.0, action: indices:admin/template/get] READ: 134B [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s0] [70][indices:admin/template/get] received request [2023-11-15T19:32:50,545][TRACE][o.o.t.TaskManager ] [node_s0] register 226 [transport] [indices:admin/template/get] [] [2023-11-15T19:32:50,545][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.indices.template.get.GetIndexTemplatesRequest@1dfa4350] with cluster state version [27] [2023-11-15T19:32:50,545][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 226 [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16475] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61561} [length: 476, request id: 70, type: response, version: 3.0.0, header size: 48B] WRITE: 476B [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [35] [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s0] [70][indices:admin/template/get] sent response [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-35] bytes, now [0] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61561, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 476, request id: 70, type: response, version: 3.0.0] READ: 476B [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s1] [70][indices:admin/template/get] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,545][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 164 [2023-11-15T19:32:50,545][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 731 [transport] [indices:admin/template/delete] [] [2023-11-15T19:32:50,545][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.indices.template.delete.DeleteIndexTemplateRequest@7fecb100] with cluster state version [27] [2023-11-15T19:32:50,545][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: indices:admin/template/delete [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61551, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 157, request id: 513, type: request, version: 3.0.0, header size: 79B, action: indices:admin/template/delete] WRITE: 157B [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s5] [513][indices:admin/template/delete] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,545][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adding [55b][indices:admin/template/delete] to used bytes [new used: [55b], limit: 4294967296 [4gb], estimate: 110 [110b]] [2023-11-15T19:32:50,545][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61551} [length: 157, request id: 513, type: request, version: 3.0.0, action: indices:admin/template/delete] READ: 157B [2023-11-15T19:32:50,545][TRACE][o.o.t.T.tracer ] [node_s0] [513][indices:admin/template/delete] received request [2023-11-15T19:32:50,545][TRACE][o.o.t.TaskManager ] [node_s0] register 227 [transport] [indices:admin/template/delete] [] [2023-11-15T19:32:50,545][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.indices.template.delete.DeleteIndexTemplateRequest@3bff75c8] with cluster state version [27] [2023-11-15T19:32:50,546][TRACE][o.o.c.s.MasterService ] [node_s0] will process [remove-index-template [random_index_template]] [2023-11-15T19:32:50,546][DEBUG][o.o.c.s.MasterService ] [node_s0] executing cluster state update for [remove-index-template [random_index_template]] [2023-11-15T19:32:50,546][INFO ][o.o.c.m.MetadataIndexTemplateService] [node_s0] removing template [random_index_template] [2023-11-15T19:32:50,546][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to compute cluster state update for [remove-index-template [random_index_template]] [2023-11-15T19:32:50,546][TRACE][o.o.c.s.MasterService ] [node_s0] cluster state updated, source [remove-index-template [random_index_template]] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: false meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,546][DEBUG][o.o.c.s.MasterService ] [node_s0] publishing cluster state version [28] [2023-11-15T19:32:50,546][TRACE][o.o.c.c.CompressedStreamUtils] [node_s0] serialized writable object for node version [3.0.0] with size [285] [2023-11-15T19:32:50,546][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] serialized cluster state diff for version [28] in for node version [3.0.0] with size [285] [2023-11-15T19:32:50,546][TRACE][o.o.c.c.CoordinationState] [node_s0] handleClientValue: processing request for version [28] and term [1] [2023-11-15T19:32:50,546][TRACE][o.o.c.c.LeaderChecker ] [node_s0] setCurrentNodes: nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,546][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] publishing PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: false meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned } to [PublicationTarget{discoveryNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}, PublicationTarget{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, state=NOT_STARTED, ackIsPending=true}] [2023-11-15T19:32:50,546][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,546][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,546][TRACE][o.o.t.T.tracer ] [node_s0] [385][internal:cluster/coordination/publish_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,546][TRACE][o.o.t.T.tracer ] [node_s0] [385][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,546][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,546][TRACE][o.o.t.TaskManager ] [node_s0] register 228 [direct] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,546][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,546][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 404, request id: 386, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 404B [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s0] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 404, request id: 387, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 404B [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s0] [386][internal:cluster/coordination/publish_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [288] bytes, now [288] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 404, request id: 386, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 404B [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,547][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s0] [387][internal:cluster/coordination/publish_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 404, request id: 388, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 404B [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [288] bytes, now [288] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s2] [386][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 404, request id: 387, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 404B [2023-11-15T19:32:50,547][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,547][TRACE][o.o.t.TaskManager ] [node_s2] register 154 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s1] [387][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s0] [388][internal:cluster/coordination/publish_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 404, request id: 389, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 404B [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [288] bytes, now [288] [2023-11-15T19:32:50,547][TRACE][o.o.t.TaskManager ] [node_s1] register 165 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 404, request id: 388, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 404B [2023-11-15T19:32:50,547][TRACE][o.o.c.c.PublicationTransportHandler] [node_s0] sending cluster state diff for version [28] to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s2] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s4] [388][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s0] [389][internal:cluster/coordination/publish_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,547][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/publish_state [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [288] bytes, now [288] [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s1] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.t.TaskManager ] [node_s4] register 137 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 404, request id: 389, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 404B [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 404, request id: 390, type: request, version: 3.0.0, header size: 93B, action: internal:cluster/coordination/publish_state] WRITE: 404B [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s5] [389][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s4] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.t.TaskManager ] [node_s5] register 732 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s0] [390][internal:cluster/coordination/publish_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.CoordinationState] [node_s2] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [288] bytes, now [288] [2023-11-15T19:32:50,547][TRACE][o.o.g.PersistedClusterStateService] [node_s2] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,547][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 404, request id: 390, type: request, version: 3.0.0, action: internal:cluster/coordination/publish_state] READ: 404B [2023-11-15T19:32:50,547][TRACE][o.o.c.c.CoordinationState] [node_s1] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,547][TRACE][o.o.g.PersistedClusterStateService] [node_s1] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s5] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,547][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s0] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: false meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.t.T.tracer ] [node_s3] [390][internal:cluster/coordination/publish_state] received request [2023-11-15T19:32:50,547][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,547][TRACE][o.o.g.PersistedClusterStateService] [node_s0] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,547][TRACE][o.o.t.TaskManager ] [node_s3] register 288 [transport] [internal:cluster/coordination/publish_state] [] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.CoordinationState] [node_s4] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,547][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] queuing cluster state update (setting cluster state to 28) [2023-11-15T19:32:50,547][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s4] resetting newStateQueued [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s4] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,547][TRACE][o.o.c.c.CoordinationState] [node_s5] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,547][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,547][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] queuing cluster state update (setting cluster state to 28) [2023-11-15T19:32:50,547][TRACE][o.o.d.PeerFinder ] [node_s4] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,547][TRACE][o.o.g.PersistedClusterStateService] [node_s4] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,547][DEBUG][o.o.c.c.PublicationTransportHandler] [node_s3] received diff cluster state version [28] with uuid [_5esdxhhRs-Fh4d2QFm-kA], diff size [285] [2023-11-15T19:32:50,547][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,547][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,547][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,547][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s5] resetting newStateQueued [2023-11-15T19:32:50,547][TRACE][o.o.c.c.Coordinator ] [node_s5] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,548][TRACE][o.o.d.PeerFinder ] [node_s5] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,548][TRACE][o.o.d.PeerFinder ] [node_s5] not active [2023-11-15T19:32:50,548][TRACE][o.o.c.c.PreVoteCollector ] [node_s5] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,547][TRACE][o.o.d.PeerFinder ] [node_s4] not active [2023-11-15T19:32:50,548][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,548][TRACE][o.o.g.PersistedClusterStateService] [node_s5] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,548][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 732 [2023-11-15T19:32:50,548][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16728] [2023-11-15T19:32:50,548][TRACE][o.o.c.c.PreVoteCollector ] [node_s4] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,548][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: handling [PublishRequest{term=1, version=28, state=cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned }] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,548][TRACE][o.o.c.c.CoordinationState] [node_s3] handlePublishRequest: accepting publish request for version [28] and term [1] [2023-11-15T19:32:50,548][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] queuing cluster state update (setting cluster state to 28) [2023-11-15T19:32:50,548][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,548][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61584} [length: 480, request id: 389, type: response, version: 3.0.0, header size: 47B] WRITE: 480B [2023-11-15T19:32:50,548][TRACE][o.o.g.G.AsyncLucenePersistedState] [node_s3] resetting newStateQueued [2023-11-15T19:32:50,548][TRACE][o.o.c.c.Coordinator ] [node_s3] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-288] bytes, now [16440] [2023-11-15T19:32:50,548][TRACE][o.o.d.PeerFinder ] [node_s3] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,548][TRACE][o.o.g.PersistedClusterStateService] [node_s3] currentTerm [1] matches previous currentTerm, writing changes only [2023-11-15T19:32:50,548][TRACE][o.o.d.PeerFinder ] [node_s3] not active [2023-11-15T19:32:50,548][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,548][TRACE][o.o.c.c.PreVoteCollector ] [node_s3] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,548][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 137 [2023-11-15T19:32:50,548][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 288 [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16728] [2023-11-15T19:32:50,548][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] updating global metadata doc [2023-11-15T19:32:50,548][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61578} [length: 481, request id: 390, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-288] bytes, now [16440] [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16728] [2023-11-15T19:32:50,548][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61585} [length: 481, request id: 388, type: response, version: 3.0.0, header size: 48B] WRITE: 481B [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,548][TRACE][o.o.t.T.tracer ] [node_s5] [389][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-288] bytes, now [16440] [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,548][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,548][TRACE][o.o.t.T.tracer ] [node_s3] [390][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,548][TRACE][o.o.t.T.tracer ] [node_s4] [388][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,548][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61584, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 480, request id: 389, type: response, version: 3.0.0] READ: 480B [2023-11-15T19:32:50,548][TRACE][o.o.t.T.tracer ] [node_s0] [389][internal:cluster/coordination/publish_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,549][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61578, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 481, request id: 390, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,549][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61585, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 481, request id: 388, type: response, version: 3.0.0] READ: 481B [2023-11-15T19:32:50,549][TRACE][o.o.t.T.tracer ] [node_s0] [390][internal:cluster/coordination/publish_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,549][TRACE][o.o.t.T.tracer ] [node_s0] [388][internal:cluster/coordination/publish_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,549][TRACE][o.o.g.P.MetadataIndexWriter] [node_s4][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s4/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,549][TRACE][o.o.g.P.MetadataIndexWriter] [node_s1][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s1/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,549][TRACE][o.o.g.P.MetadataIndexWriter] [node_s2][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s2/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,549][TRACE][o.o.g.P.MetadataIndexWriter] [node_s0][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s0/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,549][TRACE][o.o.g.P.MetadataIndexWriter] [node_s3][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s3/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,550][TRACE][o.o.g.P.MetadataIndexWriter] [node_s5][NIOFSDirectory@/private/var/folders/z2/wzcpdgsx7zv3m3p57fk3g41w0000gr/T/org.opensearch.search.query.SearchQueryIT_D490856B1361B346-022/tempDir-002/node_s5/nodes/0/_state lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bcde55d] flushing [2023-11-15T19:32:50,574][TRACE][o.o.i.b.request ] [node_s4] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,575][TRACE][o.o.i.b.request ] [node_s5] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,575][TRACE][o.o.i.b.request ] [node_s1] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,578][TRACE][o.o.i.b.request ] [node_s3] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,578][TRACE][o.o.i.b.request ] [node_s2] [request] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,581][DEBUG][o.o.g.PersistedClusterStateService] [node_s4] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,581][DEBUG][o.o.g.PersistedClusterStateService] [node_s5] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,581][DEBUG][o.o.g.PersistedClusterStateService] [node_s1] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,581][TRACE][o.o.c.c.Coordinator ] [node_s1] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,581][TRACE][o.o.d.PeerFinder ] [node_s1] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,581][TRACE][o.o.d.PeerFinder ] [node_s1] not active [2023-11-15T19:32:50,582][TRACE][o.o.c.c.PreVoteCollector ] [node_s1] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,582][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 165 [2023-11-15T19:32:50,582][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16728] [2023-11-15T19:32:50,582][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61567} [length: 492, request id: 387, type: response, version: 3.0.0, header size: 48B] WRITE: 492B [2023-11-15T19:32:50,582][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-288] bytes, now [16440] [2023-11-15T19:32:50,582][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,582][TRACE][o.o.t.T.tracer ] [node_s1] [387][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,582][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61567, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 492, request id: 387, type: response, version: 3.0.0] READ: 492B [2023-11-15T19:32:50,582][TRACE][o.o.t.T.tracer ] [node_s0] [387][internal:cluster/coordination/publish_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,584][DEBUG][o.o.g.PersistedClusterStateService] [node_s3] writing cluster state took [204ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,584][DEBUG][o.o.g.PersistedClusterStateService] [node_s2] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,584][TRACE][o.o.c.c.Coordinator ] [node_s2] handlePublishRequest: coordinator remaining FOLLOWER of [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] in term 1 [2023-11-15T19:32:50,584][TRACE][o.o.d.PeerFinder ] [node_s2] deactivating and setting leader to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,584][TRACE][o.o.d.PeerFinder ] [node_s2] not active [2023-11-15T19:32:50,584][TRACE][o.o.c.c.PreVoteCollector ] [node_s2] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,584][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 154 [2023-11-15T19:32:50,584][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16728] [2023-11-15T19:32:50,584][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61576} [length: 492, request id: 386, type: response, version: 3.0.0, header size: 48B] WRITE: 492B [2023-11-15T19:32:50,585][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-288] bytes, now [16440] [2023-11-15T19:32:50,585][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,585][TRACE][o.o.t.T.tracer ] [node_s2] [386][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,585][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61576, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 492, request id: 386, type: response, version: 3.0.0] READ: 492B [2023-11-15T19:32:50,585][TRACE][o.o.t.T.tracer ] [node_s0] [386][internal:cluster/coordination/publish_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,588][DEBUG][o.o.g.PersistedClusterStateService] [node_s0] writing cluster state took [0ms]; wrote global metadata [true] and metadata for [0] indices and skipped [0] unchanged indices [2023-11-15T19:32:50,588][TRACE][o.o.c.c.PreVoteCollector ] [node_s0] updating with preVoteResponse=PreVoteResponse{currentTerm=1, lastAcceptedTerm=1, lastAcceptedVersion=28}, leader={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,589][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 228 [2023-11-15T19:32:50,589][TRACE][o.o.t.T.tracer ] [node_s0] [385][internal:cluster/coordination/publish_state] sent response [2023-11-15T19:32:50,589][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,589][TRACE][o.o.t.T.tracer ] [node_s0] [385][internal:cluster/coordination/publish_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,589][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,589][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [28] and term [1] from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,589][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,589][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,589][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: accepted publish response for version [28] and term [1] from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,589][TRACE][o.o.c.c.CoordinationState] [node_s0] handlePublishResponse: value committed for version [28] and term [1] [2023-11-15T19:32:50,589][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,589][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,589][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 322, request id: 391, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,589][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,589][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s0] [391][internal:cluster/coordination/commit_state] sent to [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 322, request id: 392, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 322, request id: 391, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s0] [392][internal:cluster/coordination/commit_state] sent to [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s2] [391][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 322, request id: 392, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s1] [392][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,590][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,590][TRACE][o.o.t.TaskManager ] [node_s2] register 155 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,590][TRACE][o.o.c.c.Coordinator ] [node_s2] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][TRACE][o.o.c.c.CoordinationState] [node_s2] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,590][TRACE][o.o.t.TaskManager ] [node_s1] register 166 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 322, request id: 393, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,590][TRACE][o.o.c.c.Coordinator ] [node_s1] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][TRACE][o.o.c.c.CoordinationState] [node_s1] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,590][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s0] [393][internal:cluster/coordination/commit_state] sent to [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 322, request id: 393, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,590][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 322, request id: 394, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s4] [393][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,590][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] cluster state updated, version [28], source [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, local {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,590][TRACE][o.o.t.TaskManager ] [node_s4] register 138 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,590][TRACE][o.o.c.s.ClusterApplierService] [node_s2] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][TRACE][o.o.c.c.Coordinator ] [node_s4] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] cluster state updated, version [28], source [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, local routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s0] [394][internal:cluster/coordination/commit_state] sent to [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,590][TRACE][o.o.c.c.CoordinationState] [node_s4] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,590][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 322, request id: 394, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,590][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,590][TRACE][o.o.t.T.tracer ] [node_s3] [394][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,590][TRACE][o.o.c.s.ClusterApplierService] [node_s1] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,590][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.t.TaskManager ] [node_s3] register 289 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,590][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.c.Coordinator ] [node_s3] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,590][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s1] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s2] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] applying settings from cluster state with version 28 [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] apply cluster state with version 28 [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] applying settings from cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.repositories.RepositoriesService@2e256902] with change to version [28] [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] apply cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.indices.cluster.IndicesClusterStateService@58f1b39a] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.repositories.RepositoriesService@417b8324] with change to version [28] [2023-11-15T19:32:50,590][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 322, request id: 395, type: request, version: 3.0.0, header size: 92B, action: internal:cluster/coordination/commit_state] WRITE: 322B [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.indices.cluster.IndicesClusterStateService@45440fd3] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.script.MockScriptService@b2181c5] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.RestoreService@57f9c669] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.script.MockScriptService@46891f] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.ingest.IngestService@5211c8ed] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.RestoreService@2cf36cc8] with change to version [28] [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] cluster state updated, version [28], source [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, local {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.ingest.IngestService@310ca8d1] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.search.pipeline.SearchPipelineService@729cc56] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.search.pipeline.SearchPipelineService@545313b4] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s4] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.ingest.IngestActionForwarder@60697650] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@574bd3b9] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.tasks.TaskManager@3d5b3a1] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.SnapshotsService@4f57dedc] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] set locally applied cluster state to version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.InternalClusterInfoService@6427ea90] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@6df82bb6] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@439ed513] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@324e4282] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResponseCollectorService@69665d31] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.node.ResourceUsageCollectorService@456f6e30] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksClusterService@70ec0dc3] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.cluster.routing.DelayedAllocationService@30f60b0e] with change to version [28] [2023-11-15T19:32:50,590][TRACE][o.o.c.c.CoordinationState] [node_s3] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.persistent.PersistentTasksNodeService@6b2257f4] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [207] bytes, now [207] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s2] calling [org.opensearch.gateway.GatewayService@5582aa4e] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 322, request id: 395, type: request, version: 3.0.0, action: internal:cluster/coordination/commit_state] READ: 322B [2023-11-15T19:32:50,591][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handling join within publish response: Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, targetNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s2] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.t.TaskManager ] [node_s2] unregister task for id: 155 [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,591][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] handlePublishResponse: handling [PublishResponse{term=1, version=28}] from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}]) [2023-11-15T19:32:50,591][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [55] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.ingest.IngestActionForwarder@814adf] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.T.tracer ] [node_s5] [395][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,591][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][TRACE][o.o.t.T.tracer ] [node_s0] [395][internal:cluster/coordination/commit_state] sent to [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@19521e28] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.tasks.TaskManager@45d8fb4f] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.TransportLogger ] [node_s2] NioSocketChannel{localAddress=/127.0.0.1:61512, remoteAddress=/127.0.0.1:61569} [length: 71, request id: 391, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.SnapshotsService@cdac4de] with change to version [28] [2023-11-15T19:32:50,591][DEBUG][o.o.t.TransportService ] [node_s0] Action: internal:cluster/coordination/commit_state [2023-11-15T19:32:50,591][TRACE][o.o.c.NodeConnectionsService] [node_s4] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] set locally applied cluster state to version 28 [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] applying settings from cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.InternalClusterInfoService@6a31ad84] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.TaskManager ] [node_s5] register 733 [transport] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] apply cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.repositories.RepositoriesService@5daadbc8] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@659248ee] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.cluster.IndicesClusterStateService@599544a5] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@3abdd4fd] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@23792777] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.i.b.in_flight_requests] [node_s2] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResponseCollectorService@4443e311] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.T.tracer ] [node_s2] [391][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.node.ResourceUsageCollectorService@3361164f] with change to version [28] [2023-11-15T19:32:50,591][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] cluster state updated, version [28], source [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, local {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksClusterService@672f36e2] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.c.Coordinator ] [node_s5] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.cluster.routing.DelayedAllocationService@d07e7ef] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.t.T.tracer ] [node_s0] [396][internal:cluster/coordination/commit_state] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,591][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61569, remoteAddress=127.0.0.1/127.0.0.1:61512} [length: 71, request id: 391, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,591][TRACE][o.o.t.T.tracer ] [node_s0] [396][internal:cluster/coordination/commit_state] received request [2023-11-15T19:32:50,591][TRACE][o.o.c.c.CoordinationState] [node_s5] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.persistent.PersistentTasksNodeService@5687d635] with change to version [28] [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s3] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,591][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.script.MockScriptService@1c97d1f3] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [391][internal:cluster/coordination/commit_state] received response from [{node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s1] calling [org.opensearch.gateway.GatewayService@6d6f71c4] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.ingest.IngestService@26be5633] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s1] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s0] register 229 [direct] [internal:cluster/coordination/commit_state] [] [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s1] unregister task for id: 166 [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.search.pipeline.SearchPipelineService@2d41adb7] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.action.ingest.IngestActionForwarder@5e8274d6] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.tasks.TaskManager@7cb82ebc] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: execute [2023-11-15T19:32:50,592][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=28} applied version 28, max now 28 [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] set locally applied cluster state to version 28 [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s1] NioSocketChannel{localAddress=/127.0.0.1:61517, remoteAddress=/127.0.0.1:61565} [length: 71, request id: 392, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResponseCollectorService@28e3e0d6] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.snapshots.SnapshotShardsService@8efdc68] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.node.ResourceUsageCollectorService@45863d67] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.store.IndicesStore@78f636a5] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.c.Coordinator ] [node_s0] handleApplyCommit: applying commit ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}} [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s3] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.persistent.PersistentTasksNodeService@395e152d] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.c.CoordinationState] [node_s0] handleCommit: applying commit request for term [1] and version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] applying settings from cluster state with version 28 [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s1] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@4d476a64] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] apply cluster state with version 28 [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s1] [392][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 229 [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s4] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@44d2e77] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [396][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.repositories.RepositoriesService@6032f45b] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] cluster state updated, version [28], source [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: true meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, local {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s4] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [396][internal:cluster/coordination/commit_state] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61565, remoteAddress=127.0.0.1/127.0.0.1:61517} [length: 71, request id: 392, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s4] unregister task for id: 138 [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.cluster.IndicesClusterStateService@640d6c85] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [392][internal:cluster/coordination/commit_state] received response from [{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s4] NioSocketChannel{localAddress=/127.0.0.1:61514, remoteAddress=/127.0.0.1:61586} [length: 71, request id: 393, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.script.MockScriptService@210ffe60] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.ingest.IngestService@67038f1b] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.search.pipeline.SearchPipelineService@10e43f8c] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.action.ingest.IngestActionForwarder@1f021051] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s4] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.tasks.TaskManager@4120012f] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.NodeConnectionsService] [node_s5] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s4] [393][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] set locally applied cluster state to version 28 [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] applying settings from cluster state with version 28 [2023-11-15T19:32:50,592][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=28} applied version 28, max now 28 [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] apply cluster state with version 28 [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResponseCollectorService@3a202547] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61586, remoteAddress=127.0.0.1/127.0.0.1:61514} [length: 71, request id: 393, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.snapshots.SnapshotShardsService@b0965d0] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.repositories.RepositoriesService@11ab1a37] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [393][internal:cluster/coordination/commit_state] received response from [{node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.cluster.IndicesClusterStateService@15948cda] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.node.ResourceUsageCollectorService@4067c78] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.store.IndicesStore@456e3cae] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.script.MockScriptService@5096100c] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.ingest.IngestService@28bfec03] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.persistent.PersistentTasksNodeService@7e5f3359] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.search.pipeline.SearchPipelineService@610c2181] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@3e38eea7] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.action.ingest.IngestActionForwarder@50cf8326] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s3] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@96aa0a4] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.tasks.TaskManager@30b9ef43] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28} applied version 28, max now 28 [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] set locally applied cluster state to version 28 [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResponseCollectorService@6731c2a3] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s3] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.snapshots.SnapshotShardsService@6731e7ed] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.node.ResourceUsageCollectorService@4bcb8e9e] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 289 [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.store.IndicesStore@34f7b353] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.persistent.PersistentTasksNodeService@5d3a5bb4] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.recovery.PeerRecoverySourceService@35dcd171] with change to version [28] [2023-11-15T19:32:50,592][TRACE][o.o.c.s.ClusterApplierService] [node_s5] calling [org.opensearch.indices.replication.SegmentReplicationSourceService@2274062a] with change to version [28] [2023-11-15T19:32:50,592][DEBUG][o.o.c.s.ClusterApplierService] [node_s5] processing [ApplyCommitRequest{term=1, version=28, sourceNode={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61516, remoteAddress=/127.0.0.1:61580} [length: 71, request id: 394, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,592][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 733 [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16647] [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61513, remoteAddress=/127.0.0.1:61579} [length: 71, request id: 395, type: response, version: 3.0.0, header size: 48B] WRITE: 71B [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s3] [394][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-207] bytes, now [16440] [2023-11-15T19:32:50,592][TRACE][o.o.i.b.in_flight_requests] [node_s5] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s5] [395][internal:cluster/coordination/commit_state] sent response [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61580, remoteAddress=127.0.0.1/127.0.0.1:61516} [length: 71, request id: 394, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,592][TRACE][o.o.t.T.tracer ] [node_s0] [394][internal:cluster/coordination/commit_state] received response from [{node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,592][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61579, remoteAddress=127.0.0.1/127.0.0.1:61513} [length: 71, request id: 395, type: response, version: 3.0.0] READ: 71B [2023-11-15T19:32:50,593][TRACE][o.o.t.T.tracer ] [node_s0] [395][internal:cluster/coordination/commit_state] received response from [{node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,593][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28} applied version 28, max now 28 [2023-11-15T19:32:50,593][TRACE][o.o.c.c.LagDetector ] [node_s0] NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28} applied version 28, max now 28 [2023-11-15T19:32:50,593][TRACE][o.o.c.c.C.CoordinatorPublication] [node_s0] onPossibleCompletion: [Publication{term=1, version=28}] was successful [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=28}]: execute [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] cluster state updated, version [28], source [Publication{term=1, version=28}] cluster uuid: 0G3YDLHTTpmWJZrGFEBIwA [committed: true] version: 28 state uuid: _5esdxhhRs-Fh4d2QFm-kA from_diff: false meta data version: 23 coordination_metadata: term: 1 last_committed_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} last_accepted_config: VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} voting tombstones: [] metadata customs: index-graveyard: IndexGraveyard[[[index=[test/RxQIvuKTRjSPkdTQFWDwqA], deleteDate=2023-11-16T01:32:48.222Z], [index=[test/VYTqcztNQ8mZP5spA2-HmQ], deleteDate=2023-11-16T01:32:50.437Z]]] nodes: {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, local, cluster-manager {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} routing_table (version 13): routing_nodes: -----node_id[pWGsJ26ZTSmfrCNxbu-KoQ][V] -----node_id[9SWyj4IxQoKQhSS_1-SQNQ][V] -----node_id[OhN8Eeh3SuiVD7m3P0ly7Q][V] ---- unassigned [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] connecting to nodes of cluster state with version 28 [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.NodeConnectionsService] [node_s0] still connected to {node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] applying settings from cluster state with version 28 [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] apply cluster state with version 28 [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.repositories.RepositoriesService@4b52a152] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.indices.cluster.IndicesClusterStateService@4f6d2ce0] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.script.MockScriptService@a6fab69] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.RestoreService@65a9c473] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.ingest.IngestService@3b2fa1a5] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.search.pipeline.SearchPipelineService@1d03900c] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.ingest.IngestActionForwarder@6a405426] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2411/0x000000013165b540@41e19f41] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.tasks.TaskManager@7052c51] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.SnapshotsService@6075f4d] with change to version [28] [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] set locally applied cluster state to version 28 [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.InternalClusterInfoService@23a71628] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.snapshots.InternalSnapshotsInfoService@7c26f627] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@73b53b39] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.metadata.TemplateUpgradeService@170fb472] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResponseCollectorService@5cd3ff7a] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.node.ResourceUsageCollectorService@660cd06c] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksClusterService@3cba4367] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.cluster.routing.DelayedAllocationService@56fcdca6] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.r.DelayedAllocationService] [node_s0] no need to schedule reroute - no delayed unassigned shards [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.persistent.PersistentTasksNodeService@1597d12c] with change to version [28] [2023-11-15T19:32:50,593][TRACE][o.o.c.s.ClusterApplierService] [node_s0] calling [org.opensearch.gateway.GatewayService@1518ec42] with change to version [28] [2023-11-15T19:32:50,593][DEBUG][o.o.c.s.ClusterApplierService] [node_s0] processing [Publication{term=1, version=28}]: took [0s] done applying updated cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) [2023-11-15T19:32:50,593][DEBUG][o.o.c.c.C.CoordinatorPublication] [node_s0] publication ended successfully: Publication{term=1, version=28} [2023-11-15T19:32:50,593][TRACE][o.o.c.c.Reconfigurator ] [node_s0] Reconfigurator{autoShrinkVotingConfiguration=true} reconfiguring VotingConfiguration{qyM6N6JnTzC20m5eT6YiNQ,VewvEbaXQMeeqnL9EVM1cw,xeiWmyoIStSZl02qvB3WsA} based on liveNodes=[{node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, {node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}, {node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}], retiredNodeIds=[], currentClusterManager={node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true} [2023-11-15T19:32:50,593][TRACE][o.o.c.c.LagDetector ] [node_s0] lag detection for version 28 is unnecessary: [NodeAppliedStateTracker{discoveryNode={node_s1}{xeiWmyoIStSZl02qvB3WsA}{yI5sjc5dRxmWyVaz3t3pxA}{127.0.0.1}{127.0.0.1:61517}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=28}, NodeAppliedStateTracker{discoveryNode={node_s5}{pWGsJ26ZTSmfrCNxbu-KoQ}{ykC9RMfESaGahq9O7sFFyw}{127.0.0.1}{127.0.0.1:61513}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28}, NodeAppliedStateTracker{discoveryNode={node_s4}{OhN8Eeh3SuiVD7m3P0ly7Q}{_UyNQsmGSXehWfb1VP1XxA}{127.0.0.1}{127.0.0.1:61514}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28}, NodeAppliedStateTracker{discoveryNode={node_s3}{9SWyj4IxQoKQhSS_1-SQNQ}{YIS1ZCFbQROMwtYM1dq4xg}{127.0.0.1}{127.0.0.1:61516}{dir}{shard_indexing_pressure_enabled=true}, appliedVersion=28}, NodeAppliedStateTracker{discoveryNode={node_s2}{VewvEbaXQMeeqnL9EVM1cw}{f9oUv8jDQpux45DKb3cF2A}{127.0.0.1}{127.0.0.1:61512}{imr}{shard_indexing_pressure_enabled=true}, appliedVersion=28}] [2023-11-15T19:32:50,593][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 227 [2023-11-15T19:32:50,593][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16495] [2023-11-15T19:32:50,594][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61551} [length: 72, request id: 513, type: response, version: 3.0.0, header size: 48B] WRITE: 72B [2023-11-15T19:32:50,594][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-55] bytes, now [16440] [2023-11-15T19:32:50,594][DEBUG][o.o.c.s.MasterService ] [node_s0] took [0s] to notify listeners on successful publication of cluster state (version: 28, uuid: _5esdxhhRs-Fh4d2QFm-kA) for [remove-index-template [random_index_template]] [2023-11-15T19:32:50,594][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,594][TRACE][o.o.t.T.tracer ] [node_s0] [513][indices:admin/template/delete] sent response [2023-11-15T19:32:50,594][TRACE][o.o.t.TransportLogger ] [node_s5] NioSocketChannel{localAddress=/127.0.0.1:61551, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 72, request id: 513, type: response, version: 3.0.0] READ: 72B [2023-11-15T19:32:50,594][TRACE][o.o.t.T.tracer ] [node_s5] [513][indices:admin/template/delete] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,594][TRACE][o.o.t.TaskManager ] [node_s5] unregister task for id: 731 [2023-11-15T19:32:50,594][TRACE][o.o.t.TaskManager ] [testCommonTermsQuery] register 290 [transport] [cluster:admin/repository/get] [] [2023-11-15T19:32:50,594][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [testCommonTermsQuery] starting processing request [org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesRequest@1f8c320a] with cluster state version [28] [2023-11-15T19:32:50,594][DEBUG][o.o.t.TransportService ] [testCommonTermsQuery] Action: cluster:admin/repository/get [2023-11-15T19:32:50,594][TRACE][o.o.i.b.in_flight_requests] [testCommonTermsQuery] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16440] [2023-11-15T19:32:50,594][TRACE][o.o.t.TransportLogger ] [testCommonTermsQuery] NioSocketChannel{localAddress=/127.0.0.1:61540, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 138, request id: 69, type: request, version: 3.0.0, header size: 78B, action: cluster:admin/repository/get] WRITE: 138B [2023-11-15T19:32:50,594][TRACE][o.o.i.b.in_flight_requests] [node_s3] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [0] [2023-11-15T19:32:50,594][TRACE][o.o.t.T.tracer ] [node_s3] [69][cluster:admin/repository/get] sent to [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] (timeout: [null]) [2023-11-15T19:32:50,594][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adding [37b][cluster:admin/repository/get] to used bytes [new used: [37b], limit: 4294967296 [4gb], estimate: 74 [74b]] [2023-11-15T19:32:50,594][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61540} [length: 138, request id: 69, type: request, version: 3.0.0, action: cluster:admin/repository/get] READ: 138B [2023-11-15T19:32:50,594][TRACE][o.o.t.T.tracer ] [node_s0] [69][cluster:admin/repository/get] received request [2023-11-15T19:32:50,594][TRACE][o.o.t.TaskManager ] [node_s0] register 230 [transport] [cluster:admin/repository/get] [] [2023-11-15T19:32:50,594][TRACE][o.o.a.s.c.TransportClusterManagerNodeAction] [node_s0] starting processing request [org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesRequest@423a75af] with cluster state version [28] [2023-11-15T19:32:50,594][TRACE][o.o.t.TaskManager ] [node_s0] unregister task for id: 230 [2023-11-15T19:32:50,595][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [16440] bytes, now [16477] [2023-11-15T19:32:50,595][TRACE][o.o.t.TransportLogger ] [node_s0] NioSocketChannel{localAddress=/127.0.0.1:61515, remoteAddress=/127.0.0.1:61540} [length: 4002, request id: 69, type: response, version: 3.0.0, header size: 48B] WRITE: 4002B [2023-11-15T19:32:50,595][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-16440] bytes, now [37] [2023-11-15T19:32:50,595][TRACE][o.o.t.TransportLogger ] [node_s3] NioSocketChannel{localAddress=/127.0.0.1:61540, remoteAddress=127.0.0.1/127.0.0.1:61515} [length: 4002, request id: 69, type: response, version: 3.0.0] READ: 4002B [2023-11-15T19:32:50,595][TRACE][o.o.t.T.tracer ] [node_s3] [69][cluster:admin/repository/get] received response from [{node_s0}{qyM6N6JnTzC20m5eT6YiNQ}{9x3UO5FBTu6JcDHHjxzGAQ}{127.0.0.1}{127.0.0.1:61515}{imr}{shard_indexing_pressure_enabled=true}] [2023-11-15T19:32:50,595][TRACE][o.o.t.TaskManager ] [node_s3] unregister task for id: 290 [2023-11-15T19:32:50,595][TRACE][o.o.t.T.tracer ] [node_s0] [69][cluster:admin/repository/get] sent error response org.opensearch.repositories.RepositoryMissingException: [*] missing at org.opensearch.action.admin.cluster.repositories.get.TransportGetRepositoriesAction.clusterManagerOperation(TransportGetRepositoriesAction.java:140) [classes/:?] at org.opensearch.action.admin.cluster.repositories.get.TransportGetRepositoriesAction.clusterManagerOperation(TransportGetRepositoriesAction.java:65) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.masterOperation(TransportClusterManagerNodeAction.java:144) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.clusterManagerOperation(TransportClusterManagerNodeAction.java:153) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.lambda$doStart$3(TransportClusterManagerNodeAction.java:271) [classes/:?] at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [classes/:?] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [classes/:?] at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:271) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.tryAction(TransportClusterManagerNodeAction.java:206) [classes/:?] at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:137) [classes/:?] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [classes/:?] at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341) [classes/:?] at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:115) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:167) [classes/:?] at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:79) [classes/:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [classes/:?] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [classes/:?] at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:102) [classes/:?] at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:98) [classes/:?] at org.opensearch.ratelimitting.admissioncontrol.transport.AdmissionControlTransportHandler.messageReceived(AdmissionControlTransportHandler.java:63) [classes/:?] at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) [classes/:?] at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) [classes/:?] at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [classes/:?] at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [classes/:?] at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [classes/:?] at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [classes/:?] at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [classes/:?] at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [classes/:?] at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343) [classes/:?] at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246) [classes/:?] at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59) [classes/:?] at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152) [classes/:?] at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167) [classes/:?] at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438) [classes/:?] at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264) [classes/:?] at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) [classes/:?] at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) [classes/:?] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] [2023-11-15T19:32:50,596][TRACE][o.o.i.b.in_flight_requests] [node_s0] [in_flight_requests] Adjusted breaker by [-37] bytes, now [0] [2023-11-15T19:32:50,597][INFO ][o.o.s.q.SearchQueryIT ] [testCommonTermsQuery] [SearchQueryIT#testCommonTermsQuery [p0={"search.concurrent_segment_search.enabled":"false"} seed=[D490856B1361B346:5B09F6EE7D3C6926]]]: cleaned up after test [2023-11-15T19:32:50,598][INFO ][o.o.s.q.SearchQueryIT ] [testCommonTermsQuery] ["search.concurrent_segment_search.enabled":"false"] after test REPRODUCE WITH: ./gradlew 'null' --tests "org.opensearch.search.query.SearchQueryIT" -Dtests.method="testCommonTermsQuery [p0={"search.concurrent_segment_search.enabled":"false"} seed=[D490856B1361B346:5B09F6EE7D3C6926]]" -Dtests.seed=D490856B1361B346 -Dtests.opensearch.logger.level=TRACE -Dtests.locale=bg -Dtests.timezone=America/Mexico_City -Druntime.java=17 java.lang.AssertionError: Count is 2 hits but 1 was expected. Total shards: 1 Successful shards: 1 & 0 shard failures: at __randomizedtesting.SeedInfo.seed([D490856B1361B346:5B09F6EE7D3C6926]:0) at org.junit.Assert.fail(Assert.java:89) at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:303) at org.opensearch.search.query.SearchQueryIT.testCommonTermsQuery(SearchQueryIT.java:421) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.base/java.lang.Thread.run(Thread.java:840)