ConcurrentModificationException accessing map in KubernetesCrudDispatcher

[MikeEdgar]

Describe the bug

The MockWebServer occasionally crashes due to a ConcurrentModificationException caused from KubernetesCrudDispatcher. Here, this occurs from handleWatch, but appears to be possible elsewhere.

2023-06-23 13:54:00  okhttp3.mockwebserver.MockWebServer  SEVERE  MockWebServer[54552] connection from /127.0.0.1 crashed 
java.util.ConcurrentModificationException
	at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756)
	at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:788)
	at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:786)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at io.fabric8.kubernetes.client.server.mock.KubernetesCrudDispatcher.lambda$handleWatch$6(KubernetesCrudDispatcher.java:298)
	at io.fabric8.kubernetes.client.server.mock.WatchEventsListener.onOpen(WatchEventsListener.java:63)

Fabric8 Kubernetes Client version

6.7.2

Steps to reproduce

Reproducible by running this test [1] in the Strimzi cluster operator test suite.

[1] https://github.com/strimzi/strimzi-kafka-operator/blob/016c98afdd0e173d3f07f523c73f9b9fdb418020/cluster-operator/src/test/java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperatorWithPoolsMockTest.java#L621

Expected behavior

No ConcurrentModificationException :-)

Runtime

other (please specify in additional context)

Kubernetes API Server version

other (please specify in additional context)

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

N/A for runtime/server version. This occurs in unit tests with the mock web server.