Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConcurrentModificationException from ThreadStatePropagationChannelInterceptor #9623

Closed
gautham-kishtapuram opened this issue Oct 31, 2024 · 1 comment

Comments

@gautham-kishtapuram
Copy link

This issue has been observed with Spring Integration Core version 6.3.0.

Detailed Versioning Summary
  • Before Upgrade:
    • Java version: 17
    • Spring Boot version: 3.0.1
    • spring-boot-starter-integration: 3.0.1
    • spring-integration-core: 6.0.1
  • After Upgrade:
    • Java version: 21
    • Spring Boot version: 3.3.0
    • spring-boot-starter-integration: 3.3.0
    • spring-integration-core: 6.3.0

Background and Problem Details
Our project was updated from Java 17 to Java 21 and from spring-integration-core:6.0.1 to 6.3.0.

So in our Spring Integration setup, we implemented a custom ThreadStatePropagationChannelInterceptor to manage MDC (Mapped Diagnostic Context) propagation across asynchronous message channels. However, after the update -- especically spring-integration-core:6.3.0, we began encountering ConcurrentModificationException errors during message processing.Additionally, this only occurs when the application is in debug mode.

Exception Stack Trace
   Caused by: java.util.ConcurrentModificationException
  at java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:977) ~[?:?]
  at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:899) ~[?:?]
  at java.base/java.util.AbstractCollection.toString(AbstractCollection.java:458) ~[?:?]
  at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:467) ~[?:?]
  at org.springframework.integration.channel.interceptor.ThreadStatePropagationChannelInterceptor$MessageWithThreadState.toString(ThreadStatePropagationChannelInterceptor.java:134) ~[spring-integration-core-6.3.0.jar!/:6.3.0]
  at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:467) ~[?:?]
  at org.springframework.integration.channel.AbstractMessageChannel.sendInternal(AbstractMessageChannel.java:381) ~[spring-integration-core-6.3.0.jar!/:6.3.0]
  at org.springframework.integration.channel.AbstractMessageChannel.sendWithMetrics(AbstractMessageChannel.java:349) ~[spring-integration-core-6.3.0.jar!/:6.3.0]
  at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:329) ~[spring-integration-core-6.3.0.jar!/:6.3.0]
  at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:302) ~[spring-integration-core-6.3.0.jar!/:6.3.0]
  
Screenshot from where this expection is being raised
This Screen shot shows that when the application runs in debug mode, a toString method on **messageToSend** is invoked. where, messageToSend contains an implementation of ThreadStatePropagationInterceptor

Screenshot from 2024-10-30 13-21-56


Analysis :

  • According to this PR: https://github.com/Fix ThreadSPropagationChInterceptor for stacking #8735, an update to ThreadStatePropagationChannelInterceptor introduced a design change that stores each context state in stateQueue within MessageWithThreadState.
  • This issue appears to stem from concurrent access to ThreadStatePropagationChannelInterceptor.MessageWithThreadState.stateQueue, which is currently implemented as a LinkedList and is not thread-safe.
  • Because In this new approach, the postReceive() method calls poll() on the stateQueue to retrieve and clear the oldest context while preserving the order of interceptors. However, this fix appears to have introduced a concurrency issue. Since the poll() method modifies the **stateQueue** object, which can lead to a ConcurrentModificationException when multiple threads access stateQueue simultaneously. In my case, I observed that while one thread performs a toString operation on a LinkedList, another thread calls poll(), resulting in this CMException

  • To Reproduce
    Ensure your project is using Spring Integration Core version 6.3.0.
    Implement a custom ThreadStatePropagationChannelInterceptor for context propagation.
    Run the application in debug mode and send messages through an integration flow that routes messages to couple of asynchronous channels. Monitor for any ConcurrentModificationException during message processing.

    Expected behavior
    Concurrent access to stateQueue should not result in a ConcurrentModificationException. Ideally, stateQueue would be thread-safe to support multi-threaded access during message processing.

    @gautham-kishtapuram gautham-kishtapuram added status: waiting-for-triage The issue need to be evaluated and its future decided type: bug labels Oct 31, 2024
    @gautham-kishtapuram gautham-kishtapuram changed the title ConcurrentModificationException from ThreadStatePropagationChannelInterceptor when accessing stateQueue concurrently ConcurrentModificationException from ThreadStatePropagationChannelInterceptor Oct 31, 2024
    @artembilan
    Copy link
    Member

    I think I'll fix it use a LinkedBlockingQueue instead.

    As a workaround I suggest to look into a solution with a message header.
    See SecurityContextPropagationChannelInterceptor, for example.

    And I mentioned that in the PR you've linked here: #8735 (comment)

    @artembilan artembilan added this to the 6.4.0 milestone Oct 31, 2024
    @artembilan artembilan added in: core (EOL) for: backport-to-6.2.x for: backport-to-6.3.x and removed status: waiting-for-triage The issue need to be evaluated and its future decided labels Oct 31, 2024
    spring-builds pushed a commit that referenced this issue Oct 31, 2024
    Fixes: #9623
    Issue link: #9623
    
    The `ConcurrentModificationException` is thrown from the `ThreadStatePropagationChannelInterceptor.MessageWithThreadState.stateQueue`
    which is a not thread-safe `LinkedList`
    
    * Fix `ThreadStatePropagationChannelInterceptor.MessageWithThreadState.stateQueue` to be a `LinkedBlockingQueue` instead
    
    (cherry picked from commit ba57ee8)
    spring-builds pushed a commit that referenced this issue Oct 31, 2024
    Fixes: #9623
    Issue link: #9623
    
    The `ConcurrentModificationException` is thrown from the `ThreadStatePropagationChannelInterceptor.MessageWithThreadState.stateQueue`
    which is a not thread-safe `LinkedList`
    
    * Fix `ThreadStatePropagationChannelInterceptor.MessageWithThreadState.stateQueue` to be a `LinkedBlockingQueue` instead
    
    (cherry picked from commit ba57ee8)
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants