Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-4006] In long running contexts, we encountered the situation of double registe... #2886

Closed
wants to merge 5 commits into from

Commits on Oct 21, 2014

  1. In long running contexts, we encountered the situation of double regi…

    …ster without a remove in between. The cause for that is unknown, and assumed a temp network issue.
    
        However, since the second register is with a BlockManagerId on a different port, blockManagerInfo.contains() returns false, while blockManagerIdByExecutor returns Some. This inconsistency is caught in a conditional statement that does System.exit(1), which is a huge robustness issue for us.
    
        The fix - simply remove the old id from both maps during register when this happens. We are mimicking the behavior of expireDeadHosts(), by doing local cleanup of the maps before trying to add new ones.
    
        Also - added some logging for register and unregister.
    tsliwowicz committed Oct 21, 2014
    Configuration menu
    Copy the full SHA
    f48bce9 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2014

  1. Code review comments fixed

    tsliwowicz committed Oct 23, 2014
    Configuration menu
    Copy the full SHA
    df9d98f View commit details
    Browse the repository at this point in the history
  2. whitspaces fix

    tsliwowicz committed Oct 23, 2014
    Configuration menu
    Copy the full SHA
    7bcfc3d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    41a2217 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    094d508 View commit details
    Browse the repository at this point in the history