Implement Offline-To-Bootstrap state transition in StorageManager #1326

jsjtzyy · 2019-11-30T01:20:56Z

implemented Offline-To-Bootstrap logic in storage manager
introduced State Transition Exception and Transition Error Code
introduced bootstrap-in-progress file to determine if a store is new added and still in bootstrap process
added a config to switch on/off state change listeners in partition state model

codecov-io · 2019-11-30T01:46:49Z

Codecov Report

❗ No coverage uploaded for pull request base (master@b9fa880). Click here to learn what that means.
The diff coverage is 66.66%.

@@            Coverage Diff            @@
##             master    #1326   +/-   ##
=========================================
  Coverage          ?   72.19%           
  Complexity        ?     6496           
=========================================
  Files             ?      471           
  Lines             ?    37220           
  Branches          ?     4696           
=========================================
  Hits              ?    26871           
  Misses            ?     9108           
  Partials          ?     1241

Impacted Files	Coverage Δ	Complexity Δ
...main/java/com.github.ambry.server/AmbryServer.java	`78.87% <ø> (ø)`	`10 <0> (?)`
...b.ambry.clustermap/StaticClusterAgentsFactory.java	`40% <0%> (ø)`	`4 <0> (?)`
...rc/main/java/com.github.ambry.store/BlobStore.java	`89.77% <100%> (ø)`	`100 <1> (?)`
...java/com.github.ambry/config/ClusterMapConfig.java	`100% <100%> (ø)`	`3 <0> (?)`
...hub.ambry/clustermap/StateTransitionException.java	`100% <100%> (ø)`	`2 <2> (?)`
...ithub.ambry/clustermap/StateModelListenerType.java	`100% <100%> (ø)`	`1 <1> (?)`
...ry.replication/CloudToStoreReplicationManager.java	`57.03% <20%> (ø)`	`10 <0> (?)`
...hub.ambry.clustermap/AmbryPartitionStateModel.java	`21.95% <25%> (ø)`	`2 <1> (?)`
.../com.github.ambry.clustermap/HelixParticipant.java	`72.09% <39.13%> (ø)`	`25 <4> (?)`
...ithub.ambry.clustermap/AmbryStateModelFactory.java	`81.81% <60%> (ø)`	`3 <3> (?)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9fa880...b099ee5. Read the comment docs.

This changes integrates replica state transition logic with server regular startup. For new added replica, state transition should perform some bootstrap work. For existing replicas, their regular startup won't be affected.

jsjtzyy · 2019-12-02T01:30:42Z

ambry-clustermap/src/test/java/com.github.ambry.clustermap/MockHelixManagerFactory.java

+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+
+
+public class MockHelixManagerFactory extends HelixFactory {


I moved MockHelixManagerFactory out of HelixParticipantTest.java. No specific change in this file. (This single file contributes 426 lines and changes in rest files should be less than 600 lines, which is not hard to review)

lightningrob

Initial comments added.

ambry-api/src/main/java/com.github.ambry/server/StateModelListenerType.java

ambry-api/src/main/java/com.github.ambry/server/StateTransitionException.java

ambry-api/src/main/java/com.github.ambry/server/TransitionErrorCode.java

ambry-store/src/main/java/com.github.ambry.store/BlobStore.java

ambry-store/src/main/java/com.github.ambry.store/DiskManager.java

ankagrawal

a couple of minor comments. Looks good to me otherwise.

ankagrawal · 2019-12-03T22:25:00Z

ambry-clustermap/src/main/java/com.github.ambry.clustermap/HelixParticipant.java

@@ -43,25 +43,26 @@
 /**
 * An implementation of {@link ClusterParticipant} that registers as a participant to a Helix cluster.
 */
-class HelixParticipant implements ClusterParticipant, PartitionStateChangeListener {
-  private final Logger logger = LoggerFactory.getLogger(getClass());
+public class HelixParticipant implements ClusterParticipant, PartitionStateChangeListener {
  private final String clusterName;
  private final String zkConnectStr;
  private final HelixFactory helixFactory;


MINOR: since we are already in this file, looks like helixfactory is not being used. We can remove it.

right, removed.

ankagrawal · 2019-12-03T23:21:02Z

ambry-clustermap/src/main/java/com.github.ambry.clustermap/HelixParticipant.java

-  public void onPartitionStateChangeToStandbyFromLeader(String partitionName) {
-    for (PartitionStateChangeListener partitionStateChangeListener : partitionStateChangeListeners) {
-      partitionStateChangeListener.onPartitionStateChangeToStandbyFromLeader(partitionName);
+  public void onPartitionBecomeStandbyFromLeader(String partitionName) {


MINOR:

maybe we can replace this by
if(partitionstateChangeListeners.containsKey(StateModelListenerType.CloudToStoreReplicationManagerListener) {
partitionStateChangeListeners.get(StateModelListenerType.CloudToStoreReplicationManagerListener).onPartitionBecomeStandbyFromLeader(partitionName)
}

Appreciate your suggestion but I feel like it loses a little bit of readability. What do you think?

ankagrawal

lgtm

…rs (#1327) introduced ClusterParticipant into ReplicationManager and StatsManager to register state change listeners implemented Offline-To-Bootstrap logic in Replication/Stats Manager. (Note: this change is remaining part of PR #1326)

jsjtzyy self-assigned this Nov 30, 2019

jsjtzyy force-pushed the startup-state-transition branch from 07de3a1 to 0a605eb Compare November 30, 2019 22:31

WIP: Integrate state transition logic with server startup

12f625a

This changes integrates replica state transition logic with server regular startup. For new added replica, state transition should perform some bootstrap work. For existing replicas, their regular startup won't be affected.

jsjtzyy force-pushed the startup-state-transition branch from 0a605eb to 12f625a Compare December 1, 2019 00:50

jsjtzyy changed the title ~~WIP: Integrate state transition logic with server startup~~ WIP: Implement Offline-To-Bootstrap state transition in StorageManager Dec 1, 2019

java docs and simplify tests

ee8d452

jsjtzyy marked this pull request as ready for review December 1, 2019 19:07

jsjtzyy changed the title ~~WIP: Implement Offline-To-Bootstrap state transition in StorageManager~~ Implement Offline-To-Bootstrap state transition in StorageManager Dec 1, 2019

jsjtzyy mentioned this pull request Dec 2, 2019

Implement Offline-To-Bootstrap transition in Replication/Stats Managers #1327

Merged

jsjtzyy commented Dec 2, 2019

View reviewed changes

jsjtzyy requested review from ankagrawal and lightningrob December 2, 2019 17:55

lightningrob requested changes Dec 3, 2019

View reviewed changes

comments

6247328

ankagrawal reviewed Dec 3, 2019

View reviewed changes

addressed comments

b099ee5

ankagrawal approved these changes Dec 4, 2019

View reviewed changes

lightningrob approved these changes Dec 5, 2019

View reviewed changes

lightningrob merged commit 4bf702f into linkedin:master Dec 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Offline-To-Bootstrap state transition in StorageManager #1326

Implement Offline-To-Bootstrap state transition in StorageManager #1326

jsjtzyy commented Nov 30, 2019 •

edited

Loading

codecov-io commented Nov 30, 2019 •

edited

Loading

jsjtzyy Dec 2, 2019 •

edited

Loading

lightningrob left a comment

ankagrawal left a comment

ankagrawal Dec 3, 2019 •

edited

Loading

jsjtzyy Dec 4, 2019

ankagrawal Dec 3, 2019 •

edited

Loading

jsjtzyy Dec 4, 2019

ankagrawal left a comment

		import org.apache.helix.store.zk.ZkHelixPropertyStore;


		public class MockHelixManagerFactory extends HelixFactory {

Implement Offline-To-Bootstrap state transition in StorageManager #1326

Implement Offline-To-Bootstrap state transition in StorageManager #1326

Conversation

jsjtzyy commented Nov 30, 2019 • edited Loading

codecov-io commented Nov 30, 2019 • edited Loading

Codecov Report

jsjtzyy Dec 2, 2019 • edited Loading

Choose a reason for hiding this comment

lightningrob left a comment

Choose a reason for hiding this comment

ankagrawal left a comment

Choose a reason for hiding this comment

ankagrawal Dec 3, 2019 • edited Loading

Choose a reason for hiding this comment

jsjtzyy Dec 4, 2019

Choose a reason for hiding this comment

ankagrawal Dec 3, 2019 • edited Loading

Choose a reason for hiding this comment

jsjtzyy Dec 4, 2019

Choose a reason for hiding this comment

ankagrawal left a comment

Choose a reason for hiding this comment

jsjtzyy commented Nov 30, 2019 •

edited

Loading

codecov-io commented Nov 30, 2019 •

edited

Loading

jsjtzyy Dec 2, 2019 •

edited

Loading

ankagrawal Dec 3, 2019 •

edited

Loading

ankagrawal Dec 3, 2019 •

edited

Loading