In the same datahost, disable readHost first, then disable writeHost, report ERROR message, but disable is successful

[Dinosauria]

• dble version：
  5.6.29-dble-2.20.04.99-76eddf968f3fa3236c08c61c389f9bf0c3d2330f-20200504004651
• preconditions ：
  Use automation code to build a 3-node dble zk cluster
• configs：

schema.xml

<!DOCTYPE dble:schema SYSTEM "schema.dtd">
<dble:schema xmlns:dble="http://dble.cloud/">
    <schema name="schema1" sqlMaxLimit="100" dataNode="dn5">
        <table name="test" dataNode="dn1,dn2,dn3,dn4" type="global"/>
        <table name="sharding_2_t1" dataNode="dn1,dn2" rule="hash-two"/>
        <table name="sharding_4_t1" dataNode="dn1,dn2,dn3,dn4" rule="hash-four"/>
    </schema>
    <dataNode name="dn1" dataHost="ha_group1" database="db1"/>
    <dataNode name="dn2" dataHost="ha_group2" database="db1"/>
    <dataNode name="dn3" dataHost="ha_group1" database="db2"/>
    <dataNode name="dn4" dataHost="ha_group2" database="db2"/>
    <dataNode name="dn5" dataHost="ha_group1" database="db3"/>
    <dataHost balance="0" maxCon="1000" minCon="10" name="ha_group1" slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <writeHost host="hostM1" url="172.100.9.5:3306" password="111111" user="test"/>
    </dataHost>
    <dataHost balance="1" maxCon="1000" minCon="10" name="ha_group2" slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <writeHost host="hostM2" url="172.100.9.6:3306" password="111111" user="test" disabled="true" id="hostM2" weight="0">
            <readHost host="slave1" url="172.100.9.2:3306" password="111111" user="test" disabled="true" id="slave1" weight="0"/>
        </writeHost>
    </dataHost>
</dble:schema>

rule.xml

<!DOCTYPE dble:rule SYSTEM "rule.dtd">
<dble:rule xmlns:dble="http://dble.cloud/">
    <tableRule name="hash-two">
        <rule>
            <columns>id</columns>
            <algorithm>two-long</algorithm>
        </rule>
    </tableRule>
    <tableRule name="hash-three">
        <rule>
            <columns>id</columns>
            <algorithm>three-long</algorithm>
        </rule>
    </tableRule>
    <tableRule name="hash-four">
        <rule>
            <columns>id</columns>
            <algorithm>four-long</algorithm>
        </rule>
    </tableRule>
    <tableRule name="hash-string">
        <rule>
            <columns>id</columns>
            <algorithm>hash-into-two</algorithm>
        </rule>
    </tableRule>
    <function name="two-long" class="Hash">
        <property name="partitionCount">2</property>
        <property name="partitionLength">1</property>
    </function>
    <function name="three-long" class="Hash">
        <property name="partitionCount">3</property>
        <property name="partitionLength">1</property>
    </function>
    <function name="four-long" class="Hash">
        <property name="partitionCount">4</property>
        <property name="partitionLength">1</property>
    </function>
    <function name="hash-into-two" class="StringHash">
        <property name="partitionCount">2</property>
        <property name="partitionLength">1</property>
    </function>
</dble:rule>

server.xml

<!DOCTYPE dble:server SYSTEM "server.dtd">
<dble:server xmlns:dble="http://dble.cloud/">
    <system>
        <property name="processors">1</property>
        <property name="processorExecutor">1</property>
    </system>
    <firewall/>
    <user name="test">
        <property name="password">111111</property>
        <property name="schemas">schema1</property>
    </user>
    <user name="root">
        <property name="password">111111</property>
        <property name="manager">true</property>
    </user>
</dble:server>

• steps：

1. dataHost @@disable name='ha_group2' node='slave1'; --succeed
2. dataHost @@disable name='ha_group2' node='hostM2';

• expect result：
  step2 succeed without any ERROR code
• real result：

mysql> dataHost @@disable name='ha_group2' node='hostM2';
ERROR 1003 (HY000): 3:No enum constant com.actiontech.dble.config.loader.zkprocess.zookeeper.process.HaInfo.HaType.

• supplements：

2. dble logs:

• dble1.log
• dble2.log
• dble3.log

[sunsun314]

the dble3.log shows that when the DataHostResponseListener get the notify from zk the data in PathChildrenCacheEvent is ''.Then the dble try to get the right data for once and go on the data parse as the code shows below:

               if (!"".equals(data)) {
                    response(data, childData.getPath());
                } else {
                    CuratorFramework zkConn = ZKUtils.getConnection();
                    String newData = new String(zkConn.getData().forPath(childData.getPath()), "UTF-8");
                    response(newData, childData.getPath());
                }

But the data in zk was still empty, that cause this issue
the error happen on time 2020-05-06 18:00:40.467
But look at the other follower, it got the notify bit latter and get the right data from zk at around 2020-05-06 18:00:40.468

As a result, we thought the issue is caused by the unsynchronized zk data, we may try to fix it in the future

[dcy10000]

For developer：

  public static void setStatusToZK(String nodePath, CuratorFramework curator, String value) throws Exception {
        Stat stat = curator.checkExists().forPath(nodePath);
        if (null == stat) {
            ZKPaths.mkdirs(curator.getZookeeperClient().getZooKeeper(), nodePath);   //the reason caused error
        }
        LOGGER.debug("ZkMultiLoader write file :" + nodePath + ", value :" + value);
        curator.setData().inBackground().forPath(nodePath, value.getBytes());
    }

"ZKPaths.mkdirs" with two arguments will create the last node with empty string.
use this instead:

ZKPaths.mkdirs(curator.getZookeeperClient().getZooKeeper(), nodePath , false);  

Or, replace "setStatusToZK" with "ClusterHelper.setKV".

For user：

This bug is already fixed in #1849.

[quexiuping]

mysql> dbGroup @@disable name='ha_group2' instance='slave1';
Query OK, 0 rows affected (0.02 sec)

mysql> dbGroup @@disable name='ha_group2' instance='hostM2';
Query OK, 0 rows affected (0.01 sec)

verified on :5.7.21-dble-9.9.9.9-2b7c5cc536ee9b092eb51492eacc711ca0e7d20e-20210323051816