RegionCache Enhancement

[zyguan]

Region cache is the core component of kv client. It maintains the information of regions and stores, which is required to send requests to the right TiKV nodes.

Currently, there are some stability issues related to region cache:

• follower read may access a slow store, which leads to remarkable performance degradation.
• region cache can not reload region properly when there is down peer, which leads to issues like unexpected continuous cross AZ traffic after tikv shutdown #879 .
• region cache may invoke GetRegion too frequently, which leads to heavy load on PD.

In order to resolve these issues and build an unified health controller, here are some preparatory tasks related to region cache:

• Store Management
  • Simplify store life cycle (unify state and liveness).
  • Extract store management from region cache.
    • region_cache: restrict access to store related fields #1106
    • region_cache: move store related code to a standalone file #1196
    • region_cache: extract store related fields to store cache #1279
  • Maintain health feedback from each store (shall be the input of replica selection).
    • Support receving health feedback #1153
• Region Management
  • Simplify region state (optimize fields).
    • improve region reload strategy #1122
  • Refine management of background cron jobs.
    • enhance background job management of region cache #1171
  • Let region cache be aware of down peers.
    • improve region reload strategy #1122
  • Reload region properly according to its state and peers' state.
    • improve region reload strategy #1122
    • introduce a random jitter to region cache ttl #1148
  • Investigate the root cause of too frequent GetRegion problem.