2024 July APAC Community Sync

[ChenSammi]

July 5th
Attenders: Hao, Mingyu, JIanghua, Xi, Sammi

Jianghua:

• HDDS-11083. Avoid duplicate creation of RunningDatanodeState #6886 reuse RunningDatanodeState in datanode
• Reported that there are hundreds of RaftServer objects on Datanodes. Need further investigation.

Hao/Mingyu:

• Discussed one optimization of container balancer target datanode selection criteria. If the policy is
• Observed high tail latency(>1s) of getBlock request due to LOCK in ChunkUtils#processFileExclusively. ReentrantReadWriteLock like alternative solution is discussed.

Xi:

• Since Ratis 3.1.0 is released. Will try to cherry-pick all existing commits which targets for 1.4.1 to Ozone 1.4.1 release branch in the following weeks.
• Discussed the idea to support S3 storage class feature.

[ChenSammi]

July 19th
Attenders: Hao, Mingyu, Hongbin, Jianghua, Wei-Chiu, Sammi

Hongbin/Jianghua:

• The further investigation shows that "hundreds of RaftServer objects on Datanodes" is because there are so many pipeline directories under the Datanode ratis directory. Besides they will consume some Datanode resource to maintain the state, there is no impact to the integrity of data read/write. It would be nice if Datanode can have some logic to clearly close these RaftServer objects.
• It's found that it took hours for SCM to reflect the new state of specific container replica. Suggest to check the SCM metrics, and tuning the SCM container report handler configuration "ozone.scm.event.Incremental_Container_Report.thread.pool.size" or ozone.scm.event.Container_Report.thread.pool.size" if needed.
• Client failed to read EC data during SCM restart, due to the EC pipeline lack of enough DN information during this period. And the pipeline is cached in OM, and will only be refreshed after 6h currently. Refer to last issue in https://docs.google.com/document/d/1g1h-63fvA-be-clvyVRAHLWehoadCnjNRWyX8Bp-UIU/edit.

Hao:

• RocksDB block cache hit rate on one Datanode is about 15%. Currently the block cache is fixed 1GB in Ozone. Want to improve the block cache hit rate.
• Discussed one common EC reconstruction failure reason, when there is failed and uncommitted block data in some EC container replica due to client side write failure and retry.