[DO NOT MERGE] Backport some fixes from master to ozone-1.4

[xichen01]

What changes were proposed in this pull request?

Backport the following fixes from master to ozone-1.4 for 1.4.1 (Do not include Etag-related PRs, these will be backported in separate PRs):

HDDS-10680. Duplicate delete key blocks sent to SCM (#6513)
HDDS-10692. ozone s3 getsecret prints some internal details (#6531)
HDDS-10630. Add missing parent directories deleted between initiate and complete MPU (#6496)
HDDS-10677. Bump plotly.js to 2.25.2 (#5915)
HDDS-10676. Bump browserify-sign to 4.2.3 (#6509)
HDDS-10674. Bump ip to 1.1.9 (#6243)
HDDS-10673. Bump es5-ext to 0.10.64 (#6284)
HDDS-10672. Bump express to 4.19.2 (#6442)
HDDS-10669. Bump axios to 0.28.0 (#6242)
HDDS-9200. [Snapshot] Added logs and metrics for snapshot purge and set property APIs (#6453)
HDDS-10590. [Snapshot] Synchronized snapshot purge, set snapshot property and SstFilteringService (#6456)
HDDS-9426. Calculate Exclusive size for deep cleaned snapshot's deleted directories (#6099)
HDDS-10477. Make Rocksdb tools native lib compatible with all chipset with the same arch (#6341)
HDDS-10552. Downgrade Surefire to 3.0.0-M4 (#6406)
HDDS-9802. Tool to fix corrupted snapshot chain (#6386)
HDDS-10496. Fetch dependencies for cache using actual build (#6359)

What is the link to the Apache JIRA

How was this patch tested?

https://github.com/xichen01/ozone/actions/runs/8732721413

[xichen01]

@adoroszlai, @hemantk-12, @swamirishi, @aswinshakil, @SaketaChalamchala, @sodonnel, @ChenSammi please check that your commits are correctly cherry-picked

[adoroszlai]

Thanks @xichen01 for working on the backport. There is already a PR for EC-related fixes: #6551. Let's rebase this one when that's merged.

[xichen01]

https://issues.apache.org/jira/browse/HDDS-10681
https://issues.apache.org/jira/browse/HDDS-10682
https://issues.apache.org/jira/browse/HDDS-10704

Remove above PRs which have been merged in EC-related fixes: #6551

[xichen01]

Thanks @xichen01 for working on the backport. There is already a PR for EC-related fixes: #6551. Let's rebase this one when that's merged.

Done

[adoroszlai]

Thanks @xichen01 for updating the PR, my part LGTM. (There is some minor difference in HDDS-10552 due to HDDS-10013 not being backported, but I think that's OK.)

Please remember to push from CLI instead of merging in GitHub UI. Also, please do not cherry-pick each commit to ozone-1.4 from your own branch. We should preserve the commit hashes from this PR:

git checkout ozone-1.4
git pull
git merge --ff-only ozone-1.4-backport
git push

To be able to do that, this PR needs to be rebased on top of origin/ozone-1.4 (1f16f43 currently). The two branches are now diverged:

$ git log --oneline --decorate --graph HEAD origin/ozone-1.4
* e2359a0c45 (HEAD -> ozone-1.4-backport) HDDS-10680. Duplicate delete key blocks sent to SCM (#6513)
* 1c181d298f HDDS-10692. ozone s3 getsecret prints some internal details (#6531)
* a38906f6ad HDDS-10630. Add missing parent directories deleted between initiate and complete MPU (#6496)
* 4a98099dda HDDS-10677. Bump plotly.js to 2.25.2 (#5915)
* 6cf513b9b9 HDDS-10676. Bump browserify-sign to 4.2.3 (#6509)
* 3c35e1f7eb HDDS-10674. Bump ip to 1.1.9 (#6243)
* 7cf6ce0752 HDDS-10673. Bump es5-ext to 0.10.64 (#6284)
* 161b3c5a35 HDDS-10672. Bump express to 4.19.2 (#6442)
* c14eb106a9 HDDS-10669. Bump axios to 0.28.0 (#6242)
* 9ca6d2c880 HDDS-9200. [Snapshot] Added logs and metrics for snapshot purge and set property APIs (#6453)
* b75fddb0c6 HDDS-10590. [Snapshot] Synchronized snapshot purge, set snapshot property and SstFilteringService (#6456)
* 5e9e8d7653 HDDS-9426. Calculate Exclusive size for deep cleaned snapshot's deleted directories (#6099)
* 572e3aed1b HDDS-10477. Make Rocksdb tools native lib compatible with all chipset with the same arch (#6341)
* cc73e01f77 HDDS-10552. Downgrade Surefire to 3.0.0-M4 (#6406)
* 9a2bde1c70 HDDS-9802. Tool to fix corrupted snapshot chain (#6386)
* 986be02a53 HDDS-10496. Fetch dependencies for cache using actual build (#6359)
| * 1f16f430d2 (origin/ozone-1.4, ozone-1.4) HDDS-10704. Do not fail read of EC block if the last chunk is empty (#6540)
| * e453c61d0b HDDS-10682. EC Reconstruction creates empty chunks at the end of blocks with partial stripes (#6515)
| * b58be97410 HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused (#6514)
|/  
* f31dad3ef9 HDDS-9976. Memory leak for DeleteBlocksCommand when queue is full (#5845)
...

[adoroszlai]

Let's merge #6557 first, which is already based on top of ozone-1.4 while this PR is not. Rebase this one only afterwards.

[hemantk-12]

Thanks @xichen01 for the backporting the changes.

I reviewed snapshot related following changes and left one comment. Otherwise it looks good to me.

1. HDDS-9200. [Snapshot] Added logs and metrics for snapshot purge and set property APIs (HDDS-9200. [Snapshot] Added logs and metrics for snapshot purge and set property APIs #6453)
2. HDDS-10590. [Snapshot] Synchronized snapshot purge, set snapshot property and SstFilteringService (HDDS-10590. [Snapshot] Synchronized snapshot purge, set snapshot property and SstFilteringService #6456)
3. HDDS-9426. Calculate Exclusive size for deep cleaned snapshot's deleted directories (HDDS-9426. Calculate Exclusive size for deep cleaned snapshot's deleted directories #6099)
4. HDDS-10477. Make Rocksdb tools native lib compatible with all chipset with the same arch (HDDS-10477. Make Rocksdb tools native lib compatible with all chipset with the same arch #6341)
5. HDDS-9802. Tool to fix corrupted snapshot chain (HDDS-9802. Tool to fix corrupted snapshot chain #6386)

[ivandika3]

Thank you @xichen01 for the backport.

Could you also help to backport the recent fix HDDS-10717?

[xichen01]

Thank you @xichen01 for the backport.

Could you also help to backport the recent fix HDDS-10717?

Done.

[SaketaChalamchala]

Thanks @xichen01 . My part looks good.

[adoroszlai]

Thanks @xichen01 for completing this backport.