feat(ingestion/iceberg): Improve iceberg connector logging

[skrydal]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[sgomezvillamor]

have you considered using the existing TopKDict?

datahub/metadata-ingestion/src/datahub/utilities/stats_collections.py

Line 17 in c6bb65f

class TopKDict(DefaultDict[_KT, _VT]):

[skrydal]

Yes but it didn't fit my solution in 100%, I wanted to log some more information than just key of the dictionary, I would need to either make dictionary key a tuple (making it even more prone to errors) or keep reference table, which needs to be printed somehow anyway. I decided this would be the simplest solution.

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...datahub/ingestion/source/iceberg/iceberg_common.py	93.75%	2 Missing ⚠️

Files with missing lines	Coverage Δ
...on/src/datahub/ingestion/source/iceberg/iceberg.py	96.95% <100.00%> (ø)
...tahub/ingestion/source/iceberg/iceberg_profiler.py	91.30% <100.00%> (ø)
...datahub/ingestion/source/iceberg/iceberg_common.py	97.41% <93.75%> (-1.52%)	⬇️

... and 6 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6cd995...eb14f5c. Read the comment docs.

[sgomezvillamor]

Even if it's not the TopKDict, keeping a collection of items sorted by a specific field is a common functionality and so it might be beneficial to define it generically so it can be shared. Keeping it within the iceberg source makes it harder to locate and so used.

I agree this can be understood as early not-needed-yet optimization/generalization. Being this more about personal preference than strong arguments, I'm approving.

Anyway, adding unit tests for the new class, regardless of the approach, would be great for maintaining coverage.