JMX metrics for HDFS operations

[findepi]

Similarly to what we do with S3 file system, we should have JMX-exported metrics for HDFS operations/invocations.

[Anurag870]

@findepi Would like to work on this

[findepi]

@Anurag870 sure, thanks! if you need guidance, please join the #dev channel of our slack https://prestosql.io/slack.html

[findepi]

For the record, S3 stats are collected in PrestoS3FileSystemStats class.

[amommendes]

Hey @findepi, is this PR still to be worked? Inspecting the code, we already have NamenodeStats being exported in HiveHdfsModule. Probably I'm missing something, but here do we need to implement more statistics (as we have for PrestoS2FileSystem)?

[findepi]

NamenodeStats expose only few operations (file metadata queries). We could want HDFS stats like file open, or read operations (data throughput).
cc @sopel39

[amommendes]

NamenodeStats expose only few operations (file metadata queries). We could want HDFS stats like file open, or read operations (data throughput).
cc @sopel39

Great!

[amommendes]

Hey @findepi, I would like to work on this!

[tnatssb]

Hi @amommendes, any progress on this?

[amommendes]

Hi @amommendes, any progress on this?

Hey @tnatssb , not yet! I had a discussion about these statistics in the slack a time ago.
If someone wants to take this PR, no problem!

[tangjiangling]

@findepi As a newcomer, I want to start integrating into the community and contributing from this issue.

[tangjiangling]

NamenodeStats expose only few operations (file metadata queries). We could want HDFS stats like file open, or read operations (data throughput).

@findepi I have an idea to implement this feature, and I hope you can review it:

Let me introduce some background first:

1. Get Hadoop FileSystem will be implemented through the Cache, as shown in the code in io.trino.plugin.hive.HdfsEnvironment:
   FileSystemManager.registerCache(TrinoFileSystemCache.INSTANCE);
2. The value in the cache is io.trino.plugin.hive.fs.TrinoFileSystemCache.FileSystemWrapper, which indirectly inherits from Hadoop FileSystem and overrides some of the interfaces (e.g., open, append, create, etc.)

Therefore, the simplest way to achieve this feature is to extend io.trino.plugin.hive.fs.TrinoFileSystemCache.FileSystemWrapper, override more methods (e.g., delete, etc.), add interface metrics (e.g., latency, error count, etc.).
By the way, we also need a class similar to TrinoS3FileSystemStats, can be called TrinoHdfsFileSystemStats, and merge the metrics in NamenodeStats to TrinoHdfsFileSystemStats.

[tangjiangling]

Therefore, the simplest way to achieve this feature is to extend io.trino.plugin.hive.fs.TrinoFileSystemCache.FileSystemWrapper, override more methods (e.g., delete, etc.), add interface metrics (e.g., latency, error count, etc.).

After thinking about it for a while, I shouldn't extend io.trino.plugin.hive.fs.TrinoFileSystemCache.FileSystemWrapper directly, because io.trino.plugin.hive.fs.TrinoFileSystemCache. FileSystemWrapper is the base class of Hadoop ecoFileSystem (e.g., S3, HDFS), extend a class similar to io.trino.plugin.hive.s3.TrinoS3FileSystem, called TrinoHdfsFileSystem, to better monitor operations (e.g., open, create, delete, etc.).

@findepi @electrum

[weijiii]

@findepi Hi I'd like to try taking on this :)

[hashhar]

Seems to be addressed by #17078