FileTailSource for reading Parquet files,

[Susmit07]

FileTailSource for reading Parquet files, we'd be working with a lower-level stream of bytes, and it's not ideal for structured formats like Parquet. FileTailSource is designed for streaming content like logs or text files where new data is appended. Since Parquet files are columnar and binary, parsing them as raw byte streams, would Pekko Streams File connectors be ideal for the use case?

[Susmit07]

@pjfanning @raboof can you guys please help here

[raboof]

I'm not sure I understand the use case, could you explain that some more?

(there is https://pekko.apache.org/docs/pekko-connectors/current/avroparquet.html , but without understanding the use case I'm not sure it applies)

[Susmit07]

yes started using it ... will keep you folks updated in the thread?

[pjfanning]

I've come across tailing raw HDFS files (eg raw text stored in an HDFS file that is being appended to) - hadoop fs -tail ...

I haven't come across this for Parquet. I am no expert on Parquet but I wasn't aware that such files could be (i) appended to or (ii) tailed.

This link seems to indicate that Parquet files can't be appended to:
https://stackoverflow.com/questions/39234391/how-to-append-data-to-an-existing-parquet-file

[Susmit07]

Hey @pjfanning / @raboof

Thanks with the response ..

Update

https://gist.github.com/Susmit07/e39fd14ca2516f14e922020bf0d2c503

With the above code snippet i am able to read the parquet files, we will only have the access to read the data, our app will be listening to specified directories containing parquet files only, it should read all current files and deltas with time, basically on creation and modification event, with this code things works fine, appending to a pre-existing parquet file i believe that shouldn't happen in our case.

Do you guys see any issues with the above approach?