How to display progress bar?

[hjazz6]

Hi,

How do I display the progress bar, say, when I'm opening a file? I have

df = vaex.open(<arrow file>, progress=True)

But there is no progress bar. Is there any package I need to install, or configuration I need to set? I have already installed progressbar2, and I'm running vaex 4.7.0 in Jupyter Notebook.

Thank you.

[JovanVeljanoski]

Opening memory mapped files (arrow, hdf5) is instantaneous, so no progress bar is needed. Thus vaex.open does not support a progress bar (no need for it.. ).

[maartenbreddels]

This is also documented at https://vaex.io/docs/api.html#vaex.open But I think it makes sense for files that take some time to load, is that happening for your arrow file? (from mobile phone) Op do 20 jan. 2022 11:38 schreef Jovan Veljanoski ***@***.***

…

: Opening memory mapped files (arrow, hdf5) is instantaneous, so no progress bar is needed. Thus vaex.open does not support a progress bar (no need for it.. ). — Reply to this email directly, view it on GitHub <#1846 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANPEPKBFUIVXVOHXSTQVLDUW7Q2VANCNFSM5MMJYWOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hjazz6]

Yes, I saw from the documentation that the progress bar can be shown for vaex.open. It takes about 6 seconds to open the file (a few microseconds when timing using %time, but the cell continues running for a few seconds (%%time), so it would be useful to see the progress.

I didn't see progress for vaex.open_many, can it show the progress bar? This would really help because it would take a couple of minutes to open the multiple files I have.

[maartenbreddels]

Does the arrow file contain compressed data? (I guess that would make it a feather file). Or maybe dict encoded data?

[hjazz6]

I don't think so, I didn't explicitly specify for compressed or dict encoded data to be stored in the arrow file. It just contained the results of an elasticsearch query (extracting 29 columns of data). But is there a way I can verify from the arrow file to be sure?

[maartenbreddels]

you can check df.dtypes, if it contains a dictionary type. For compression I don't know. What you can try is to export it to hdf5 or arrow again using vaex, and see if that improves load times, that might give you a hit what the issue is.

[hjazz6]

Thanks, I'll try that next week.

[hjazz6]

Thanks for the suggestion on exporting the arrow file to hdf5. I tried that, and I can now open the file in less than 300ms, and the memory usage seems to be minimal too. I'll convert all my arrow files to hdf5 then.