Is there a way to get progress on long running queries?

[houstonhaynes]

I'm curious about getting a progress value while long running queries execute. Is that possible?

[Giorgi]

There is nothing built-in the .NET library. DuckDB itself has duckdb_query_progress to get the query progress so you can use it to implement yourself and hopefully send a PR!

[houstonhaynes]

I worked with a peer on it and aping the C API by looking at what similar commands were doing. Most of the observables I've build have been in the Meadow IoT ecosystem - so it was kind of a wag. The API is surprisingly light on details (whether it returns ints or a decimal value - only mentioning returning a '-1' in a boundary condition - so I guessed int) I planned on checking it in the next day or so.

We also built the pg_duckdb extension and are going to look into simply using the Npgsql driver so this might be a moot point either way.

[Giorgi]

The API may not have the best documentation but at least it says how many parameters it has and what object it returns. I don't understand why you declared the function with two parameters when it has only one parameter. Looks like AI hallucination to me :) Also, can't understand why the comments are removed from code.

[houstonhaynes]

Bad copy-pasta from other work. The basic idea of posting this without having run it and picked up that flub was to in-effect ask if 1) we're "barking up the right tree" with this approach, and/or 2) whether we're structuring where things reside in the code in a way that doesn't muck up you plan for maintainability.

I wasn't sending this to you as a finished product - I was putting it in front of you to make sure I don't waste any more time on something that you would reject if we had gold-plated it.

Feel free to ignore it. We've got a lot of other spinning plates and this isn't our only option. I really like what you've done, and I would want to keep the conversation if there's an actual constructive exchange to have. All I've gotten so far is opaque critique and not a lot of direction. If that's the way you run things then more power to you. Everyone's busy.

[Giorgi]

You are right - everyone's busy. That's why you should respect other people's time and not send code that will throw an error the moment it runs. No one expects perfect code from the first commit, but you could at least have run to verify that it works.

Anyway, I marked the PR as draft and will provide further feedback once the issues that I mentioned in my previous comment are resolved. By the way, you can look up the function in duckdb.h to get a bit more information about it.

[houstonhaynes]

Thanks - and apologies for getting ahead of our skis. I could have been more clear in what I was asking in the PR. We're also looking at shortcut with pg_duckdb that pre-runs queries that force reads through the hive structure. (and use stats to get a baseline) We really like the idea of using duckdb "directly" but we're in prototyping mode so scouting for different options.