Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include an option to treat Arrow binary columns as String #283

Open
lfkpoa opened this issue Jun 10, 2022 · 0 comments
Open

Include an option to treat Arrow binary columns as String #283

lfkpoa opened this issue Jun 10, 2022 · 0 comments

Comments

@lfkpoa
Copy link

lfkpoa commented Jun 10, 2022

Hi,
I'm really impressed by Arquero and the use of the Arrow format.
I'm trying to use ClickHouse database (which is really fast) with Arquero using Arrow format, since ClickHouse can return the output in different formats including Arrow, and I can send the sql query to their http endpoint using fetch, such as:

data_promise = aq.loadArrow('http://myserver:8123/?add_http_cors_header=1&user=default&password=&default_format=Arrow', 
    { fetch: { method: 'POST', body: 'SELECT col_A, count(*) as cnt from mytable group by col_A' }});

The problem is that strings are returned as binary type and Arquero does not recognize them as strings, so I need to use "derive" on every string column to decode them.
I know the problem is that ClickHouse is not returning the type correctly but I think Arquero and ClickHouse make such a great match that it would be worth it to make it possible to load Arrow tables and convert binary types to strings automatically.
Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant