Snowflake pandas

[fdosani]

Tagging @jfyu also

Would love to have a bit of a discussion about this PR before taking any action to see if folks align on the approach here. I've overridden the to_pandas function for Snowflake. If no size is provided it will use the internal snowflake function to get the pandas df. If size is provided to will use the existing inherited database functionality.

I was reluctant to change to the insert_dataframe_to_table to use write_pandas(). The 2 are slightly different in their approach. Our method is not using COPY like the internal snowflake method but inserting directly via a INSERT statement, and provides table creation and schema abilities. I updated the docs to explain this. I think if people are really keep on using write_pandas they should be able to call it directly from the cursor.

Any thoughts on the above?

[dasjyao]

Instead of replacing, I think it might be a better idea to combine insert_dataframe_to_table and write_pandas() . More specifically, we should keep the functionality of table creation, but replace the INSERT statements to use write_pandas(). Also, we need to do some performance tests to see how write_pandas() improves the speed.

[jfyu]

I really like the create table feature because that's a very big use case. If it's between one and the other, then I choose to keep the create table feature that we currently have. However, an alternative is to add an argument in insert_dataframe_to_table for large operations using COPY, that could be the simplest solution.

As an aside, it does look like snowflake-connector-python has an insertion method as well (https://docs.snowflake.com/en/user-guide/python-connector-api.html#pd_writer) but only for SQLAlchemy

[jborchma]

Looks great. Two small comments for the docstrings

[jborchma]

lgtm

[fdosani]

There is a bigger question about the insert_dataframe_to_table method. Should we add a flag like use_snowflake_copy which will use the underlying snowflake feature if selected vs our version? It should be simple enough and can do it in the next PR if we all agree on that?

This would provide the best of both worlds :)