Bug/ch155055/emily mo inconsistent to carto error

[Mmoncadaisla]

Context

In this small PR from Support we aim to avoid hitting the Batch API 16 kb payload limit when using the _truncate_and_drop_add_columns function, which is called from to_carto (through copy_from) with if_exists='replace' collision strategy if the table structure changes used and the table has a considerable amount of columns.

Further context can be found here: https://app.clubhouse.io/cartoteam/story/155055/emily-mo-inconsistent-to-carto-error

Proposed solution

1. Create a temporary PostgreSQL user defined function through a standard SQL API call with the required changes
2. Apply this function in the transaction (after the TRUNCATE operation and before the cartodbfication process)
3. Drop the temporary function

Example output with the current logic

CartoException: ['Your payload is too large: 18745 bytes. Max size allowed is 16384 bytes (16kb). Are you trying to import data?. Please, check out import api http://docs.cartodb.com/cartodb-platform/import-api/']

Example output with the changes implemented

Success! Data uploaded to table "test_to_carto_truncate_drop_add_" correctly
'test_to_carto_truncate_drop_add_'

We have also verified that the temporary function is dropped as expected.

PR changes

Relevant

• io/managers/context_manager.py where the logic referred above is implemented. Add helper functions for query definition, add functions to create and drop UDFs and perform necessary changes in the _truncate_and_drop_add_columns function

Minor

• utils/utils.py add a function create_tmp_name to create temporary-like names using uuid
• /data/services/service.py adapt the _new_temporary_table_name method to make use of the create_tmp_name function

[shortcut-integration]

This pull request has been linked to Clubhouse Story #155055: [emily-mo] Inconsistent to_carto error.

[Jesus89]

Nice job. It is a very ingenious solution :)

However, I would check the payload size before going that way because it requires two extra queries. So for every normal situation, it would work the same (with no extra requests) but then the len(query) > X this system can be done.

Regarding the X, the length of the query and the payload of the request are very similar. but instead of using "16384" (which is the limit of the Batch API) we could use 12000 just to make sure there are no false negatives.

[jgoizueta]

LGTM, nice job!

[Jesus89]

LGTM.

I have added a comment to improve readability

[Mmoncadaisla]

Performed different Q&A tests on both an enterprise and a free plan accounts and it seems to be working as expected. Thank you a lot for the great CR @Jesus89 @jgoizueta ❤️