Feature/ch157435/cartoframes to carto performance evaluation #1735
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request has been linked to Clubhouse Story #157435: CARTOframes to_carto performance evaluation. |
|
cc @pablomoniz |
Jesus89
approved these changes
Jun 15, 2021
Mmoncadaisla
deleted the
feature/ch157435/cartoframes-to-carto-performance-evaluation
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
In this small PR from Support we aim the following two things:
to_cartoperformance, further context can be found here: https://app.clubhouse.io/cartoteam/story/157435/cartoframes-to-carto-performance-evaluationCDB_TableMetadata_Triggerfrom the CARTOframes side (except if usingto_cartowithif_exists='append'on a cartodbfied table), further context could be found here: https://app.clubhouse.io/cartoteam/story/149748/farmers-insurance-admin-duplicate-key-value-violates-unique-constraint-cdb-tablemetadata-pkey-using-to-carto#activity-159580Proposed solution
Change the order in which the cartodbfication process is being performed, to be performed after the COPY from operation is done to prevent triggers from 1) reducing the overall upload process performance, and 2) potentially raise an unexpected error due to the
CDB_TableMetadata_Trigger.PR changes
io/managers/context_manager.pyremove the cartodbfication step from the corresponding functions (e.g:create_table_from_query) and perform this step through the Batch API after the copy_from is performed (except if'append'is used or the user explicitly setscartodbfy=Falsein theto_cartofunction call.tests/unit/io/managers/test_context_manager.pyadapt corresponding tests (e.g:test_copy_from_exists_replace_truncate)