feat: logging during ingest #219
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joshwong-cs
changed the title
michalc
force-pushed
the
feat/adding-login-during-ingest
branch
from
b51e6c9 to
2608380
Compare
michalc
force-pushed
the
feat/adding-login-during-ingest
branch
from
2608380 to
92e9ab7
Compare
michalc
approved these changes
May 30, 2024
michalc
force-pushed
the
feat/adding-login-during-ingest
branch
from
92e9ab7 to
556a57c
Compare
In some cases people think pg-bulk-ingest is either doing nothing during ingest, or loading all the data into memory. Adding in a bit more logging to make it clearer data is going into the database, including interleaved with fetching data from source (due to the magic of iterables/generators). The complex logic on when to log is an attempt to make the logging suitable for both small and large numbers of rows without per ingest configuration, and keeping the noise down in the logs. It logs every 100 rows initially, going up to logging every million rows later in the ingest. For example it could log: Ingesting from source into the database... Ingested 100 rows... Ingested 200 rows... Ingested 300 rows... Ingested 400 rows... Ingested 500 rows... Ingested 600 rows... Ingested 700 rows... Ingested 800 rows... Ingested 900 rows... Ingested 1000 rows... Ingested 2000 rows... Ingested 3000 rows... Ingested 4000 rows... Ingested 5000 rows... Ingested 6000 rows... Ingested 7000 rows... Ingested 8000 rows... Ingested 9000 rows... Ingested 10000 rows... Ingested 20000 rows... Ingested 30000 rows... Ingested 40000 rows... Ingested 50000 rows... Ingested 60000 rows... Ingested 70000 rows... Ingested 80000 rows... Ingested 90000 rows... Ingested 100000 rows... Ingested 200000 rows... Ingested 300000 rows... Ingested 400000 rows... Ingested 500000 rows... Ingested 600000 rows... Ingested 700000 rows... Ingested 800000 rows... Ingested 900000 rows... Ingested 1000000 rows... Ingested 2000000 rows... Ingested 3000000 rows... Ingested 4000000 rows... Ingested 5000000 rows... Ingested 6000000 rows... Ingested 7000000 rows... Ingested 8000000 rows... Ingested 9000000 rows... Ingested 10000000 rows... Ingested 11000000 rows... Ingested 12000000 rows... Ingested 12312312 rows in total Co-authored-by: Tash Boyse <57753415+nboyse@users.noreply.github.com> Co-authored-by: Michal Charemza <michal@charemza.name> Co-authored-by: Mohizur Khan <mohizurkhan@digital.trade.gov.uk> Co-authored-by: Josh Wong <166488409+joshwong-cs@users.noreply.github.com>
michalc
force-pushed
the
feat/adding-login-during-ingest
branch
from
556a57c to
a927012
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In some cases people think pg-bulk-ingest is either doing nothing during ingest, or loading all the data into memory. Adding in a bit more logging to make it clearer data is going into the database, including interleaved with fetching data from source (due to the magic of iterables/generators).
The complex logic on when to log is an attempt to make the logging suitable for both small and large numbers of rows without per ingest configuration, and keeping the noise down in the logs. It logs every 100 rows initially, going up to logging every million rows later in the ingest. For example it could log: