Save file transfer error histories for further processing #169
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a file download fails, preserve a dated and abbreviated error description in a new column error_history. This gives external scripts the information needed to deal with incorrect urls or bad checksums.
For large-scale replication, it is impossible to manually deal with problems affecting individual files, such as an incorrect checksum reported by the data node. The Synda database as LLNL has over 16,000 files with bad checksums and over 260,000 files which have been needed some kind of individual attention. To do this, we need (and I have) an external script.
Usually a file should be made to "disappear" if it cannot be downloaded after repeated attempts, and if the transfer failures are for reasons specific to that file. By "disappear" I mean that there will be no more attempts to transfer it, even after any possible "synda install..." or "synda retry" command. An easy way to make that happen is to change the database so that the file has a nonstandard status such as "bad_checksum". In order to do this, you have to know how many transfer attempts there have been, and when. The changes in this pull request will add this information to the database.
For an example of such an external script, see https://github.com/painter1/Synda-scripts/blob/fad3c1bb5d131bc0a5bb79d1467d789fd4cc28d1/permanent_error_status.py