Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add monthly cron to run Cover Server Archival #8278

Closed
Tracked by #6822
mekarpeles opened this issue Sep 9, 2023 · 0 comments · Fixed by #9296
Closed
Tracked by #6822

Add monthly cron to run Cover Server Archival #8278

mekarpeles opened this issue Sep 9, 2023 · 0 comments · Fixed by #9296
Assignees
Labels
Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Cover Service Cover Store (book covers service) Module: Data dumps Priority: 2 Important, as time permits. [managed] State: Blocked Work has stopped, waiting for something (Info, Dependent fix, etc. See comments). [managed]

Comments

@mekarpeles
Copy link
Member

mekarpeles commented Sep 9, 2023

Related to #6822 and #7691

Proposed Recipe

For now, until there's a fully automatic cron and the code is robust enough to be run stand-alone without oversight, here is a way for us to catch-up on our backlog of non-archived covers:

ssh -A ol-covers0
cd /opt/openlibrary

CONTAINER=$(docker ps | grep -oP 'openlibrary_covers_run_\S+')
if [ -z "$CONTAINER" ]; then
    COMPOSE_FILE="compose.yaml:compose.production.yaml" HOSTNAME=$HOSTNAME docker compose run --rm -d covers bash
fi

CONTAINER=$(docker ps | grep -oP 'openlibrary_covers_run_\S+')
# Will reattach to an existing session else create a new one
# Container will start bash, run python, & return to bash on exit
docker exec -it $CONTAINER screen -DR python bash -c 'python && bash'

from openlibrary.coverstore import config
from openlibrary.coverstore.server import load_config
load_config("/olsystem/etc/openlibrary.yml")
load_config("/olsystem/etc/coverstore.yml")
from openlibrary.coverstore import archive
archive.archive()
archive.Batch.process_pending(upload=True, finalize=True, test=False)

Longer-term

We likely want one cron that performs archival, e.g.

from openlibrary.coverstore import config
from openlibrary.coverstore.server import load_config
load_config("/olsystem/etc/openlibrary.yml")
load_config("/olsystem/etc/coverstore.yml")
from openlibrary.coverstore import archive
archive.archive()

And then another cron which runs upload + cleanup (i.e. "finalize")

archive.Batch.process_pending(upload=True, finalize=True, test=False)
@mekarpeles mekarpeles added this to the Sprint 2023-09 milestone Sep 9, 2023
@mekarpeles mekarpeles added Module: Cover Service Cover Store (book covers service) Module: Data dumps Priority: 2 Important, as time permits. [managed] Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] labels Sep 11, 2023
@mekarpeles mekarpeles self-assigned this Sep 11, 2023
@mekarpeles mekarpeles added the State: Blocked Work has stopped, waiting for something (Info, Dependent fix, etc. See comments). [managed] label Oct 2, 2023
@mekarpeles mekarpeles changed the title Update Open Library cron to run Cover Server Archival & Upload Daily Add Open Library monthly cron to run Cover Server Archival Oct 15, 2023
@mekarpeles mekarpeles changed the title Add Open Library monthly cron to run Cover Server Archival Add monthly cron to run Cover Server Archival Oct 15, 2023
mekarpeles added a commit that referenced this issue May 17, 2024
For closing #8278

To be run with:
```
PYTHONPATH=. python /openlibrary/openlibrary/coverstore/archive.py /olsystem/etc/openlibrary.yml /olsystem/etc/coverstore.yml
```
scottbarnes pushed a commit that referenced this issue May 22, 2024
* Prepares cover archival to be run as cron #8278

For closing #8278

To be run with:
```
PYTHONPATH=. python /openlibrary/openlibrary/coverstore/archive.py /olsystem/etc/openlibrary.yml /olsystem/etc/coverstore.yml
```
---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Cover Service Cover Store (book covers service) Module: Data dumps Priority: 2 Important, as time permits. [managed] State: Blocked Work has stopped, waiting for something (Info, Dependent fix, etc. See comments). [managed]
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant