Run cron jobs in Docker #4387

cclauss · 2021-01-08T12:01:49Z

Blocked by: ~~internetarchive/olsystem#110~~
Closes #4383, #4542
Subtask of #4060

We are running on an oldev-based node but we want to run the cronjobs that are in /olsystem/etc/cron.d.

What about Caution on the double run?

https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container

Technical

Testing

Screenshot

Stakeholders

docker-compose.yml

cclauss · 2021-01-28T19:19:30Z

Add ia_db_pw_file for 5 * * * * python $SCRIPTS/openlibrary-server /olsystem/etc/openlibrary.yml runmain openlibrary.solr.process_stats --load-from-db > /var/log/openlibrary/loanstats.log 2>&1

And python $SCRIPTS/manage-imports.py --config /olsystem/etc/openlibrary.yml add-new-scans

docker-compose.production.yml

cclauss · 2021-02-05T12:39:42Z

Added the solrwriter change where both strip_bad_char() and unicodedata.normalize() expect value to be a str.

cclauss · 2021-02-05T18:15:36Z

<html>

<head>
    <title>Apache Tomcat/6.0.39 - Error report</title>
    <style>
        <!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}
        -->
    </style>
</head>

<body>
    <h1>HTTP Status 400 - ERROR: [doc=loans/loan-1944635007] Error adding field
        \'duration_hours_i\'=\'0.0008333333333333334\'</h1>
    <HR size="1" noshade="noshade">
    <p><b>type</b> Status report</p>
    <p><b>message</b> <u>ERROR: [doc=loans/loan-1944635007] Error adding field
            \'duration_hours_i\'=\'0.0008333333333333334\'</u></p>
    <p><b>description</b> <u>The request sent by the client was syntactically incorrect.</u></p>
    <HR size="1" noshade="noshade">
    <h3>Apache Tomcat/6.0.39</h3>
</body>

</html>

They were erroring trying to access db

cdrini

Lgtm! One last change: Can you update our deploy script to make sure it starts this service?

cclauss · 2021-02-11T20:52:38Z

I made that change on #4457 but that is still not landed.

mekarpeles

lgtm

mekarpeles · 2021-02-12T10:42:49Z

I see restart always on the cron and #4457 has landed, going to merge this as well.

cclauss added Affects: Admin/Maintenance Issues relating to support scripts, bots, cron jobs and admin web pages. [managed] Module: Docker Issues related to the configuration or use of Docker. [managed] labels Jan 8, 2021

cclauss added this to the Active Sprint milestone Jan 8, 2021

cclauss requested review from mekarpeles and cdrini January 8, 2021 12:01

cclauss commented Jan 8, 2021

View reviewed changes