Instead of scraping directly from mnonboard, write sync_content.sh file

[iannesbitt]

For repositories that contain many records, it's necessary to run nohup ./sync_content.sh & so that an inadvertent ssh hangup does not cancel the scrape. Therefore, mnonboard should write the sync_content.sh file rather than run scrapy directly.

sync_content.sh should look like this, where the only dynamic content is the node name in line 3:

#!/bin/bash

NODE="mnTestDVNO"

HOME_DIR="/home/mnlite"
MNLITE_DIR="${HOME_DIR}/WORK/mnlite"
NODE_DIR="${MNLITE_DIR}/instance/nodes/${NODE}"

ENV_DIR="${HOME_DIR}/.virtualenvs/mnlite"
LOG_DIR="/var/log/mnlite"

cd "${MNLITE_DIR}"
source "${ENV_DIR}/bin/activate"
LOG_FILE="${LOG_DIR}/${NODE}-crawl.log"
logger "Start crawl on: ${NODE} logfile: ${LOG_FILE}"
scrapy crawl --logfile=${LOG_FILE} JsonldSpider -s STORE_PATH=${NODE_DIR}
logger "End crawl on ${NODE}"