updating_inventory.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU","gpuClass":"standard"},"cells":[{"cell_type":"markdown","source":["# Updating the Inventory\n","---\n","This notebook will provide the code necessary to perform an update to the Biodata Resource Inventory, using trained models.\n","\n","The steps include:\n","* Run new query on EuropePMC\n","* Classify new articles\n","* Run NER to get resource names for predicted positives\n","* Get URLs for predicted positives\n","* Gather other metadata\n","\n","\n","\n","# Setup\n","---\n","### Mount Drive\n","\n","First, mount Google Drive to have access to files necessary for the run:\n"],"metadata":{"id":"x4whPVjZZa7x"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"BmwESzXcjXTb","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1675109206770,"user_tz":420,"elapsed":1702,"user":{"displayName":"Kenneth Schackart","userId":"14619721059788161882"}},"outputId":"c938f5b9-9033-4ebd-c8a4-d7cc008390f0"},"outputs":[{"output_type":"stream","name":"stdout","text":["Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n","/content/drive/MyDrive/GitHub/inventory_2022\n"]}],"source":["from google.colab import drive\n","drive.mount('/content/drive')\n","%cd /content/drive/MyDrive/GitHub/inventory_2022/"]},{"cell_type":"markdown","source":["Run the make target to install Python dependencies.\n","\n","You may see the error: `ERROR: pip's dependency resolver does not currently take into account all the packages that are installed`, but the code should run regardless.\n","\n"],"metadata":{"id":"6a7pMnIVbKXE"}},{"cell_type":"code","source":["! make setup_for_updating"],"metadata":{"id":"iBMUW3C0YIz4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["If you need to download the model checkpoints for the best classifier and NER models, run the cell below. If the `train_and_predict` pipeline was run, then the models are already present."],"metadata":{"id":"UleWUNAcqoL-"}},{"cell_type":"code","source":["# Get trained article classification model\n","# Create output directory\n","! mkdir -p out/classif_train_out/best\n","# Print name of model to the necessary file\n","! CLASSIFIER=\"out/classif_train_out/article_classifier.pt\"\n","! echo $CLASSIFIER > out/classif_train_out/best/best_checkpt.txt\n","# Download the model\n","! wget -O $CLASSIFIER https://huggingface.co/globalbiodata/inventory/resolve/main/article_classifier.pt\n","# Check that it downloaded properly\n","! echo \"5718a7f70becacb46d46501734c83aab81c86feec563594f6a25c116aa31b521 $CLASSIFIER\" | sha256sum -c\n","\n","# Get trained NER model\n","! mkdir -p out/ner_train_out/best\n","! NER=\"out/ner_train_out/named_entity_recognition.pt\"\n","! echo $NER > out/ner_train_out/best/best_checkpt.txt\n","! wget -O $NER https://huggingface.co/globalbiodata/inventory/resolve/main/named_entity_recognition.pt\n","! echo \"dc0bc8b4929e33da52bc92e12720260b392421883889e0a36c809cb0b5c40f5d $NER\" | sha256sum -c"],"metadata":{"id":"RcQ3mQhiqoi-"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Setting up Configurations\n","\n","Before running the automated pipelines, first update the configuration file `config/update_inventory.yml`. It can be accessed in Google Drive, though you may need to download it and edit it in a text editor such as Notepad, then reupload it.\n","\n","* **Europe PMC query publication date range**: These are stored as variables `query_from_date` and `query_to_date` in that file. Note that the dates are inclusive. For example to get papers published in 2022, both of those varibles should be 2022.\n","* **Previous inventory file**: During strict deduplication and flagging for manual review, the results of the previous inventory are taken into account. Specify the location of the most recent inventory output file in the variable `previous_inventory`."],"metadata":{"id":"CFB8BHYk8AwS"}},{"cell_type":"markdown","source":["# Running the pipeline\n","---\n","Now, we are ready to run the pipeline\n","\n","## Run it\n","\n","The following cell will run the pipeline described above. It may take a while, but GPU will speed it up a lot."],"metadata":{"id":"XG8imhT0bms7"}},{"cell_type":"code","source":["! make update_inventory"],"metadata":{"id":"zFSmOvuUnSPE"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Selective Manual Review\n","\n","After running the initial pipeline, the inventory has been flagged for selective manual review.\n","\n","The file to be reviewed is located at:\n","\n","`out/new_query/for_manual_review/predictions.csv`\n","\n","Review the flagged columns according to the instruction sheet ([doi: 10.5281/zenodo.7768363](https://doi.org/10.5281/zenodo.7768363)), then place the manually reviewed file in the following folder:\n","\n","`out/new_query/manually_reviewed/`\n","\n","The file must still be named `predictions.csv`\n","\n","# Processing Manual Review\n","\n","Next, further processing is performed on the manually reviewed inventory."],"metadata":{"id":"mdwb9NveMdP0"}},{"cell_type":"code","source":["! make process_manually_reviewed_update"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"z_c4ZbgONIoo","executionInfo":{"status":"ok","timestamp":1675109651846,"user_tz":420,"elapsed":42670,"user":{"displayName":"Kenneth Schackart","userId":"14619721059788161882"}},"outputId":"5c69eb01-f62f-4df6-cd20-cd7512d0bba6"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["snakemake \\\n","-s snakemake/update_inventory.smk \\\n","--configfile config/update_inventory.yml \\\n","-c 1 \\\n","--until process_countries\n","\u001b[33mBuilding DAG of jobs...\u001b[0m\n","\u001b[33mUsing shell: /usr/bin/bash\u001b[0m\n","\u001b[33mProvided cores: 1 (use --cores to define parallelism)\u001b[0m\n","\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n","\u001b[33mJob stats:\n","job                  count    min threads    max threads\n","-----------------  -------  -------------  -------------\n","check_urls               1              1              1\n","get_epmc_meta            1              1              1\n","process_countries        1              1              1\n","total                    3              1              1\n","\u001b[0m\n","\u001b[33mSelect jobs to execute...\u001b[0m\n","\u001b[32m\u001b[0m\n","\u001b[32m[Mon Jan 30 20:13:29 2023]\u001b[0m\n","\u001b[32mrule check_urls:\n","    input: out/new_query/processed_manual_review/predictions.csv\n","    output: out/new_query/url_checking/predictions.csv\n","    jobid: 11\n","    reason: Missing output files: out/new_query/url_checking/predictions.csv\n","    resources: tmpdir=/tmp\u001b[0m\n","\u001b[32m\u001b[0m\n","/usr/local/lib/python3.8/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.1\n","  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n","Done. Wrote output to out/new_query/url_checking/predictions.csv.\n","\u001b[32m[Mon Jan 30 20:13:59 2023]\u001b[0m\n","\u001b[32mFinished job 11.\u001b[0m\n","\u001b[32m1 of 3 steps (33%) done\u001b[0m\n","\u001b[33mSelect jobs to execute...\u001b[0m\n","\u001b[32m\u001b[0m\n","\u001b[32m[Mon Jan 30 20:13:59 2023]\u001b[0m\n","\u001b[32mrule get_epmc_meta:\n","    input: out/new_query/url_checking/predictions.csv\n","    output: out/new_query/epmc_meta/predictions.csv\n","    jobid: 10\n","    reason: Missing output files: out/new_query/epmc_meta/predictions.csv; Input files updated by another job: out/new_query/url_checking/predictions.csv\n","    resources: tmpdir=/tmp\u001b[0m\n","\u001b[32m\u001b[0m\n","/usr/local/lib/python3.8/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.1\n","  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n","Done. Wrote output to out/new_query/epmc_meta/predictions.csv.\n","\u001b[32m[Mon Jan 30 20:14:06 2023]\u001b[0m\n","\u001b[32mFinished job 10.\u001b[0m\n","\u001b[32m2 of 3 steps (67%) done\u001b[0m\n","\u001b[33mSelect jobs to execute...\u001b[0m\n","\u001b[32m\u001b[0m\n","\u001b[32m[Mon Jan 30 20:14:06 2023]\u001b[0m\n","\u001b[32mrule process_countries:\n","    input: out/new_query/epmc_meta/predictions.csv\n","    output: out/new_query/processed_countries/predictions.csv\n","    jobid: 9\n","    reason: Missing output files: out/new_query/processed_countries/predictions.csv; Input files updated by another job: out/new_query/epmc_meta/predictions.csv\n","    resources: tmpdir=/tmp\u001b[0m\n","\u001b[32m\u001b[0m\n","/usr/local/lib/python3.8/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.1\n","  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n","Done. Wrote output to out/new_query/processed_countries/predictions.csv.\n","\u001b[32m[Mon Jan 30 20:14:10 2023]\u001b[0m\n","\u001b[32mFinished job 9.\u001b[0m\n","\u001b[32m3 of 3 steps (100%) done\u001b[0m\n","\u001b[33mComplete log: .snakemake/log/2023-01-30T201328.761519.snakemake.log\u001b[0m\n"]}]},{"cell_type":"markdown","source":["## Final inventory\n","\n","The final inventory, including names, URLS, and metadata is found in the file:\n","*    `out/new_query/processed_countries/predictions.csv`"],"metadata":{"id":"AV4p2VA_NUfi"}}]}