Add script to download papertrail logs

[jjnesbitt]

This script aids in the download of papertrail logs, as past 2 weeks, we can't easily search papertrail from the UI, and must download the hourly logs. To aid in this, this script will concatenate the logs into one file that can be easily searched.

[yarikoptic]

FWIW: oh cool! as I have mentioned before, since sometime in 2021 I was downloading them on drogon via this bash script

(base) dandi@drogon:/mnt/backup/dandi/papertrail-logs$ cat fetch.sh 
#!/bin/bash

set -eu

for a in dandi-api dandi-api-staging; do
	echo "Working on $a"
	source .git/$a-secrets.env
	mkdir -p $a
	rm -f $a/archives.json
	curl -sH "X-Papertrail-Token: $PAPERTRAIL_KEY" https://papertrailapp.com/api/v1/archives.json | jq . >| $a/archives.json
	grep filename $a/archives.json | sed -e 's,.*: "\(\(.*\).tsv.*\)".*,\1 \2,g' | while read filename datestamp; do
		d=$a/$(echo $datestamp | sed -e 's,-,/,g' -e 's,/[^/]*$,,g')
		f="$d/${filename//.gz/}"
		if [ ! -e "$f" ]; then
			echo -n "  Fetching archive for $datestamp"
			mkdir -p $d
			curl -L -sH "X-Papertrail-Token: $PAPERTRAIL_KEY" "https://papertrailapp.com/api/v1/archives/$datestamp/download" | gunzip > "$f"
			echo -e "\r  Fetched archive for $datestamp: $(wc -l $f)"	
		fi
	done
done

I will check later if it is the same logs . Meanwhile -- let's merge the PR??

[jjnesbitt]

FWIW: oh cool! as I have mentioned before, since sometime in 2021 I was downloading them on drogon via this bash script

(base) dandi@drogon:/mnt/backup/dandi/papertrail-logs$ cat fetch.sh 
#!/bin/bash

set -eu

for a in dandi-api dandi-api-staging; do
	echo "Working on $a"
	source .git/$a-secrets.env
	mkdir -p $a
	rm -f $a/archives.json
	curl -sH "X-Papertrail-Token: $PAPERTRAIL_KEY" https://papertrailapp.com/api/v1/archives.json | jq . >| $a/archives.json
	grep filename $a/archives.json | sed -e 's,.*: "\(\(.*\).tsv.*\)".*,\1 \2,g' | while read filename datestamp; do
		d=$a/$(echo $datestamp | sed -e 's,-,/,g' -e 's,/[^/]*$,,g')
		f="$d/${filename//.gz/}"
		if [ ! -e "$f" ]; then
			echo -n "  Fetching archive for $datestamp"
			mkdir -p $d
			curl -L -sH "X-Papertrail-Token: $PAPERTRAIL_KEY" "https://papertrailapp.com/api/v1/archives/$datestamp/download" | gunzip > "$f"
			echo -e "\r  Fetched archive for $datestamp: $(wc -l $f)"	
		fi
	done
done

I will check later if it is the same logs . Meanwhile -- let's merge the PR??

Sounds good to me!

[dandibot]

🚀 PR was released in v0.3.19 🚀