Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backup: update rsnapshot configuration #2637

Merged
merged 2 commits into from
May 4, 2021
Merged

Conversation

richardlau
Copy link
Member

@richardlau richardlau commented Apr 29, 2021

The first commit updates the copy of rsnapshot.conf we have in this repository with what is currently deployed on the backup server, taken via:

ssh backup cat /opt/local/etc/rsnapshot.conf > rsnapshot.conf

(FWIW it looks like we no longer take mysql dumps from the benchmark machine -- I think this was part of #2485 with the de-chartering of the benchmarking WG (nodejs/TSC#955).)

The second commit is a proposed change (not yet made) to also backup /etc from both CI servers. This will include nginx and telegraf (for grafana) configuration. Currently the whole of /etc is around 7mb on each server. I've left the existing Jenkins backup where it is to not disturb the diffs to the existing backups since those are large. We would end up with a directory structure for each backup (e.g. under /backup/periodic/daily.0):

directory backup of
ci.nodejs.org existing /var/lib/jenkins from ci.nodejs.org
ci-release.nodejs.org existing /var/lib/jenkins from ci-release.nodejs.org
etc/ci.nodejs.org /etc from ci.nodejs.org
etc/ci-release.nodejs.org /etc from ci-release.nodejs.org
etc/nodejs.org /etc from (direct.)nodejs.org

cc @nodejs/build-infra

Copy link
Member

@jbergstroem jbergstroem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rvagg
Copy link
Member

rvagg commented Apr 30, 2021

can we add /etc/ off nodejs.org too? way more useful stuff going on in there than even the CI machines that we'd be sad to lose .. mostly we'd cross our fingers that we've kept ansible in sync if we had to restore that but I don't have great faith in that.

@richardlau
Copy link
Member Author

Sure, that'd be /etc from direct.nodejs.org, right? I need to get a few hours sleep first but I'll add that later today. Something like:

backup root@direct.nodejs.org:/etc/ etc/nodejs.org/

(I'll sleep first because the file is tab sensitive so I'd rather do that in front of a proper computer with a proper editor rather than try to get it right on a tablet through the GitHub web interface.)

@richardlau
Copy link
Member Author

Added a backup for /etc/ for direct.node.org.

Sync `rsnapshot.conf` with what is currently running on the backup
server:
`ssh backup cat /opt/local/etc/rsnapshot.conf > rsnapshot.conf`
Adds backup of `/etc/` to `etc/<server>/`. The existing backups of
`/var/lib/jenkins/` are left in `<server>/` to avoid creating new
inodes for unchanged files/directories.
@richardlau
Copy link
Member Author

I've deployed the changes now and they took effect for the most recent backup.

before (previous backup)

[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]# ls -al /backup/periodic/daily.1/
total 130
drwxr-xr-x  6 root root    6 Apr 30 00:19 .
drwxr-xr-x 20 root root   20 May  1 00:32 ..
drwxr-xr-x 24 root root   81 Apr 29 06:52 ci-release.nodejs.org
drwxr-xr-x 25 root root  154 Apr 30 00:07 ci.nodejs.org
drwxr-xr-x  2 root root    4 Apr 30 00:19 iptables
drwxr-xr-x  9 root root 1956 Apr 29 06:43 www-logs
[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]#

after

[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]# ls -al /backup/periodic/daily.0/
total 131
drwxr-xr-x  7 root root    7 May  1 00:32 .
drwxr-xr-x 20 root root   20 May  1 00:32 ..
drwxr-xr-x 24 root root   81 Apr 30 06:04 ci-release.nodejs.org
drwxr-xr-x 25 root root  154 May  1 00:10 ci.nodejs.org
drwxr-xr-x  5 root root    5 May  1 00:31 etc
drwxr-xr-x  2 root root    4 May  1 00:32 iptables
drwxr-xr-x  9 root root 1957 Apr 30 06:45 www-logs
[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]# ls -al /backup/periodic/daily.0/etc/
total 28
drwxr-xr-x   5 root root   5 May  1 00:31 .
drwxr-xr-x   7 root root   7 May  1 00:32 ..
drwxr-xr-x  98 root root 190 Apr 29 06:46 ci-release.nodejs.org
drwxr-xr-x 108 root root 198 Apr 21 04:03 ci.nodejs.org
drwxr-xr-x 108 root root 204 Apr  8 05:17 nodejs.org
[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]# du -hs /backup/periodic/daily.0/etc/*
3.4M    /backup/periodic/daily.0/etc/ci-release.nodejs.org
3.9M    /backup/periodic/daily.0/etc/ci.nodejs.org
5.0M    /backup/periodic/daily.0/etc/nodejs.org
[root@3a355104-c5d6-405f-863b-9ce5948ba77b ~]#

Copy link
Member

@mhdawson mhdawson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@richardlau richardlau merged commit 85abedf into nodejs:master May 4, 2021
@richardlau richardlau deleted the backup branch May 4, 2021 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants