Exclude BiT files from the backups by default?

[aryoda]

As part of discussions regarding #1003 the idea came on the table if BiT files should be excluded (by default) from backups:

Consider excluding BiT files from the backups

BiT uses rsync and this is not meant to backup all files of the whole system since open files may cause problems.
but users want to to do this anyhow so ignoring BiT files in backups would be a good idea.

From what I see in the code

backintime/common/config.py

Lines 184 to 189 in bff7aeb

	HOME_FOLDER = os.path.expanduser('~')
	DATA_FOLDER = '.local/share'
	CONFIG_FOLDER = '.config'
	BIT_FOLDER = 'backintime'
	self._DEFAULT_LOCAL_DATA_FOLDER = os.path.join(HOME_FOLDER, DATA_FOLDER, BIT_FOLDER)
	self._LOCAL_CONFIG_FOLDER = os.path.join(HOME_FOLDER, CONFIG_FOLDER, BIT_FOLDER)

the following folders should be excluded too by default (and manually as workaround for older versions of BiT):

• /root/.local/share/backintime
• /<currentuser>/.local/share/backintime

Ignoring specific files kept open by BiT would be even better since ignoring the whole folder means that

• the configs (backup profiles) and
• the user-callback script

are not contained in the backup 😲

This would require to add a positive list of all file names (or extensions) that are uses in the config.py (e.g. worker*.* files).

The default config can be changed here in the source code:

backintime/common/config.py

Lines 132 to 135 in bff7aeb

	DEFAULT_EXCLUDE = [ '.gvfs', '.cache/*', '.thumbnails*',
	'.local/share/[Tt]rash*', '*.backup*', '*~', '.dropbox*', '/proc/*',
	'/sys/*', '/dev/*', '/run/*', '/etc/mtab', '/var/cache/apt/archives/*.deb',
	'lost+found/*', '/tmp/*', '/var/tmp/*', '/var/backups/*', '.Private' ]

[Germar]

BiT will backup the configuration in an extra step even if it is not included

backintime/common/snapshots.py

Line 905 in 110d82a

def backupConfig(self, sid):

[aryoda]

TODO: We should test if the existing config file is overwritten if it is contained in the snapshot and the home folder is restored

[Hitro147]

TL;DR: Configuration File Overwrite Behavior in Back In Time

• Issue: Post-snapshot modifications to the .config/backintime/config file are overwritten upon restoring from a snapshot.
• Test: Conducted a test involving a snapshot, modification of the config file, and a subsequent restore operation.
• Implications: Important user configurations made post-snapshot can be lost without warning.
• Recommendations: Implement user warnings, allow selective file restoration, and enhance documentation to address this issue.

---

Machine Configuration and Test Procedure

Machine Configuration

• Operating System: Ubuntu 22.04.3 LTS
• Back In Time Version: 1.4.2-dev
• Docker Version: Docker version 24.0.6, build ed223bc
• FileSystem Layout: Using Docker volumes for /root/snapshots
• Additional Notes: Running Back In Time without GUI; all operations performed via command line

Detailed Steps

Step 1: Initial Backup Creation

• Environment setup and verification of snapshot creation.
• Command: /backintime/common/backintime --config /root/.config/backintime/config --profile-id 1 backup

Step 2: Post-Snapshot Modification

• Modified /root/.config/backintime/config by adding a comment # Modified for Issue #1382 testing and altering a configuration setting (Changed profile1.snapshots.automatic_backup_mode from 0 to 1).
• Documented changes for comparison.

Step 3: Restoration Process

• Initiated restore from the specific snapshot, observing the overwrite of post-snapshot modifications.
• Command: /backintime/common/backintime --config /root/.config/backintime/config --profile-id 1 restore

Observations

Upon restoration, the .config/backintime/config file reverted to its snapshot state, indicating that modifications made after the snapshot were overwritten.

Implications

This behavior poses a risk of unintentional data loss, especially for users who might not expect configuration changes to be overwritten during restoration.

Recommendations

1. User Warnings: Introduce warnings or prompts during the restoration process to alert users to potential overwrites.
2. Selective Restoration: Develop options to selectively restore or exclude files, providing users more control.
3. Documentation Enhancements: Clearly describe overwrite behavior in documentation, offering guidelines to protect important changes.

Conclusion

The test confirms that Back In Time's current restoration process overwrites files with their snapshot versions, specifically the configuration file. Addressing this through improved warnings, selective restoration options, and better documentation can enhance the tool's usability and reliability.

[emtiu]

This actually fits very well here. We should check whether we consider this a bug.

[buhtz]

That was what I had in mind. It is related to the current Issue but an Issue of its own.

Hitro147: Do I understand it correct, that the config file was not an entry in the regular "Include" list of that backup? But it was included by default because of the bahvior described by Germar? I am asking because I am not able to reproduce this behavior here.

Germar do you remember why this behavior (restoring config files) was introduced?

[Hitro147]

The config file was indeed included in the snapshot by default behavior. It was not explicitly listed in the regular "Include" list for the backup profile. The inclusion happened as part of the default behavior described by Germar, where Back In Time includes the entire home directory (/root in this case) by default, which contains the .config/backintime/ directory and its contents, including the config file.

Perhaps ensuring that the entire home directory is included by default in your test setup might help replicate the behavior?

[emtiu]

The config file was indeed included in the snapshot by default behavior. It was not explicitly listed in the regular "Include" list for the backup profile. The inclusion happened as part of the default behavior described by Germar, where Back In Time includes the entire home directory (/root in this case) by default, which contains the .config/backintime/ directory and its contents, including the config file.

That's not what Germar was talking about, though. He is referring to a functionality that includes the config even if $HOME/.config/backintime is not part of the included paths. If you inspect your snapshot directories, you will find this file like host/user/profile/20230131-230001-815/config, written by this function that Germar is referring to:

backintime/common/snapshots.py

Line 905 in 110d82a

def backupConfig(self, sid):

As I see it, we have established two useful facts:

1. BackInTime's config is stored with any snapshot, as in host/user/profile/20230131-230001-815/config. This is useful and poses no problem.
2. If the home directory is part of a snapshot, and it is restored with default settings, this will overwrite $HOME/.config/backintime, potentially causing trouble (as tested by @Hitro147). This is something we might want to address.

(Side note: Might the config also live somewhere other than $HOME/.config/backintime? If yes, we need to account for that.)

How might we proceed?

1. Warning the user of the consequence of overwriting the config location: probably a bit tricky to code, also the user would be presented with this at a stressful time (=when restoring $HOME). Also, should the user then be presented with any choices—and if yes, which? Not a great solution.
2. Hardcoding an exclude for the config location: not cool, potentially unexpected behavior.
3. Including the config location with the default excludes, manageable by the user: minimally invasive, easy to understand, probably the most elegant solution.

I prefer 3. Which is what @aryoda has been proposing right from the start ;)

[Hitro147]

Oops! My bad. I totally got the wrong idea. Thanks for clearing that up and helping me see it right 😄

[buhtz]

Thanks emtiu for your summarize. My apologize I was on the total wrong track here.

1. Warning the user of the consequence of overwriting the config location: probably a bit tricky to code, also the user would be presented with this at a stressful time (=when restoring $HOME). Also, should the user then be presented with any choices—and if yes, which? Not a great solution.
2. Hardcoding an exclude for the config location: not cool, potentially unexpected behavior.
3. Including the config location with the default excludes, manageable by the user: minimally invasive, easy to understand, probably the most elegant solution.

I would like to add a number 4 proposal similar to number 3.

Don't add the config location to the default exclude. But when the users add new Include entries to the list, BIT should detect if its own config file is involved. In that case the user should get a question-dialog-box with an explanation (BIT config files are included), a warning (overwritten on restore) and a decision (add config files to Exclude list? Yes/No). The dialog should show up directly when the "Add Include" was done.
To my understanding this should be easy to implement.

[emtiu]

Don't add the config location to the default exclude. But when the users add new Include entries to the list, BIT should detect if its own config file is involved. In that case the user should get a question-dialog-box with an explanation (BIT config files are included), a warning (overwritten on restore) and a decision (add config files to Exclude list? Yes/No). The dialog should show up directly when the "Add Include" was done.

I like that.

The dialog box could basically say: „Attention: You're including BiT's own config in the snapshots. Please be aware: Restoring from these snapshots to your running system later might overwrite BiT's configuration with an older state, leading to unexpected behavior. If you know what you're doing, you may continue. Otherwise, BiT can add the location of its own configuration from the snapshots for you. You can always adjust this behavior through the Include/Exclude lists.“

Two buttons: „Snapshots should include BiT config“ and „Exclude BiT config from snapshots“.

The wording is not great yet: we must be very carfeul to make it understandable and not too complicated. I'll be happy to do later—writing understandable words about complicated things is my profession ;)