Object lock support for repositories on S3/B2/GCS

[kapitainsky]

It is only idea at this stage but I think rustic already has some building blocks available to at least simplify such repository.

I have used it in kopia - https://kopia.io/docs/advanced/ransomware-protection/#even-more-protection - and got overall concept from there.

I can achieve it in fully "manual" mode today with rustic:

1. configure S3 bucket with object lock and default retention in COMPLIANCE mode set to N number of days - it implies that versioning is enabled on this bucket too
2. configure S3 bucket life cycle rule to delete any old versions older than N days
3. Use this bucket for rustic backups - business as usual
4. At least every (N-1) days run script extending all current objects' lock time:

for this I use rclone to list all objects and aws cli to do the job:

rclone lsf S3remote:bucket_name -R --files-only | xargs -P 16 -n 1 extend_lock.sh

where extend_lock.sh executes aws cli command:

aws s3api put-object-retention --endpoint-url url --bucket bucket_name --key S1 --retention '{ "Mode": "COMPLIANCE", "RetainUntilDate":"now+N days" }'

5. In order to restore from such repo in case of e.g. ransomware attack I use rclone again:

rclone mount S3remote:bucket_name /mount/point  --s3-version-at=`DATE`

it mounts read only bucket as it was at some past DATE - I can use now rustic to just point to this repo and restore what I need.

BTW this is actually one of the reason I started looking at rustic. restic with its need to have writable locks directory is a bit more complicated to use here.

=======

Now point (5) would be rather hard to implement at the moment without proper S3 API - let's forget it. There is rclone and this is something user expects never to use - or once in a lifetime, so can be a bit "manual". I think rclone way is fully acceptable.

Points (1...3) - are configuration to be documented how to run such protected repo.

But rustic could help with point (4). In similar way like it today invokes cold repo restore command it could have option to invoke lock extension command (for S3, B2, GCS remotes etc.) when run e.g. as rustic extendlock or maybe generic hook for actions on all repo objects rustic action --all.

This command would pass every repo object to toml defined command - ideally in multithread fashion as e.g. aws cli is slow (this is why I use xargs -P 16 in my manual way)

I see questions about ransomware protection popping up from time to time on various forums and some rustic support here would make it easier for people to actually implement it.

[aawsome]

About 4.) rustic has already added a creation date for each pack file to the index. This is also used when marking the pack for deletion. Actually I think that for something like extendable object lock, we should use a similar mechanism to determine which packs need to get a lock update. In fact, I wouldn't say that we need to update the lock for all packs, but depending on a "lock retention" policy could also figure out which packs are still locked, which need a lock extenstion and which may be open for removal
Note that there is already the option -keep-delete for prune which allows to not touch packs until they reach the given age.

I have to think about what we could do with index files. It might be Ok, though, to not lock them as they anyway only contain redundant data which can always be recreated by the repair index command from pack files.

For snapshot files, we could use the snapshot date and do similar things in forget.

So, in summary: Yes I think rustic can add lots of support in this case! It is a bit of work and redesign in the backend trait and forget/ prune, but that's all! Let's put it onto the issue list and I see when I or someone else finds time to add these features.

About 5.) I think rclone also supports this with the serve command. So, I think using

[repository.options]
rclone_command = "rclone serve restic --s3-version-at=`DATE` --addr localhost:0" 

in your restore-config should also work.
Native support using opendal needs more support from opendal. But it may be available soon, see apache/opendal#2611

[kapitainsky]

We should not use any date based logic to determine which lock to extend - it is counterproductive to the whole idea of lock protected repo. When you run lock extension you want to make sure that ALL files are from this moment protected for next N days.

What makes only sense is to skip files which are not needed. What you want is situation that from a moment of time lock extension is run all repo is protected for the next N days. In ideal situation it is something you run daily - or maybe for long lock (like 1 year) weekly.

[kapitainsky]

Yes I think rustic can add lots of support in this case! It is a bit of work and redesign in the backend trait and forget/ prune, but that's all!

hmmm, IMO it does not require any redesign. In general some repos are better candidates for lock protection than others. If you have tones of data changing daily you can still do it but of course it will come with storage cost attached. It is user decision to use it or not.

[kapitainsky]

About 5.) I think rclone also supports this with the serve command. So, I think using

[repository.options]
rclone_command = "rclone serve restic --s3-version-at=`DATE` --addr localhost:0" 

yes it would be another way - still it requires rclone. But as I explained I do not think it is any problem IMO. It is pure disaster recovery - as long as documented even if requires extra hoops it is perfectly OK.

[kapitainsky]

I posted this idea to see what others are thinking - myself I think it would be interesting and very useful thing to have. As I explained I can achieve it today without any rustic support - but if backup software can cooperate it makes it much easier to create easy solution for more people to use.

When I have lock protected repo I run backup daily - if lock is for 100 days I will have 100 daily backups. There is no need to prune anything earlier - it would be almost against all idea:)

[kapitainsky]

I would say let it sink in:) There are many bright people out there - hopefully some will share their thought on this subject:)

There is definitely requirement for lock protected repos - maybe niche and a bit specialised but very solid.

This is also something what can differentiate rustic compared to "others". Catering for a bit "pro" needs. I call it "pro" because based on some forums discussions I am sure a lot of people do not realise how strong COMPLIANCE lock protection is.
Only way to cancel it is to terminate your S3 account contract. There is no other way to delete such protected data. If enabled without care for huge amount of data for very long time it can be disastrous. But at the same time it is mandatory at least in some industries - to show that your data is effectively indestructible for amount of time required by regulator, hence storage provider offering such functionality.

[aawsome]

I thought a bit about this proposal and would like to propose that we implement a rustic lock command:

• The command would be called with a duration until when the lock should be done and a selection of snapshots whose data shall be locked - analog e.g. to the rustic snapshots command. So it could be rustic lock --duration 30d a736s, rustic --duration 2d lock (lock all snapshots) or rustic --duration 1y --filter-label "my_important_snapshots".
• To actually lock the needed files, we must extend the internal backend traits with a lock method. This could be implemented directly by the backend (e.g. using opendals blocking as soon as it is implemented) or by an adapter which e.g. calls a given script (to be specified in the config file) to do the actual locking.
• Locked repository files would be the snapshots files from the snapshot selection and all the pack files which are needed for all those snapshots. (index files won't be locked as they can be re-generated using rustic index repair in case they are lost).
• To let rustic know that snapshots and pack files are locked, I would save the actual locked-until-date in the snapshot and in the index for the pack files. Using that information we can assure that forget and prune wouldn't even try to remove/repack data which in fact is covered by a lock
• Lock extension would be a simple rustic lock --duration <new-duration> for some snapshots which are already locked. rustic can even detect if some locks are already long enough in place using the locked-until date (we could also add an option --always-lock to always call the locking on the backend). Besides calling the lock on the backend, the index locked-until entries can be just updated. For snapshots it is a bit more involved. The actual snapshot file could be still locked while we want to add a longer lock. In this case, the snapshot would be doubled (with different locked-until) and we need to handle how we want to display these doubled snapshots e.g. in the snapshots command (e.g. only display the one with the latest locked-until). But in the end, once we reach some locked-until date of former snapshotfiles, they can be simply deleted using forget (and the more recent duplicate(s) still exist).

UPDATE: we can of course also use something like rustic lock --until <DATE> or even offer both possibilities. In the end, the actual backend locking call and the saved locked-until information will be always a date.

UPDATE2: If the backend supports it, we can also add the possibility to do a rustic lock --until forever

[kapitainsky]

Definitely will play with it. Thank you!

But how I am supposed to use it?

My S3 compatible storage command to extend lock is:

aws s3api put-object-retention --bucket test-bucket-lock --key test.file --retention '{ "Mode": "COMPLIANCE", "RetainUntilDate":"2024-02-06T14:00:00+00:00" }'

I guess it should be put somewhere in restic.toml file? And rustic will use it substituting key and RetainUntilDate values?

[aawsome]

Exactly - I definitively will add support for something like this (syntax is not fixed yet):

[repository.options]
lock-command = "aws s3api put-object-retention --bucket test-bucket-lock --key %repofile --retention '{ \"Mode\": \"COMPLIANCE\", \"RetainUntilDate\":\"%date\" }'"

(or you put everything in a script and call lock-command = "extend_lock.sh %repofile %date")

I also do hope that opendal once directly supports it.

[kapitainsky]

From my experience aws cli is real slow. It is written in Python I think and takes few 100 ms to initialise its execution.

When I do it manually I run it in multithread mode - I achieve it by:

list_all_files.sh| xargs -P 16 -n 1 extend_lock.sh

Is rustic also using multithreading here? For any larger repo doing it sequentially would be painfully slow otherwise.

[aawsome]

Yes, rustic does spawn 20 threads which call the internal API in parallel. With calling a command that would mean that 20 commands are called concurrently.
We have the same issue with warming up where you can also provide a command. The 20 is currently fixed for both warm-up and locking (but may be customizable in future)

[kapitainsky]

this is real good news that it uses multithreading. IMO it should definitely be configurable as some of external tools like aws cli can be real memory hungry and 20 processes might be too much for less powerful systems (raspberry pi for example). At the same time sometime you can run it on 1GiB internet Cx with 64 cores CPU - then 20 seems like far too little:)

I will definitely test the feature you implemented. It will take time due to the nature of locking (days not hours possible only) and my way of testing:) I try to make things difficult and simulate various scenarios - especially around how I can restore my locked protected data when shit hits the fan. As at the end it is the whole purpose of this feature.

[kapitainsky]

I managed to compile #1066 and have new lock option available

$ ./rustic --help
rustic - fast, encrypted, deduplicated backups powered by Rust

Usage: rustic [OPTIONS] <COMMAND>

Commands:
....
  lock         Lock snapshots
...

However this rustic version fails when trying to run simple backup:

$ ./rustic backup --use-profile rustic --tag test_rustic
...
[00:00:00] backing up...                  █████████████████████░░░░░░░░░░░░░░░░░░░ 326.07 MiB/592.87 MiB 1.76 GiB/s   (ETA 0s)                                                              The application panicked (crashed).
Message:  misaligned pointer dereference: address must be a multiple of 0x8 but is 0x7fb16201f40d
Location: /Users/kptsky/.cargo/registry/src/index.crates.io-6f17d22bba15001f/nix-0.27.1/src/unistd.rs:3794

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
thread caused non-unwinding panic. aborting.

It works perfectly fine with release version.

[kapitainsky]

I have tried many times - it always happens. As fix is already on the way I will just wait a moment.

[aawsome]

Ah - just realized that it is macOS-specific. That explains why I never got the error...

[kapitainsky]

I did backup with release version and now back to locking part - it works.

I have lock command in toml:

[repository.options]
root = "/"
bucket = "test-dal-lock"
endpoint = "https://p7v1.ldn.idrivee2-40.com"
region = "gb-ldn"
access_key_id = "XXX"
secret_access_key = "XXX"
lock-command = "aws s3api put-object-retention --endpoint-url https://p7v1.ldn.idrivee2-40.com --bucket test-dal-lock --key %repofile --retention '{ \"Mode\": \"COMPLIANCE\", \"RetainUntilDate\":\"%date\" }'"

All run without any errors:

rustic lock repository --duration 1d
[INFO] using config ./rustic.toml
[INFO] repository opendal:test-dal-lock: password is correct.
[INFO] using cache at /Users/kptsky/Library/Caches/rustic/e48130ed29692fcfc60d586c479f84cfa4bf6dd92ed553ebe6a4f3b0e5a3d41e
[00:00:00] listing Key files..
[00:00:00] locking Key files..            ████████████████████████████████████████          1/1
[00:00:00] listing Snapshot files..
[00:00:00] locking Snapshot files..       ████████████████████████████████████████          2/2
[00:00:00] listing Index files..
[00:00:00] locking Index files..          ████████████████████████████████████████          3/3
[00:00:00] listing Pack files..
[00:00:00] locking Pack files..           ████████████████████████████████████████         19/19

but no lock is set, e.g.:

$ aws s3api  get-object-retention --endpoint-url https://p7v1.ldn.idrivee2-40.com --bucket test-dal-lock --key data/09/09b46c93055c80dc2c0c76f20ed542a3ee3ecb9fa9405b5a75280c397a390dcf

An error occurred (NoSuchObjectLockConfiguration) when calling the GetObjectRetention operation: The specified object does not have a ObjectLock configuration

Other think I noticed is that even when lock-command is missing from toml file it runs without any errors...

When I do locking manually it works - so bucket has locking enabled:

$ aws s3api put-object-retention --endpoint-url https://p7v1.ldn.idrivee2-40.com --bucket test-dal-lock --key data/09/09b46c93055c80dc2c0c76f20ed542a3ee3ecb9fa9405b5a75280c397a390dcf --retention '{ "Mode": "COMPLIANCE", "RetainUntilDate":"2024-02-21T12:00:00+00:00" }'

$ aws s3api get-object-retention --endpoint-url https://p7v1.ldn.idrivee2-40.com --bucket test-dal-lock --key data/09/09b46c93055c80dc2c0c76f20ed542a3ee3ecb9fa9405b5a75280c397a390dcf
{
    "Retention": {
        "Mode": "COMPLIANCE",
        "RetainUntilDate": "2024-02-21T12:00:00+00:00"
    }
}

Would it be possible to add more debug info to rustic lock? Showing e.g. what lock command is executed and its output? At the moment debug only shows what files are listed.

[aawsome]

Actually the command call is not yet implemented.
So far lock only shows what it would do without in fact doing anything.

I'll come back to you when I've added it.

[kapitainsky]

OK - clear.

What is not listed though is for example config file? It should be included in lock too IMO

[kapitainsky]

• To let rustic know that snapshots and pack files are locked, I would save the actual locked-until-date in the snapshot and in the index for the pack files. Using that information we can assure that forget and prune wouldn't even try to remove/repack data which in fact is covered by a lock

I am not sure what is the purpose of this. Myself I want to use the repo in normal way - with forget and prune etc. Locking has only one purpose - to protect data from attempted corruption. There is no point to mix locking and append only logic. At the end it should be up to end user how to use it. This way it also simplifies all logic - any extra logic means only more chances for bugs.

What I would like rustic lock to do is to help me to extend lock on ALL repo files only. If you think that extra options like locking only specific snapshots are needed then fine - as long as basic operation to apply specific lock on all repo files is available. I do not see how from practical perspective I could use specific snapshots locking - then I would need complex wrapper to extend locks to only some snapshots I locked in the past etc. Unless you envision it as one off operation. In my usage it is periodic task keeping all files protected for the next N days.

[aawsome]

It is somehow strange that you want to do forget or prune if you before tell your backend to lock the files. Maybe this works with your backend as locking there means to enable versioning and keeping the current version. But this is only a special case of a backend.

Other backends may really "lock" the repository files. In that case a forget or prune will fail with an "not allowed" error, but only in a late step. So, for these kind of backends it helps if rustic does not even try to remove the locked files. In case of a totally locked repository it would mean to forbid forget and prune etc. In case of only some locked snapshotfiles and packfiles, it would mean to only remove unlocked snapshots and to remove or repack only unlocked pack files.

[kapitainsky]

Hmmm - theoretically you are probably right but all backends I know - S3/Azure/GCS work with versioning and do not prevent you from e.g. deleting files (they are retained as versions or delete markers anyway). If rustic will support even more exotic options then only better.

What I am only talking about here is that rustic should not limit end user from decision how to use it. Yes with 100 days lock in place you can say it does not make any sense deleting anything not old enough and keep all last 100 days backups. But at the same time it might be not practical from performance perspective to keep all backups for very long period of time (1 year lock and hourly backups for example). I can still restore all repo as it was e.g. 90 days ago if needed.

[aawsome]

Actually "locking" could also be implemented by removing rights to overwrite or delete the giving file. With a good ACL system you could even allow to remove those rights but forbid to re-add them. IMO this is the better locking concept as it removes the need to handle versions.
But anyway, rustic is supposed to support both versioning-locking and non-versioning-locking.

[kapitainsky]

I am only loudly thinking here BTW - by no means trying to tell you what and how to do:)

[kapitainsky]

Whatever solution what will help enormously with me testing is as much debug as possible. Then I can troubleshoot many things myself and only seek help in real problem cases.

[aawsome]

But at the same time it might be not practical from performance perspective to keep all backups for very long period of time (1 year lock and hourly backups for example).

Actually you do keep all hourly backups for 1 year if you run rustic lock repository --duration 1y. It might just be that a following prune makes them inaccessible (in an easy way) for rustic as they only exist as older versions on your backend. Same is with pack files. If you do repack packs, you in fact save duplicate data - the blobs in the original pack (in older version(s)) and the still-used blobs in a newly generated pack. If you want to save space it might be better not to repack those.
But yes, users should decide what they are doing, which tradeoffs they want to make and finally what is most convenient for them!

[kapitainsky]

100% correct. But with long lock and many snapshots I might hit the memory limits or unacceptable performance deterioration on my specific system. So it is great option if I can "off-load" it to the lock/versioning storage subsystem.

At the end what I need is ability to restore my data as it was in the past - some evil happened, forensics shows my system was compromised 50 days ago. But I know that no evil actor could wipe/modify past data (in case of S3 compliance locking). So I can simply get my repo as it was 51 days ago.

[kapitainsky]

What I can commit to is that when it is working I will create foolproof step by step manual how to configure rustic repo protected with S3 COMPLIANCE lock (probably something most hobbyist/casual user today need - given cheap S3 providers like IDrive). Including recovery - at this stage it will be rclone based but I imagine people wandering into locking should not have much issue with using it.

[aawsome]

Just wanted to mention #1078 which (e.g. in an emergency case) is able to handle a missing/incomplete index. It is an alternative for rustic repair index in the case of read-only repositories.

[kapitainsky]

effectively it means that all indexes are not needed at all? And only kept for performance reason? As all data in indexes is redundant?

[aawsome]

Yes, that's what it is.

In detail: the index redundantly keeps the information in which pack file which blobs are kept and how to access them (position / compression state).
In restic this is basically only a performance reason as index files are also cached locally.
In rustic, there is support for cold storages. Here having this information on hot storage allows to only warm-up needed pack files instead of all pack files. Moreover rustic saves some extra information about packs in the index like a date (used to keep packs for a minimum time) or a delete or lock (in the current WIP locking implementation) mark. But these informations are all not neccessary to access the data within a repository.

[kapitainsky]

and what exactly --check-index option does in case of read only repo? Allow to operate on a repo (e.g. restore) without index files? The option name suggests that it only does some data check.

Still I think that lock should (at least optionally) allow to lock indexes as well (in case of lock/versioning storage). Storage savings but not doing it are minimal and in case of restoration the last thing I would like to do is to worry about rebuilding missing things - which might be also slow for large repos.

[aawsome]

and what exactly --check-index option does in case of read only repo? Allow to operate on a repo (e.g. restore) without index files? The option name suggests that it only does some data check.

It does exactly what repair index does - except it doesn't modify the repository, but use the information to perform the command. (This also means if an index needs to be repaired, this will be done by every command call).
If you have a better name than --check-index I'm open to rename the option!

[kapitainsky]

Thank you. Actually now I see that this name makes sense:) It is just optional additinal index check. Using it with missing index file(s) is just the edge case:)

[kapitainsky]

I have found aws cli rather slow and experimented with minio mc cli - many times faster - like 10x +. However this tool requires different command parameters, e.g.:

mc retention set MODE VALIDITY S3/mybucket/key

for example:

mc retention set compliance 1d S3/mybucket/key

VALIDITY has to have format Nd or Ny only - https://min.io/docs/minio/linux/reference/minio-mc/mc-retention-set.html

Hence I can not use %date variable in lock-command. Would it be possible to add another variable %validity which would simply pass retention value from rustic lock "as it is" so when I run:

rustic lock repository --duration 1d

%validity will have 1d value, then I could use:

lock-command = "mc retention set compliance %validity S3/mybucket/%repofile"

I think minio is popular enough to justify supporting their tools too - especially that mc is so much faster