Unable to restore encrypted backup - Bad MAC #8355
Comments
|
So basically now coming from #8347 it seems i evolved to this one here. With the latest 4.30.7 on Nexus 6P it finishes after 2 hours my backup and my Pixel3 - 4.30.7 Signal tells me that my password is wrong. Same problem as above. Tried https://github.com/xeals/signal-back and worked with the same file and same password. Since i reached from crashing Signal on Pixel with restoring backup, i think the next Level is succesfully restoring a backup. :D Pixel 3 - https://debuglogs.org/8afd5e209b77295d13b983b38b06c71fd0cbe32f04e4521f5f5aad5b2f87a4d3 Nexus 6P - https://debuglogs.org/703329cfc7d727fab8c4cc4087552e37cde3d45abd8f88428ef92ea91d3ee583 |
|
Same here. Running LineageOS 16. Interestingly, with the wrong password specified, Signal will immediately claim that the password is incorrect. With the right password, it reads around 2000 messages and then suddenly has a change of mind and claims that the password is incorrect after all. Seems legit. ***I tried both 4.26.2 and 4.30.8. 4.26.2: 4.30.8: I have restored backups (not this specific file) in the past a couple of times with no issues. The only change is that when it worked it was with Android 7.x and now I'm on 9.x. Probably not relevant though. Clearly, the backup was a) generated by signal itself and b) in good health since signal-back can properly decode it. |
But maybe this is the relevant part |
Indeed, seems to be a common thing... |
|
Not valid anymore! I had different Signal versions! Sorry for any confusion.
|
|
^^ There goes the 9.x theory. |
|
By the way, I did not see this issue restoring on LineageOS 15.1 (Android 8.1). Different backup file though (another account). |
|
Actually, it may be possible that the backup was not restored. This was a few weeks ago and my memory is flimsy. Sorry about the spam, not possible to edit comments anymore it seems.. |
|
I had different Signal versions! Sorry for any confusion. |
|
Also experiencing this migrating from Nexus 6 Android 7.1.1 to Moto X4 Android 8.1.0 Decrypts messages (as it is counting the correct number to be restored) Then when it fails it throws Incorrect password. Log cat shows the Bad Mac error as well. Has anyone found a work around? Stock Roms (Nexus 6, Moto X4 Android One edition) |
|
Had the chance to day to try on Android 8.1 AOSP. In one swift strike, it seems that we can rule out the culprit being Android 9 or LineageOS. Perhaps @moxie0 has something to add at this stage? |
|
This is the very same backup file from last month by the way, which signal-back can perfectly read.. |
|
Hi, I also have this issue coming from an Android 7.1.2 and going to an Android 9.0.2 As far as I understand, this seems to be related to #8327 @greyson-signal @elkhadiy can maybe help us as they were the ones involved in #8327 I am using the 4.31.6 version which seems to be corresponding to the master version. |
|
Hi, Just skimmed through this thread, I'll be back later on to help. What I can say for now is that before 4.30.2, when restoring a backup, Signal was happily decrypting all messages without checking their message authentication code (MAC). #8327 fixes this check. Now it seems we are dealing with some corrupt data in the backup. I'll hack together a tool to try and find the offending frames. EDIT(2018-12-16): So I'm a little puzzled by the fact that OP can decrypt the backup file with signal-back while he has a EDIT(2018-12-18): Removed superfluous information irrelevant to the issue at hand. |
|
I ran your tool and got the following exception: Traceback (most recent call last):
File "/home/royalvein/bin/miniconda/envs/atom/bin/signal-bkp-decrypt", line 11, in <module>
load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')()
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/cli.py", line 23, in run
bkp = SignalBackup(args.backup_file, args.passphrase)
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/signal_backup.py", line 22, in __init__
+ datetime.now().strftime("%Y-%m-%d--%H-%M-%S")
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/osfs.py", line 311, in makedir
_path = self.validatepath(path)
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/base.py", line 1429, in validatepath
else "paths must be str (not bytes)"
TypeError: paths must be unicode (not str)
Exception AttributeError: "SignalBackup instance has no attribute 'preference_file'" in <bound method SignalBackup.__del__ of <signal_backup_manager.signal_backup.SignalBackup instance at 0x7f77295e3200>> ignoredI do not know if it is the expected behavior... However, I did not try signal-back before so i did it at the same time and I was not able to extract my messages and got: Maybe my issue is not the same after all even if the behavior seems similar (bad mac error + count of messages is good) to @spospartan104 So I try again to import my message and get the exact exception which is not the same as @sarevok-anchev (in term of the line numbers but maybe the version differs) W RegistrationActivity: null
12-19 23:45:39.169 4632 7965 W RegistrationActivity: java.io.IOException: Bad MAC
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.lang.Thread.run(Thread.java:764) |
|
Oh my bad, that's because it's written in python3 so you can go ahead and Though ~$ echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) )) |
|
Well... My bads... The file I get to run the command was corrupted. I redownload it and now there is no error with signal-back and your tool. echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) ))gives me |
|
Are you sure Signal tries to restore from the file you checked? For example you only have that one in |
|
Yep I try with 2 different backups but each time, only one backup was available in this folder. |
|
Hi, Well today, I choose to reset my phone because I mess with it to do some tests with multiple apps. I don't know what was the cause of this issue but it seems resolved for me after the reset. Anyway thanks for the help. Signal is really a cool app and I missed it ! |
|
@elkhadiy I tried your tool and there's an error: Note that:
I must say it's a little bit frustrating that the option to set a very long passphrase and encrypt the app data was removed in previous versions, I never lost a single backup that way over the years, now I'm forced to relay on the signal backup functionality, which apparently in certain unknown circumstances creates corrupted backups, and it's already the second time I'm losing data. Personally I brought more than 100 people into Signal over the years, and now I can't reach any of them for more than a month because of this bug. A bit offtopic, but I needed to vent - I had a good grip on backup flow, now need to rely on a dumbed down version, and it's the second time that it screws my data up. Not good. Anyway.. Let's look towards a possible solution. Can you offer some guidance on what the best next steps are from this position? I don't want to lose the conversation history yet again, and since there's roughly a month and a half of undelivered messages now, I would like to keep the same encryption keys so that they get delivered (eventually). The error must be in some corrupted attachment. signal-back has (on quick glance) all of the messages. It would be way less catastrophic to lose some attachments rather than lose the whole conversation history. Nevertheless, signal-back can recover a lot of images and audio from this supposedly corrupted backup. Maybe even all of them - I wouldn't know. Is it possible to make signal ignore a corrupted attachment rather than abort the backup in an all-or-nothing fashion? Perhaps even have an option to ignore such errors altogether (at the users' peril) ? How about extending your tool to ignore the errors and discard the file/message in question, and outputting a new backup file minus the "bad frames" ? That could be a fast solution to get signal going quickly again. What do you think @elkhadiy? |
|
Oh yeah sorry I derped in my tool, forgot to catch the potential frame construction exception.
Hmmm weird... I'll try and see what he does (but I'm not that good with golang).
Yeah pretty sure you won't restore from this backup with the current state of affairs since at least this frame seems to be pretty borked, if protobuf can't parse it...
Yep, was thinking about proposing this in the app's backup restore process. Something flexible that would report bad messages or attachments and restores anyway... I was also planning on writing a module that rebuilds the backup using the same passphrase anyway for a friend of mine that wants to import his Facebook messenger history. Pretty sure we can fix your backup file! I'll get back to you hopefully very soon. |
|
Beautiful, thank you. |
|
@elkhadiy Thanks for the tool - was struggling a bit to get it working on Windows. Been a while since I ran stuff in a shell. Getting some odd errors: Full console output: Don't know what to make of it to be honest. And no idea whether signal-back does actually decrypt the whole thing. I ended up getting a huge xml file though. |
|
Any updates @elkhadiy ? |
|
i'm also seeing this on a friend's phone, trying to restore a fresh backup from an old phone on a new one. @elkhadiy your tool is great, and indeed it complains about a corrupted frame. could it be extended to re-encrypt the backup to a new file with all corrupted frames dropped? |
|
I've just seen this on my phone; I was unable to send (#8602) and figured a reinstall would probably do the trick, but I'm unable to restore my messages. Fairly chunky backup at 60,000 messages. Can't get debruitage to run on my desktop as Ubuntu doesn't have the fts5 extension for sqlite packaged, but will try again on Arch later. Having Signal ignore (with some kind of message) corrupt MAC frames would be the ideal outcome, I think - losing 60,000 messages for one glitch isn't fabulous. |
|
Just wanted to let you guys know, I'm working on this. I've been able to decrypt (and interpret) the backup data for a long time and just started working on reencryption last week and making good progress. I think a lot of the past messages shouldn't be in this github issue as they do not relate to any bug in Signal (supporting corrupted backups would be a feature request). The only possible bug here is that signal pops up a wrong password error when a bad Mac is encountered, while I think a 'file corrupted' popup would be way more likely if it happens any time after the first frame is decoded. So, if I have something to test for you, I will create a thread on the forum and post a link here. Note it might take a long time (think months) , as I sometimes don't have spare time for an extended period. Also, I do not know how Signal handles backups with frames missing, so there are no guarantees this will work. Frames are not necessarily independent things (attachment data belongs to an entry in the 'part' database, which belongs to an entry in the 'mms' database. The restoration might for all I know still fail if one of those frames is missing. Also, random corruption in a random location might result in anything, that is, if you're unlucky, a single bit flipped might make not just one frame, but all following frames unreadable. Anyway, I didn't intend for this message to get this long. Just wanted to let you know, a tool is coming though it might be a while. |
we were able to reliably reproduce 'corrupted' files. no matter how often we made new backups on the one phone, they always turn out to be corrupted on the other one. i wouldn't rule out the possibility that signal's backup routine has a hidden issue leading to corrupted frames in backups which triggered it, and the restore problems are merely a symptom. |
|
I'd +1 that suggestion, @unDocUMeantIt - my phone has dozens of GB free, backups were allowed to complete fully onto internal storage with no sleep/screenlock, and restore attempts likewise. I can't think of any potential source for corruption of the backup other than in creation by Signal. If the error is caused because the contents themselves are also being checked during restore, but in normal operation such corrupted contents are tolerated, then the behaviour should be modified to catch what would cause corruption at "point of entry" to storage and abort then, and to permit restore behaviour to ignore those corrupt frames on restoration. |
|
@abstractThought I'm not sure what you mean. In normal operation, all (unencrypted) data only lives in memory, but I suppose you could imagine some malware exists that gets to it, but I'd say it's far more likely that type of malware exists for android and is looking for the signal app, than for PC looking for my program. Personally, I actually trust my PC more than I do my phone. @santa-klaus I have some initial backup merging support. It works on the small files I tried, but definitely needs testing. Are you interested in trying? The method is a little involved as it actually turned out to be quite a difficult procedure. Example, click to open if you dare... ;)
Assuming a current backup Then, you can import a selection of threads from the source file into your current backup and export to a new backup file. Expect tons of output (I really need to clean that up sometime): The program automatically tries to determine into which thread of the current db the old messages should be inserted. This might fail if contact have changed numbers or if one of the backups has a contact with a country code (+316.....) and the other omits it (06....). |
|
@bepaald been fighting with coinbase to release my dust without me sending in a pic of my ID. Finally happened, didn't want you to think I forgot about you. Thank you again. |
|
@oneof3holes O wow, thanks! I thought you had just decided against it (which would have been fine of course), but this is cool. I really appreciate it. Thanks again! |
|
@bepaald I am sorry, I got drowned in work. I might have some more time to try things in October and would like to do it. Also, my phone broke, so I will probably have a good reason to merge backups soon. I'll let you know when I get around to do some testing. |
|
@santa-klaus Absolutely not a problem of course. However, just a heads up: I was looking at the source of the 4.48 beta that is currently out and it will change the internal sql database in such a way that it will certainly break my current merging code. I do plan to rewrite the code to be able to merge backups, but there is no telling how long it will take me. Also, it might take even longer (if ever) to get to a point where I can merge an old style db into a new one which would probably also be a use case after your current install updates to 4.48. Obviously, you don't have to do it if you don't want to or don't have time, but I just wanted to warn you that the window of opportunity might be closing. |
|
Oh, that is really good to know, thank you for the heads up! To be
honest, for me it doesn't sound like rewriting stuff to be able to merge
old and new backups is worth the effort, since this will have some, but
very limited use cases...
I'll definitely write if I manage to look into this!
Am 25.09.2019 20:39 schrieb bepaald:
… @santa-klaus [1] Absolutely not a problem of course. However, just a
heads up: I was looking at the source of the 4.48 beta that is
currently out and it will change the internal sql database in such a
way that it will certainly break my current merging code.
I do plan to rewrite the code to be able to merge backups, but there
is no telling how long it will take me. Also, it might take even
longer (if ever) to get to a point where I can merge an old style db
into a new one which would probably also be a use case after your
current install updates to 4.48.
Obviously, you don't have to do it if you don't want to or don't have
time, but I just wanted to warn you that the window of opportunity
might be closing.
--
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub [2], or mute the
thread [3].
Links:
------
[1] https://github.com/santa-klaus
[2]
#8355?email_source=notifications&email_token=ACTFAB5LXJRUNB2334ODHLLQLOV4TA5CNFSM4GC73FNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7S5I7Q#issuecomment-535155838
[3]
https://github.com/notifications/unsubscribe-auth/ACTFAB3ATVCOA66YK2JLMNTQLOV4TANCNFSM4GC73FNA
|
|
Same issue, android 10. Adding my info and a screen cast in case it helps any. Actual result: I have followed Signal's instructions on backing up and restoring a backup. I am moving from a OnePlus 5t to a OnePlus 7t, both using Signal beta, latest. When I type in the wrong backup code, it fails to restore the backup, which is expected, but when I type in the right code, it proceeds to loop through all messages and then throw a toast error saying "Incorrect backup passphrase", and does not import the backup. Expected result: Backup would restore messages ScreenshotsScreencast: https://imgur.com/a/TirPqVa Device infoDevice: OnePlus 7t (HD1905) Link to debug loghttps://debuglogs.org/3bb67a2279313eb2d71a190b82d949b32684407fb8371d9bdae64cf00b06ea52 |
|
I'll add to this, I had almost 70,000 messages, and when I trimmed conversations to 15,000 messages, I removed around 20,000 total messages, and then the import worked. So it's something with the contents of the message themselves, perhaps that attachment issue mentioned above. |
|
I am having the same issue. I have some 19K+ messages in a backup file that's about 2GB. I've made very sure the passphrase is correct. These were my steps:
Signal almost immediately stops and pops up a toast that says "Incorrect backup passphrase". Using signal-back's |
|
I have the same issue. :(
|
|
@dcormier Make sure all 30 digits are there. When pasting a passphrase containing spaces, the last 5 digits can be left off. Either copy the digits without spaces, or manually add those last 5. |
I'll try that later, but perhaps the app should handle that differently? Seems like a UX problem. Maybe the app should accept the pasted value and remove the whitespace internally? Or try it both ways (with the pasted whitespace included and then with it removed)? |
Instead of aborting the entire backup restore. Related to #8355
|
For those of you who are in a situation where you still have your old (broken) backup and would like to try to restore it, I've has made an experimental branch that has been able to restore these broken backups (or at least what we think they look like). You can reach out to me at alan@signal.org. I can get you a release-signed build of this branch and we can see if that fixes the problem. Alternatively, if you’re competent at Android development, the source for the fix is at: https://github.com/signalapp/Signal-Android/tree/alan/backup-restoration where the procedure would be to restore the bad backup using that branch, make a new backup, then get back on official releases and restore once more with the new backup. A debug log from successful or failed attempts to restore the bad backup (either using that branch or an officially signed build of it from Greyson) would be very useful. |
|
I see this is still open, but no new comments in a year or so, so hoping to find out about the current status. I'm experiencing the same problem, and my online searching has yielded some other users with the issue, but no solution. Passphrase definitely entered correctly (as mentioned elsewhere, the behavior is different than when entered incorrectly--the counter increases as it loops through the old messages until eventually stopping with the invalid passphrase error.) @alan-signal It looks like the branch you had created for users to workaround this problem is now gone (though at a year old, I'm not positive it would've imported messages from a newer version of Signal?) I'm a developer (though not Android), so I shouldn't be completely useless at figuring things out if there were a version I could make work. FWIW, there are definitely loads of attachments of many different file types--which I'm willing to lose if that's what it were to take. Also, here's a link to logcat output, though I don't see anything relevant myself. I'm really hoping to get this resolved, as the backup contains almost 20k messages that I actually need to reference somewhat often. Worst case scenario, I suppose I can leave Signal with its backed up messages on my old device to reference when needed, but that really seems non-ideal. Anyway, |
|
@travisterrell I had this problem before; my issue was much like someone said above and their was a borked multimedia file in an old message (signal message not a carrier mms, but carrier could be a similar issue) I can't get on to my main system today (moving) But I may be able to dig up the application that was mentioned here as those links have expired. #8355 (comment) Alternatively you can use the media view in a few different of your larger conversations to see if there are any broken files (could just be blacked out images as thumbnails) |
|
@spospartan104 (and @travisterrell): the program is here: https://github.com/bepaald/signalbackup-tools I do believe however that the app is a lot more tolerant of bad attachments in the backup these days, so if it fails to import the backup file, the damage to the file might be bigger than you would hope. My program should still be able to get all information that is in the backup out of it and into a new, working backup, but it is by no means guaranteed to be an automated process anymore, it often requires some custom coding. Let me know if I can help. And please forgive any slow response from me, I have some busy work periods sometimes, especially this time of year. |
|
Thank you very much! I didn't get an opportunity to start manually scanning through my attachments yesterday, so I'll give the tool a try in a bit and see what happens. My c++ knowledge has probably gotten a bit rusty since I last had to write in it a decade ago, so hopefully it'll work without too much custom coding.
It makes sense that Signal itself has gotten better at backups, given I found less reports of this issue in recent times than a year ago when searching before.
Update: Success with the command line tool! Also, this is pretty cool; I'm going to play around with it some more, like the export tool. Thanks!
…On Thu, Dec 3, 2020, 10:19 AM bepaald ***@***.***> wrote:
@spospartan104 <https://github.com/spospartan104> (and @travisterrell
<https://github.com/travisterrell>): the program is here:
https://github.com/bepaald/signalbackup-tools
I do believe however that the app is a lot more tolerant of bad
attachments in the backup these days, so if it fails to import the backup
file, the damage to the file might be bigger than you would hope.
My program should still be able to get all information that is in the
backup out of it and into a new, working backup, but it is by no means
guaranteed to be an automated process anymore, it often requires some
custom coding. Let me know if I can help. And please forgive any slow
response from me, I have some busy work periods sometimes, especially this
time of year.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8355 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADMUG5WF6KTTIE54XIP2YZ3SS63ARANCNFSM4GC73FNA>
.
|
|
@bepaald With the save again; :) |
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
and others
Bug description
Unable to restore signal backup on new phone.
Similar to #7637 but I don't have xposed installed.
Signal stable closing without error after reading 11401 messages.
Signal beta giving error message about incorrect passphrase after reading 11401 messages.
I am certain the passphrase and the backup are correct as they are working fine using https://github.com/xeals/signal-back
Steps to reproduce
Actual result:
Expected result: Backup is successfully restored.
Screenshots
beta error (in German due to phone language)
Device info
Device: Nokia 7 Plus Dual-Sim
Android version: 9
Kernel version: 4.4.146-perf+
Signal version:
Beta: 4.30.2
also tested on stable: 4.29.7
Link to debug log
Working on grabbing a logcat log but struggling with it. Will add through edit.
Logcat for version 4.29.7:
logcat for version 4.30.2:
The text was updated successfully, but these errors were encountered: