Deduplicate messages

[devurandom]

Currently TextSecure (2.0.1) can only import messages from the default Android SMS application once. Afterwards doing the same again results in duplicate messages. It would be nice to be able to deduplicate them.

I assume that going through the list of messages and comparing from, to, time and text would be sufficient. Then just delete everything except the first occurance.

Please excuse me, if this is a duplicate of another issue. It was hard to find good search terms, since "duplicate" occurs extremely often as "duplicate of issue #...".

[perguth]

I would like to see this feature implemented too. I guess a bunch of us have old sms backup files lying around that may contain duplicates and wait to be consolidated by importing.

[perguth]

If you can afford to lose embedded media like pictures you could try the following route, @devurandom:

1. Export TextSecure as plaintext
2. Install SMSBackup And Restore
3. Backup SMS for security using SMSBackup And Restore
4. Import the TextSecure backup into the SMS app via SMSBackup And Restore while letting it check for duplicates

If you are now free of duplicates in your SMS app: win. Go ahead erase the messages in TextSecure and reimport from your SMS app (maybe just reinstall).
If not: clear your SMS apps messages via SMSBackup And Restore and import your SMS backup.

[McLoo]

I'm interessted in that too.Even tried some aproaches...
Not deduplicatiing already merged messages, but avoiding of inersting duplicates (depending on the sender, date and message)
If this is something that @moxie0 is ok with, I'll give it another try.

[moxie0]

@McLoo seems like a good idea

[devurandom]

@McLoo Thanks for taking this on! Deduplicate on import is a very good step in the right direction. If someone wants to deduplicate the existing messages, he could hopefully use your code and submit a patch for an enhanced feature.

[riyapenn]

Related user action: "I clicked the import button and while waiting I then clicked on the notification progress bar and it imported again. I assumed the notification bar would bring me to a different screen not kick off a load."

https://gist.github.com/1fd6592e5848fb3a4e76163ee0e4da3a (though looks like this log may have been captured too late)

[automated-signal]

GitHub Issue Cleanup:
See #7598 for more information.