Merge data #840
Comments
|
From Dominik: short outline of the merge algorithm is here: |
|
The data merge will be done in two steps
What do you think about it? |
|
There are many, many people that have been waiting well over a year for something like this Jeremy - I can't overstate how welcome a feature this is and I'm personally looking forward to your successful progress in closing the issue :) I have felt for some time that this is the single most important issue that has KeePassDX has needed support for and most all of the reviews I've read have zeroed in on this one feature that other, and otherwise less capable Android KeePass clients were enjoying. Last but not least, I think your solution is elegant too, and keeps inline with the principles you have stated many times regarding the correct and official jobs of the Android external file managers vs the app KeePassDX. I know I've corrupted several databases using my git-sync script methods that I derived from Using regular old UNIX based Pass and the Android version, Password Store, which requires OpenKeychain (On Windows, it's Pass-Winmenu - all three platforms sync perfectly via private git repo). The git based sync works flawlessly with the Pass family, but I could never get my own derivitive to work with kdbx files (That would be the ideal solution IMO, since anyone can obtain and use a free private git repo). I also tried working with NextCloud, but I could never be sure if I was overwriting or just not getting the updates, and often, when comparing kdbx files on the local devices found that they were both woefully out of sync with each other (You're putting in a merge that should work nicely. Of course, and considering that every single thing I've mentioned above is FOSS - it goes without saying that any solution using Dropbox, Google Drive, Box, OneDrive, etc., are wholly unaccaptable, and categorically dismissed - as they should be. I will not support or deploy application software that isn't FOSS where it can be avoided in any way, and that includes on Platforms like Apple and Microsoft which themselves are closed source proprietary OSes. Eventually, I just had to give up though, and since I've deployed a few dozen VaultWarden installations for customers, almost decided to just throw in the towel and go with that for myself, but instead, discovered AuthPass, an Android client which provides a reliable syncing solution for NextCloud and my kdbx database. They've recently launched clients for Windows and Linux, but I'll stick with KeePassXC, and I'm excited to be able to begin using KeePassDX again soon on Android with NextCloud as well. In a nutshell.... This is great news Jérémy, and thank you for building this support for your KeePassDX! |
|
I think this is a great idea, especially the idea of a second app. You could make it a premium feature. |
He could, but IMO that would mean many of us would go elsewhere considering it to be a decision to drop support for the FOSS community. I know that I personally will only ever contribute directly to a dev or project, and then only if it's FOSS. In both cases, Jérémy has accommodated this. Many folks simply will never use the Google play store or any of it's infra, and Jérémy has direct contrib vehicles via LiberaPay, PayPal and a crypto tipbot. I'm still not clear why a git synchronization method won't work for kdbx files while it does work seamlessly for pass files, but I am delighted that he's addressing at least one method that will work for a large number of folks in synchronizing their kdbx files. Considering that I'm actually one of the people who has contributed directly via PayPal to the project, I wouldn't even mind if there was a (so-called premium or not) version of this second app's apk available for d/l directly from the Kunzisoft site for a fee.... as long as it's FOSS - such is permissible under the GPL as the method of distribution being compensated for during the delivery of the free software. He wouldn't actually ever even need to put it up on the play store anyway, since it would presumably only exist to service the extended needs of folks who use KeePassDX anyway. |
|
The idea I had for this new application was pretty simple. Still under GPL license, the application will integrate access to local files and webDAV, there will be a smart cache manager to prevent synchronization problems and it will be free and accessible to everyone. But what bothers me is to be forced to integrate APIs from products that don't respect my philosophy but also don't make the effort to update their own application to be compatible with the standard Android URIs. It's a lot of time and investment for workarounds that I don't even want to do so I'll make a paid plugin system completely optional for each integrated API, for example if a user wants a Dropbox connection through the API he'll have to pay the plugin. I think this is the best solution as it doesn't harm anyone, contributors will have access to the available content, those who only use protocols based on an open philosophy will not have any restrictions on functionality and those who absolutely want to use private clouds normally don't have a problem with the notion of payment. I haven't thought about the git implementation yet, but I think the best way to do it is to do it natively with a folder and password encryption, so that the git protocol itself handles the conflicts. And as KeePassDX is not made for that because it manages only one file and as it is an advanced feature, I redirect you rather to https://github.com/android-password-store/Android-Password-Store for the moment) |
This is fair. Though, even with a correct Android URI integration, synchronizing local files and expect it to be compatible with drives is wrong. I long knew it, but a recent issue in Keepass2Android confirmed that. What happens is you will have a 2-layer synchronization system. The first between the open database and the local database, handled by KeepassDX, and the second between your local file and your drive file, handled by the cloud client. You can manage what happens on the first layer, but not on the second one. If your local file is not up-to-date with the drive file when you synchronize the local file with KeepassDX, since the cloud client does not know how to manage discrepancies, the latest file (i.e. local in this case) will overwrite the cloud one, and you will lose the changes made in the cloud file. I'm not saying this should be handled. In fact it can't be handled without integrating the cloud APIs directly, to get rid of the second layer synchronization, and this is totally the opposite of your stance (that I respect). But I'm saying you should be well aware of this issue, and have it documented at the least :) On my side, I'm being more pragmatic. If your stance make your software useless, then the endeavour itself is useless. I find your soft really fresh, but not being able to synchronize with public clouds makes it useless for me. I will still continue to follow your project though, because I find it really cool. 👍 |
|
I'm glad you wrote this comment, I'll be able to bounce back on it because I've already studied the problem. I think you just didn't understand the overall thoughts on the future of the project or it is simply not your need in the very short term.
Indeed, there are 2 levels, but not exactly the ones you describe and that changes everything, I will explain.
Here what you call local file implicitly indicates that it is managed manually and stored in a way visible to the user. But as the basic principle of file management on Android are URIs (by using the SAF system), it allows to point to any file location (even virtual). So the file you consider as "local" can in fact be just a cached file offered from a cloud application, KeePassDX doesn't care if it's a real local file or not and it doesn't have to know because it's an editor. Besides, we don't really know it ourselves, when we select a file from any Cloud application and it opens KeePassDX through the ACTION_VIEW intent, the only thing we know is that the file is proposed and its data stream too, but it could be a file stored directly online on a remote server, if the flow is well managed, it's technically possible and not local to the device.
Yes you are right, but is this behavior specific to a KeePass file? Of course not, it just means that cloud implementation clients don't handle synchronization of files that are synchronized from multiple locations well. If they were plain text files, it would be the same, and in fact it's a design problem of their template and it's not up to the editor who opens the file to solve this problem. (And surprisingly, I don't like doing the work of multinational companies that offer only half-functional solutions in closed applications.) That's why I've been thinking about how to do this work with a separate generic cloud file manager app, the so-called FileSync project which is still only in the discussion stage. The goal of this project is to propose a generic solution for the synchronization of any type of file while keeping a cache and history system visible in real time for the user, who will see the potential conflicts between files and will be able to choose the one best suited for the editor. It will be possible to reassign a different version of the file to the same URI, and as KeePassDX will be able to detect changes after a change of data in the same URI, it will be enough to press the merge button to have an updated file with all data.
Here you confuse all-in-one solution and encapsulation of components. It is of course possible to integrate everything, which KeePass2Android has done.
That's the trick, with this second application, it is possible to manage the second layer you are talking about with the right encapsulation of components I want to keep in KeePassDX. And as I said in my previous comment, cloud API plugins will be added to the FileSync application. And besides it solves the problem of synchronization for any kind of cloud files, it's a big project, but I think it's the right solution. Afterwards, if you still think that pragmatically this way of doing things and that the program is useless, nothing prevents you from using other solutions. |
|
Hey calm down, I did not say your app was useless :) When I say that an app is useless, it's always when its mission is not met. I'm not saying KeepassDX mission is to handle cloud drives. That being (re)-said, Your detailed answer is really interesting, and I agree with almost everything. Some remarks though, if I may. 1/ You are right that the sync issue with drive is not an app issue. That's not the point. The point is from a user perspective. However, a developper can not handled every issue either, of course. 2/ True that the issue is not related to kdbx. But that's true with the current cloud drives/clients. Some drives know how to manage diff, but they don't exist on Android. So it does not change the state of fact for sure, but this is not an absolute fact either. 3/ Even when/if a cloud client knows how to manage merging, for sure it can't do it for all formats, especially kdbx that is encrypted. That leads to: 4/ I find your strategy really interesting. To have a dedicated app to handle the issue. However, it raises more questions to me than it solves. Will it handle merge itself? If yes, how will it solve the issues above? Or is it dedicated to KeepassDX, so it knows how to handle kdbx? If yes, how will it handle sync without key, if not, how different will it be from cloud drives that already manage discrepancies by keeping the multiple versions side by side? From your answer, it leaves me no doubt that your are well aware of the issues, so I'll just continue to keep on eye on all that. 5/ And yes, sorry to not being aware of all that happens here. I did try KeepassDX, but since it does not answer my needs (triple warning, my needs and only mine). My intention was not to say you're doing it wrong, just to have and share another point of view. If I did find your app so useless, I would not have been following if for a few years :) |
|
Ok I understand, I try to be as factual as possible but it's true that the word was strong for me. It must be the consequence of having not always benevolent messages from users by email. 2/ It is not the goal to create a synchronization application that manages the diff, but once all the objectives have been achieved, why not. 3/ Yes that's why a generic solution with a manual selection system in case of conflict is for me a viable solution at first. 4/
No, that would be too much to handle at first. As I said in the previous message, proposing a file version then another through the same URI allows KeePassDX to manage the merge once decrypted. For other formats, it is to be studied.
It does not need to know the format. Sending all data from the file to the stream a second time is enough. As the association, file / URI is done manually, there is no real merge in the sync application. KeePassDX will receive two different types of data, and when saving, the flow entering the FileSync application will created a new version (merged) of the file.
It will be similar to clouds which keep versions of files side by side but the user can knowingly determine if a version is up to date or not. For me, the important thing is to offer a very clear interface that makes the user understand that there may be conflicts, and how to resolve them with a manipulation. Not just hiding the problem and getting heaps of feedback that the data is never up to date. 5/ I understand. I am also open to criticism. I just need to get things in order and fully manage the editor before moving on to sync. At first KeePassDX was just KeePassDroid, then there were lots of additions and refactorings. I still remember when users said it sucked because there was no autofill. I understand the needs, just my time is not infinite. :D |
To someone that leaves people with a broken plugin for keepass, it's reaaaaaaaaally understood... |
|
I realized that this is not easy to understand for everyone, here is an explanatory diagram which allows to see the global functioning of a synchronization and a merge with KeePassDX. |
|
The version 3.2.0 which contains the merge has been deployed in beta on the Play Store. |
|
For those who are testing the beta, do you have any problems with the data merge? |
|
Oh my! Where to start? Well I suppose the first thing I should mention is that I share much of the same frustrations as @Karreg . Earlier (a few months ago) I myself mentioned something similar by saying I would have to turn to another solution in the meantime. That has happened for almost all of the people I support... myself? As you know I really really really like coupling KeePassDX w/ KeyPassXC - it's a major pain in the derriere but when I like something I can be quite stubborn. I'm glad that much of this has been hashed out w/o me in the conversation as of late. I didn't want to come across as one of those asshats you mention that appear in your email inbox @J-Jamet :) Okay, I'm going to go and follow the new FileSync software project. I suspect it will be some time before it's worth bothering to pay attention to, so in the meantime I guess I should cover where I am personally, restating some of my firm philosophies, what might be useful to me, and what I will probably always find as completely useless, as a staunch FOSS and Privacy advocate (Please realize these are my own hard limits based on my personal prejudices): Completely Useless:Anything having to do with, or bothering to accommodate the following:
I simply "Don't know, don't care, don't wanna know, and won't pay attention anyway". What is this?Will this / Can this, be of any benefit to you @J-Jamet ?
What I think is going on w/KeePassDX v3.2.0:I use F-Droid, so I'm still running KeePassDX v3.0.4Okay correct me if I'm wrong (Please), but I think this is what the behavior is with your current release.
It must be very prominent and user friendly - the users see mismatch messages, older than, newer than, and their brains explode - I get phone calls, we all get grief, and people say bad things to you in your email inbox lolz. Telling the user: "IS SAFE TO DO" is very important. Further, since you say it is safely merged using your latest patches, I don't actually see any utility in telling the user that the files are different anyway - that's more of an IT person's thang. Users use, they most often don't understand and that causes tickets to be opened, phone calls to happen, etc. So is that basically what is happening here in steps 1 - 3c above? If so, that's great, but... I don't see why that would ever be necessary or even happen until the separate FileSync App actually exists, except for in the rare case I clobber my .kdbx on disk with a mv -f or cp -f command from Termux, etc., when KeePassDX is already open. Why? Because right now, I'm so afraid to attempt any kind of sync of my .kdbx files unless they're completely manual, by closing KeePassDX/XC in the event that I do anything on either side of the device boundaries. I know when I do, so when I do, I stop, go through the whole laborious procedure to run a script that rsync's the file up/down, etc., then I open the app on the other machine and there's no discrepencies - This will never happen for a regular user who is a typical office worker or sales rep for bicycle parts, or high school students that own a laptop and an Android. I can't use KeePass for these folks. If steps 1-3c is what essentially happens, I don't see why the user needs to be informed anyway (you said the merges should be safe and that's why the Play Store folks are the beta testers lolz). Why won't rsync work?If the merge function is safe it should, right? I'm going to nudge you on....Git 🥇 Somewhere way back in one of these issue #'s with the filesync flag, I posted some scripts I had set up to use Git, and my application (not that it really matters - they'll all either work the same or they won't work at all) was using a private repo on one of my Gitea servers (I realize it actually doesn't REALLY matter that it's a private repo, since .kdbx files are encrypted, which is why they're supposedly safe to use on Gdrives, for those folks who have sold their souls and have been subjugated by the Google). I didn't really understand it at the time, and after scouring over the Serverfault family of sites finally gave up myself. But my reasoning was this, which is something I already use for somethings like SSH admin stuff that I need when using JuiceSSH or Termux related activities, because it syncs on everything, my BSD, linux machines, even a Windows laptop I use sometimes (on Android, requires OpenKeyChain to be installed from F-Droid too). Here's your latest quote and this actually excites me!
Now... I don't wish to beleaguer you, especially because I know that waaaay back, you did address what I was trying to do (I thought I was doing basically the same thing that Android-Password-Store does - https://f-droid.org/en/packages/dev.msfjarvis.aps/ Here's what the pass site says:
On UNIX, everything is always easy. On Android, well.... it's not always true, but more often than not, "There's an app for that." lolz. In this case, as you pointed out, there actually is an app for that (two, working in conjunction together at F-Droid, Password Store for On Windows, there's pass-winmenu (The best I've personally tried - not listed at the Now, please accept my apologies in advance, because I do know that several months ago (a year and a half maybe?) you tried to explain to me why this wouldn't work. Over and over I read and tried to undertand what you wrote, and I just never did comprehend why not - it's just an encrypted file, right? How is a .kdbx any different than a gpg encrypted For example, in the scripted procedure I posted here so long ago, all I was basically doing was mimicking the action of And from the other device:
I don't see why a trigger of some sort couldn't be incorporated on the android side to do the push pull everytime the hash of the .kdbx file changes. I know that might sound dumb. Regardless, I'm just elated that you're looking at that now, because I believe that once you finish your https://github.com/Kunzisoft/FileSync a Git based sync option analagous to what Look, I know this was a long read, and Jérémy, I so do appreciate you taking the time to go over it (if you have). Hopefully you didn't tl;dr because as you know I am a big fan of using .kdbx files for password stores and especially KeePassDX :) Kindest regards, Bradley ⛵ . |
|
I understand that certain behaviors are a source of frustration. I will try to be as clear as possible so that there is no misunderstanding or misinformation.
Concretely, it is the same operation as on KeePassDX, the Storage Access Framework allows to create a link to a file and to retrieve or send data through the stream provided. The choice of not keeping any KeePass persistence data was made because it allows KeePassDX to be completely data independent and behaves as a simple editor. On the other hand, there is the disadvantage of synchronization, I will come back to this later.
Basically, the workflow goes well as you describe it with points 1, 2 and 3 even if in point 3 it is up to the user to choose the merge button because he can choose save or reload. If the content has been modified by a buggy synchronization application (which is not uncommon), or simply that the user does not want the changes he had made previously, he must have the choice. But in most cases, I agree that this is the best thing to do, that's why I'm thinking of putting a shortcut. If users don't understand what they are doing, they must be educated, KeePassDX is above all intended for people who want to manage their passwords manually, otherwise you might as well go for a turnkey server solution. Everyone knows the http protocol (at least in name), a merge is not difficult to understand. :p
You don't have to use the FileSync application. The purpose of creating this application is to remove the synchronization bugs that occur when two files are modified in two places at the same time. But as I said above it is not magic, you will have to send a version, then another one to merge in KeePassDX for the saved data to be complete. You are afraid to do the synchronization because conflicts are not well managed by the cloud file managers. If there is a history and visually there is a conflict indicated, just manually merge two files.
You can't be editing an entry and all of a sudden, the application goes on hold and reloads to do the merge.
This is where it gets interesting. If the need is to have a perfect synchronization of files from several places in an automatic way, then I say right away, KeePass is not made for that. It's a single file, think of it as a small encrypted virtual hard disk. You cannot merge two encrypted files together without decrypting them. There are other more suitable projects for this purpose, but this obviously requires client/server synchronization systems that somehow recognize the architecture of the data that is passing through. In the case of the Android-Password-Store project, there is not only one file, but one file per password. And each of these files defines a credential in a tree visible in the associated repo. https://github.com/android-password-store/Android-Password-Store/wiki/Frequently-Asked-Questions#is-my-repository-safe-as-a-public-repo In the case of KeePass, the merge must be performed once the data is decrypted, which limits synchronization solutions because two contents must be open at the same time but is much more flexible in the choice of encryption and can be used completely offline. It is a single file in encrypted binary form. In the case of Bitwarden, no surprise, it is a server to be deployed that synchronizes data between several access points. It requires a centralized infrastructure.
It's not dumb, it will just be done in the FileSync application to cache files coming from multiple places. In the case of Android-Password-Store, the merge is done by keeping the files whose timestamp is higher. Just take the time to look at what really suits your needs. ;) |
|
Currently the data merge does not automatically save the merged content even if the "Autosave database" option is enabled, do you think it is useful to automatically save the database if the merge went well? I ask because if we are not careful, the content of the first opened database is replaced if this is the case, is this a behavior you want or you prefer to save manually after checking that a merge is really OK? |
|
Unanimously, the merge does not save automatically in 3.3.0. I close the issue now that the merge from an external file is implemented. |
Ask to merge data if "Reload database" button is clicked and file was modified externally.
Actually, saving database overwrite external changes.
The text was updated successfully, but these errors were encountered: