Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Web clipper overwrites existing database entries #623

Open
m0lentum opened this issue Apr 19, 2024 · 2 comments · Fixed by Zeroo28/Allusion-CS#1 · May be fixed by #624
Open

Web clipper overwrites existing database entries #623

m0lentum opened this issue Apr 19, 2024 · 2 comments · Fixed by Zeroo28/Allusion-CS#1 · May be fixed by #624

Comments

@m0lentum
Copy link

m0lentum commented Apr 19, 2024 •
edited
Loading

Describe the bug
Occasionally when adding an image with the web clipper add-on, it replaces an existing entry instead of creating a new one. The added image receives the tags and time of import of the old entry, which disappears from the gallery until Allusion is restarted, at which point it reappears as a newly imported entry with no tags. Any subsequent images added with the web clipper will all replace this same entry, and all tags added via the web clipper will appear on this entry. This behavior persists even across restarts.

I've also noticed that the next time a new image is added to the directory by saving it there without using the web clipper, it also replaces the same entry, but the next image after that will create a new entry. Edit: After some debugging it seems that files added with the clipper are interpreted as being moved/renamed when a new file is added after them. Continuing to investigate.

To Reproduce
Steps to reproduce the behavior:

  1. Choose a download directory and enable the web clipper in settings
  2. Use the web clipper context menu to add some images from a browser
  3. Eventually an image will replace an existing entry

Strangely, this only sometimes happens on my main Allusion collection, but on a development build with a newly created database it seems to happen every single time I use the web clipper. This also happens if I add an image by manually sending an http request to the clip server endpoint, so it's definitely an issue with the host process, not the browser extension.

Expected behavior
All added images receive new database entries and old ones are unchanged.

Allusion version

  • v1.0.0-rc.10

Desktop OS

  • NixOS 23.11
@m0lentum
Copy link
Author

After a bunch of debugging I believe I've found the root cause in the way inode numbers are used to detect moved/renamed files:

  • when the file added via the clipper is initially downloaded, it gets saved to disk, caught by the file watcher, and its inode number stored in the database
  • after this, exiftool is used to add the page url to the image metadata. This causes the image to be rewritten to another inode, but the database still has the inode number it was originally downloaded into
  • the inode the first image was downloaded to is now vacant. When the clipper is used again, the same inode is used to store the new image, and again caught by the file watcher and stored into the database. This causes the new image to be interpreted as a moved version of the previous one, even though in reality both files are now at new inodes

Relatedly, any operation modifying file metadata with exiftool causes the file's inode number to change, breaking subsequent renames. If you change an image's metadata in Allusion and then rename the file, a new entry is created and the old one is still visible as a missing image.

m0lentum added a commit to m0lentum/Allusion that referenced this issue Apr 19, 2024
@m0lentum
fix: Write in place with exiftool
3ac37f5 
Fixes allusion-app#623

This fixes issues with the way renamed files
are detected from inode numbers,
which would become desynced whenever exiftool was used.
Any files that had their metadata edited would not be correctly detected as moved,
and the web clipper would occasionally overwrite existing database entries,
mistaking them for moved files.
m0lentum added a commit to m0lentum/Allusion that referenced this issue Apr 19, 2024
@m0lentum
fix: Write in place with exiftool
c5f8fa1 
Fixes allusion-app#623

This fixes issues with the way renamed files
are detected from inode numbers,
which would become desynced whenever exiftool was used.
Any files that had their metadata edited would not be correctly detected as moved,
and the web clipper would occasionally overwrite existing database entries,
mistaking them for moved files.
@m0lentum m0lentum linked a pull request Apr 19, 2024 that will close this issue
Open
@m0lentum
Copy link
Author

Turns out exiftool has a parameter that makes it write in place, which prevents files being moved. I've made a PR applying it in #624. The same issue can still be caused by replacing files manually, so it may be worth considering alternative solutions that don't involve inode numbers at all, but with this modification Allusion no longer breaks itself (in cases I've tested anyway).

@Zeroo28 Zeroo28 mentioned this issue Jul 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@m0lentum