-
Notifications
You must be signed in to change notification settings - Fork 352
[Feature Request] Split and merge documents #335
Comments
Hi, welcome to GitHub! As of right now, I don't have any plans to support editing PDF documents. If you really need that, it might be worth giving Papermerge a shot, they do have some editing tools over there. Although I don't know if they support document merging specifically. This would also be a very big change, since
Not going to happen (anytime soon). |
Agree, feels kinda out of the scope of this app IMHO, and so many tools can do this, even native PDF / image viewers... |
Just curious : what tools are you using for merging/splitting/rotating pdf documents ? |
PDFArranger gets the job done, and has everything I need. Apart from that, I usually use gscan2pdf for scanning, and discard unwanted pages with that, or scan multiple pages into a single document. No further editing needed, usually. |
Some request here. I feed scanned TIFFs from an ADF document scanner. Of course I could do a manual preprocessing outside of paperless, but for non tech users, it might be interesting to get an preprocessing-inbox where you could merge, remove, reorder pages, turn pages, ... before they are fully processed within paperless-ng? But yes, I see the point that it's a lot of work to support it for all the different (multi-page) formats ... 🤔 |
In #426 I had a similar idea (I did not found this already existing issue) about how to implement this:
The TAR/ZIP approach would allow paperless-ng to prevent losing the old original documents, while, as I see it, allow it "in theory relatively simply" (haven't seen the code of this project yet) to allow this with all existing documents. |
I wonder where the need for merging generally comes from. For me it's because my printer's scan to mail function can't put more than 4-5 pages into a single document. I wonder if it's comparable for most of you. If so, we might not need (much) UI for this or even change much about the one document == one file paradigm. Instead we could allow defining rules for merging at consumption time. For mail this is "easy", we could have a mail rule to merge all attachments in a mail into one document for example (convert to PDF if needed, sort by name, then merge PDFs). For manual upload it could be a checkbox or a separate button "upload selected files as one". API wise that could be a new endpoint, "upload multiple as one", to be integrated into any frontends that want to support this. The most tricky bit is for the folder drop, since there's no rule system for that yet as far as I noticed. One could imagine something alike the mail filtering system though based on filenames, match all files with pattern, do action on that set of files. The only action at first would be merge of course. So, to summarize I wonder if "merge at consumption" solves the needs of most people here already. |
@jhass Your ideas on how to implement this feature seems great. To answer your question, my printer may has such a feature, but because I'm currently scanning over 600 documents to store even older documents on the computer, I decided not to use such features because I wanted to scan these amounts of pages using a feed scanner without thinking about them in the first place. I wanted to do the sorting/merging of documents only digitally but create an index over my offline documents by sorting them by an incrementing scan id they gain after scanning. I think this is much easier in my case. |
Not too different story here, just maybe a little less to go through and I'm also throwing out stuff where I feel fine at retaining only the digital copy, so going by the recommended ASN system :) I'm just running paperless-ng on a server so using the mail function of my feed scanner rather than some scan tool on a PC is easiest. Now my pain is that I have multiple documents for the same ASN 😅 , othewise I'm not too worried about, it's indeed easy to find the "other parts" by date or title. |
Chiming in here and sharing some further ideas and comments. I'm pretty busy right now and don't have all that much time except for critical stuff.
This would certainly be possible. However, does your scanner actually support sending multiple scanned files in one mail? Also, I'd like to have the merging logic available to all users, not just users who use the mail functionality.
This is in fact the most tricky part. Once paperless detects new files in the consumption folder, it sends them to the task queue for processing immediately. How does paperless detect when the last document of a batch has arrived? I don't think there's a good solution here. If we do this, I'd like to have the merging functionality available to everyone, and the consumption folder is still the most commonly used way to upload documents. Therefore, some ideas on what could actually work for everyone, given the current architecture.
After that works, we could also think about adding support for selecting individual pages from the selected documents. (This is something I'd find useful as well, since I've got lots of documents with empty pages that my scanner detected as not empty) The most critical part is making the backend work, so that should be the focus. How does that sound? This all is also very isolated functionality and can be added without affecting anything else. If someone wants to take a stab at that, I can give some more detailed instructions on how to do it. |
No it does not, it doing the opposite, splitting the document into multiple PDFs inside one mail (I think it's one mail, I actually never checked 😅) if it gets too big. Yes, a merging tool that just creates a new document sounds like a great idea 👍 :) As a third alternative I think something like a meta-document which groups several documents in a defined order but keeping the members untouched and their own entries could also already help a lot of usecases and might be a little bit less effort. |
We'd still need some UI to define these meta documents, which is about the same as the one for a merge tool. We also need support in the back end for that, documents now have ordered child documents? Also, many components of paperless have to take this new data structure into account (search index should not return documents that are part of a meta document, import+export, metadata matching should use the concatenated content of all documents in a meta document, ...) Compared to building a feature that uses already existing data structures and abides to already defined contracts (and in doing so is compatible with all existing features), this is actually a lot more work. And then there's also the test suite. Changing features requires changing associated test cases. Adding a new isolated feature just requires new test cases for that feature. |
I didn't mean it to be that invasive, child documents could appear normally still and meta documents could not appear in full text search etc, child documents would "just" provide a quick link to go to the meta document they're part of. I felt building the merge background task could potentially prove to be quite the rabbit hole 😅 |
I want to do things properly :)
Actually not, use pikepdf to produce a new pdf document, submit that to the consumer just as we do with other new documents, and optionally delete some documents when done. The consumer will take care of the rest. The actual merging and editing is very straight forward (https://pikepdf.readthedocs.io/en/latest/topics/pages.html). |
I just put a new API endpoint together, and the actual merging process on the server side is straight forward. Reordering documents, keeping only selected pages, that's all simple. The merged document will appear as a new document to paperless, with notifications and all that. Now I need to get this implemented properly and figure out how the UI is supposed to work. |
Very cool! Didn’t even realize you were actively working on this. Let me know if / when / where I can help, have some UI ideas |
Well, I don't exactly communicate what I'm actively working on, that's true. If you want to work out a UI for this, go for it. I've also got some UI ideas, not sure if they align with yours, here goes.
These are just ideas. If you got better ideas, go for it, while keeping the following in mind:
The API will essentially accept an ordered list document of ids, and for each document id an optional page range. I don't have the details down yet. As long as the UI is able to provide that, we're good. It will be possible to specify the same document twice, in case you want to add pages from a document somewhere in the middle of another document. The API will also have an option to download the resulting document as a preview without actually adding it to paperless. |
Yea thats pretty similar to what I imagined. And I agree as this will be a not-every day and probably even not every-user kinda tool the button shouldnt be too prominent / take up too much space, maybe inside a menu or something. And yep exactly what I was thinking about getting there from document detail or bulk edit, and it opens a modal with the UI. As for the actual UI, definitely agree on visual drag + drop, preview does sound cool too. And then when the user is done do they hit "Save" does it create a new document? And what about metadata? Im sure we'll have to figure out lots of stuff once we dig in. Mobile might be a challenge, etc. |
Not necessarily a modal, I think this should be a full page view. You may want to go from the merge tool to the list again, and add more documents.
It will create a new document. Maybe a checkbox that will cause the source files to be removed on success.
Options for either keeping info from the first document (which should be most representative for the resulting file), or run it through the matching algorithms again.
It's okay to have certain functionality not available on mobile. |
Hmm, just now this makes me think about whether it will be frustrating if the actual merge UI has no way to add documents, like a “picker” of some kind. Like if you added 2 docs but realized you need a third you’d have to go back to the list and find the other, hightlight it and add it. A little odd. Then again maybe people will mostly be merging two docs so it’s no big deal? Just kinda asking / thinking out loud. I personally haven’t needed this so I’m trying to put myself in the mindset of a user of this |
The picker would essentially be a list view (with filtering) as well, or something similar, and we already have that. |
Yea. Should be fun challenge, LMK when I can start playing with it. |
While you're currently working on this topic, would it make sense to directly consider some kind of UseCase: It generates from 2 documents (D1 & D2) with multiple pages (D1P1, D1P2, .... & D2P1, D2P2, ....) a new document with (D1P1, D2P1, D1P2, D2P2,.....). This mitigates the single side scan a bit. |
Chromium.
I manually dragged a separator to the end after splitting.
I think it's okay that way. |
Ok these small things should be addressed, I'll tackle the rest when I have some more time later and reply directly to your code review. Thanks |
Any updates on this? This seems like a fantastic feature. |
I just need to find some time to address a few remaining issues. |
First of all a big thank you, amazing Software, was searching since longer a solution for a paperless easy to use tool, now I found it. |
New to Paperless, just saw this project last week, and I'm very happy with this piece of software, it's great! Just picked up a cheap scanner today, but i also ran into problems that it would make 2 pdf's is it is a double sided letter, no option to merge from te scanner itself... Glad to see there is already a solution being worked on. If you need more people to test, I'm also willing to test some documents here. |
Is there an approximate ETA on the availability of this feature on the stable main branch? |
Hi, yes all so here highly wanted :-) |
+1 :) |
2 similar comments
+1 :) |
+1 :) |
I’d also like to see this functionality. Especially the ability to merge a pdf with another one in reverse-alternate order. (For those of us who have non-duplex scanners who end up with alternate order with the second pdf being reversed). Right now doing it with pdfsam but would be great to have this built in. |
Hm, lol. the first function, what i've missed after uploading a few documents - merging :-D So, here i will give a +1 |
+1 also from my end :) |
+1 In my opinion, its the only thing missing from this program. |
I do not urgently need this feature, but I can give a use case case where it comes in handy. I need to create an account of charges. I used to directly scan to paperless and have not prepared any other 'scan-option'. As new bills occur regularly, they could be scanned to paperless-ng and then appended to the existing document containing all the bills. So this feature helps to keep 'work in progess documents' up to date. For now, I scan all the bills to paperless and once the document can be finished, I download all bills and merge them locally. Not a big deal, to be honest. But maybe this use case will help for the documentation. |
For some of you, this may help: |
First of all, thanks for this great piece of software! I also miss this feature after using paperless for one hour, as my scanner doesn’t easily support scanning multiple pages to a network share. Had to use a RPi with saned and scanbd to listen for the scanner’s hardware buttons to be pressed. So I’m quite happy that it works after all. Seeing that it’s been 9 months or so without progress, I’m asking myself where and how we could help to get this thing finished? |
See here: |
Thanks for the heads up! |
May I leave one more idea: Use OCR and a special page (with a unique text, like, PAPERLESS-SPLIT-PAGE-PAPERLESS-SPLIT-PAGE on it), to automatically split a scan there (and remove the seperator page). My scanner is really slow on loading the scan preset, but quite fast on the actual scanning. Therefore, I would love to add a bunch of documents at once, but get them split automatically. |
@maugsburger - Paperless-NGX just released Version 1.7.0 that include the ability to use a PATCHT file for separating documents into different files. Its disabled by default, but if you read thru the documentation, it's quite easy to turn on. |
Hi, for me it's the opposite: I scan over the year bill's into single documents and in the new year, I would like to select all bill's of the last year and merge them into one big file. So I can handle them better. c-c |
As a user I would like to merge different scan into one document.
Example: I scan the front and back side of an ID card, it uploads as different documents into paperless. I can merge the 2 documents into one.
The text was updated successfully, but these errors were encountered: