Add a searchable comment field as document properta

[christofkac]

I have many cases where I would like to add a comment to a document to later be able to search for parts of it.
Although the tag feature helps in many cases, it does not work in all cases, e.g. if I want to write "Document received from xxx in order to yyy".

Would this be possible to add?
Thanks
Christof

[tim-vogel]

I also find this extension very useful. I had also already tried to do the development myself. However, I have no experience with Django and Angular. I still have a few problems in the implementation. But I'm definitely trying to complete the implementation for me as an exercise.

I have provided a one to many relation for the comments, so that multiple comments can be added to a document.

If you can wait a bit more, I hope to submit the extension as a pull request.

[christofkac]

Thanks a lot for the feedback. I’m not desperate on that feature therefore I for sure am happy to wait until you completed the implementation. In case you need a beta-tester, don’t hesitate to contact me 😉 Von: tim-vogel ***@***.***> Gesendet: Dienstag, 12. April 2022 21:33 An: paperless-ngx/paperless-ngx ***@***.***> Cc: christofkac ***@***.***>; Author ***@***.***> Betreff: Re: [paperless-ngx/paperless-ngx] Add a searchable comment field as document properta (Discussion #687) I also find this extension very useful. I had also already tried to do the development myself. However, I have no experience with Django and Angular. I still have a few problems in the implementation. But I'm definitely trying to complete the implementation for me as an exercise. I have provided a one to many relation for the comments, so that multiple comments can be added to a document. If you can wait a bit more, I hope to submit the extension as a pull request. — Reply to this email directly, view it on GitHub <#687 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIDBAVVJY6QJTJIVV4BSX7LVEXFVRANCNFSM5TCYAJ4A> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tim-vogel]

Hello,
short sign of life to my attempts to implement a comment function. I had not much time for private things in the last weeks. Also I had to acquire more knowledge than expected. But it's slowly going on ;-)

[christofkac]

Looking forward to your extension ;-)

[itsdaveit]

I think this has been discussed before in #141

[torwag]

What would happen if you simply add your comment on the part of the OCR text. You could add whatever you want. The text, like all other text will be used during searches.
Only drawback, if you redo the OCR process, your manual entered text would be gone. However, I never did this and can live with that.

To extend that feature, instead of adding more and more fields for possible other use cases, maybe the current OCR text field together with some syntax (org-mode style, markdown, etc.) could be used to add whatever info one might want to add in addition and the syntax helps to differentiate better during search.

E.g.

Receiver: Tom
Sender: Mia
Subject: important document
Signage: Mr Riddle
Supervised: Mr Foo
Team: Orcs
Colour: Red
Status: Todo
Place: Office
Notes: Received from Mr Jumbo, don't forget his birthday!

OCR:
<----------- This is automatic OCR it, everything below will be replaced if the OCR process is triggered again-------->
Lorem Ipsum....

In the search field I could use:

Place: office

to indicate to look in a customized special field instead of all text

It would not clutter the GUI and people could come up with whatever system serve their needs.

[tim-vogel]

You have already listed the biggest disadvantage yourself. The use of OCR text would only be a workaround until a stable solution is implemented.

To maintain the clarity of the GUI, one could consider making the display of the comments tab optional. This way everyone can decide for themselves if they want to have it or not.

[torwag]

Yes, but my proposal would be to use one big "notes" field, which contains the OCR text but can be extended by any user to contain more than that. This give the biggest flexibility. The text field is there already, all it needs is a simple parser (which are plentiful available as a modules), a change in the ocr part to replace only the ocr text if ocr gets a redo and some modification on the search pattern system.

[tim-vogel]

I am not a professional software or UI developer. But as a user, I wouldn't want to use your kind of implementation. For me, content must be separated. Your suggestion is to mix all information. Since you could remove any info tab in the document detail and just write the text in the ocr part. One big text field for everything. Clear and organized is something else. But here every user has his own preferences. I would not want to use your suggestion. Others might. I'm trying to develop the comment function for myself, because I want to learn how Python, django and Angular work together. The topic is new to me and I see it as a good introduction to the topic.

But I will implement the comment tab display as an option. So users who don't want this feature can disable it. It is also questionable whether the project team will even accept my development and include it in the project.

[tooomm]

OCR and comments are two very different things and should not be merge in my opinion.

[dstampfer]

I'd love this feature!

[tim-vogel]

I have implemented all functions. These are now tested by good friends. After that I update peperless to 1.71 and put the code online. Interested people can do a beta test then.

If you already have customization requests based on the screenshot, please post them here.

[Carunga]

This looks great, I would surely use that - actually I was about to suggest the same idea.

[dstampfer]

Great thing! Do you also export the comments when using the exporter? It would be great to have a $DOCUMENTNAME.comments.txt exported with the comments.

[tim-vogel]

Very good hint. I will try to implement it with.

[mtettke123]

That looks really great. Would love to give it a try!

[tim-vogel]

the feedback from the alpha testers is back and there seems to be significant problems with the search function. I have therefore decided to delete all the code for the search and ask for help here again.

@shamoon
where is the correct point in the python code to search via the general search box and via the document search. If I analyzed it correctly, there are two REST endpoints for this. Once/api/search/autocomplete and once /api/documents/?page=1&page_size=50&ordering=score&query=keyword.
Do both endpoints have a standalone implementation, or is there a central code for the search?

It would be nice if someone could help me find the right entry point for the search. Thanks a lot

[torwag]

@tim-vogel: I don't want to hijack this thread. Just, as you are knee-deep into the internals. What do you think, how complex would it be to implement the saving of different versions of a file, very similar like the comments you added here. In fact, an archived version could be equivalent to a comment resp. added to a comment (when adding a comment, click on "archive current version".
The reason I ask, I sign a lot of documents and maybe annotate them otherwise. I would love to keep an "original" and a "signed"/"annotated" version, within the same entry.

[tim-vogel]

I can understand your request very well. But I suspect that the current architecture of paperless will not make your project possible without deep changes.

I would also be the wrong person to implement it. I have already struggled with the current implementation. Even the simple extension of fulltestsearch has brought me to my current limits in understanding Django.

You should open your request as a separate thread so that it can be discussed properly.

I hope to release my code this week. But this will not contain a fulltext search. Maybe another user will add it.

[tim-vogel]

Hello,

I have published a paperless version with comment function on my Github account. This version is still based on paperless 1.5, but I will adapt it to the current code state of paperless in the next days and publish it again on my Github account for testing.

By default the comment function is disabled. To enable it, the environment variable PAPERLESS_COMMENTS_ENABLED must be set to true. Whether the feature will be enabled by default in the future will be decided by the paperless project team. Also about the inclusion of the code in the project.

It was not possible for me to implement the full test search. This feature is still pending. Here the support of an experienced paperless developer is necessary.

Feature list:

• - Tab for displaying comments in the document detail view.
• - List all comments for a document in descending order (newest comments first)
• - add new comments
• - delete comments
• - full text search over comments
• - integration of comments into the standard import and export function of paperless
• - activate / deactivate the comment function (only the view is hidden or shown)
• - new service to retrieve backend environment variables.

[Solverz-0]

+1 for this, would be very helpful to have a notes field, searchable or not, it would still be great to have.

[gador]

Hi @tim-vogel ,

this looks really good! I would suggest you rebase your code not on 1.7.1, but on current dev. Only bugfixes and documentation should go into a current or an old version.

[tim-vogel]

The implementation for paperless ngx 1.7.1 is coming. I currently have very little time to finish it.

[mbay-ODW]

Any news here? I really would love to see that feature as well.

[tim-vogel]

I have just submitted the pull request for the comment function. (#1375)

[tim-vogel]

In the pull request (#1375) there is currently a discussion about where the input field for new comments should be placed. Above the comment list or below it. The more opinions are given, the better the final solution will represent the wishes of the user community. So please give your opinion on this. Thank you!

[efelon]

I have added my thumbs up in the pull request. Just for reference here:

Definitely +1 for entering a new comment at the top and sorting comments newest (top) to oldest (bottom)