Issues with search and groups and associated feature request (floating mode)

[AEgit]

Fixes #9073

JabRef 5.0-dev--snapshot--2018-07-25--master--077fdacc2
Windows 10 10.0 amd64
Java 1.8.0_181

Steps to reproduce:

1. Select the group "All entries" in the groups panel
2. Use the article search (not the groups search/filter!) to find an article
3. Articles that match the search term are found
4. Select another group in the groups panel, that does not contain any of the articles that match the search term of (2)
5. Carry out another search using the search term of (2)
6. No articles at all are displayed in the maintable, despite the fact we know, that they exist (they just don't belong to the selected group)

This behaviour is probably not intended and is probably related to the fact that the floating mode was removed in newer JabRef versions.
To find an article I will now always have to select either "All entries" or the group that contains the article. If I have selected another group, the respective article will not be shown in maintable view. Especially in large databases with many groups this can be very inconvenient.

Therefore I propose to enable the floating mode again for the maintable. This way, if an article matches the search terms it will be always shown. If it belongs to the group, that has been selected, it is shown in the usual colours in the maintable. If it does not belong to the selected group, it should appear darker in the maintable like in previous JabRef versions.

[Siedlerchr]

In addition there's a toggle button for group union and intersection. I just don't know if it has influence. The floating mode is on the bucket list but as far as I remember some technical problems need to be solved before

[AEgit]

JabRef 5.0-dev--snapshot--2018-10-28--master--05047f32a
Windows 10 10.0 amd64
Java 1.8.0_191

I have seen that the current development snapshot contains a "Groups"-specific preference (Options -> Preferences -> Groups), which should address this problem. The following options are given:

"Hide non-hits" vs. "Gray our non-hits"

"Display only entries belonging to all selected groups" vs.
"Display all entries belonging to one or more of the selected groups"

However, changes made to these options do not change the behaviour of JabRef. I just wanted to let you know about this, since I was not sure, whether these options are currently just not implemented (i. e. the preferences are just place holder) or whether this is a bug.

Also note, that it is currently possible to set preferences that are mutually exclusive, i. e. you can select both "Hide non-hits" and "Gray out non-hits".

[tobiasdiez]

Thanks @AEgit for going through the bugs related to groups and check if they still apply to the most recent dev version. This is a huge help!

[AEgit]

Yes, sorry, for not having been active in a while - I was just so busy (and I am right now for the next couple of months), that I could not test and help more.

Thanks again for developing this amazing piece of software!

[AEgit]

Note, that the floating mode is still missing in:

JabRef 5.0-dev--snapshot--2019-03-15--master--7ed15bc4a
Windows 10 10.0 amd64
Java 1.8.0_201

So this issue is still relevant with the current development version.

[AEgit]

Links to information on how to implement the float mode using JavaFX (which I heavily rely on - one of the reasons I currently still have to stick to version 3.8.2) are found here:
#3621 (comment)

[AEgit]

As far as I can tell one of the reasons as to why the floating mode still has not been implemented is due to this issue:

https://bugs.openjdk.java.net/browse/JDK-8091326

The last comment in that report is from 2014 (!). Is there a way to speed up that feature requirement and/or can JabRef maybe get around this problem in JavaFX, e. g., by using this approach:
https://stackoverflow.com/questions/37768326/group-by-function-in-javafx-table
(mentioned in #3621 (comment)).

I would really like to see the floating mode. To be honest, I am not sure, how people can use the current version of JabRef without the floating mode IF their database also contains groups and IF they use the search feature. This is not meant as a criticism but as a genuine question: Maybe I misunderstand the current JabRef implementation and there is a different way of using it, that does not required the floating mode (?). But I am lost, as to how to circumvent the problem described in initial post of this thread with the current version of JabRef. Am I really the only one that has this problem? You would think there is someone else using the current JabRef version whose database contains groups and who uses the article search feature (?).

[AEgit]

Trying to push this issue again. I would really like to see the return of the floating mode. Based on questions in the JabRef forum (https://discourse.jabref.org/t/maintain-entry-selection-when-switching-groups/1932) I do not seem to be the only one, that needs the floating mode.

Edit: Could this link maybe help with the implementation?
https://stackoverflow.com/questions/56267183/can-i-group-rows-in-a-javafx-tableview

[AEgit]

JabRef 5.3--2021-01-24--ae43548
Windows 10 10.0 amd64
Java 14.0.2
JavaFX 15.0.1+1

This is still a (major) problem in the current developer version. Note, that it is impossible for me to switch to version 5 before the floating mode is added (unless I misunderstand how JabRef is meant to be used without a floating mode - see here: #4237 (comment) - if that is the case, please let me know, what I should do). For this reason I still have to use version 3.8.2 for my actual work.

[sauliusg]

Therefore I propose to enable the floating mode again for the maintable. This way, if an article matches the search terms it will be always shown. If it belongs to the group, that has been selected, it is shown in the usual colours in the maintable. If it does not belong to the selected group, it should appear darker in the maintable like in previous JabRef versions.

I can confirm that the issue exists in the current development version and I support suggestion of @AEgit to restore the "darker display for other items" which was very useful for me as well.

[AEgit]

Here again my request to address this problem (missing floating mode). As long, as this issue is not solved, I cannot switch from JabRef 3.8.2 to JabRef 5. Since JabFox requires JabRef 5, this is quite unfortunate and makes working with JabRef much slower for me.

As far as I understand the links mentioned here can help in solving this problem:
#4237 (comment)
#4237 (comment)
#4237 (comment)

I note, that the MainTable JavaFX migration has been marked as "DONE":
#3621
However, as you can see in the initial post, there are still a couple of known (!) bugs. Obviously, I cannot force the developers to devote their time to the problems that I think deserve most attention. But, at least from my user perspective, I would prefer if they were to focus on these bugs and regressions relative to previous versions rather than implement new features.
Again, this is not meant as an angry critique - I really like JabRef. But, unfortunately, I need this feature to be able to work with JabRef properly. Or is there another way I should use JabRef to overcome this problem (see comment here: #4237 (comment))?

[ThiloteE]

The following statements are based on the most current version of JabRef.
(JabRef 5.4--2021-08-15--96061b7
Windows 10 10.0 amd64
Java 16.0.2
JavaFX 16+8)

I would like to argue that the problem description as described in the first post is currently not a bug, but a feature. Changing this now would be one step forward, but one step back.

• As far as i am aware, there is no other way to search for entries within a specific group other than to first select the group and then search.
• I personally use this quite a bit. Sometimes i think of groups as separate projects and a search REALLY is just supposed to bring forth the entries within that specific group.
• Whereas there exist search commands like author=x and title=y there currently exists no search command for group=z

Notwithstanding i think what you had in mind @AEgit seems like a valid enhancement of JabRef.
I think there are more options other than floating mode though.
The same outcome could be achieved IF there were to be a rework of the search bar. Your feature-request is named perfectly for that xD

I personally am a big fan of the Thunderbird search options. They do not have one searchbar but two! AND they do have a separate window for additional more precise search operations. Please note how they made the shortcut for the search bars (e.g. strg+shift+k) visible in light grey. See here:

To be honest, having two search bars sounds not all too hard to implement. Correct me, if i am wrong.

• Adding another one next to or beneath the existing search-bar could be sufficient. I think there could be just enough space if the other shrinks a little, although i also really like it to be long :D
• For users to not get confused with too many search bars, they should be named adequately and the name has to show at first glance. The name should not be hidden. I am thinking of something along the lines of: (1) Search... (2) Search with Options (3) Search for a group
• To be honest i don't think the name of "Filter Groups" reflects is usage all too well, because it as a matter of fact does NOT filter groups for entries or other fields, but just searches for a specific group.

The following screenshot shows the search-bar that could be modified.

Another source of inspiration could be Citavi. Their standard settings are just like JabRefs, but they offer very detailed additional search options:

[ThiloteE]

Turns out Citavi has even more additional search options:

(I post this not as advertisement, but as inspiration for a possible solution to our problems.)

[AEgit]

If adding an additional search bar solves the problem, I am fine with that as well. Though note, that old versions of JabRef (e.g., 3.8.2) had a floating mode which worked perfectly. Also note, that there is already a search command for groups (which in my case usually do NOT represent separate projects), namely "groups" (but maybe that does not do, what you want?).

The advantage of the floating mode was, that you could search for any item you wanted in JabRef, no matter which group was selected. If you had a group selected, that did not contain the items you searched for, that would give you additional information. The items, that would match the search request, would still be found, but would be listed in grey colour in the main table view. This would indicate immediately, that the respective item was not part of the group you selected.
I imagine people with small databases do not realise the advantage this brings - but with databases that contain thousands of groups and tens of thousands of items, you REALLY want this feature.

Anyway, as long as this issue is not solved, I cannot switch to JabRef 5.

[Siedlerchr]

@AEgit have you tried this new search results in external window?

[AEgit]

@Siedlerchr The search results in the external window are better, but they do not solve the problem.

I still don't know, whether an item belongs to a group or not (as indicated by the gray colour in the old floating mode) when doing a search. Furthermore, and probably even more important - I cannot drag an drop items found with the external window to a new group.
That was the big advantage of the floating mode - you would search for something, then select a group of interest, then realise that this respective item had not been added to that group, and then just drag and drop this item to the group of interest.
Currently this wokflow is impossible with JabRef 5 - and it is a major problem with big databases.

[AEgit]

@Siedlerchr Furthermore, could you please re-open the old issues I went through just today (where the issue persists)? Sorry, I was inactive for a while (too much to do), but I had time again now to check old issues which were closed by the bot. Nearly all of them still persist.

[tobiasdiez]

What you describe above is possible, at least to some degree. You can assign colors to groups now, and these colors are displayed in front of the entry in the main table; and you can sort the maintable by the groups the entry belong to. Thus, you can assign different colors to your groups (e.g. only temporary if you like) and then search as before.

Another idea would be to implement searching for group membership...

[AEgit]

Thanks for the quick reply, but I am afraid, this again is a solution that does not work on large databases. If you have thousands of groups it is not feasible to assign colours to each group (let alone distinguish these colours).

[AEgit]

To further expand on this - the usercase is basically a database with thousand of entries and groups.

You are searching for a keyword which you know/think is contained in one of your entries/items. And you think that some (NOT ALL!) of the respective entries should be assigned to a certain group. If they are not, you want to be able to assign these items (that appear in your global search) to that particular group.

This workflow was possible with the old JabRef (thanks to the floating mode) - it is not, with the new implementation.

[Siedlerchr]

Just to be clear. We would have implemented this already if it was easily possible. The current Filter Architecture works as follows: We can only filter the data source of the table, not really the rows in the table itself. I personally, see no easy way to achieve this. And unfortunately the Table Control has is a bit of an neglected child of the JavaFX people...

[AEgit]

Thanks for the quick reply again - hmm, do you think this will ever be implemented in JabRef 5? Because if not, I will probably seriously have to consider to move to a different reference manager - which would be a shame since otherwise I really like JabRef. But I reckon version 3.8.2 will not work forever and I already have to live with some limitations of that version (e.g, the JabFox plugin does not work with that old version anymore, which makes adding new articles quite time-consuming).
Again, I really appreciate all the work that is being put into JabRef and if it was possible to get that feature back, that would be great. But if you can categorically exclude this possibility, then I fear have to think at least how I will manage my references in the future.

What I still wonder is, whether my user case is really that niche. I might try to upload an example video using the old version (to show what was possible and what version 5 is missing at the moment). Maybe I am just really missing something elementary and there is a different way to achieve the same in JabRef 5.

Thanks again for your help!

[tobiasdiez]

I think it will still take some time (read years) until the UI framework that we use supports something like the old floating mode, if it ever will be. So, yes, for your own interest please do evaluate other options and see what works best for you. Of course, we would be very sad to loose such an active and helpful member of the community. But in the end we develop JabRef to support users doing their tasks more effectively and its only normal that other tools and features work better for some people; and also that the direction of development does not always align with all users' needs and wishes.

That being said there are many things that we can do change without that much work. Some ideas that might work for your use case:

• Add the possibility to invert the group membership filtering, i.e. only show entries that do not belong to the given group.
• Add the possibility to filter by groups via the search string, so that title='jabref' and not in 'reference manager' would show only entries that have jabref as title but that are not in the group 'reference manager'. @koppor would that be easy or hard with Lucene?

What do you think? Maybe its an idea if you think how you would do this task in an ideal world (but without the floating mode).

---

We might also be able to emulate the float mode without support by javafx. Some ideas for this:

1. When in "float mode", don't filter by group here
   

   jabref/src/main/java/org/jabref/gui/maintable/MainTableDataModel.java

   Lines 46 to 49 in 9c5b8ab

   	entriesFiltered = new FilteredList<>(entriesViewModel);
   	entriesFiltered.predicateProperty().bind(
   	EasyBind.combine(stateManager.activeGroupProperty(), stateManager.activeSearchQueryProperty(), (groups, query) -> entry -> isMatched(groups, query, entry))
   	);

   
   by using something like

EasyBind.combine(stateManager.activeGroupProperty(), stateManager.activeSearchQueryProperty(), stateManager.isFloatMode, (groups, query, isFloatMode) -> entry -> isMatched(groups only if !isFloatMode, query, entry))

2. Apply a pseudo-class to the table row when its not matched by the group. Code: https://stackoverflow.com/a/41239849/873661
3. Add different styling on that pseudo-class to say reduce the opacity.

That would give some version of the old floating mode, but without grouping entries in the maintable in "matched" vs "unmatched".

[AEgit]

@tobiasdiez Thank you for your open words! Hmm, I will have to think what to do then.

As for the proposals made: Inverting group membership (or filtering the groups the way you described) could potentially already be an advantage - it is still not as convenient, as the original floating mode, but it would be a step forward.

I attach here a very simple example video of what is possible with JabRef 3.8.2 using the floating mode which (as far as I can tell) is not possible at the moment with JabRef 5:

Jabref.382.Floating.Mode.Usage.Example.mp4

(1) I search for a term,
(2) then select the group which contains the term,
(3) then see which items belong to the respective groups and which items do NOT belong to the group (but contain the search term!),
(4) and then add one of the items containing the search term (but not belonging to the group) to the respective group.

Now imagine having to do this multiple items on a database with thousands of entries and groups (i. e. not all groups are visible at the same time in the groups panel - neither are the items). Is there a "JabRef 5" way of doing things that I am missing to replicate such a task. If not, am I really the only one having this problem :D?

Thank you for your help!

[AEgit]

I am glad to see that this issue has been added to the GSoC project and I am hopeful that the problem can be fixed by using Lucene as suggested here #8206 (comment)

[AEgit]

Just as a quick update: I have noted that for the Lucene search approach (#8963 (comment)), the following solution has been suggested to the problem:

Use gray background for score==0. See #4237 (comment).

I am not familiar with the technicalities of Lucene, but I reckon a score of 0 represents an item, that does not match the search criteria at all. That is not (!) the same as what the old floating mode in JabRef 3.8.2 offered - the items in gray did match the search criteria to a certain degree. The only difference between the items with white background and those with gray background was, that the white ones belonged to the group, that the user had selected at the moment. The gray ones did not (!) belong to the selected group.

Hopefully, this old behaviour of JabRef can be restored with the new Lucene search.