Refresh journal lists periodically

[koppor]

Before a release, the journal lists have to be updated manually. With this PR, that is happening periodically. Similar to our CSL files (refs #5718)

[tobiasdiez]

What's actually the reason of having them in an extra repository, and not here in the main one?

[koppor]

I stumbled upon this while reading our howto for releasing a new version:

Currently, we collect the data at https://abbrv.jabref.org/. We split the different lists up to fields:

For instance, we have a script for fetching the latest abbreviations from https://mathscinet.ams.org. See https://github.com/JabRef/abbrv.jabref.org/blob/master/update_mathscinet.py for details.

The split of of the lists helps

• users contributing abbreviations to their field of knowledge
• including separate lists of different fields (e.g., MathSciNet)
• ease re-using the data in other tools (somehow refs Extract libraries out of JabRef ("Toolkit") / Offer JabRef as library / jar / maven central / JabKit #110)

What's changed in journalLists.txt

(Didn't you ask that - I just made the effort. Mhh...)

• journalLists.txt: v5.0-alpha and before merge of [WIP] Convert abbreviation data to CSV and adapt JabRef accordingly #5538: 15206 lines
• journalLists.csv: right after merge of [WIP] Convert abbreviation data to CSV and adapt JabRef accordingly #5538: 15205 lines
• combined journal lists in JabRef/abbrv.jabref.org@85db72bdd0e34ff2c7: 117575
• combined journal lists in master in https://github.com/JabRef/abbrv.jabref.org/: 117543
• in this PR we also have 117543 lists. That makes 102.337 more (!) entries.

TODO:

Backport PRs on journalLists.csv / check whether they are included in abbrv.jabref.org

• check 38afd66
• check 2ca0084
• check 40baaa4
• port Update IEEEJournalList #2579 to abbrv.jabref.org
• port Linux: ad a file to an article -> JabRef freezes #3301 to abbrv.jabref.org
• check Extend list of journal abbreviations by a few math journals #4376
• check Add a few journal abbreviations #4412
• check Correct PNAS abbrev. #4524
• check Wrong journal name "Astronomy Journal" should be "Astronomical Journal" #4529
• check f1bbbb7
• check Freeze when double click entry to edit entry 5.0DEV #4462

[tobiasdiez]

What could also work is to include the abbreviation files directly here in this rep, and then run the combine script as part of the build progress. Might reduce the management overhead concerning backporting.

Did you checked the performance and memory consumption of the current version? I'm a bit worried that 100k new abbreviations take their toll.

[koppor]

I think, maintenance in a different repo is easier for users. Not for us, but I think, there are more potential users and we can get along maintaining it at two places. I agree though that JabRef should be more intelligent: - keep list separate to allow users to relate the available lists with the list in Jabef - JabRef should recommend a list automatically. In case I open a bib, JabRef switches the abbreviation list to the related one. In case there is none set, auto-select the best fitting. As an intermediate solution, I would get rid off the combined list and just import all lists. This combination is a left-over from the early days of JabRef. Performance is unchecked. Need to do. Tobias Diez <notifications@github.com> schrieb am Mo., 16. Dez. 2019, 22:50:

…

What could also work is to include the abbreviation files directly here in this rep, and then run the combine script as part of the build progress. Did you checked the performance and memory consumption of the current version? I'm a bit worried that 100k new abbreviations take their toll. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5749?email_source=notifications&email_token=AAKNU7WVPA7NBZD6AKUMCYTQY7Z2LA5CNFSM4J3FAWNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHAHBPA#issuecomment-566259900>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKNU7QLONT4FJXAMO2UUR3QY7Z2LANCNFSM4J3FAWNA> .

[tobiasdiez]

Do we have that many contributions to the abbreviation list? I thought it was something like twice per year...

[koppor]

Even though, it is not that often, I find the homepage https://abbrv.jabref.org more welcoming than a sub directory in the JabRef source with a README.md explaining the contribution. The target group of contributors to JabRef is different from the group contributing to abbrv.jabref.org. The former are people knowing how to code, the latter are just users of JabRef with some IT skills.

Maybe, it is no issue for the second group to overlook that they don't need to setup a workspace, they don't need to run tests, ... (see https://github.com/JabRef/jabref/blob/master/CONTRIBUTING.md and https://github.com/JabRef/jabref/blob/master/.github/PULL_REQUEST_TEMPLATE.md).

I would keep the abbrv repo separate to adress the other target group.

Maybe, I should do a study using the GenderMag method by @mendezc1 to more scientifically undermine (or reject) my argument. WDYT @igorsteinmacher?

[koppor]

Performance test is not necessary any more (#5769)

I just checked the performance and think the list is too big. Memory increased from about 400mb to 1gb after abbreviating one journal title.

I wonder about the quality - need to find out when the files diverged. We had the requirement create the list based on our abbreviation list.

Moreover, the quality of the data seemed way worse than it was before. A lot of abbreviations don't have dots anymore.

[github-actions]

Please do not update journalList.csv directly. Please update the list at https://github.com/JabRef/abbrv.jabref.org/tree/master/journals

[jlaehne]

I modified combine_journal_lists.py to separately merge each of the lists on abbrv.jabref with the combined 'journalList.csv'. Checking how much the list increases gives a good idea, where all the extra entries come from:

journals/journal_abbreviations_acs.csv: 1717
Combined key count: 15531, Not in journalList.csv: 326

journals/journal_abbreviations_ams.csv: 2547
Combined key count: 17366, Not in journalList.csv: 2161

journals/journal_abbreviations_annee-philologique.csv: 848
Combined key count: 16047, Not in journalList.csv: 842

journals/journal_abbreviations_dainst.csv: 1590
Combined key count: 16782, Not in journalList.csv: 1577

journals/journal_abbreviations_entrez.csv: 19506
Combined key count: 33601, Not in journalList.csv: 18396

journals/journal_abbreviations_geology_physics.csv: 681
Combined key count: 15273, Not in journalList.csv: 68

journals/journal_abbreviations_ieee.csv: 262
Combined key count: 15463, Not in journalList.csv: 258

journals/journal_abbreviations_lifescience.csv: 9744
Combined key count: 15238, Not in journalList.csv: 33

journals/journal_abbreviations_mathematics.csv: 3312
Combined key count: 18281, Not in journalList.csv: 3076

journals/journal_abbreviations_mechanical.csv: 4759
Combined key count: 15478, Not in journalList.csv: 273

journals/journal_abbreviations_medicus.csv: 3168
Combined key count: 15970, Not in journalList.csv: 765

journals/journal_abbreviations_meteorology.csv: 119
Combined key count: 15205, Not in journalList.csv: 0

journals/journal_abbreviations_sociology.csv: 94
Combined key count: 15290, Not in journalList.csv: 85

journals/journal_abbreviations_webofscience-dots.csv: 87224
Combined key count: 91085, Not in journalList.csv: 75880

journals/journal_abbreviations_webofscience.csv: 87225
Combined key count: 91086, Not in journalList.csv: 75881

journals/journal_abbreviations_general.csv: 4538
Combined key count: 16927, Not in journalList.csv: 1722

From this, ams, annee-philologique, dainst, entrez, ieee, mathematics, sociology and webofscience were probably not part of the repository when journalList.csv was created.

However, also acs, mechanical, medicus, general have several hundred entries added after the lists were last merged.

The biggest increases come from entrez.csv and the webofscience files. Leaving them out gives still a moderate database size (instead of the 15205 entries currently in journalList.csv):
Combined key count: 24606

However, merging all but these three files with journalList.csv gives
Combined key count: 25492
So also journalList.csv still contains about 900 abbreviations which were probably added after the last merge.

[jlaehne]

The huge size of the webofscience files can be probably explained by the fact that it contains A LOT of conference proceedings, where several conference names lead to the same abbreviation. The most striking case is P. Soc. Photo-opt. Ins., which is the abbreviation for 6044 entries! (See retorquere/zotero-better-bibtex#1436 (comment))

[jlaehne]

Another issue is that abbr.jabref combines both lists with dots in the abbreviation and some lists without dots. I don't think it makes sense to merge them into a single journalList, as you normally want either, but not a mixed style of abbreviations.

So it could make sense to maintain two journalLists in the future.

[jlaehne]

To backport the changes to journalList.csv, I would propose to find out the about 900 entries that are not in any of the other lists with dots and add these entries to general.csv using a python script.

Afterwards, I would propose to maintain two lists
a) without dots, merging entrez, index-medicus
Combined key count: 22451
(+ webofscience: 101891)
b) with dots, merging general, ams, acs, geology-physics, lifescience, mathematics, mechanical, meteorology, sociology
Combined key count: 21627
(+ webofscience: 97024)

If you don't want to have too huge indices due to the memory issue, leave out the webofscience lists and people can load them separately if they want this huge extra chunk.

ieee seems to be added to JabRef separately anyway.

The external lists astronomy (very small anyway) and economics are not in the csv format yet.

daist, annee-philologique use a non-ISO abbreviation style without dots and spaces or largely even just a string of capitalized letters. I addedd a PR for a comment to the readme.md.

[jlaehne]

Another note on the webofscience lists. They contain some odd variations of journal names and with their size are hard to maintain in a clean state. Particularly, capitalization is wrong on abbreviations such as AIP, IOP, IEEE, ..., which are Aip, Iop, Ieee, .... Also e.g. Journal of Physics D: Applied Physics becomes Journal of Physics D-applied Physics in the webofscience lists.

I guess this behaviour is due to the ISI lists being fully capitalized, which was then translated using a first letter capitalized scheme (and I don't see a better way unless they provide an interface to get the non-capitalized lists).

So I would argue in favour of keeping them as optional files that are not integrated into the standard journal list(s). Edit: Maybe even add a note on abbrv.jabref site to mention the caveats of these lists!

[jlaehne]

Do you know this project: https://github.com/marcinwrochna/abbrevIso
With frontend: https://marcinwrochna.github.io/abbrevIso/
And API: https://tools.wmflabs.org/abbreviso/

It takes the official list of ISO4 abbreviations of single words, plus the general rules defined in the ISO4 specifications to deduce the abbreviation for any journal name you input.

Could be an alternative or complementary (when missing in the lists) approach to abbreviate journal names. But of course, it does not handle unabbreviation, for which there is no alternative to lists. It can also be a way to check the consistency of existing lists and it might make sense to link to the frontend on the abbrv.jabref website, so that people who want to add abbreviations can check for the correct one.

[koppor]

I need to dive into the comments and the delta between the journal lists here and the other repo. Need to think whether we need a combined journal list. Did not have the time formit yet.

[koppor]

Still need to dive into the comments. Complicated stuff, needs more than one hour concentration.

[koppor]

We need to work on this for the next release as this (somehow) blocks contributors using IntelliJ 2020.2

• because each gradle build takes a long time
• because IntelliJ 2020.2 works with gradle only (see Refine IntelliJ howto to enable built-in compiler again #6794)

[koppor]

Thanks to the work by @jlaehne, this was "just" to implement the steps described by @tobiasdiez.