IEEE Xplore title em dash cleanup

[InAnYan]

Closes JabRef#286.

I added a test to DoiFetcherTest for doi https://doi.org/10.1109/PERCOMW.2015.7133989.

The parsed title of the article should be:

Towards situation-aware adaptive workflows: SitOPT — A general purpose situation-aware workflow management system

But previously it was:

Towards situation-aware adaptive workflows: SitOPT — A general purpose situation-aware workflow management system

In order to make the cleanup, I added some code to the format method of the FieldContentFormatter. (Is that the right file for cleanup?).

Mandatory checks

• Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
• Tests created for changes (if applicable)
• Manually tested changed features in running JabRef (always required)
• Screenshots added in PR description (for UI changes)
• Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
• Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

[InAnYan]

Oops, sorry. The test should be in field format test, but not in DoiFetchTester. Moved

[InAnYan]

I've created a new formatter, but can't understand where to add it to format fields. Currently, the title for article https://doi.org/10.1109/PERCOMW.2015.7133989 is not formatted. But I've made the class and tests. Maybe I should've not added it to the Formatters class (NormalizeEnDashesFormatter is not added there too)

[Siedlerchr]

There should be a method doPostcleaup where you can add the formatter
If you want to apply for all fields, you can use InternalField. INTERNAL_ALL_FIELD

[koppor]

I think, running https://github.com/JabRef/jabref/blob/main/src/main/java/org/jabref/logic/formatter/bibtexfields/HtmlToUnicodeFormatter.java twice should be tested instead of introducing a new cleaning

[koppor]

Seeing the input, isnt that a double encoded unicode character?

Should the importer run https://github.com/JabRef/jabref/blob/main/src/main/java/org/jabref/logic/formatter/bibtexfields/HtmlToUnicodeFormatter.java twice for the title field?

[InAnYan]

But that will leave the ampersand.

I've tried to add additional HtmlToUnicodeFormatter.java to DoiFetcher.doPostCleanup. Is that the right place?

It replaced the & with ampersand, but #x2014; was left as is. So, probably there should be a separate formatter?

I read the original issue and you have provided a sketch of the solution. You said to add a new cleanup class and integrate it with IEEE class. But how to add the cleanup to the IEEE? There seems to be no doPostCleanup method. What if we just change the parseJsonResponse and directly cleanup from there?

[Siedlerchr]

For the IEEE doPostCleanup method, it's just a single method that is called manually because only the EntryBasedParserFetcher has it defined as interface method.
See thee DoiFetcher for example, it defines it manually

[InAnYan]

It seems that the JabRef doesn't use IEEE class when importing the DOI https://doi.org/10.1109/PERCOMW.2015.7133989

[InAnYan]

I've changed the class for the formatter.

Calling HtmlToUnicodeFormatter twice doesn't solve the problem completely.

And it seems IEEE class is not used for fetching https://doi.org/10.1109/PERCOMW.2015.7133989. (Maybe that's because this is an old issue, and the way JabRef works has changed?).

I've added the formatter to DoiFetcher.postCleanup. It works.

[Siedlerchr]

Pleease the failing tests here

just click on details
for openrewrite you can execute the rewriteRun task

[koppor]

This is Unicode or plain ascii? I think, we fetch in plain BibTeX, thus, it should be --, shouldn't it?

[InAnYan]

The source code is in Unicode.

In NormalizeEnDashesFormatter en dash was replaced with --, em dash should be replaced with it too or something longer?

[koppor]

Let's keep --.

Normally, it would be ---, but this is very strange in Paper titles. Never saw it. Therefore, I proposed --.

[koppor]

Calling HtmlToUnicodeFormatter twice doesn't solve the problem completely.

There needed to be a small fix. I coded it in #11091. Since the new code is much more smaller than the code proposed in this PR (and also handles more cases), I close this PR.

Thank you for digging out where to call the formatter! 👍