ris import not working - required fields

[glorious1]

ABT Version: 4.11.3

PHP Version: 5.6

Theme: GeneratePress

Plugin List:

• Loginalyzer

Browser: Firefox 57

Expected behavior: Export RIS file from EndNote, then import into ABT.

Actual behavior: Error when selecting file: "Error!
The selected file could not be processed". I will attach the RIS file (changed extension to txt), maybe you'll see what's wrong with it. Thanks!
ff.txt

Console messages:

// Paste messages here

[dsifford]

Thanks for the report.

Issue is likely with https://github.com/dsifford/astrocite , but since that's also one of my projects, we can keep it here.

Will check on this later on. The file you attached appears to be a valid RIS file from just scanning over visually, so not sure exactly what could be causing this.

[dsifford]

Related: #384

[glorious1]

As a test, I went to PUBMED and downloaded a file with two random references. It came as type .nbib, but it looks like a ris file. (Maybe I need to get PUBMED to send it some other way?) I changed the type to .ris and tried to import - got same error. PUBMED file attached.
citations.txt

Thanks!

[dsifford]

Yeah, although nbib and ris appear to be of an identical format, the fields that they use to represent data are not the same. Thus, they are not interchangeable.

The best way to get references from PubMed is to just paste a list of PMIDs into the "Add Reference" dialog.

So, for example...

If you wanted to cite the references from your citations.txt file, paste this: 29053945, 28782269

[glorious1]

Thanks, but I wasn't actually trying to get references from PubMed, I was just trying to get a known valid RIS file to test importing. The wiki mentions PubMed as a source of RIS files.

[dsifford]

Whoops! That's an error. Sorry for the confusion.

See here for a few known valid RIS files: https://github.com/dsifford/astrocite/tree/master/packages/astrocite-ris/src/__tests__/cases

[glorious1]

Hmm, couldn't figure out how to download one of those files, but I copied the contents of long.ris and pasted it into a new text document, saved it with encoding utf-8, and got the same error when I tried to import into ABT.

Then I went to a different site and did the same with sample.ris, and it worked. I have no idea what the controlling difference is. I'll attach sample.ris and my long.ris.
sample (works).txt
long (doesnt work).txt

[glorious1]

This just gets stranger. I have sample.ris, which works, and my.ris, which doesn't. If I open my.ris with NotePad or WordPad, delete the contents, and paste one of the refs from sample.ris and save, I STILL can't get my.ris to work. So seems it has nothing to do with the content, but some invisible property of the file.

[dsifford]

@glorious1 Windows computer?

If so, it's likely an issue with windows line endings. Windows ends lines with both a carriage return and line feed character (i.e. <CRLF>), whereas all other systems end their lines with a single line feed character only (i.e. <LF>).

This is probably what's going on. I don't have a windows computer so I would have never caught it! Thanks for reporting.

[glorious1]

I took the two files home and looked at them on my Mac with Textwrangler. You're right, the one that doesn't work has Windows CRLF, but it also has encoding "UTF-8 with BOM" (no idea what that is). The one that works has just UTF-8. When I changed the encoding to UTF-8, it worked!

That's leaving the Windows line ending, so apparently either kind of line ending works. Sorry to take your time. I LOVE the plug-in, just left a nice review. Thank you.

[dsifford]

Aha.. Thanks for reporting back with that info. Hadn't considered the BOM character. That's probably the ticket for sure.. I'll investigate as soon as I get some free time.

Thanks for your kindness and patience. Glad you're enjoying the plugin. Thanks for taking the time to submit a review as well. I greatly appreciate it!

[glorious1]

Sigh . . . sorry to add to the confusion. I had the two files confused and imported the wrong one. You were right the first time — the CRLF is what prevents import. The presence of the BOM has no effect.

I think I once found a way to convert CRLF to LF at work; just have to remember what that was . . .
Jim

[dsifford]

Got it. Thanks for clarifying.

Re: How to convert from CRLF to LF -- most text editors nowadays have that capability. (Atom, Visual Studio Code, Notepad++, etc)

[glorious1]

Just FYI, I noticed one other, possibly related thing with the RIS files. The online data sources (like Web of Science) often have fields that have mutliple lines in them (like author's addresses, or keywords that are each on a separate line). When EndNote exports such references as a RIS file, it cannot be imported by ABT. If I edit the file so there is only one line per field (even if it wraps), ABT imports it fine.

[dsifford]

Good to know, thanks. I'll try to look into both issues soon.

[mjluser1]

This plugin looks to be very useful. However, I am getting the same error message as user glorious1 on .ris file exported from JabRef. This occurs even when I reduce the file to only one citation and removed all carriage returns. I have no idea what to try next. Please advise, and thanks.
1222onereference.txt

[glorious1]

There are a couple of things wrong with that file. The first two lines (and blank line after) look like some kind of header JabRef puts in, not a reference. I removed them. The ER - line has to be after the reference, not before, so moved that. Finally for some reason the blank line in the middle of the reference was a problem, removed. Then it imported fine.

[dsifford]

Thanks @glorious1

I can probably add lenience to references having blank lines separating fields, however (as @glorious1 already reported) every TY field that begins a reference must be ended with an ER field. So, even with the lenience, the RIS file you shared would still not work as it does not follow RIS specifications.

[glorious1]

Yes, allowing blank lines would be nice. I think both of the reference managers I've used (EndNote and Bookends) will export blank lines if a reference doesn't have data for a given field. I talked to Bookends support and they said blank lines in RIS file are perfectly acceptable.
Thanks!

[dsifford]

@glorious1 Just to be sure, can you confirm that blanks are currently not supported? I can't recall if I originally added support for that or not. It's been a while since I wrote it!

[glorious1]

I just tested again to be sure. It looks like blank lines after an "ER -" record before the next reference are OK, but any blank lines inside a reference seem to prevent it from importing.

[dsifford]

Noted, thanks. I'll fix that soon.