Citavi Importer should import comments and quotations (knowledge items)

[Siedlerchr]

Requested in the forum:

Is your suggestion for improvement related to a problem? Please describe.
The citavi importer in JabRef does not import comments and quotations and those are also not exported by citavi when chosing bib

Describe the solution you'd like
I would like to have comments in citavi displayed as comments in JabRef and maybe

Additional context
https://discourse.jabref.org/t/citavi-data-migrate-to-jabref-including-all-notes-qutotations/3081/13

** Implementation**
The data is stored in the knowledge item xml tag.

Zotero imports them and merges them together:
https://github.com/zotero/translators/blob/1d1ce6111ac17a4064c1d8836f6beec54404c32b/Citavi%205%20XML.js#L206-L231

Implementation details

 <KnowledgeItem id="84c2aab6-1ac9-4de1-81c6-8d4a21bb148c">
      <CreatedBy>s</CreatedBy>
      <CreatedOn>2017-05-17T12:07:51</CreatedOn>
      <ModifiedBy>s</ModifiedBy>
      <ModifiedOn>2017-05-17T12:07:51</ModifiedOn>
      <CoreStatement>Summary of NGO definition and its problems (NGOisation, exclusion)</CoreStatement>
      <CoreStatementUpdateType>1</CoreStatementUpdateType>
      <KnowledgeItemType>0</KnowledgeItemType>
      <PageRange><![CDATA[<sp>
  <n>153</n>
  <in>true</in>
  <os>153</os>
  <ps>153</ps>
</sp>
<ep>
  <n>154</n>
  <in>true</in>
  <os>154</os>
  <ps>154</ps>
</ep>
<os>153-154</os>]]></PageRange>
      <PageRangeNumber>153</PageRangeNumber>
      <PageRangeNumeralSystem>30</PageRangeNumeralSystem>
      <QuotationIndex>8</QuotationIndex>
      <QuotationType>1</QuotationType>
      <ReferenceID>c2d35798-c0e2-48b4-9d4d-534daad53de0</ReferenceID>
      <Relevance>0</Relevance>
      <TextSourceTextFormat>1</TextSourceTextFormat>
      <Text>"In everyday English usage, the term ‘NGO’ connotes a non-governmental, non-profit, cause-driven association motivated by altruistic intent rather than pecuniary or political gain. NGOs are the good guys. However, scholars who scrutinized the habits of North–South NGO funding already warned us not to think that ‘non-governmental’ is synonymous with ‘democratic.’6 Many scholars and activists have critically interrogated prototypes of NGOs or CSOs as units of civic engagement: this construc- tion, they argue, constitutes a potentially potent classificatory scheme that excludes mass or spontaneous mobilization. One perceptive critic of devel- opment aid argued that civil society promotion amounts to the profession- alization and institutionalization of certain patterns of knowledge.7 The construct of the NGO, originally meant to distinguish independent advocacy from inter-governmental transactions, is problematic.8 The//governmental/non-governmental binary is a convenient dummy variable defining civic energies aphophatically for what they are not. Inside the industry there are a range of rhyming variations distinguishing para- statal qua-NGOs, government-organized GONGOs, royally-organized RONGOs, donor-oriented DONGOs, World Bank initiated BINGOs, and even entrepreneurial B-Y-O (bring-your-own) “bringos.”9 But they all call themselves NGOs, even in Arabic, where in lieu of translation the identical acronym is often rendered in text in Latin alphabet letters or in speech as pronounced in English. Overall, the ubiquitous neologism of the NGO is an imprecise linguistic expression that is left, as we will see, for various governments and donor agencies to define bureaucratically and ideologically. In practice, an NGO is something registered as such with national governments, the United Nations, or donor agencies. In compar- ing and contrasting criteria for inclusion and patronage, this chapter illustrates the politics and paradoxes of ‘NGOization.’ Many observers deduced that political aid stimulated a proliferation of professional, rather than grassroots, NGOs."</Text>
      <TextIsComplex>false</TextIsComplex>
      <TextFormatted>{\rtf1\ansi\ansicpg1252\uc0\stshfdbch0\stshfloch0\stshfhich0\stshfbi0\deff0\adeff0{\fonttbl{\f0\froman\fcharset0 Times New Roman;}{\f1\fnil\fcharset0 Segoe UI&#xD;
;}}{\colortbl;}{\stylesheet{\s0\snext0\fi0\sb0\sa120\ltrpar\li0\lin0\ri0\rin0\ql\faauto\rtlch\afs20&#xD;
\ltrch\fs18\f1 Normal;}{\s1\snext0\keepn\fi0\sb240\sa480\outlinelevel0\ltrpar\li0\lin0\ri0\rin0\ql\faauto\rtlch\afs20\ltrch\b\fs28\f1 heading 1;}{\*\cs10\additive Default Paragraph Font;}}{\*\generator Aspose.Words for .NET 11.11.0.0;}{\info\version0\edmins0\nofpages0\nofwords0\nofchars0\nofcharsws0}{\mmathPr\mbrkBin0\mbrkBinSub0\mdefJc1\mdispDef1\minterSp0\mintLim0\mintraSp0\mlMargin0\mmathFont0\mnaryLim1\mpostSp0\mpreSp0\mrMargin0\msmallFrac0\mwrapIndent1440\mwrapRight0}&#xD;
\deflang2057\deflangfe2052\adeflang1025\jexpand\showxmlerrors1\validatexml1\viewscale100\fet0\ftnbj\aenddoc\aftnnrlc\nogrowautofit\htmautsp\noultrlspc\nouicompat\nofeaturethrottle1\formshade\dghspace180\dgvspace180\dghorigin1800\dgvorigin1440\dghshow1\dgvshow1&#xD;
\dgmargin\pgbrdrhead\pgbrdrfoot\sectd\pgndec\sbkpage\pgwsxn11906\pghsxn16838\marglsxn1800\margrsxn1800\margtsxn1440\margbsxn1440\headery708\footery708\ltrsect\pard\plain\itap0\s0\fi0\sb0\sa120\widctlpar\ltrpar\li0\lin0\ri0\rin0\ql\faauto\sl240\slmult1\rtlch\afs20\ltrch\fs18\f1{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 "In }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 everyday}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1 English}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 usage}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1&#xD;
\hich\f1 , }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 the}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 term}{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  \u8216 NGO\u8217  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 connotes}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  a }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 non-governmental}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 , }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 non-profit}{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 , }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 cause-driven}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1 association}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 motivated}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1&#xD;
\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 by}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 altruistic}&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 intent}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 rather}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 than}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 pecuniary}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1&#xD;
\hich\f1 or}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 political}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 gain}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 . NGOs }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 are}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 the}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 good}&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 guys}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 . }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 However}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 , }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 scholars}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 who}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 scrutinized}&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 the}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 habits}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 of}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 North\u8211 South}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  NGO }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1&#xD;
\dbch\af1\hich\f1 funding}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 already}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 warned}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 us}{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 not}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1 to}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 think}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1&#xD;
\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 that}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  \u8216 }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 non-governmental}&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 \u8217  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 is}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24&#xD;
\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 synonymous}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1  }{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1 with}{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031&#xD;
\loch\af1\dbch\af1\hich\f1  \u8216 democratic.\u8217 6 Many scholars and activists have critically interrogated prototypes of NGOs or CSOs as units of civic engagement: this construc- tion, they argue, constitutes a potentially potent classificatory scheme that excludes mass or spontaneous mobilization. One perceptive critic of devel- opment aid argued that civil society promotion amounts to the profession- alization and institutionalization of certain patterns of knowledge.7 The construct of the NGO, originally meant to distinguish independent advocacy from inter-governmental transactions, is problematic.8 The//governmental/non-governmental binary is a convenient dummy variable defining civic energies aphophatically for what they are not. Inside the industry there are a range of rhyming variations distinguishing para- statal qua-NGOs, government-organized GONGOs, royally-organized RONGOs, donor-oriented DONGOs, World Bank initiated BINGOs, and even entrepreneurial B-Y-O (bring-your-own) \u8220 bringos.\u8221 9 But they all call themselves NGOs, even in Arabic, where in lieu of translation the identical acronym is often rendered in text in Latin alphabet letters or in speech as pronounced in English. Overall, the ubiquitous neologism of the NGO is an imprecise linguistic expression that is left, as we will see, for various governments and donor agencies to define bureaucratically and ideologically. In practice, an NGO is something registered as such with national governments, the United Nations, or donor agencies. In compar- ing and contrasting criteria for inclusion and patronage, this chapter illustrates the politics and paradoxes of \u8216 NGOization.\u8217  Many observers deduced that political aid stimulated a proliferation of professional, rather than grassroots, NGOs."}&#xD;
{\rtlch\af0\afs24\ltrch\fs18\lang1031\langnp1031\loch\af1\dbch\af1\hich\f1\cs10\par}{\*\latentstyles\lsdstimax267\lsdlockeddef0\lsdsemihiddendef1\lsdunhideuseddef1\lsdqformatdef0\lsdprioritydef99{\lsdlockedexcept}}}</TextFormatted>
    </KnowledgeItem> 

[ryan-carpenter]

In the XML Citavi backup file, <QuotationIndex> is the knowledge item's order of appearance (sort-order), which can be rearranged in the application. The tag<KnowledgeItemType> has the value 0 or 1 for text and image quotations, respectively. Fundamentally, the knowledge items are all 'quotations' rather than 'knowledge items', so the the basic designation of type is <QuotationType> .

Knowledge item	<KnowledgeItemType>	<QuotationType>
Image quotation (D)	1	0
Direct quotation (D)	0	1
Indirect quotation (F)	0	2
Summary (S)	0	3
Comment (C)	0	4
Highlight (Y)	0	5
Highlight in red (Q)	0	6

[Siedlerchr]

@ryan-carpenter Thanks for the help. I followed the zotero implementation, and just put the contents in the comment field: with markdown formatting
You can test a version here: https://builds.jabref.org/pull/9033/merge/
It uses markdown formatting for the comment field,
The knowledge-items have an id for the reference, to that was no problem.

[ryan-carpenter]

Here is the xml file containing the sample knowledge items. A few things to be aware of are:

• The file was made from a local project, not Citavi Web, not DB Server. Those products may or may not have the same structure.
• Yellow highlights to no have any accompanying user input like the rest of the quotation types.
• There is an option in Citavi to always save annotations with the attached PDF. By default, annotations are overlaid on the file but not saved in it. I am not sure how or if this affects backups and exports, but presumably someone wishing to retrieve the maximum possible data from a Citavi project would want to save annotations in the PDF.
• Including the Core statement field in Citavi exports is optional and off by default. I have it enabled.
• The attached file includes core statements. In this case, all the statements are short and thus included in full. I made them a little too short for the test-output but found in another test that the field gets truncated after a few more characters than in my examples here. The 'true' core statement field, if enabled by the user, contains the complete statement and is in the CoreStatement tag. The truncated copy is in the tag with sort... in the name where the text is converted to lowercase.
• The Text field contains multi-line rich text. The Core statement field contains sinlge-line text in the GUI, but also accepts some special characters. While the Text field retains formatting, the Core statement field converts the characters to ASCI (or similar).

knowledge-items-example.ctv6.txt

Edit: made with Citavi v6.14 (Windows 10)

[ryan-carpenter]

To include Citavi cores statements in exports (Windows) click : Tools > Options > Formatting > select Include core statements

[ryan-carpenter]

@Siedlerchr, a test import pulled in the reference only without knowledge items. I clicked File > Import > Into new and selected the original backup file (matching the XML file I posted here earlier). JabRef is set to Biblatex format. Have I missed a step?

JabRef 5.8-PullRequest9033.18--2022-08-10--a978702
Windows 10 10.0 amd64
Java 18.0.2
JavaFX 18.0.1+2

[Siedlerchr]

@ryan-carpenter Thanks for the hint,it was only the first found knowledge item for the reference was used. I was not aware that you could have multiple per references I am working on a follow up.

[Siedlerchr]

@ryan-carpenter Please try https://builds.jabref.org/pull/9043/merge/
I am not sure if we need to further distinguish in the comments the type of knowledge item as you listed, we just add all of them to the comment fields without any further indication. Maybe it's helpful to have the type listed with it?

[ryan-carpenter]

@ryan-carpenter Please try https://builds.jabref.org/pull/9043/merge/

It works.

I am not sure if we need to further distinguish in the comments the type of knowledge item as you listed, we just add all of them to the comment fields without any further indication. Maybe it's helpful to have the type listed with it?

The type, i.e., QuotationType, is important but not as important as the fields within a knowledge item (if populated).

• The first priority is the primary data in CoreStatement, Text or both (substitute Caption and Description, respectively, for image quotations). Note that the Core statement is optional, but populated automatically with text selected text in the PDF, and this may be the only field populated.
• Following in a collective second priority are the keywords, categories, and groups.

Thinking my own use case, I probably would have made nonstandard fields for each item or type, and then applied some naming scheme within for the keywords, categories, and groups. This way, I could use regex searches to find specific types of information. For instance, I might want to find all the direct quotes assigned to a particular group.

Keep in mind that knowledge items may have a completely different set of keywords, categories, and groups than the parent reference. It seems quite likely that Citavi users have dedicated valuable hours to organizing their knowledge items using this capability and will want to preserve their work.

Of course, this would also work within the comment field. Wherever the data goes, I would vote for keeping some indication of the quotation type, and ideally, combined with the QuotationIndex to preserve distinct associations between the quotation and its related keywords, categories, and groups. In other words, I would like to know that quote1 has keywords a and b while quote2 for the same reference has instead keywords b and c.