CVOC : Indexed field accuracy (Ontoportal integration) #10505

luddaniel · 2024-04-18T14:42:06Z

Following agreements on next steps of #10145

What this PR does / why we need it:

This PR allows admins to specify which indexed fields must be use for a :CVocConf configured service.

Suggestions on how to test this:

Related documentation : gdcc/dataverse-external-vocab-support#21
Ontoportal CVocConf example : cvoc-agroportal-ontoportal-conf.json

Default behaviour (with indexIn):

With indexIn:

Does this PR introduce a user interface change?
Indirectly, it changes search result snippet.

…iguration + handle ontoportal json formats from externalvocabularyvalue

coveralls · 2024-04-18T15:03:46Z

coverage: 20.65% (+0.08%) from 20.574%
when pulling 4a6d3e4 on Recherche-Data-Gouv:9276-allow-mapping-of-indexable-fields-in-cvoc-conf
into 5bf6b6d on IQSS:develop.

qqmyers · 2024-04-23T16:36:34Z

doc/release-notes/9276-doc-cvoc-index-in.md

+
+### Updates on Support for External Vocabulary Services
+
+#### Indexed field accuracy


Suggested change:

Updates on Support for External Vocabulary Services

Multiple extensions of the External Vocabulary mechanism have been added. These extensions allow interaction with services based on the Ontoportal software and are expected to be generally useful for other service types.

These changes include:

Improved Indexing with Compound Fields

When using an external vocabulary service with compound fields, you can now specify which field(s) will include additional indexed information, such as translations of an entry into other languages. This is done by adding the indexIn in retrieval-filtering.
For more information, please check GDCC/dataverse-external-vocab-support documentation.

Broader Support for Indexing Service Responses

Indexing of the results from 'retrieval-filtering' responses can now handle additional formats including Json Arrays of Strings and values from arbitrary keys within a JSON Object.

(+ entries from the other Ontoportal PRs)

qqmyers · 2024-04-23T16:44:32Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

-                        for (JsonString elm : childFields.getValuesAs(JsonString.class)) {
-                            dft = findByNameOpt(elm.getString());
-                            logger.info("Found: " + dft.getName());
+                    if (jo.containsKey("managed-fields")) {


Just confirming - this was a bug? - the schema requires managed-fields these days

Not a bug, I would say old code, child-fields does not exists anymore so I replaced it with managed-fields. I think this code is not useful but at least the key is not obsolete.

Right - looks like it is just for logging purposes/to warn about incorrect field choices.

qqmyers · 2024-04-23T16:45:45Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

-                                  + elm.getString());
+                                        + managedFields.getString(s));
+                            } else {
+                                logger.info("Found: " + dft.getName());


change logger.info() to fine()

qqmyers · 2024-04-23T16:50:34Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

-     * 
+     * Retrieves indexable strings from a cached externalvocabularyvalue entry filtered through retrieval-filtering configuration.
+     * <p>
+     * This method externalvocabularyvalue entries have been filtered and contains a single JsonObject.


Suggested change:
This method assumes externalvocabularyvalue entries have been filtered and that they contain a single JsonObject.

qqmyers · 2024-04-23T16:51:26Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

+     * Retrieves indexable strings from a cached externalvocabularyvalue entry filtered through retrieval-filtering configuration.
+     * <p>
+     * This method externalvocabularyvalue entries have been filtered and contains a single JsonObject.
+     * Is handled : Strings, Array of Objects with "lang" and ("value" or "content") keys, Object with Strings as value or Object with Array of Strings as value.


Suggested change: Is handled : - > Cases Handled : A String, an Array of Strings, an Array of Objects with "value" or "content" keys, an Object with one or more entries that have String values or Array values with a set of String values.

(Trying to wordsmith as well as removing the "lang" requirement which isn't in the code.)

qqmyers · 2024-04-23T16:52:06Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

+     * <p>
+     * This method externalvocabularyvalue entries have been filtered and contains a single JsonObject.
+     * Is handled : Strings, Array of Objects with "lang" and ("value" or "content") keys, Object with Strings as value or Object with Array of Strings as value.
+     * The string, or the "value/content"s for each language are added to the set.


Suggested change:
The string(s), or ...

qqmyers · 2024-04-23T16:54:47Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

+     * This method externalvocabularyvalue entries have been filtered and contains a single JsonObject.
+     * Is handled : Strings, Array of Objects with "lang" and ("value" or "content") keys, Object with Strings as value or Object with Array of Strings as value.
+     * The string, or the "value/content"s for each language are added to the set.
+     * This method can retrieve string values to be indexed in term-uri-field (parameter defined in CVOC configuration) or in "indexIn" field (optional parameter of retrieval-filtering defined in CVOC configuration).


Suggested change:
Retrieved string values are indexed in the term-uri-field (parameter defined in CVOC configuration) by default, or in the field specified by an optional "indexIn" parameter in the retrieval-filtering defined in the CVOC configuration).

qqmyers

Looks good overall. I suggested some doc changes and noted one place where a logger.info should be dropped to a fine(). Otherwise the code and tests looks good.

W.r.t. docs - across all of these PR, it seems like adding 'services based on Ontoportal' to the list of SkosMos and ORCID in https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#using-external-vocabulary-services would be good. Maybe adding 'how indexing is handled' to the sentence starting with "Configuration involves " as well.

…ntoportal integration) IQSS#10505'

luddaniel

All documentation suggestions had been taken in account.

luddaniel · 2024-05-02T09:01:50Z

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

-                        for (JsonString elm : childFields.getValuesAs(JsonString.class)) {
-                            dft = findByNameOpt(elm.getString());
-                            logger.info("Found: " + dft.getName());
+                    if (jo.containsKey("managed-fields")) {


Not a bug, I would say old code, child-fields does not exists anymore so I replaced it with managed-fields. I think this code is not useful but at least the key is not obsolete.

…-cvoc-conf

qqmyers

Looks good.

…ntoportal integration) IQSS#10505'

sekmiller · 2024-06-10T20:03:40Z

@luddaniel can you please update this branch with the latest from develop. There are merge conflicts. Thanks!

…-cvoc-conf

9276 - CVOC : allow customized mapping of indexed fields of cvoc conf…

e9bf15e

…iguration + handle ontoportal json formats from externalvocabularyvalue

luddaniel changed the title ~~9276 - CVOC : Indexed field accuracy (Ontoportal integration)~~ CVOC : Indexed field accuracy (Ontoportal integration) Apr 18, 2024

Add unit tests for "getIndexableStringsByTermUri" method

6b2e111

qqmyers reviewed Apr 23, 2024

View reviewed changes

qqmyers requested changes Apr 23, 2024

View reviewed changes

qqmyers added the Size: 10 A percentage of a sprint. 7 hours. label Apr 23, 2024

Update documentations related to PR 'CVOC : Indexed field accuracy (O…

adf5074

…ntoportal integration) IQSS#10505'

luddaniel commented May 2, 2024

View reviewed changes

Merge branch 'develop' into 9276-allow-mapping-of-indexable-fields-in…

836d722

…-cvoc-conf

qqmyers approved these changes May 6, 2024

View reviewed changes

luddaniel added a commit to Recherche-Data-Gouv/dataverse that referenced this pull request May 14, 2024

Update documentations related to PR 'CVOC : Indexed field accuracy (O…

72c1f61

…ntoportal integration) IQSS#10505'

sekmiller self-assigned this Jun 10, 2024

Merge branch 'develop' into 9276-allow-mapping-of-indexable-fields-in…

4a6d3e4

…-cvoc-conf

sekmiller merged commit ad58f3e into IQSS:develop Jun 13, 2024
12 checks passed

qqmyers mentioned this pull request Jun 28, 2024

Feature Request/Idea:Implement mechanism to support integration with OntoPortal #9276

Closed

2 tasks

pdurbin added this to the 6.3 milestone Jul 3, 2024

luddaniel deleted the 9276-allow-mapping-of-indexable-fields-in-cvoc-conf branch September 19, 2024 14:43

luddaniel mentioned this pull request Sep 23, 2024

6.3 : Indexing issue while using CVOC setting #10869

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CVOC : Indexed field accuracy (Ontoportal integration) #10505

CVOC : Indexed field accuracy (Ontoportal integration) #10505

luddaniel commented Apr 18, 2024

coveralls commented Apr 18, 2024 •

edited

Loading

qqmyers Apr 23, 2024 •

edited

Loading

qqmyers Apr 23, 2024

luddaniel May 2, 2024

qqmyers May 6, 2024

qqmyers Apr 23, 2024

qqmyers Apr 23, 2024 •

edited

Loading

qqmyers Apr 23, 2024 •

edited

Loading

qqmyers Apr 23, 2024

qqmyers Apr 23, 2024

qqmyers left a comment

luddaniel left a comment •

edited

Loading

luddaniel May 2, 2024

qqmyers left a comment

sekmiller commented Jun 10, 2024


		### Updates on Support for External Vocabulary Services

		#### Indexed field accuracy

CVOC : Indexed field accuracy (Ontoportal integration) #10505

CVOC : Indexed field accuracy (Ontoportal integration) #10505

Conversation

luddaniel commented Apr 18, 2024

coveralls commented Apr 18, 2024 • edited Loading

qqmyers Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Updates on Support for External Vocabulary Services

Improved Indexing with Compound Fields

Broader Support for Indexing Service Responses

qqmyers Apr 23, 2024

Choose a reason for hiding this comment

luddaniel May 2, 2024

Choose a reason for hiding this comment

qqmyers May 6, 2024

Choose a reason for hiding this comment

qqmyers Apr 23, 2024

Choose a reason for hiding this comment

qqmyers Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

qqmyers Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

qqmyers Apr 23, 2024

Choose a reason for hiding this comment

qqmyers Apr 23, 2024

Choose a reason for hiding this comment

qqmyers left a comment

Choose a reason for hiding this comment

luddaniel left a comment • edited Loading

Choose a reason for hiding this comment

luddaniel May 2, 2024

Choose a reason for hiding this comment

qqmyers left a comment

Choose a reason for hiding this comment

sekmiller commented Jun 10, 2024

coveralls commented Apr 18, 2024 •

edited

Loading

qqmyers Apr 23, 2024 •

edited

Loading

qqmyers Apr 23, 2024 •

edited

Loading

qqmyers Apr 23, 2024 •

edited

Loading

luddaniel left a comment •

edited

Loading