Evaluation

This page reports all details to perform the experiments, in order to evaluate SeMi against different approaches for the semantic model generation.

Software X Journal

The results presented in the manuscript submitted to the Software X Journal (currently "Under Review") are obtained performing the experiment described in this section.

Evaluation data are currently available in the Semi repository, however you can run the scripts reported in page entitled "Data Preparation", in order to create all the data from scratch.

Let's see in details which data are involved in the semantic models generation performed with SeMi. The starting directory is semi/data/taheriyan/task_04/. This directory comprises the following folders (in alphabetical order):

• background: it stores the background linked data related to each target source;
• learning_datasets: it stores the splitted version of background linked data (training, validation, and test data set) and includes the output of the best model resulting from the training process:
  • entity embeddings;
  • object property embeddings;
  • scores of the reconstructed facts included in the test dataset;
• ontology: it stores the ontology and a list of the related classes;
• refined_semantic_models: it includes the semantic models obtained exploiting embeddings produced by the training process;
• semantic_models: it includes the plausible and the initial semantic models obtained before the the training process;
• semantic_types: it includes the semantic types, already available for the experiment;
• sources: it includes the target sources.

SeMi has been evaluated against different systems and approaches for the semantic model generation. Semantic models produced by these approaches (and by SeMi) are available in the semi/evaluation/taheryan2016/task_04 directory. This directory includes the following folders (in alphabetical order):

• semantic_models_gt: it includes the ground-truth semantic models;
• semantic_models_occ: it includes the semantic models generated using only the frequency of semantic relations (no heuristics to extract and rank complex the graph patterns, as done by Taheriyan);
• semantic_models_semi: it includes the semantic models generated by semi;
• semantic_models_steiner: it includes the semantic models generated performing a steiner tree detection on a graph weighted according to the ontology structure (no background knowledge).

In order to perform the evaluation you can run the following command:

node evaluation/taheriyan2016/perform_eval.js task_04