Triplestsore indexer #28

dannylamb · 2017-02-01T19:13:48Z

Islandora/documentation#394

This adds a small camel connector to index RDF from CLAW into a triplestore.

To Test

Assuming you're starting from scratch here (it's easier than getting lost in Karaf)

Pull in Install changes for triplestore indexer islandora-deprecated/claw_vagrant#8 into your claw_vagrant repo.
- $ cd /path/to/claw_vagrant
- $ git pull https://github.com/dannylamb/claw_vagrant triplestore-indexer
Update alpaca.sh to say git clone -b triplestsore-indexer https://github.com/dannylamb/Alpaca.git instead of cloning from origin/master. Be mindful of the spelling on that branch name. I derped pushing it up to Github
vagrant up
When it's finished, uninstall the islandora module (you will have to disable islandora_collection too)
Log into the box with vagrant ssh
Pull in the changes from Don't re-add drupal to document root on a re-provisioning documentation#34, which will enable GET requests on FedoraResource entities.
- $ cd /var/www/html/drupal/web/modules/contrib/islandora
- $ git pull https://github.com/dannylamb/islandora.git triplestore-indexer
Then re-install islandora and islandora_collection modules.

Once installed:

Visit http://localhost:8000/admin/content/fedora_resource and create a new Fedora resource
Visit http://localhost:8080/bigdata/#query and issue a select ?s ?p ?o where { ?s ?p ?o } query to get all triples. You should see the triples from the rdf mapping for the bundle you created.
Visit your resource in Drupal and update it
Go back to the triplestore and run the query again. At a minimum, modified dates and vclock should have updated.
Visit your resource in Drupal again, and delete it.
Back in the triplestore, the triples should be gone and the query will return no results.

DiegoPino

If bumping activeMQ version here to 5.14.1 claw vagrant needs to be bumped here also
https://github.com/Islandora-CLAW/claw_vagrant/blob/8eb5513a561d444463fd36eddb4f7b9ae0d320e8/configs/karaf/fcrepo_camel_toolbox.script#L2 ?

dannylamb · 2017-02-07T15:44:36Z

@DiegoPino I have updated the install scripts to rely on the features.xml provided by Alpaca. We no longer have to add the feature repos for camel and activemq by hand. islandora-deprecated/claw_vagrant#8

dannylamb · 2017-02-07T15:58:44Z

@Islandora-CLAW/committers @Natkeeran This is ready for review, with install instructions. These changes cover three PRs, so if something's not clear and you're having problems with installing it, tag me.

whikloj · 2017-02-07T21:56:02Z

I'm doing it the hard way, I'm building Alpaca inside and replacing the islandora-indexer-triplestore feature. We'll see if I can do it without getting lost.

whikloj · 2017-02-08T02:09:13Z

Ok that didn't do anything, I'll try by the instructions here.

dannylamb · 2017-02-08T03:56:33Z

For the record, you'd have to

$ sudo service karaf-service stop
$ sudo rm -rf /opt/karaf/data/*
$ sudo rm /opt/karaf/etc/edu.amherst.*
$ sudo rm /opt/karaf/etc/ca.islandora.*
$ sudo rm /opt/karaf/etc/org.fcrepo.*
$ sudo rm /opt/karaf/etc/org.ops4j.datasource*
$ sudo service karaf-service start
$ cd /path/to/Alpaca
$ ./gradlew install
Then feed karaf the same instructions in https://raw.githubusercontent.com/dannylamb/claw_vagrant/56777a3d84d8c598ee99be103257e650220043fe/configs/karaf/alpaca.script. That is, copy/paste that into a file named alpaca.script and then $ /opt/karaf/bin/client -f alpaca.script
Touch up config by running for all the deployed bundles a la https://github.com/dannylamb/claw_vagrant/blob/56777a3d84d8c598ee99be103257e650220043fe/scripts/config.sh

Basically, you're resetting karaf, getting rid of any old config, and then re-installing the recompiled Alpaca. Afterwards, you gotta re-touch up the config.

DiegoPino

property placeholders in blueprint should load their values from the provided islandora.alpaca.indexing.triplestore.cfg file. Does not seem to be a good idea to have settings/URL hardcoded in blueprint properties if we are already providing a .cfg file for this.

DiegoPino · 2017-02-08T15:29:38Z

islandora-indexing-triplestore/src/main/resources/OSGI-INF/blueprint/blueprint.xml

@@ -8,18 +8,23 @@
       http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

  <!-- OSGI blueprint property placeholder -->
-  <cm:property-placeholder id="properties" persistent-id="ca.islandora.indexing.triplestore" update-strategy="reload" />
+  <cm:property-placeholder id="properties" persistent-id="ca.islandora.alpaca.indexing.triplestore" update-strategy="reload" >


Since we have already an islandora.alpaca.indexing.triplestore.cfg file that defines these properties' values, we should load them from here instead of hardcoding. See http://camel.apache.org/properties.html#Properties-Usinga.cfgor.propertiesFileForBlueprintPropertyPlaceholders

@DiegoPino The properties placeholder element in blueprint.xml is what loads the config file, and the 'hardcoded' values you're referring to are the defaults in case someone modifies a config file and removes an entry. It also is what bridges the gap and allows me to refer to values from config within routes, and not just in the blueprint file.

DiegoPino · 2017-02-08T15:31:30Z

islandora-indexing-triplestore/src/test/resources/OSGI-INF/blueprint/blueprint-test.xml

+       http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
+
+  <!-- OSGI blueprint property placeholder -->
+  <cm:property-placeholder id="properties" persistent-id="ca.islandora.indexing.triplestore" update-strategy="reload" >


Same as above: tests settings should be loaded from .cfg file. See http://camel.apache.org/properties.html#Properties-Usinga.cfgor.propertiesFileForBlueprintPropertyPlaceholders

whikloj · 2017-02-08T16:10:10Z

@dannylamb Trying a vagrant up with alpaca.sh checking out your branch threw an error.

==> default: FAILURE: Build failed with an exception.
==> default: * What went wrong:
==> default: Execution failed for task ':islandora-indexing-triplestore:install'.
==> default: > Could not publish configuration 'archives'
==> default:    > 
==> default: Cannot publish artifact 'ca.islandora.indexing.triplestore-configuration.cfg (ca.islandora.alpaca:islandora-indexing-triplestore:0.1.1-SNAPSHOT)' (/home/ubuntu/Alpaca/islandora-indexing-triplestore/build/cfg/main/ca.islandora.indexing.triplestore.cfg) as it does not exist.
==> default: 
==> default: 
==> default: * Try:
==> default: Run with --stacktrace option to get the stack trace. Run with --info or --debug
==> default:  option to get more log output.

dannylamb · 2017-02-08T16:11:23Z

@whikloj ok, looks like that config file needs a .alpaca. in the middle of it. Moment.

DiegoPino · 2017-02-08T16:15:51Z

@dannylamb ok, I got the same issue as @whikloj . Naming for id of placeholder was not the same on both and differed from the actual config file, so I kinda extrapolated to whole stuff (wrongly). If you fix that I'm 👍

dannylamb · 2017-02-08T16:19:09Z

@whikloj @DiegoPino My last test properly deployed the config file, and this PR addresses that here. Is your Alpaca on your vagrant box cloned from my branch and repo?

Update alpaca.sh to say git clone -b triplestsore-indexer https://github.com/dannylamb/Alpaca.git instead of cloning from origin/master.

DiegoPino · 2017-02-08T16:23:45Z

@dannylamb yeah, that is the correct file:

<configfile finalname="/etc/ca.islandora.alpaca.indexing.triplestore.cfg">mvn:ca.islandora.alpaca/islandora-indexing-triplestore/${project.version}/cfg/configuration</configfile>

But the tests are referencing at
https://github.com/Islandora-CLAW/Alpaca/pull/28/files/2a851f9773e789d287bdeb219889e8204c4e8770#diff-6c7fc65ffa4096d22a789cff61e01f56R11,

<cm:property-placeholder id="properties" persistent-id="ca.islandora.indexing.triplestore" update-strategy="reload" >

I can be getting this all wrong of course so please excuse me if so.

whikloj · 2017-02-08T16:30:31Z

Yeah I also see the correct file path in the karaf features.xml.

whikloj · 2017-02-08T16:33:32Z

got it
https://github.com/Islandora-CLAW/Alpaca/pull/28/files#diff-709f0c89fb4718eaf3e1ea6fb036e0bcR36

ruebot · 2017-02-08T16:38:13Z

vagrant up

whikloj · 2017-02-08T17:29:02Z

Still doesn't work, removed and tried reloading inside vagrant. Still didn't work.

dannylamb · 2017-02-08T17:51:34Z

@whikloj I haven't pushed the changes up yet. Give me a minute.

dannylamb · 2017-02-08T19:19:00Z

I'm doing another fresh vagrant up on my end to verify before setting folks loose on this again.

dannylamb · 2017-02-08T20:42:42Z

And another one. vagrant uping again.

ruebot · 2017-02-08T21:24:50Z

@dannylamb build came up clean for me now. I'll proceed with testing further.

Natkeeran · 2017-02-08T21:27:48Z

@dannylamb I'll give it a try as well.

Followed the steps. It does not index it for me! I cannot see any triples related to the resource that was created.

ruebot · 2017-02-08T22:16:12Z

All test procedures work for me. I'm comfortable merging.

whikloj · 2017-02-09T15:17:42Z

I see objects created, I see JSON-LD REST responses, but nothing in the triplestore.

Karaf says

2017-02-09 15:14:52,807 | ERROR | ing-triplestore] | TriplestoreIndexer               | 187 - ca.islandora.alpaca.islandora-indexing-triplestore - 0.1.1.SNAPSHOT | Error indexing http://localhost:8000/fedora_resource/1 in triplestore: HTTP operation failed invoking http://localhost:8000/fedora_resource/1?_format=jsonld with statusCode: 403
org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://localhost:8000/fedora_resource/1?_format=jsonld with statusCode: 403        at org.apache.camel.component.http4.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:284)
        at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:192)
...

Do I need to make REST requests available to all users?

whikloj · 2017-02-09T15:18:52Z

Oh wait the configuration changes did not take effect. I might have this.

whikloj · 2017-02-09T17:13:10Z

Ok using the instructions this PR, islandora-deprecated/claw_vagrant#8 and Islandora/documentation#34 all work.

I'm merging all 3. 👍

dannylamb added 7 commits January 29, 2017 04:32

Rewiring triplestore indexer

ffc8513

First pass after initial testing. Functional, but requires tests.

bc84be4

Triplestore indexer tests

d785695

Conding standards fixes

51453b2

Removing dummy creds from config

b43f5a8

Fixing config mis-deployment

2e61108

Bumping activemq

2a851f9

DiegoPino reviewed Feb 7, 2017

View reviewed changes

dannylamb mentioned this pull request Feb 7, 2017

Install changes for triplestore indexer islandora-deprecated/claw_vagrant#8

Merged

DiegoPino suggested changes Feb 8, 2017

View reviewed changes

Tests are now pointing to the same config

e28c733

Found another missing .alpaca.

5da158f

DiegoPino approved these changes Feb 8, 2017

View reviewed changes

ruebot approved these changes Feb 8, 2017

View reviewed changes

whikloj merged commit 6c38bc4 into Islandora:master Feb 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Triplestsore indexer #28

Triplestsore indexer #28

dannylamb commented Feb 1, 2017 •

edited

Loading

DiegoPino left a comment •

edited

Loading

dannylamb commented Feb 7, 2017

dannylamb commented Feb 7, 2017

whikloj commented Feb 7, 2017

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017 •

edited

Loading

DiegoPino left a comment

DiegoPino Feb 8, 2017

dannylamb Feb 8, 2017 •

edited

Loading

DiegoPino Feb 8, 2017

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017

DiegoPino commented Feb 8, 2017

dannylamb commented Feb 8, 2017

DiegoPino commented Feb 8, 2017 •

edited

Loading

whikloj commented Feb 8, 2017

whikloj commented Feb 8, 2017

ruebot commented Feb 8, 2017

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017

dannylamb commented Feb 8, 2017

dannylamb commented Feb 8, 2017

ruebot commented Feb 8, 2017

Natkeeran commented Feb 8, 2017 •

edited

Loading

ruebot commented Feb 8, 2017

whikloj commented Feb 9, 2017

whikloj commented Feb 9, 2017

whikloj commented Feb 9, 2017

Triplestsore indexer #28

Triplestsore indexer #28

Conversation

dannylamb commented Feb 1, 2017 • edited Loading

To Test

DiegoPino left a comment • edited Loading

Choose a reason for hiding this comment

dannylamb commented Feb 7, 2017

dannylamb commented Feb 7, 2017

whikloj commented Feb 7, 2017

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017 • edited Loading

DiegoPino left a comment

Choose a reason for hiding this comment

DiegoPino Feb 8, 2017

Choose a reason for hiding this comment

dannylamb Feb 8, 2017 • edited Loading

Choose a reason for hiding this comment

DiegoPino Feb 8, 2017

Choose a reason for hiding this comment

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017

DiegoPino commented Feb 8, 2017

dannylamb commented Feb 8, 2017

DiegoPino commented Feb 8, 2017 • edited Loading

whikloj commented Feb 8, 2017

whikloj commented Feb 8, 2017

ruebot commented Feb 8, 2017

whikloj commented Feb 8, 2017

dannylamb commented Feb 8, 2017

dannylamb commented Feb 8, 2017

dannylamb commented Feb 8, 2017

ruebot commented Feb 8, 2017

Natkeeran commented Feb 8, 2017 • edited Loading

ruebot commented Feb 8, 2017

whikloj commented Feb 9, 2017

whikloj commented Feb 9, 2017

whikloj commented Feb 9, 2017

dannylamb commented Feb 1, 2017 •

edited

Loading

DiegoPino left a comment •

edited

Loading

dannylamb commented Feb 8, 2017 •

edited

Loading

dannylamb Feb 8, 2017 •

edited

Loading

DiegoPino commented Feb 8, 2017 •

edited

Loading

Natkeeran commented Feb 8, 2017 •

edited

Loading