Dataverse discovery in Google - Machine Readable Sitemaps

[eugene-barsky]

Hello:

AS per Philip's request, we sat in a Google session about data discoverability, and Dr. Natasha Noy @google was mentioning Google preference to include sitemaps for better data indexing.

They also talked about Google preference for schema.org coding for the landing pages.

Thanks,

Eugene

[jggautier]

Thanks @eugene-barsky! Some more info for this issue: Google's "Build and submit a sitemap" guide

[pdurbin]

Yep, as I mentioned at #2717 (comment) it was about 24 minutes into the video at https://www.rd-alliance.org/making-data-discoverable-web-search-engines that sitemaps were mentioned and the slide above appeared. Thanks for opening this issue @eugene-barsky

[mheppler]

In issue #4555 @pameyer commented:

"Why don't my datasets show up in google?" seems like a question that comes up relatively commonly (but completely out of scope for this issue).

This would be the appropriately scoped issue for that question.

[djbrooke]

We'll estimate this and work on this. I'm removing the schema.org part of the title, since that was delivered in 4.8.4. We still need to add a sitemap. This is more important since Google Dataset Search is now a thing. :)

[pdurbin]

I'm playing around with Documentation Driven Development (DDD), if that's a thing, by making pull request #5084 which for now is only a stub of the direction I think we're going. See d2ccf59

At standup I mentioned that for my family site I recently switched from Jekyll to Hugo, which creates a sitemap at http://thedurbins.com/sitemap.xml . Hugo creates an XML file but at https://support.google.com/webmasters/answer/183668 Google indicates they support multiple formats:

• XML
• RSS, mRSS, and Atom 1.0
• Text

My assumption is that we want to create an XML file. I assume we'll be using an EJB timer to control how often the XML file is updated. If I'm misunderstanding any requirements, please advise.

[kcondon]

Sending back, here's your requested "Punch List" @pdurbin, mostly what we had discussed last evening:

1. Need logging to indicate something is running for admins when job takes longer than a few minutes, eg. start, finish, object counts. Other ideas welcome.

2. Running endpoint on copy of prod finished in around an hour, shows a sitemap page but not sure it actually finished correctly since the result output on the command line was different from a successful one:

"Phil, sitemap seems like it finished in under an hour but I was not watching it.
weirdly it said this:
[root@dvn-vm5 tmp]# curl -X POST http://localhost:8080/api/admin/sitemap

<?xml version='1.0' encoding='UTF-8' ?>

this is what it says when it works:
[root@dvn-vm4 tmp]# curl -X POST http://localhost:8080/api/admin/sitemap
{"status":"OK","data":{"message":"Sitemap updated."}}
however, there is now a sitemap.xml file on vm5"

3. Make operation non-blocking on the command line: current blocking call continues to run anyway if ctrl-c is my understanding.

4. Make call check if already running and return already running rather than execute again, provides feedback and potentially corrupting sitemap. Some suggested approaches:
   -Mozilla download model, create sitemap.xml as a diff file until completed/verified, then copy to sitemap.xml. Temp file acts as lock to check.
   -Db table entry with lock.
   -In memory singleton with state to check if in progress.

5. Document or provide a script to add as a validation cron job that would validate sitemap.xml after completion, otherwise not sure whether it completed successfully or was interrupted by service restart, etc.

6. Document fact that file exists on node on which endpoint was run so in a multi web node environment, this file needs to be shared or replicated.

Many of the above are design/usability issues rather than strictly QA functional testing.

[pdurbin]

@kcondon after we chatted a few minutes ago I added the final thing we agreed would be nice: feedback from the curl command if the staged file exists. Over to you. Thanks.

[kcondon]

@pdurbin
Found a couple things:

1. Path to sitemap.xml is listed as two different places in multiple web server (logos) versus sitemap instructions (sitemap). It appears the sitemap instructions are correct based on the log messages.
2. When no initial sitemap exists, endpoint fails in server log:

[2018-10-04T15:09:57.296-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.sitemap.SiteMapUtil] [tid: _ThreadID=147 _ThreadName=__ejb-thread-pool5] [timeMillis: 1538680197296] [levelValue: 800] [[
  Writing staged sitemap to /usr/local/glassfish4/glassfish/domains/domain1/docroot/sitemap/sitemap.xml.staged]]

[2018-10-04T15:09:57.397-0400] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.sitemap.SiteMapUtil] [tid: _ThreadID=147 _ThreadName=__ejb-thread-pool5] [timeMillis: 1538680197397] [levelValue: 900] [[
  Unable to update sitemap! Unable to write staged sitemap to /usr/local/glassfish4/glassfish/domains/domain1/docroot/sitemap/sitemap.xml.staged. TransformerException: java.io.FileNotFoundException: /usr/local/glassfish4/glassfish/domains/domain1/docroot/sitemap/sitemap.xml.staged (No such file or directory)]]