Add the ability to "Explore in Whole Tale" (External Tool)

[amoeba]

Hey all, @pdurbin and I met the other week and we talked about Dataverse and WholeTale. WholeTale is a reproducibility-oriented environment for science and it would integrate really well with Dataverse's External Tools. WholeTale isn't in production just yet so this issue is intended to raise awareness between our projects and keep track of this idea for later. See https://github.com/whole-tale/dashboard/issues/269#issuecomment-422155299 for a bit more detail.

@pdurbin please feel free to add any details you deem appropriate.

[pdurbin]

@amoeba hi! Thanks for creating this issue and for linking to my comment over on the Whole Tale side. I guess there are a few details I'd like to emphasize here:

• It was absolutely fantastic to hang out with you all at the 2018 Whole Tale workshop. I've been thinking I should blog about it or something but for now people can check out the slides from my lightning talk call "Reproducibility and Dataverse" at https://twitter.com/philipdurbin/status/1040267883090202625 (update: video at https://www.youtube.com/watch?v=SuyQTsOGugc )
• Integration with Whole Tale seems so similar to the proposed idea of integrating with Binder in Binderverse: integrating Binderhub with Dataverse (using docker+kubernetes) #4714 that I can imagine similar levels of excitement from the Dataverse community as when the idea was announced on our mailing list at https://groups.google.com/d/msg/dataverse-community/VG6gTMEd_Ps/Xy7jDhVoBwAJ . Whole Tale brings not only Jupyter notebooks but RStudio and other tools, from what I gather.
• I'm almost certain that Whole Tale wants an "Explore in Whole Tale" button within Dataverse at the dataset level rather than the file level, which means that this issue is dependent on Dataset Explore #5028 because Dataverse currently only supports file level exploration (and only tabular files at that), as explained at http://guides.dataverse.org/en/4.9.2/installation/external-tools.html
• That said, Whole Tale could go ahead and make a pull request to add a file level External Tool for now. Again, it would only be for tabular files but it would be a way to add the button today. The pull request you would make would look something like pull request Removed the dataExplorer config file and updated the external tool in… #4443 and you could emphasize the experimental nature of the tool and integration. You'll be linking to instructions that you host and maintain yourself anyway, so you can update them whenever you want. If you're interested in making a pull request now for adding a button at the file level for tabular files, just let us know and we can coach you through the process. For now we'd only be able to pass a database id of the file and the URL of the Dataverse installation that sent it but that would be enough for Whole Tale to uniquely identify and download the file. And make it available it a Jupyter notebook or whatever, presumably.

I have more thoughts, but I'd better "trim the tail" for now. 😄 Thanks again!

[pdurbin]

I'm just noting that over at whole-tale/whole-tale#43 there's a plan to meet on October 24th.

[pdurbin]

I noticed that @craig-willis just opened this related issue: whole-tale/girder_wholetale#171 - "Add import endpoint to support launching Tales using external identifiers"

[craig-willis]

Thanks you, @pdurbin. Indeed -- been talking with @Xarthisius and we are moving ahead with this concept with both Dataverse and DataONE in mind now. We will initially support only file-level identifiers.

[pdurbin]

@craig-willis ah, file level for now is great because Dataverse doesn't currently support dataset level external tools, which we're tracking at #5028. Please note that only files that have gone through a successful tabular ingest are supported. Right now the identifier that's sent for a file is a database id, an integer, and Dataverse installations can choose whether or not they mint DOIs (or Handles) at the file level.

I noticed that you tentatively have "PUT" rather than "GET" external tools in Dataverse only open a new browser window, a GET. I realize that http://guides.dataverse.org/en/4.9.4/installation/external-tools.html doesn't explain this very well but there is a note about how only tabular files are supported.

[craig-willis]

@pdurbin we're close to having a prototype of our endpoint working. I have a preliminary dev Dataverse instance up (per whole-tale/whole-tale#49 !) and was able to very easily add my external tool definition -- this is a great feature.

I wanted to share one thought based on our assumption of how this would work. On paper, we were planning to have our endpoint accept a single identifier/URI and use the external tools field replacement to pre-compose the URL. Something like:

{
  "displayName": "Whole Tale",
  "description": "Explore in Whole Tale",
  "type": "explore",
  "toolUrl": "https://dashboard.wholetale.org/import",
  "toolParameters": {
    "queryParameters": [
      {
        "url": "{siteUrl}/api/access/datafile/{fileId}?key={apiToken}"
      }
    ]
  }
}

This resulted in the error Unknown reserved word: {siteUrl}/api/access/datafile/{fileId}, which after looking at ExternalTools.java, it looks like this is a 1:1 mapping, not a general field replacement. (I'll also acknowledge the recommendation that we use the X-Dataverse-key header in place of the key parameter).

We can obviously work with it as-is, but thought that you might be interested. When thinking about a general solution for multiple platforms (e.g., DataONE, Dataverse) the URL approach seemed cleanest on paper. Still, adding the tool this way is so much more convenient than a PR to the main codebase!

[pdurbin]

@craig-willis thanks, this is great feedback and I just copied it over to #5028 (comment) which is an issue in which we are more likely to work on it (or punt until an issue beyond that one).

It sounds like you got your external tool manifest working, which is great. Do you or @amoeba want to go ahead and make a pull request to add a link to the Dataverse "external tools" page? At this point you'll probably want to indicate that it's experimental. Your pull request would look something like pull request #4443 which is when the author of Data Explorer last adjusted the text in on that page. Here's how the text looks as of this writing:

That's from http://guides.dataverse.org/en/4.9.4/installation/external-tools.html

[craig-willis]

Thanks @pdurbin. We still need to release/deploy changes on our end, which is probably a few weeks away. At that point, I'll issue the PR as suggested.

[pdurbin]

@craig-willis sounds good. I just invited you to join https://github.com/orgs/IQSS/teams/dataverse-readonly/members so I can assign this issue to you but for now I'll assign it to @amoeba as I drag it to the "Community Dev" column at https://waffle.io/IQSS/dataverse

[pdurbin]

@craig-willis @Xarthisius thanks for joining today's community call and announcing that you're working on integration between Whole Tale and Dataverse. I just left a more detailed comment at https://github.com/whole-tale/dashboard/issues/269#issuecomment-440399337 with a number screenshots but here I'll post the final screenshot after data has been downloaded by DOI from Dataverse and is ready to be used with a Jupyter notebook or RStudio in Whole Tale:

My take on http://irclog.iq.harvard.edu/dataverse/2018-11-20#i_79567 is that there's still work to be done on the UI in whole-tale/dashboard#287 but getting the data from Dataverse into Whole Tale by DOI is excellent progress!

[pdurbin]

@craig-willis also, not to muddy the waters for your future pull request but the "Analysis and Computation" section should be updated in a future version of http://guides.dataverse.org/en/4.9.4/admin/integrations.html to say something like "From Whole Tale you can download data from Dataverse by DOI or other persistent ID for use in a Jupyter notebook or RStudio."

But you should still edit the external tools page too, please!

[pdurbin]

I just left a similar comment at #4714 (comment) but I realized I should leave a comment here too.

A number of us were on a call on Wednesday ( whole-tale/whole-tale#50 ). I attempted to give a demo of clicking a "Whole Tale" button (right under "Data Explorer") and being sent to an installation of Whole Tale with Jupyter notebooks and RStudio and other tools. I wasn't logged to Whole Tale at the time so it wasn't the best demo but I just tried again from https://dev1.dataverse.org/dataset.xhtml?persistentId=doi:10.5072/FK2/EPNUKP&version=1.0 and it worked better this time. Here are some screenshots. Great stuff! Thank you @craig-willis and @Xarthisius!

[pdurbin]

@craig-willis is it safe to use https://github.com/whole-tale/wt-design-docs/blob/b3ed01560f600ba79510b067d393a7eeb8d9b1c1/users_guide/wholetale.json (also at https://wholetale.readthedocs.io/users_guide/integration.html#dataverse-external-tools ) if I wanted to advocate for loading up the Whole Tale external tool manifest as part of the "definition of done" for this issue?

Over at whole-tale/dashboard#327 I see that Whole Tale 0.5 RC is being tested and we can wait until it has been deployed to production, as you indicated at whole-tale/wt-design-docs#61 (comment)

Finally, develop...craig-willis:5097-wholetale-integration looks great! The only thing I would consider mentioning on the "Integrations" page is that from Whole Tale it's possible to download data by DOI (or Handle?) that's hosted in Dataverse. Just a thought. Pinging @dlmurphy to take a look as well.

[craig-willis]

@pdurbin reviews PRs before they're created! Amazing!

Loading the wholetale.json seems like a reasonable completion criteria -- assuming the release is deployed. However, we're not yet happy with that JSON and are planning to change the hostname. I'll comment here when both are complete.

[pdurbin]

@craig-willis thanks for making pull request #5417. It's currently in QA.

I wanted to raise an issue from whole-tale/dashboard#327 that we've been using my "dev1" server for testing the Whole Tale "explore" button but I'm about to drop the database on that server because I'm using it for another project (Make Data Count). In that issue I suggested that maybe we could add the Whole Tale "explore" button to https://demo.dataverse.org I'm curious if @craig-willis @djbrooke and others think this is a good idea.

[craig-willis]

@pdurbin This is a perfect next step from my perspective. With the PR issued and release 0.5 deployed, we're as ready as we're going to be for now.

[pdurbin]

@craig-willis great news! I wasn't sure if you had deployed on your side yet.

[pdurbin]

@craig-willis I spoke with @djbrooke about the Dataverse demo site this morning and he seemed fine with adding the Whole Tale button. Are you planning on announcing this integration at https://groups.google.com/forum/#!forum/dataverse-community or similar? Two integrations, actually. The "Whole Tale" button on Dataverse demo and the ability to download data by DOI from Dataverse installations from the Whole Tale web interface. I assume Whole Tale would like feedback on this stuff. 😄 This looks related: whole-tale/wt-design-docs#61

@djbrooke and I also discussed your Single Sign On (SSO) issue. Well, I told him I'd find it. This one: whole-tale/whole-tale#53 . "Authentication between the two systems" is a common sticking point we encounter. When you click the "Compute" button in Dataverse ( http://guides.dataverse.org/en/4.10.1/admin/integrations.html#compute-button ) you have a better user experience if you're already logged into the computing environment such as OpenStack ( http://guides.dataverse.org/en/4.10.1/installation/config.html#setting-up-compute ).