Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request - "catalog record" API? #4337

Closed
dustymc opened this issue Feb 17, 2022 · 23 comments
Closed

Feature Request - "catalog record" API? #4337

dustymc opened this issue Feb 17, 2022 · 23 comments
Labels
Accessibility Issue is related to Arctos accessibility. Enhancement I think this would make Arctos even awesomer! Help wanted I have a question on how to use Arctos Priority-Critical (Arctos is broken) Critical because it is breaking functionality.
Milestone
In next release

Comments

@dustymc
Copy link
Contributor

dustymc commented Feb 17, 2022

Is your feature request related to a problem? Please describe.

The current API is (mostly, probably) capable of building a replica of the catalog record detail page by specifying appropriate "columns." That of course relies on the user having some idea of what we consider to be "catalog record details."

Good:

  • There's one 'catalog record stuff' API call, no way to get lost
  • Our idea of what constitutes a catalog record is not the only valid viewpoint, the current approach allows/encourages users to build what they want how they want.
  • The current approach only uses the resources that are requested; it's generally cheaper than the alternative.

Not-so-good:

  • Users have to sort through the documentation (which is available in the API) to figure out what they want
  • We have no way of suggesting what they use or how those things interact with each other.

Describe what you're trying to accomplish

Should we offer a compiled "catalog record stuff" API call? That would provide a very easy way to get everything on the catalog record detail page, and would provide us an opportunity to organize things - eg #2450 could be delivered as one unavoidable object...

rights {
   license: whatever,
   copyright_holder: agent-object,
   whatever: morestuff
}

rather than scattering those kinds of data across multiple "columns." This would probably add some expense to most API calls, but would also allow us to predict what's going to be requested and build a cache.

Describe the solution you'd like

First, discussion - is this useful?

Describe alternatives you've considered

Don't.

Additional context

https://github.com/ArctosDB/internal/issues/51 is very related - it's difficult to build tools without knowing what sort of usage we want to allow.

If we move forward with this, the /guid/ pages in Arctos should be rebuilt to use it. This would also address the "mostly, probably" above - we don't use the current API to build those pages, so I don't know if it's actually fully capable or not.

Priority

??
@dustymc dustymc added the Enhancement I think this would make Arctos even awesomer! label Feb 17, 2022
@dustymc dustymc added this to the Needs Discussion milestone Feb 17, 2022
@campmlc
Copy link

campmlc commented Feb 17, 2022 via email

@dustymc
Copy link
Contributor Author

dustymc commented Feb 17, 2022

Who would use this, and how?

I don't know. I would, if we build it, but that's a little circular. I want to think that easier access (even with slightly more complex organization) would lead to more use, but who knows....

ALMNH

I think so, but that's a question for them - they seem a bit overwhelmed by the options in the current API, but I'm not sure that's actually true and I'm even less sure that they're a "typical" user (or even that there is any such thing).

short term

That's a point for discussion too - I'm not sure what this would take to build, or really even how it should be built, but I doubt it would be particularly quick to build.

If I follow the normal path it would also make some more things available to search and search results, which seems pretty good, and make the giant cache even giant-ier and slower to rebuild, which isn't so great. I could get a partial view up pretty quickly, but I'm not sure how far that would get anyone - certainly wouldn't fully support the /guid/ pages in Arctos.

At some level I'm probably being a little Bezos-ish with this; the only way to know you've built a good API is to use your own APIs, and we keep talking about that so maybe we should just DO it and see if anyone follows. Of course Bezos gave that sermon from atop his giant mountain of cash......

@Jegelewicz
Copy link
Member

So, if I make the call for a record I can build whatever page I want, but won't that page have a different url than the GUID?

@dustymc
Copy link
Contributor Author

dustymc commented Apr 26, 2022

page have a different url than the GUID?

Yep, and that's a conversation worth having if we ever get there. I don't think I'd have any problem with "Arctos GUIDs" not referring to Arctos, but we'd probably also want some sort of safety net (signed agreement w can wave around, whatever) so it doesn't look like Arctos is broken when you forget to renew your registration or something. (ARKs/DOIs/etc. - something persistent that can be redirected - would be an elegant technical solution, but that would obviously take some resources to build and maintain.)

@dustymc
Copy link
Contributor Author

dustymc commented Nov 29, 2022

I'm going to force this Issue.

"GUID pages" need rebuilt to be responsive and functional on mobile (along with https://github.com/ArctosDB/internal/issues/211 and #2745).

Should that be done as part of a transition to an API-based framework, or should we add a layer of duct tape to the current page? If "API" then do we need something dedicated, or should the normal API be expanded as necessary?

Lacking feedback, I'll do something random whenever this floats to the top of my pile....

@dustymc dustymc modified the milestones: Needs Discussion, Next Task Nov 29, 2022
@dustymc dustymc added Priority-Critical (Arctos is broken) Critical because it is breaking functionality. Help wanted I have a question on how to use Arctos Accessibility Issue is related to Arctos accessibility. labels Nov 29, 2022
@Jegelewicz
Copy link
Member

This needs discussion by some dedicated users. Either we start with @ArctosDB/arctos-working-group-officers as part of a meeting agenda or we set up a special meeting to discuss.

@dustymc
Copy link
Contributor Author

dustymc commented Nov 29, 2022

I don't think users (=people who get data from Arctos) would notice/have any reason to care. Someone wanting to build their own UI around Arctos data would definitely notice. I think it's more of an administrative discussion - and one that would be easier to have if we had some idea of what we wanted to allow/support/require/etc. via API, https://github.com/ArctosDB/internal/issues/51.

You can mess with my milestones if you feel you must, but I'm still going to be forced into doing something relatively soon.

@dustymc dustymc modified the milestones: Needs Discussion, Next Task Nov 29, 2022
@Jegelewicz
Copy link
Member

Then this needs some sort of urgent call for discussion. @ArctosDB/arctos-working-group-officers

@campmlc
Copy link

campmlc commented Nov 29, 2022

I can be available today, but need some background. Do we need to bring in @jcabbott72 ?

@dustymc
Copy link
Contributor Author

dustymc commented Nov 29, 2022

I was asked to summarize the recent flurry of activity, this seems as good a place as any.

We have a very expensive (in terms of CPU) and very crappy (in terms of usability) "mobile" site. This has been involved in, perhaps caused, recent outages; we have to do something. (Not supporting mobile at all has been mentioned, this seems like sustainability suicide to me.)

The technology is now in such a state that libraries and tools aren't really necessary, as demonstrated by #4349 (place search) - a single site that works on various devices can be developed in plain CSS. Getting away from the huge kludgy thing that's working on overgrowing our infrastructure is now possible; there is a real pathway out of the weeds!

#2745 will go to production tonight, but as a second way of searching catalog records. It will replace the current UI "soon" (after constructive feedback has been implemented would be nice, but I'll do what I must).

https://github.com/ArctosDB/internal/issues/211 needs addressed; the current header is not responsive (and uses very old and not-so-great technology which will eventually find a way to make problems if its own).

That will leave the primarily "guid pages" which need made responsive in some way. Whether we go to an API or not should not have any effect whatsoever on the current pages, nor is it likely to in any future in which there's one Arctos page.

NOT moving to an API would be easier in most every way.

NOT moving to an API would make things like the goofy little summary....

Screenshot 2022-11-29 at 2 45 39 PM

... much more approachable.

Moving to an API would make it easier for someone to develop alternative views.

I suppose that all adds up to me not moving towards an API unless ya'll tell me to.

  • The benefits all seem very theoretical
  • We have very finite resources
  • As above, the current API will almost-ish-probably support GUID-pages
  • As above, maybe one path to all record-related data is better than two

Not doing this still somehow feels like a lost opportunity.

@jcabbott72
Copy link

jcabbott72 commented Nov 30, 2022 via email

@dustymc
Copy link
Contributor Author

dustymc commented Nov 30, 2022

new UI

FYI that is 100% the cat record API.

This issue might lead to a new endpoint, which I'd envision being useful only for a single-record detail page.

much more difficult

Please elaborate (maybe in #2745).

@dustymc dustymc mentioned this issue Nov 30, 2022
Closed
@acdoll
Copy link

acdoll commented Nov 30, 2022 •
edited
Loading

After my first glance I also thought

choosing an institution is much more difficult now

but then I finally hit the 'choose' button and found the Check all... [Institution Acronym] option.

@dustymc
Copy link
Contributor Author

dustymc commented Nov 30, 2022

See also #5313

@jcabbott72
Copy link

jcabbott72 commented Nov 30, 2022 via email

@dustymc dustymc modified the milestones: Next Task, Active Development Dec 2, 2022
@dustymc dustymc mentioned this issue Dec 2, 2022
Closed
@dustymc dustymc mentioned this issue Dec 2, 2022
Closed
@dustymc
Copy link
Contributor Author

dustymc commented Dec 2, 2022

Abandoning this for now; #5333 will focus on the UI stuff mentioned above.

@dustymc dustymc closed this as completed Dec 2, 2022
@dustymc dustymc mentioned this issue Jun 12, 2023
Closed
@dustymc
Copy link
Contributor Author

dustymc commented Mar 22, 2024

I'm dredging this up again, it needs reprioritized and recategorized (or possibly killed off again).

It might be possible/reasonable to just cache more stuff in FLAT (and filtered flat - honoring encumbrances is also very expensive) and use that to build /guid/ pages (or most of them), rather than some new thing. That would allow paying most of the cost when something changes, rather than when a page is requested, and sorta accidentally make it possible to re-create /guid/ pages from the API.

That doesn't address one of the initial concerns (users need to understand a deep API), but: meh, there's documentation.

@dustymc dustymc reopened this Mar 22, 2024
@mkoo
Copy link
Member

mkoo commented Mar 24, 2024

yea let's discuss when I'm back-- also probably should be relevant to proposed arctosr Rpackage

@dustymc
Copy link
Contributor Author

dustymc commented Apr 9, 2024

@mkoo put something on the calendar!

should be relevant to proposed arctosr Rpackage

Yes, and I think even API access agreements in general. To recycle an initial example, we currently have...

arctosprod@arctos>> select use_license_url from flat limit 10;
                                  use_license_url                                   
------------------------------------------------------------------------------------
 http://arctosdb.org/home/data/; http://creativecommons.org/licenses/by-nc/3.0
 http://arctosdb.org/home/data/; http://creativecommons.org/licenses/by-nc/3.0
 http://arctosdb.org/home/data/
 http://arctosdb.org/home/data/; http://creativecommons.org/licenses/by-nc/3.0
 http://arctosdb.org/home/data/
 http://creativecommons.org/licenses/by/3.0/; http://mvz.berkeley.edu/mvzdata_terms
 http://arctosdb.org/home/data/
 http://creativecommons.org/licenses/by/3.0/; http://mvz.berkeley.edu/mvzdata_terms
 http://arctosdb.org/home/data/; http://creativecommons.org/licenses/by-nc/3.0
 http://arctosdb.org/home/data/

I don't think that's sufficient, those data should AT LEAST be capable of building....

Screenshot 2024-04-09 at 07 07 23

and AFAIK there's no requirements in the API agreements to do ANYTHING - I think 'rights' (along with things like GUID - which is embarrassingly "local flavored" in FLAT at the moment) should be inseparable from the other data (and I'm pretty confident that I can do all that in FLAT as JSON, then use that to make our /guid/ pages less expensive to build, along with feeding the API).

#7348 is also related - things that get shared via DWC generally need to go through FLAT - so maybe invite those folks to any meeting.

@dustymc dustymc mentioned this issue Apr 17, 2024
Closed
@dustymc
Copy link
Contributor Author

dustymc commented May 20, 2024

I added some stuff to FLAT, will let it refresh and then go from there. Please let me know ASAP if there's any reason to adjust any of this. I don't think this is quite everything I use to build GUID pages, but it may be all that should be included in this API. For example, I don't think I can add projects and loans here, but that could be it's own thing (or it could be something that's just not available except in Arctos Proper).

arctosprod@arctos>> select doi from flat where doi is not null limit 1;
               doi                
----------------------------------
 https://doi.org/10.7299/X70V8CZ1





arctosprod@arctos>> select collection_preferred_identifiers from flat where collection_preferred_identifiers is not null  limit 1;
 collection_preferred_identifiers 
----------------------------------
 ALAAC=B30716





arctosprod@arctos>> select public_accn_id from flat where public_accn_id is not null  limit 1;
 public_accn_id 
----------------
       21067392




arctosprod@arctos>> select rights from flat where rights is not null limit 1;
{
   "terms":{
      "uri":"http://mvz.berkeley.edu/mvzdata_terms",
      "display":"MVZ Terms of Use",
      "description":"MVZ General Terms of Use Statement."
   },
   "license":{
      "uri":"http://creativecommons.org/licenses/by/3.0/",
      "display":"CC BY",
      "description":"Attribution 3.0 Unported"
   },
   "loan_policy":"https://mvzhandbook.berkeley.edu/curatorial/loans-access/specimen-loans"
}


arctosprod@arctos>> select citations from flat where citations is not null limit 1;


[
   {
      "cited_name":"Vulpes vulpes",
      "type_status":"voucher",
      "publication_id":"https://arctos.database.museum/publication/10007269",
      "short_citation":"Goldsmith et al. 2015",
      "publication_doi":"https://doi.org/10.1111/mec.13509",
      "citation_remarks":null,
      "identification_id":"https://arctos.database.museum/guid/UAM:Mamm:115992/IID11544512",
      "publication_media":[
         {
            "media_id":10485452,
            "media_uri":"https://web.corral.tacc.utexas.edu/UAF/arctos/mediaUploads/aren/Goldsmith_et_al_2015_Population_structure_of_two_rabies_hosts_in_Alaska.pdf",
            "thumbnail":"https://web.corral.tacc.utexas.edu/UAF/arctos/mediaUploads/aren/Goldsmith_et_al_2015_Population_structure_of_two_rabies_hosts_in_Alaska_Page_01.jpg",
            "media_type":"text",
            "preview_uri":"https://web.corral.tacc.utexas.edu/UAF/arctos/mediaUploads/aren/Goldsmith_et_al_2015_Population_structure_of_two_rabies_hosts_in_Alaska_Page_01.jpg"
         }
      ],
      "occurs_page_number":null,
      "identification_taxa":[
         {
            "taxon":{
               "ftn":"Animalia, Chordata, Mammalia, Carnivora, Caniformia, Canidae, Vulpes, Vulpes vulpes",
               "name":"Vulpes vulpes",
               "ctrms":[
                  {
                     "psn":1,
                     "typ":"kingdom",
                     "term":"Animalia"
                  },
                  {
                     "psn":2,
                     "typ":"phylum",
                     "term":"Chordata"
                  },
                  {
                     "psn":3,
                     "typ":"class",
                     "term":"Mammalia"
                  },
                  {
                     "psn":4,
                     "typ":"order",
                     "term":"Carnivora"
                  },
                  {
                     "psn":5,
                     "typ":"suborder",
                     "term":"Caniformia"
                  },
                  {
                     "psn":6,
                     "typ":"family",
                     "term":"Canidae"
                  },
                  {
                     "psn":7,
                     "typ":"genus",
                     "term":"Vulpes"
                  },
                  {
                     "psn":8,
                     "typ":"species",
                     "term":"Vulpes vulpes"
                  }
               ],
               "nctrms":[
                  {
                     "typ":"author_text",
                     "term":"(Linnaeus, 1758)"
                  },
                  {
                     "typ":"display_name",
                     "term":"<i>Vulpes vulpes</i> (Linnaeus, 1758)"
                  },
                  {
                     "typ":"nomenclatural_code",
                     "term":"ICZN"
                  },
                  {
                     "typ":"scientific_name",
                     "term":"Vulpes vulpes"
                  },
                  {
                     "typ":"source_authority",
                     "term":"University of Alaska Museum"
                  },
                  {
                     "typ":"taxon_status",
                     "term":"valid"
                  }
               ],
               "source":"Arctos",
               "classification_id":"https://arctos.database.museum/name/Vulpes vulpes#Arctos"
            },
            "taxon_id":"https://arctos.database.museum/name/Vulpes vulpes",
            "variable":"A"
         }
      ]
   }
]



arctosprod@arctos>> select media_tags from flat where media_tags is not null limit 1;


[
   {
      "tag_id":5619,
      "media_id":10604553,
      "media_uri":"https://web.corral.tacc.utexas.edu/arctos-s3/dtilka/2019-11-12/Lemmiscus_curtatus_C45S56_122018_DORSAL.jpg",
      "thumbnail":"https://web.corral.tacc.utexas.edu/arctos-s3/dtilka/2019-11-12/tn/tn_Lemmiscus_curtatus_C45S56_122018_DORSAL.jpg",
      "media_type":"image",
      "preview_uri":"https://web.corral.tacc.utexas.edu/arctos-s3/dtilka/2019-11-12/tn/tn_Lemmiscus_curtatus_C45S56_122018_DORSAL.jpg"
   }
]

@dustymc
Copy link
Contributor Author

dustymc commented May 30, 2024

Adding collectors would be handy for record pull, and in general.

@dustymc
Copy link
Contributor Author

dustymc commented Jun 3, 2024

"collector_agents" format:

[
  {
    "agent_name": "Paul G. Heltne",
    "agent_role": "collector",
    "agent_order": 1,
    "agent_identifier": "https://arctos.database.museum/agent/21296753"
  },
  {
    "agent_name": "Jean Linsner",
    "agent_role": "collector",
    "agent_order": 2,
    "agent_identifier": "https://arctos.database.museum/agent/21319390"
  },
  {
    "agent_name": "Kayla Barlow",
    "agent_role": "preparator",
    "agent_order": 3,
    "agent_identifier": "https://arctos.database.museum/agent/21319391"
  },
  {
    "agent_name": "Tom Collins",
    "agent_role": "preparator",
    "agent_order": 4,
    "agent_identifier": "https://arctos.database.museum/agent/21317418"
  },
  {
    "agent_name": "Anastasia DeMaio",
    "agent_role": "preparator",
    "agent_order": 5,
    "agent_identifier": "https://arctos.database.museum/agent/21317512"
  },
  {
    "agent_name": "Annamarie Fadorsen",
    "agent_role": "preparator",
    "agent_order": 6,
    "agent_identifier": "https://arctos.database.museum/agent/21292439"
  },
  {
    "agent_name": "Carlos Molina",
    "agent_role": "preparator",
    "agent_order": 7,
    "agent_identifier": "https://arctos.database.museum/agent/21316660"
  },
  {
    "agent_name": "Elizabeth Nelson",
    "agent_role": "preparator",
    "agent_order": 8,
    "agent_identifier": "https://arctos.database.museum/agent/21319393"
  },
  {
    "agent_name": "Endora Roberts",
    "agent_role": "preparator",
    "agent_order": 9,
    "agent_identifier": "https://arctos.database.museum/agent/21311704"
  },
  {
    "agent_name": "Hawthorne Williams",
    "agent_role": "preparator",
    "agent_order": 10,
    "agent_identifier": "https://arctos.database.museum/agent/21319392"
  },
  {
    "agent_name": "Jensen Wong",
    "agent_role": "preparator",
    "agent_order": 11,
    "agent_identifier": "https://arctos.database.museum/agent/21316663"
  },
  {
    "agent_name": "Ellie Zahedi",
    "agent_role": "preparator",
    "agent_order": 12,
    "agent_identifier": "https://arctos.database.museum/agent/21316932"
  }
]

@dustymc
Copy link
Contributor Author

dustymc commented Jun 10, 2024

This is mostly caught up, I'll add the new stuff to the API soonish.

I pulled all results data to https://docs.google.com/spreadsheets/d/17tO8Uw-XWgKe0KaZLfX2PMsTMiiLqNOVheDFGgWBHu4/edit#gid=1980202044 and added these to the bottom, just request access/ping me if you want to adjust anything while this is available. I'll push it back to the DB whenever I get around to it.

@dustymc dustymc closed this as completed Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Accessibility Issue is related to Arctos accessibility. Enhancement I think this would make Arctos even awesomer! Help wanted I have a question on how to use Arctos Priority-Critical (Arctos is broken) Critical because it is breaking functionality.
Projects
None yet
Milestone
In next release
Development

No branches or pull requests
6 participants
@mkoo @dustymc @Jegelewicz @acdoll @campmlc @jcabbott72