Change the actionable value of loaded to "autoload" rather than NULL for the catalog record bulkloader

[Jegelewicz]

main bulkloader

If you mean the catalog record bulkloader, these are fundamentally different tools. The catalog record bulkloader is an independent tool - things in it load or error, that's it. "Component loaders" can have dependencies - things can hang around with 'autoload: ....' for weeks, then be processed after related data becomes available. That is, component loaders have three exits:

• load worked (so delete)
• load didn't work because data are a mess (so a person needs to become involved)
• load didn't work because dependent data are MIA (so try again later, no humans required)

I would be in favor of changing the actionable value of loaded to "autoload" rather than NULL for the catalog record bulkloader, but that should be addressed in a new issue.

Originally posted by @dustymc in #2974 (comment)

[Jegelewicz]

For consistency with component bulkload tools.

[Nicole-Ridgwell-NMMNHS]

I'm in favor of this. I've only just started using the catalog record bulkloader, and having the loaded value be null was a bit confusing.

[dustymc]

Yep, I still like this. Along with everything else, "autoload" doesn't look like any number of whitespace characters in various UIs.

The change would be pretty simple to implement, perhaps a couple days to chase everything down and update documentation.

MAYBE we should expand this (the finding is much of the work) and also talk about the field name - the component loaders use "status."

And "collection_object_id" shares a name with some unrelated stuff and has been confusing at times, if we want one more expansion.

[Jegelewicz]

I like changing to "status". "loaded" implies that something has happened already.

[Nicole-Ridgwell-NMMNHS]

I like changing to "status".

Agreed.

[campmlc]

Agreed. Ok also with changing collection object ID. What are options?

…

On Fri, Feb 19, 2021, 4:30 PM Nicole-Ridgwell-NMMNHS < ***@***.***> wrote: * [EXTERNAL]* I like changing to "status". Agreed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBCTSSN24C76LVX73D3S73YAHANCNFSM4XXF4QQA> .

[dustymc]

See also #3764

Bulkloader will have two actionable status values (and I don't care what the values actually are, HELP!):

• autoload_core: replaces current NULL functionality, loads the stuff in table bulkloader
• autoload_extras: autoload_core, plus also set all "component loaders" with the shared UUID to autoload

[Jegelewicz]

Those terms seem good to me.

[dustymc]

This is running at test.

Essentially everything involving table bulkloader and it's derivatives (bulkloader_stage, pre_bulkloader, bulkloader_for_download) has been touched and should be tested.

Other than the new field label (loaded is now status) and actionable values (status=="autoload_core" does what loaded==NULL used to do, status=="autoload_extras" also changes "components"), the only significant UI change (so far!) is that the mark button:

now leads to a multiple choice form.

Please also use this as an opportunity to check out the new data entry and edit screens - the old combo screen may not survive the next change!

TODO: Documentation needs cleaned before/as this goes to production.

[dustymc]

docs updated

[DerekSikes]

I want to load some records. Which button do I click now? How do I know which to click when? Thanks, Derek

…

On Mon, Oct 11, 2021 at 6:37 PM dustymc ***@***.***> wrote: Closed #3436 <#3436>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFNUM7XGQVIDIUSKQS73WTUGONOBANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

[Jegelewicz]

Please also use this as an opportunity to check out the new data entry and edit screens - the old combo screen may not survive the next change!

This needs an announcement and some time to let everyone get ready...?

[dustymc]

How do I know

https://handbook.arctosdb.org/documentation/bulkloader.html#processing

announcement and some time

#3691

[campmlc]

I support this - no reason to have inconsistent language.

…

On Tue, Oct 12, 2021 at 2:11 PM dustymc ***@***.***> wrote: * [EXTERNAL]* How do I know https://handbook.arctosdb.org/documentation/bulkloader.html#processing announcement and some time #3691 <#3691> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBB7JJ7N435QV65A5R3UGSI5DANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[DerekSikes]

Could someone answer my questions below please? Thanks! -Derek

…

On Tue, Oct 12, 2021 at 12:05 PM Derek Sikes ***@***.***> wrote: I want to load some records. Which button do I click now? How do I know which to click when? Thanks, Derek On Mon, Oct 11, 2021 at 6:37 PM dustymc ***@***.***> wrote: > Closed #3436 <#3436>. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#3436 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACFNUM7XGQVIDIUSKQS73WTUGONOBANCNFSM4XXF4QQA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > -- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

-- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

[dustymc]

I don't know how to be more clear than #3436 (comment) - can someone else chime in here, maybe help clarify the documentation?

[campmlc]

I feel there is a communication problem here. The request is for the main bulkloader?

…

On Wed, Oct 13, 2021, 12:23 PM dustymc ***@***.***> wrote: * [EXTERNAL]* I don't know how to be more clear than #3436 (comment) <#3436 (comment)> - can someone else chime in here, maybe help clarify the documentation? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBFXMGYVLVVCRPFC5SDUGXFA3ANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[DerekSikes]

I need to know which button does what the old 'LOAD' link does for the bulkloader, and when I would want to use the other button (and how to determine that). Nothing that anyone has said has answered those questions for me and I need to load specimen records. thanks, Derek

…

On Wed, Oct 13, 2021 at 10:23 AM dustymc ***@***.***> wrote: I don't know how to be more clear than #3436 (comment) <#3436 (comment)> - can someone else chime in here, maybe help clarify the documentation? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFNUM3XF35AFGLU5JM44HLUGXFA3ANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

[Jegelewicz]

I need to load specimen records.

Are there any "extras" associatied with these records (parts with attributes, multiple identifications, locality attributes)?

If not

Once a record is marked to load by making status “autoload_core” (loads data from table bulkloader)

I there are:

“autoload_extras” (also marks UUID-linked records in “component loaders” to autoload), a script periodically attempts to parse the record into the normalized core Arctos structure. This may result in two things: * the record is created and marked for cache refresh, or * an error is returned in the status column

Does that help?

[DerekSikes]

"Are there any "extras" associatied with these records (parts with attributes, multiple identifications, locality attributes)?" this is the problem - how would I know? We have parts for each record and they have attributes (eg age class = adult) but are these extras? These were all entered using the old data entry form so I'm assuming there are no extras.... ???

…

-Derek

On Thu, Oct 14, 2021 at 9:32 AM Teresa Mayfield-Meyer < ***@***.***> wrote: I need to load specimen records. Are there any "extras" associatied with these records (parts with attributes, multiple identifications, locality attributes)? If not Once a record is marked to load by making status “autoload_core” (loads data from table bulkloader) I there are: “autoload_extras” (also marks UUID-linked records in “component loaders” to autoload), a script periodically attempts to parse the record into the normalized core Arctos structure. This may result in two things: * the record is created and marked for cache refresh, or * an error is returned in the status column Does that help? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFNUM4DXT5JOFKC6A3JMMLUG4H37ANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

[Jegelewicz]

We have parts for each record and they have attributes (eg age class = adult)

Age class is a catalog record attribute, which is not an extra. Part attributes would be things like part preservation, part identifier and part location.

In any case, if there are extras, there will be something other than ) in this column:

[DerekSikes]

Thanks! Now I am understanding better. Is there any harm in clicking the load core + extras button if there are no extras? Why not just click that one all the time in case there are extras?

…

-Derek

On Thu, Oct 14, 2021 at 11:45 AM Teresa Mayfield-Meyer < ***@***.***> wrote: We have parts for each record and they have attributes (eg age class = adult) Age class is a catalog record attribute, which is not an extra. Part attributes would be things like part preservation, part identifier and part location. In any case, if there are extras, there will be something other than ) in this column: [image: image] <https://user-images.githubusercontent.com/5725767/137385199-bfbf79d3-67aa-4933-a695-ec93bf2faf54.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFNUMZW3QNAWXBHBUIJSMDUG4XPNANCNFSM4XXF4QQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- +++++++++++++++++++++++++++++++++++ *Derek S. Sikes*, Curator of Insects, Professor of Entomology University of Alaska Museum (UAM), University of Alaska Fairbanks 1962 Yukon Drive, Fairbanks, AK 99775-6960 ***@***.*** phone: 907-474-6278 he/him/his University of Alaska Museum <https://www.uaf.edu/museum/collections/ento/> - search 357,704 digitized arthropod records <http://arctos.database.museum/uam_ento> +++++++++++++++++++++++++++++++++++ Interested in Alaskan Entomology? Join the Alaska Entomological Society and / or sign up for the email listserv "Alaska Entomological Network" at http://www.akentsoc.org/contact_us

[Jegelewicz]

Why not just click that one all the time in case there are extras?

I suppose it is possible that someone may want to review the extras or download and add something to them before they get loaded. It is an opportunity to review them in their own space without all the other stuff.