component loaders (and data entry)

[dustymc]

The "data entry extras" functionality isn't as good as it could be, loading large batches of various components (identifiers, parts, identifications, etc.) causes timeouts then problems/confusion, the "claim" process for managing data entered via 'data entry extras' causes problems, etc. Let's fix it.

Very tentative suggestions, which may or may not hold up to reality:

• Attach all bulkloaders to a scheduled task (like the specimen bulkloader)
• Replace the "claim" functionality with an ability to change status in "not-yours" records (which would necessarily come with access to records created by users with whom you share collections)
• Rebuild the loaders to resolve UUIDs without first fetching guid_prefix (UUIDs are unlikely to be replicated, the second identifier isn't necessary)

A normal load would then be

• load
• validate (optional)
• set status to something (probably "autoload")
• check back later, find either
  • nothing, because it loaded and cleaned up after itself
  • errors, because you didn't validate

"Approving" records loaded by you or your students/techs/associates, via any process including data entry extras, would be

• set status
• check back later

I think that would be a significant simplification in both the code and the user experience. "Manage your..." might come with a "pick users" option (a slight increase in complexity), but most of the rest of the complexity (claim, find guid, etc.) that's been introduced for various reasons could be removed.

This has some urgency, I'd like to use #727 as a proof of concept, so I'm adding scary labels and will interpret a lack of immediate objections as enthusiastic approval.

[Jegelewicz]

I'm up for trying this method. Anything we can do to simplify and make the process consistent across tools would be nice.

[dustymc]

The basics of this are running in test with bulkload identifications. I think its worked out even better than anticipated, but timely feedback would be appreciated.

Replace the "claim" functionality with an ability to change status in "not-yours" records

The form is limited to manage_collection in order to safely (I hope!) accommodate this, and there's a new "shares collection" function which DOES NOT exclude users with locked accounts (so you can load things created by former techs & etc.).

Rebuild the loaders to resolve UUIDs without first fetching guid_prefix

This is implemented and tested, needs propagated to all other loaders

"validate" is part of the load process; there's no pre-validation. (Having this as a separate step has been a source of confusion for some time, this process facilitates a much simpler go/nogo approach.)

Todo, pending nobody finding a reason to go in a different direction:

• add component_loader and component_loader_notification to scheduler
• unschedule autoload_extras and dataentry_extras_notification
• test with Bulkloading identifications error #2936
• rebuild all component-loaders to use this system
• change "extras" notifications; need to report everything in loaders (it's all the same now) rather than attempting to pick out pieces. Use the new 'even if locked' function for this.
• figure out if we can/should merge "unloaders" into the same system (probably and probably?)

[Jegelewicz]

Now I gotta dig up some stuff to load....

[Jegelewicz]

/remind me to work on this tomorrow

[reminders]

@Jegelewicz set a reminder for Jul 31st 2020

[campmlc]

Cool. I will try too.

…

On Thu, Jul 30, 2020, 2:25 PM reminders[bot] ***@***.***> wrote: * [EXTERNAL]* @Jegelewicz <https://github.com/Jegelewicz> set a reminder for *Jul 31st 2020* — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2974 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBHA34YUBTKIKFLHGHDR6HJKJANCNFSM4PL36M5A> .

[reminders]

👋 @Jegelewicz, work on this

[dustymc]

Another major point for this model: it makes replication easy, there's now a testable locality-loader. I'll stop until I get some feedback, I don't want to replicate any problems.

The loader-scripts aren't scheduled, you can just open http://test.arctos.database.museum/ScheduledTasks/component_loader.cfm to process from the two new loaders.

[Jegelewicz]

When I follow that link - I get a white screen.

[Jegelewicz]

Let's go to Vegas!

[dustymc]

white screen.

Yea it's not very interactive - check back with the data, should be different. https://github.com/ArctosDB/internal/issues/65

Vegas

Sorry, I broke it!

[Jegelewicz]

OK, one more observation, when stuff won't load, it would help to get the error along with the csv when you download to fix stuff.

So I was able to load 10 localities - none had coordinates - I'll see if I can find a couple that do to try.

Also http://test.arctos.database.museum/ScheduledTasks/component_loader.cfm to process from the two new loaders. Needs to have some kind of interactivity...once you go there, you don't get out and we need people to understand that they have accomplished something. Assuming this will be true in production.

[dustymc]

get the error

wilco

interactivity

That's just test - it'll be on the scheduler in production, loading (or errors) will just happen (including for any number of records).

[Jegelewicz]

Clarification - So when I load a file directly to the tool, if stuff passes all the triggers, does it just load or will it always show up in the "manage" page first. Don't know why I can't decide what happens....

[dustymc]

You can load with status, and if you load with it as "autoload" then Arctos will take care of the rest (or make errors). If you follow the instructions and load from a fresh template then you'd need to set status (which gives you an opportunity to notice that you've just loaded 4582 duplicates...). How that's implemented and documented is a little waffly at the moment, but the potential for "stuff just happens" exists.

[dustymc]

This is in prod, need to integrate eg #2967 (comment) and rebuild all component-loaders under this umbrella.

Dropping priority.

[dustymc]

Need to check throttle; currently set for 10 records per run, can be upped significantly but needs monitored as things are added.

[ewommack]

Hey Arctos - Reminder to try and test this by next Thursday!

[dustymc]

See #3300 - make sure status (which can be errors) is urlencoded when necessary

[dustymc]

This has served its purpose, there's a template, it's awesome, closing.

[campmlc]

@gradyjt

[dustymc]

The next two tasks on my list (#2556, #2442) rely on this template. I can't seem to reconcile #3413 and the related AWG discussion. Do we love this or hate it? Can I keep building these things or do we need more discussion? Do I need to change something going forward? Do I need to change something with the ~dozen loaders I've already built under this model?

[Jegelewicz]

I think the tool is fine. It is just the related "documentation" that needs update, but others should weigh in.

[campmlc]

I think we could just add to the documentation, and I also suggested that rather than change the text on all the different component loaders, we change the main bulkloader to use the same terminology, e.g. mark to autoload vs mark to load etc - with a few sentences explanation on that form only.

…

On Tue, Feb 16, 2021 at 1:51 PM Teresa Mayfield-Meyer < ***@***.***> wrote: * [EXTERNAL]* I think the tool is fine. It is just the related "documentation" that needs update, but others should weigh in. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2974 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBEV7MQA2RFSXEBCW53S7LLDZANCNFSM4PL36M5A> .

[dustymc]

main bulkloader

If you mean the catalog record bulkloader, these are fundamentally different tools. The catalog record bulkloader is an independent tool - things in it load or error, that's it. "Component loaders" can have dependencies - things can hang around with 'autoload: ....' for weeks, then be processed after related data becomes available. That is, component loaders have three exits:

• load worked (so delete)
• load didn't work because data are a mess (so a person needs to become involved)
• load didn't work because dependent data are MIA (so try again later, no humans required)

I would be in favor of changing the actionable value of loaded to "autoload" rather than NULL for the catalog record bulkloader, but that should be addressed in a new issue.

[Jegelewicz]

Maybe we need to think about some way to let people jump to a specific set of data in tools like https://arctos.database.museum/tools/BulkloadOtherId.cfm

Currently there is an extra-long list of errors in there and if my username was after this person alphabetically, I'd have to scroll forever to get to my stuff.

This is just one page of it

Maybe just a table at the top that lists the usernames and lets you jump to a specific user's stuff?

[dustymc]

See https://github.com/ArctosDB/data-migration/issues/450#issuecomment-784555912 - verbose errors 400 lucee, need to POST or truncate errors or something.

Untested workaround: filter only on username, change status to something shorter.

@Jegelewicz

[Jegelewicz]

Thanks, that worked for my stuff...

[dustymc]

v1.1: csv download should include this to strip unnecessary columns

<cfset flds=mine.columnlist>
<cfif listfindnocase(flds,'key')>
<cfset flds=listdeleteat(flds,listfindnocase(flds,'key'))>	
</cfif>
<cfif listfindnocase(flds,'last_ts')>
<cfset flds=listdeleteat(flds,listfindnocase(flds,'last_ts'))>	
</cfif>
....
<cfset csv = util.QueryToCSV2(Query=mine,Fields=flds)>

[dustymc]

Moved unfulfilled requests to #3463, closing (again).