Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISSUE-21b(the return of the CSV) Islandora 7 migrations! ready to be used (almost?) #29

Merged
merged 8 commits into from
Jul 25, 2021

Conversation

DiegoPino
Copy link
Member

What? Islandora 7 importer ready for testing!

See #21 This is the second part, now full fledged with CSV header normalizer, cleaning out empty columns, parents, mapping for Compounds even if we DO not know what we will get and AMI Set creation at the end of the BATCH. Also some memory optimizations (this is heavy!)

Some stuff:

  • Better CSV header normalizing. WE get all fields first. The fetch. We keep track of the ones that actually have values per row. Once ready with the CSV we go back, make a diff on the headers and only keep the ones that are REALLY not empty and write a new clean CSV out.
  • trying to get rid of authority_marcrelator fields (may need more work?)
  • AMI set gets created and redirected to the Set.
    Tested with 7000 Objects, took 25 seconds?
  • Parent Object gets set into ismemberof
    @TODO:
  • Cleanups, probably more checks? Lots of checks here
  • Remove temp files/stuff around
  • Clear form states/steps
  • Change the message at the end (since we are redirecting we have a different message

@alliomeria this solves like 95% of the things we discussed today. the CSV is waaaayy less scary 🥧 and more 🌴

There are some cosmetics to do, but already looking better. I show at the end two CSVs. ON the message itself the original (like 700 columns!!) but in the Set itself, the attached one is clean, normalized, beautiful.

- Better CSV header normalizing. WE get all fields first. The fetch. We keep track of the ones that actually have values per row. Once ready with the CSV we go back, make a diff on the headers and only keep the ones that are REALLY not empty and write a new clean CSV out.
- trying to get rid of authority_marcrelator fields (may need more work?)
- AMI set gets created and redirected to the Set.
Tested with 7000 Objects, took 25 seconds?
- Parent Object gets set into ismemberof
`@TODO`:
- Cleanups, probably more checks? Lots of checks here
- Remove temp files/stuff around
- Clear form states/steps
- Change the message at the end (since we are redirecting we have a different message
@DiegoPino DiegoPino self-assigned this Jun 23, 2021
@DiegoPino DiegoPino added enhancement New feature or request Ingest Setup Knobs and Levers you move while thinking about feelings and metadata and CSV files UI Buttons and pixels labels Jun 23, 2021
@DiegoPino DiegoPino added this to the 0.2.0 milestone Jun 23, 2021
@DiegoPino DiegoPino linked an issue Jun 23, 2021 that may be closed by this pull request
@alliomeria
Copy link
Contributor

Hello @DiegoPino! Testing this now and really appreciating the extra csv-cleaning! 😃

Adds providesKeys and providesTypes. That way Solr data that comes back is not used for UI but only the keys we generate. Makes all cleaner and faster.
Also some UI improvements. Children mappings are only shown IF collapse is disabled. @alliomeria i could not reproduce the false collapsing? So wonder if there is something else missing in your code? Let's try this tomorrow again
Why do people make compounds and also part of a collection? Well well well
@DiegoPino
Copy link
Member Author

@alliomeria did you find any issues with this plugin? Things that I should fix before merging?

@alliomeria
Copy link
Contributor

@DiegoPino --so far all has been going very well with work related to using this plugin. ☺️ I have no major issues to report, and all the generated csv files I've reviewed have been clean and well formed.

I have a few small things I noticed and a couple questions about the pages for steps 4 & 5.

  • On the first page, in the Islandora Mappings section:

    • If you select "Collapse Multi Children Objects" and you do not fill in the "Required ADO mappings," you are not prevented from advancing to the Next page.
    • If you do not select "Collapse Multi Children Objects" and you do not fill in the "ADO mappings for Child Objects" (which has a red asterisk indicating it as required), you are prevented from advancing to the Next page.
    • In my local instance (so may be related to my particular setup), I am not seeing any error message/indication related to what's wrong/missing. If prevented from advancing, it might be helpful to have an error message display to help a user identify what they need to address.
  • Also for the first page, it may also be helpful to have either a "refresh" button or a message/note/instruction indicating how to refresh the page after selecting different options/a new collection. Making a change and hitting Next does refresh the page, but that might not always be clear, especially if none of the mappings change between different collections.

  • On the step 4 page, for "Required ADO mappings", how does the AMI generated csv file use the ADO Label mapping selected here? And are you thinking of removing the Sequence mapping? Or will this be used at some point in the future?

  • Are you also thinking of removing/disabling the step 5 page with the Zip file upload for the Solr Importer?

@DiegoPino
Copy link
Member Author

@alliomeria thanks for your feedback. Yes, most of the issues are on how I'm dealing with Form Steps validations/verifications. I think I can deal with that in another pull to make this one smaller (if you are Ok)

About your question

On the step 4 page, for "Required ADO mappings", how does the AMI generated csv file use the ADO Label mapping selected here? And are you thinking of removing the Sequence mapping? Or will this be used at some point in the future?

The label is used to provide a Label for NODE in case you forget/do not add a "label" key in your template. Its mostly redundancy and/or meant when you are using the "Direct" mode instead of the Twig template processor.

@alliomeria
Copy link
Contributor

@DiegoPino thanks to you for all of your coding developments! 🐝 Sounds good about the different pulls--whatever your own working preference of course.

Ah, thank you & got it about the ADO Label mapping, makes sense. I've been using all Custom + Twig template modes, so I did not see how/where it was being used.

@DiegoPino
Copy link
Member Author

@alliomeria will merge this so #14 can be added. Not perfect (will work on the UI issues in that new Pull) but usable and useful.

@DiegoPino DiegoPino merged commit d4b631a into 0.2.0 Jul 25, 2021
@DiegoPino DiegoPino deleted the ISSUE-21b branch July 25, 2021 22:35
@alliomeria
Copy link
Contributor

Sounds great @DiegoPino! Looking forward to using this soon! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Ingest Setup Knobs and Levers you move while thinking about feelings and metadata and CSV files UI Buttons and pixels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a FCREPO/Solr import plugin to help with Islandora 7 migrations
2 participants