Reimport Existing Project with Updated Text From USFM

[jag3773]

Story

As a checker, I want to be able to export a tC project as USFM, load it into another app, export from that app as USFM, and reimport into the same tC project so that I can make use of features in other apps without redoing work in tC.

As an aligner, I want to be able to make changes to my translation in a separate tool and then reimport the text from a USFM2/3 file so that I can realign the text that I changed.

Notes

• Existing check data should be preserved, only the text should be updated
  • For a USFM3 import with alignment data, the imported alignments should overwrite existing alignments in the project, but the importer should preserve any existing alignments which the new file does not have alignments for.
  • For a USFM2 import, the alignment data should not be affected except where the text is overwritten.
• User should be prompted to "recheck" any previous check that may have been modified by changes to the text
• Import/Export testing should happen against Paratext
• If a USFM3 is imported (not reimported) then it should also work
• Make sure "invalidated checks" are calculated during the reimport process.

Related to #3086 & #3763

[benjore]

[benjore]

• If the user tries to reimport a project, prepopulate the language field. If they change the language, then create it as a new project, not a reimport.
• Do a Git commit (so that eventually this merge can be undone).

[richmahn]

• [25] Code reimport for existing project with updated text
• [8] Write tests for reimport
• [2] PR Review for Reimport Existing Project with Updated Text From USFM #3559

[richmahn]

@benjore Is the "import" icon ok for this? The add file to folder icon is a bit misleading, plus doesn't exist in Bootstrap's icons. Here's an actual screenshot using bootstrap:

[benjore]

@richmahn I don't like that. Are there any other options?

[richmahn]

@benjore Easiest is to use the Bootstrap3 Glyphicons, but we could try to make another svg icon like we did for invalidate. https://getbootstrap.com/docs/3.3/components/

[richmahn]

@benjore I think the glyphicon-level-up is kind of similar to what you have, and has a nice "re"import feel to it (search for that icon name in the page)

[benjore]

@richmahn What I used was font awesome. I still like that one best.

[richmahn]

@benjore What I finally realize I need are how various prompts and warnings should look, from size of window, to buttons, to verbiage, and when in the process one can cancel the re-import due to them realizing things aren't quite right. Off the top of my head, realizing how much more "noisy" the reimport process is:

1. What to say when importing a bad file (not USFM) [I assume like in the import process]
2. Importing a USFM file but it isn't the same book as the project
3. Should there be a stepper like in a new project import (this is what I was originally making it to be like but now canning that):
   • Verses missing (just like with new import?)
   • how to show missing alignments and and checks for each tool (sectioned, each its own page?)
   • Anything else that needs to be displayed if wrong?

[richmahn]

@benjore Never mind about the first one, I already run the USFM validator which throws the proper message.

[benjore]

@richmahn I apologize for this, but I'm gonna change things slightly. It think it will improve the reimport process and hopefully will lessen your work.

When a user clicks "Reimport", the process should be the same as a regular import (e.g. don't automatically fill in (c) info, project info, etc.). The user should fill in the "Stepper" info as if it were a regular Import.

Currently, the user is only informed that the project is a duplicate after the whole "Stepper" is completed. This needs to change. When the user clicks "Continue" on the "Project Information" step of the "Stepper", tC should check to see if it is a reimport. That is when the reimport warning should be thrown. If I was trying to reimport an en_1th project, but accidentally select an en_2th project (and import it as such), then it should be imported as normal.

[richmahn]

I didn't even know we could delete projects. How does one go about doing that?

[richmahn]

Ah, you meaning manually from the File browser? Interesting...

[cckozie]

Going to fail this while doing further investigation.

[PhotoNomad0]

@cckozie - this may not make a difference, but are you doing "move to trash" or "delete immediately"?

[cckozie]

@PhotoNomad0 - I'm doing a 'move to trash'

[cckozie]

0.10.1 (ee3f8f5)

1. I have a local project named en_ult_act_book
   I did an online import of https://git.door43.org/tc02/en_act
   On the project details screen I set the Translation Identifier to 'ULT'
   Instead of being warning on that screen that a project with that name already exists, it lets me continue the import and at the end gives me this message:
   

2. When the user selects the 'Reimport' option on the 3dot menu, they should not be allowed to import a project that has a different language and/or book than the one selected on the project menu. Currently it works like a normal import with no messages other than the 'Project has been named..' one, and both projects now show on the projects page. (I am going to contact @benjore about moving this to a new issue that we'll deal with in the future)

3. When the user selects the 'Reimport' option on the 3dot menu, the project details screen should be pre-populated based on the meta data (language, book, translation id, nickname) for the project being reimported. (I am going to contact @benjore about moving this to a new issue that we'll deal with in the future)

4. Imported a good usfm file. Reimported the file again choosing the overwrite option, then loaded in tW. Same error encountered if changes were made to the original file before the reimport. Same error in wA. Same in 0.10.1 (b411d84) Looks like this may not be a problem on Win64 57-TIT.usfm.zip

5. Error: ** I AM NOT SEEING THIS PROBLEM IN 0.10.1 (b411d84) **

Overwrite/merge of usfm project fails when not in developer mode

Steps to reproduce

switch out of developer mode
import Titus USFM file to create new project
make a couple alignments
export as USFM3 file
reimport the USFM3 file just exported
set project details that same and click overwrite
select overwrite project
will see message that only New Testament projects are supported even though this is a Titus project.

[richmahn]

@cckozie Yes, #2 is because in Ben's diagram of this whole workflow, reimport isn't really known about until the user has selected a project and renamed it, then knowing it is a reimport. I think Ben spoke about removing it, but in my opinion it helps the user know he/she can reimport a project, yet this button only does local import, not both local and online. Probably best to do away with it. The menu "Reimport Project" simply does what the FAB does for "Import Local"

[richmahn]

@cckozie Understand all your issues on repopulating project details and such, as I brought all those up with Ben. Yet this has actually become more of an "overwrite" or "merge" feature (rather than a "reimport" feature) where the user should just go through the regular import of a local or online project, but once they give it an identifier/book/language which generates a project name, and that name exists, then they are prompted if they want to overwrite, and if they do, we keep alignment/selections as needed (importing new alignments if there is alignment data in the new import). So we really shouldn't have the "Reimport Project" option as that seem to be misleading and also what you base your expectation on as I did.

@benjore We probably need this to be considered "Overwrite" concept of imports with the same given project name. There is so much we don't know until they finally name the project and specify the book and language in the Project Details page to be able to force them to select the right book/language as Koz (and I) wanted for a reimport feature. All that is really needed is to do away with the "Reimport Project" context menu for projects and be told/prompted when existing project will be overwritten/merged.

[cckozie]

Recreate on 0.10.0 (db8cd53) with Hindi project.

[PhotoNomad0]

@richmahn - it looks like a timing problem. @RoyalSix doesn’t see the problem when he has debug tools open, but sees a crash when he doesn’t. Looks like the project is is not being deleted from the projects folder before trying to copy the new one from imports. Maybe try Spidel's fix of renaming the old project before deleting it and after that moving from imports? that way you shouldn't see any collisions when moving. And then if the renamed old project is still around, you could see if there was a permissions problem like a file/folder was locked.

[benjore]

Sorry to be late to the party!

2. When the user selects the 'Reimport' option on the 3dot menu, they should not be allowed to import a project that has a different language and/or book than the one selected on the project menu. Currently it works like a normal import with no messages other than the 'Project has been named..' one, and both projects now show on the projects page. (I am going to contact @benjore about moving this to a new issue that we'll deal with in the future)
3. When the user selects the 'Reimport' option on the 3dot menu, the project details screen should be pre-populated based on the meta data (language, book, translation id, nickname) for the project being reimported. (I am going to contact @benjore about moving this to a new issue that we'll deal with in the future)

Let's not do these for this release. We can move them into a new issue. @cckozie Seeing that this won't be possible, would you rather that we hide the "Reimport" option from the 3-dot menu?

Created #4756 to add back the Reimport option

[cckozie]

@richmahn As suggested above, please remove the 'Reimport' option from the 3 dot menu.

[cckozie]

0.10.0 (9b04f29) Using project 55-1TI.usfm.zip

The problem noted '5 days ago' above still exists and is possibly worse.

1. Import a Titus usfm2 project
2. Import the same usfm2 project giving it the same language and translation id as the previous import
3. Click the Overwrite button
4. Click the Overwrite Project button
5. Open the project in tW, see the error
   NOTE: This error occurs regardless of whether or not the second usfm2 file is identical to the first or if it has changes. It also does not matter if the first project imported is opened in tW before the second import is performed.

[cckozie]

On Win64 it did work when I imported the file a second time. However, when I imported it a third time I got the error again.

[cckozie]

0.10.0 (4566b9d)
I did a successful import, then overwrite multiple times, all with usfm2 files. Then I got this. It comes up after clicking the Overwrite button. I now get this message every time, even after restarting tC.
After deleting the Projects folder it does again work correctly.
Could this be tied to my being unable to delete the project's folder so I am renaming it?