Performance issues downloading from and uploading to Door43 in v1.1.0

[cckozie]

These timings were made using this project: https://git.door43.org/lrsallee/en_ult_act_book
Import timings ignore the time spent in the stepper.
v1.0.2
Import from Door43 - 28 seconds
Upload to Door43 - 15 seconds

v1.1.0
Import from Door43 - 61 seconds
Upload to Door43 - 279 seconds

---

v1.0.2 (new Mac 75Mbps download speed, 11Mbps upload speed)
Import from Door43 - 29 seconds
Upload to Door43 - 24 seconds
Reupload to Door43 (no changes) - 8 seconds

v.1.1.2 (new Mac, 75Mbps download speed, 11Mbps upload speed)
Import from Door43 - 48 seconds
Upload to Door43 - nearly 10 minutes

v1.1.3 (30f0e08) (With tweaks to isomorphic git)
Import from Door43 - 72 seconds
Upload to Door43 - 125 seconds
Reupload to Door43 (no changes) - 24 seconds

---

1.1.3 (64d8fb7) - Latest build
Download speed 50Mbps, Upload speed 10Mbps
Import from Door43 - 100 seconds
Upload to Door43 - 120 seconds
Reupload to Door43 (no changes) - 24 seconds

[PhotoNomad0]

@benjore @cckozie - In testing it looks like the major time consumer here is isomorphic-git (javascript replacement for separate git install). It is taking a major amount of time to do a save before push. This is why the 1.1 is taking much longer than 1.0.2. Updating the library from 0.40.4 to 0.51.12 did not improve performance significantly. Testing upload with aligned acts from imported USFM3 on a MacBook Pro:

iso-git version	save usfm3 time	git save time	git push time	total time	notes
0.51.12	4s	24s	6s	34s	1568 files, on Mac, incremental save, no changes - warning about experimental react feature
0.50.1	4s	23s	7s	34s	1568 files, on Mac, incremental save, no changes
0.40.4	4s	27s	4s	35s	1568 files, on Mac, incremental save, no changes

Note that the above is a best-case test, no alignment changes, only the first line of the USFM3 file changed (date stamp). But it is a 4MB file and there are 1568 files to check for changes.

iso-git version	save usfm3 time	git save time	git push time	total time	notes
0.50.1	5s	377s	163s	545s	3883 files, on Mac, Larry's project
0.50.1	11s	2055s	211s		3883 files, on Windows, Larry's project

[PhotoNomad0]

Short summary:
isomorphic-git is very slow on saving changes of a 4MB file with one line changed in a project with 1568 files. Also updating isomorphic-git does not measurably improve performance.

[PhotoNomad0]

Notes:

• By comparison the first commit with adding 1568 only takes 1 second longer than a commit with only usfm3 file changed.
• if we are having this problem with aligned acts, it will be much worse on psalms and Isaiah.

[cckozie]

@PhotoNomad0 - In regard to the 1.1.2 timings on the first comment above; what accounts for the extremely longer times using Larry's project rather than a new fully aligned Acts coming from USFM3 which takes about 20 seconds to upload? Is it the number of git commits or something else? I think we can all agree that a 10 minute upload for Acts on a good connection is not acceptable.

[benjore]

Thanks for this research @PhotoNomad0 ! I agree that this time is unacceptable. I'm looking forward to hearing what you find for potential solutions.

[PhotoNomad0]

Notes:
After debugging the git and talking with @neutrinog, it looks like he had to implement his own algorithms to check if a file needed to be added. Since isomorphic-git did not implement that natively. I'm looking at using a newer isomorphic-git and @neutrinog is testing a fix to his algorithm using features of a newer version of isomorphic-git.

[PhotoNomad0]

Notes:

After Joel's performance tweak to make use of newer version of isomorphic-git and its ability to stage whole project (rather than iterating a file at a time). Comparing after to before tweak times.

iso-git version	save usfm3 time	git save time	git push time	total time	notes
0.51.2	4s	10s	134s	148s	tweaks to use filesystem and Joel's performance fix, 3883 files, on Mac, Larry's file, ownership change
0.51.2	4s	347s	135s	486s	Joel's performance fix, 3883 files, on Mac, Larry's file, ownership change
0.50.1	5s	377s	163s	545s	3883 files, on Mac, Larry's file, ownership change

@benjore @cckozie - Joel is seeing much shorter times, but I am running AntiVirus checking.

[PhotoNomad0]

@cckozie - tested PR with iso-git tweaks. Probably best we can do quickly. So for review/discussion: https://tc-builds.door43.org/build/release-v1.1.3/translationCore-win-x64-1.1.3-64d8fb7.setup.exe

[cckozie]

Apples to Apples:
The import currently takes about 3 times as long as in 1.0.2
The initial upload takes about 5 times as long as in 1.0.2
The subsequent upload with no changes takes about 2 1/2 times as long as in 1.0.2
IMO we should seriously consider going back to native git or find some other way to get the performance within about 20% of the native performance.

[cckozie]

Passing this as there is a new issue to revert back to native git.