Takes too long to sync files

[lvazdela]

Hi,
I work with Ubuntu 22.04 in one computer and I used to sync just one folder with OneDrive using the following command:
onedrive --single-directory "PROYECTOS DE R" --synchronize and it worked fine. The folder currently holds 113,743 and with the previous version of onedrive it took a reasonable amount of time to synchronize (several minutes). I updated to version 2.5 and now it takes around 2 hours for fetching ant updating, what makes it impractical. I did a resync (it took around one day) but didn't solve the problem. I read carefully the documentation and I couldn´t find a solution. Am I doing something wrong? Can I do something to speed up the process?
Thanks
Luis

[abraunegg]

@lvazdela
This is 100% opposite to nearly every other comment or feedback where folk have indicated that the client is way faster in all aspects of operation.

So .. first questions are:

• How much free system memory does your system have?
• How much swap space does your system have?

The way you have described it, the application is swapping to disk - which will always make operations slower.

[lvazdela]

Thanks for your time, here is the information
How much free system memory and swap space does your system have?

lvazdela@lvazdela-Aspire-E5-551:~$ free -m

	Total	Used	free	shared	buf/cache	available
Mem	10900	2400	5613	100	2886	8118
Swap	2047	0	2047

I didn´t change any of this (I don´t know how to do it), the intriguing part is that with this configuration, the previous version worked fine

[abraunegg]

@lvazdela
In doing some testing with v2.5.0, below are the results:

System Specs

Data Size Locally

alex@ubuntu-22-04:~/OneDrive$ ls -laR | wc -l
100610
alex@ubuntu-22-04:~/OneDrive$ 

Calculations

Approx 100K items / 203 = ~500 API response bundles to process (this can be verified when using --verbose)

Execution Time for a --resync (v2.5.0)

A --resync is as intensive in operation as things can get

alex@ubuntu-22-04:~/onedrive$ time ./onedrive -s --resync --resync-auth   
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled
Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy4pdBroJVhyQpuN8F-Nxsjd .................................................................................................................................................................................................................................................................................................................................................................................................................. 
Processing 101202 applicable changes and items received from Microsoft OneDrive ............................................................................................ 
Performing a database consistency and integrity check on locally stored data .......................... 
Scanning the local file system '~/OneDrive' for new data to upload .......................... 
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy4pdBroJVhyQpuN8F-Nxsjd . 
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    12m27.994s
user    12m50.955s
sys     1m45.258s
alex@ubuntu-22-04:~/onedrive$ 

Execution Time without --resync (v2.5.0)

alex@ubuntu-22-04:~/onedrive$ time ./onedrive -s
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy4pdBroJVhyQpuN8F-Nxsjd .. 
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ......................... 
Scanning the local file system '~/OneDrive' for new data to upload .......................... 
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy4pdBroJVhyQpuN8F-Nxsjd . 
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    0m58.682s
user    0m46.087s
sys     0m15.427s
alex@ubuntu-22-04:~/onedrive$ 

Execution Time for --single-directory (v2.5.0)

alex@ubuntu-22-04:~/onedrive$ time ./onedrive -s --single-directory 'random_100k_files'
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
The OneDrive Client was asked to search for this directory online and create it if it's not located: ./random_100k_files
Generating a /delta response from the OneDrive API from this Item ID: 01PTBOWVX3VAOOCVN7PVFYBXXMJZCDBYNE .................................................................................................................................................................................................................................................................................. 
Finished processing self generated /delta JSON response from the OneDrive API
Processing 100101 applicable changes and items received from Microsoft OneDrive ......................................................................................................... 
Performing a database consistency and integrity check on locally stored data ............................ 
Scanning the local file system '~/OneDrive/random_100k_files' for new data to upload .......................... 
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Generating a /delta response from the OneDrive API from this Item ID: 01PTBOWVX3VAOOCVN7PVFYBXXMJZCDBYNE ................................................................................................................................................................................................................................................................................ 
Finished processing self generated /delta JSON response from the OneDrive API
Processing 100101 applicable changes and items received from Microsoft OneDrive ..................................................................................... 

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    16m59.721s
user    12m19.035s
sys     2m33.056s
alex@ubuntu-22-04:~/onedrive$ 

Now .. whilst 100K files and folders is not 113K files and folders ... an additional 13K is not going to add hours and/or days to how the client operates.

I am curious as to the claim your making.

[abraunegg]

Do not upgrade to v2.5.1 - it is broken.

Please wait for v2.5.2

[lvazdela]

Ok, thanks, in the meanwhile, I will work with the 2.4.25 version.
Happy weekend

[lvazdela]

Hi,
Just to finish this thread:
The documentation may be a little confusing:
1.- In https://github.com/abraunegg/onedrive/blob/master/docs/ubuntu-package-install.md the directions to download onedrive for Ubuntu 24.04 from opensuse are straight forward, but it doesn´t state what version of onedrive is going to be downloaded. only after installing onedrive you know that is the version 2.5.2
2.- In contrast, in https://github.com/abraunegg/onedrive/blob/master/docs/install.md (a link that appears in https://github.com/abraunegg/onedrive/blob/master/docs/install.md) the table advice to use onedrive 2.4.25 for ubuntu 24.04.

I was using Ubuntu 22.04 when started this thread, but I decided to update the OS to the 24.04 version. I followed the instructions from paragraph 1, and I ended up with the 2.5.2 version and the same problem I was trying to solve: takes too long to sync.
I removed the 2.5.2 version and uploaded manually the 2.4.25 version. This version seems to work pretty well for synchronizing single directories, it syncs around 5 GB from 4 directories in a couple of minutes.
Hope this information is useful
Thank you very much for your time.

[abraunegg]

The table does not advise to use v2.4.25 ...

It explicitly states do not install from the Ubuntu repositories.

That v2.4.25 is working faster is odd.

I would be interested in digging deeper as to why.

Please remember that on first use of v2.5.x there are database changes which makes the time to process longer ... but certainly not hours or days longer.

[lvazdela]

I apologize for not reading the caution part.
Please take a look to the log file with verbose activated using the 2.4.25 version:
logSinc.txt
The time it took:
real 1m1.179s
user 0m15.390s
sys 0m2.908s

The size of the Onedrive folder:
19 065 elementos, 11.4 GB en total (according to the file manager of Ubuntu 24.04)

Unfortunately, I didn´t make a log file of the performance of onedrive 2.5.2. After installing it, I made a resync, and then I did a few changes to some of the folders (editing a couple of files or adding a new one) , the sync lasted around 15 minutes, much less than before, but impractical. I did this testing twice. As you can see, for the same task , version 2.4.25 needed less than two minutes. What I remember is that it checked all the files, even those that didn't change.
If it's really important to you, I can remove the 2.4.25 version (manually) :( , install again the 2.5.2 version and send you the information about its performance.

[abraunegg]

@lvazdela
Your testing is 100% invalid for a number of reasons:

1. You are syncing with v2.4.25 , with the state of the client already in a healthy / sync state - thus, the client uses the /delta call, finds 0 changes, and immediatly goes to validation of items in the database
2. All your tests with v2.5.x has been on the first run and/or using --dry-run which is forcing a DB upgrade, thus a 100% way more intensive scan
3. When using v2.4.25 you are also using --single-directory to limit the sync scope. In v2.4.25 this still uses the /delta call but without a checkpoint, thus, it goes back to 'point 1' .. no changes , nothing transferred (which is not correct - a v2.4.25 bug)
4. When using v2.5.x , when using --single-directory the client correctly walks the 'tree' online to create a view of what should be local based on what is online .... a major operational difference to how v2.4.25 operated

So to be at least fair in any comparison:

1. Compare v2.4.25 and v2.5.2 when using this command: onedrive --synchronize --single-directory "PROYECTOS DE R" --resync
   • This will ensure that the local sync state is cleaned up, and that the actual operation is a fair comparison.
2. Compare v2.4.25 and v2.5.2 when using this command: onedrive --synchronize --single-directory "PROYECTOS DE R"
   • This will compare the 2 when the client is correctly in-sync

Now ... I 100% expect the following:

1. v2.5.2 to be slower because it is scanning the tree of your --single-directory to ensure that all online changes are correctly flagged
2. In a non-resync scenario, because of your use of --single-directory this will also be 'slower'

So to validate this, then instead of using --single-directory "PROYECTOS DE R" , use a 'sync_list' file to only scan that particular folder.

As the v2.5.2 client is now no longer performing an exhaustive scan of the online tree, there should a difference in how v2.5.2 is operating and how long the operations take. This being said - if 'PROYECTOS DE R' is a Shared Folder .. all bets are off - the exhaustive scan again will be used, because /delta cannot be used on a specific 'folder' on a shared folder with you .... this again is a major operation difference to v2.4.25 and closes a gap that changed files on shared folders were missed and/or not downloaded correctly with v2.4.25.

Lastly, you are using Ubuntu (your choice .... ) so you are using no doubt a broken curl version (got to love a distribution providing users with broken packages in the guise of stability .. again your choice). To work around these 'issues' add to a 'config' file the following options:

force_http_11 = "true"
ip_protocol_version = "1"

These are valid for v2.4.25 and v2.5.2

[lvazdela]

Hi,
Thanks for your response. Clearly, I was doing things incorrectly. Here is what I did now:
1.- Removed the old version.
2.- Uploaded the version 2.5.2
3.- Created the sync_file with the desired folders
4.- Added to the config file the options:

force_http_11 = "true"
ip_protocol_version = "1"

5.- I did a backup of the OneDrive folder and removed it
6.- I synced the desired folders with the commmand onedrive --sync
time elapsed:

real 187m17.603s
user 281m7.878s
sys 8m28.240s

At the end of the sync process, there was this warning:

Sync with Microsoft OneDrive has completed, however there are items that failed to sync.
To fix any download failures you may need to perform a --resync to ensure this system is correctly synced with your Microsoft OneDrive Account

7.- I did a resync with the command onedrive --resync
Time elapsed:

real 39m25.142s
user 28m43.152s
sys 1m8.432s
No errors appeared.

8.- I did a sync with de commmand onedrive --sync
Time elapsed:

real 0m24.502s
user 0m20.027s
sys 0m2.457s
No errors appeared.

I went to a usual working day at my office worked with some onedrive folders, came back to my house and synced my Ubuntu computer:
Time elapsed:

real 0m25.677s
user 0m21.398s
sys 0m2.089s
No errors.

In conclusion:

• I was misusing the command onedrive --single-directory option. It is not designed to select specific files to synchronize daily.
• The sync_file is the way to go when selecting files to sync.
• The Ubuntu flavor of Linux is not the best option for OneDrive users.
• onedrive is a reliable, efficient and fast option to synchronize Microsoft OneDrive.

Thank you very much for your time and patience to help me solve the problem.