onedrive --synchronize --resync --verbose fails after several 1000 files downloads

[JBrolin90]

Thank you @abraunegg I have compiled the information according to your template and you'll find it below.

I should mention that I cannot be sure if it is a bug or if it something with my setup. It runs so stable and smoothly downloading one file per second or so. The suddenly just quietly drops out.

Secondly, you only need to use --resync if the application tells you to, and even then that is only a one time deal. By doing this each time you are starting from scratch each time - which is not ideal.

Of course. Kind of stupid but I just pressed the up-arrow and ran again, without thinking obviously...

Thirdly, an exit like you are describing should have some details behind it, however because you say that no error is being generated it is probably indicative of the following:

• Some sort of flaky Internet connection somewhere between you and the OneDrive service

Yes could be. However I have a good setup with fiber and guaranteed 100x100Mbit and my is connected with a cable. I find my internet connection quite stable. Also it runs for approximately the same time before dropping out to the prompt. I would therefore suspect a memory leak or something as one possibility. But I only installed onedrive a couple a days ago and know very little about it.

• Some sort of 'broken' HTTPS transparent inspection service inspecting your traffic somewhere between you and the OneDrive service

Yeah, that is a tough one. For what it is worth, I chose my ISP as they claim to be very strict with privacy and claim to not do any traffic prioritization. I can see no indication of man in the middle or proxies of any kind. But, then again, I wouldn't know for sure and it could be between the ISP and OneDrive...

Another observation from my side: I have a folder with several wine bottles. Up until now, all drops has been while processing those directories. those bottles contains many files with weird names in deep structure. But it is still too early to draw any conclusion from that observation, I think

I have a huge OneDrive structure with several 100's of gigabytes. I'm attempting to synchronize that structure to my Debian/Bullseye server. The command "onedrive --synchronize --resync --verbose" exits to command prompt after 1-2 hours of downloading files. No error message is given. While it is running, it downloads files successfully.
When I restart the command, it continues where it left off.

onedrive v2.4.7-1, installed from the Debian repository
OneDrive Personal 1TB quota
Debian Bullseye

uname -a Linux big-bull 5.9.0-2-amd64 #1 SMP Debian 5.9.6-1 (2020-11-08) x86_64 GNU/Linux
Desktop xfce4
Curl version: curl 7.72.0 (x86_64-pc-linux-gnu) libcurl/7.72.0 OpenSSL/1.1.1h zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.8.0 nghttp2/1.41.0 librtmp/2.3, Release-Date: 2020-08-19

joachim@big-bull:~$ onedrive --display-config
Configuration file successfully loaded
onedrive version                       = v2.4.7-1
Config path                            = /home/joachim/.config/onedrive
Config file found in config path       = true
Config option 'check_nosync'           = false
Config option 'sync_dir'               = /mnt/twotb/home/joachim/onedrives/joachim/
Config option 'skip_dir'               = 
Config option 'skip_file'              = ~*|.~*|*.tmp
Config option 'skip_dotfiles'          = false
Config option 'skip_symlinks'          = false
Config option 'monitor_interval'       = 300
Config option 'min_notify_changes'     = 5
Config option 'log_dir'                = /var/log/onedrive/
Config option 'classify_as_big_delete' = 1000
Config option 'sync_root_files'        = false
Selective sync 'sync_list' configured  = false
Business Shared Folders configured     = false

The sync folder is on a local ext4 spinning drive (sda2)
My os is on sdb1, an ssd

joachim@big-bull:~$ lsblk -f
NAME   FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINT
fd0                                                                                     
sda                                                                                     
├─sda1 ntfs         System Reserved 90CE8E40CE8E1F1A                                    
└─sda2 ext4   1.0                   3607b17d-1ecf-4e12-b634-23251b304a2e    1.6T     5% /mnt/twotb
sdb                                                                                     
├─sdb1 ext4   1.0                   48dbb9d2-f5d0-4d66-a5af-6cbc1af76196   81.9G    20% /
├─sdb2                                                                                  
└─sdb5 swap   1                     ebde49e1-fbf7-4274-878b-26244ba9b3a0                [SWAP]
sdc                                                                                     
├─sdc1 ntfs         System Reserved 6C64E4D464E4A1D6                                    
├─sdc2 ntfs         Win10 64        FEAAFA3EAAF9F34F                                    
└─sdc3 ntfs                         B83AA7FF3AA7B930   

My setup is that I use OneDrive cloud storage and sync it towards my PC, running Debian according to above. I have a dual boot setup with Windows 10. I have not run Win 10 for a little more than 1 month though. In any case, only one machine access the OneDrive cloud at a time.

I'll be sure to send you the information according to https://github.com/abraunegg/onedrive/wiki/Debugging-SSL-TLS-Issues-when-the-client-stops-syncing-with-no-crash

All the best and thank for bringing onedrive to the community
Joachim

[abraunegg]

@JBrolin90
Firstly, as you have not used the Issue Template - you are missing in your first post critical information to help assist you quickly - this is why there is a Bug Report Template .. please use it. Can you please update your first post with the missing information.

Secondly, you only need to use --resync if the application tells you to, and even then that is only a one time deal. By doing this each time you are starting from scratch each time - which is not ideal.

Thirdly, an exit like you are describing should have some details behind it, however because you say that no error is being generated it is probably indicative of the following:

• Some sort of flaky Internet connection somewhere between you and the OneDrive service
• Some sort of 'broken' HTTPS transparent inspection service inspecting your traffic somewhere between you and the OneDrive service

Follow the directions listed here: https://github.com/abraunegg/onedrive/wiki/Debugging-SSL-TLS-Issues-when-the-client-stops-syncing-with-no-crash to provide more details around your issue.

[Clear2Go]

I have the same issue and submitted a bug report -- at least it looks like the same issue. Different config maybe in that I have a sync_list so I am trying to slowly get to a full sync (and remove it).
Same issue though, runs for a while stops suddenly, no error. If it exits due to a network error, then it should at least say that somewhere, or simply try again?
I have a rather large number of files, and deep directory structure as well.

[abraunegg]

@Clear2Go
No error is generated from curl, thus no error can be trapped for or articulated. The only way to see the what the internal mechanics of curl is doing is to tell curl to output debugging information.

Because this issue cannot be trapped for, the application cannot be restarted

If you are however using the systemd service files, the application will restart

[JBrolin90]

Hello, I got the error 104. Below are the few last lines of the HTTPS log. There is a link to the file below as well

So, I assume it means that onedrive is working flawlessly. But maybe OneDrive is at fault.
Maybe it would be a good idea to state this behavior in the install and configuration documentation.

I will continue to restart the initial synch until I'm through with it.

* Connection #1 to host graph.microsoft.com left intact
* Found bundle for host graph.microsoft.com: 0x560f0f315f80 [serially]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (40.126.9.113) port 443 (#1)
> GET /v1.0/drives/296971cc8ea61a62/items/296971CC8EA61A62!165954/content?AVOverride=1 HTTP/1.1
Host: graph.microsoft.com
User-Agent: ISV|abraunegg|OneDrive Client for Linux/v2.4.7-1
Accept: */*

* OpenSSL SSL_read: Connection reset by peer, errno 104
* Connection died, retrying a fresh connect(retry count: 1)
* Closing connection 1

https://drive.google.com/file/d/1IZWrRd_Rv9r_-OG47ILenNpTEDKCGXUE/view?usp=sharing

[abraunegg]

@JBrolin90

Maybe it would be a good idea to state this behavior in the install and configuration documentation.

This issue is already listed in the known issues document: https://github.com/abraunegg/onedrive/blob/master/docs/known-issues.md#application-stops-running-without-any-visible-reason

As per your logfile, the OneDrive service reset your connection .. why .. who knows. I will review your logfile in full later to see if there is any other detail which can be gleaned.

I suggest you reach out to Microsoft as well .. as it is their service which is failing you.

[abraunegg]

@JBrolin90
One point of interest in your debug log is this:

< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"SliceC","Ring":"5","ScaleUnit":"005","RoleInstance":"AGSFE_IN_41"}}

I am fairly confident that each time that this has popped up - it has mainly been against the Europe Data Centre's where OneDrive is being hosted from. Checking the debug logs:

• Issue Client stuck at Continuing the upload session #494:
  • < x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"SliceC","Ring":"5","ScaleUnit":"001","RoleInstance":"AGSFE_IN_33","ADSiteName":"WEU"}}
• Issue Monitoring stops after a while #753:
  • I must have not saved the debug log archive to check, but I am confident that this was also a EU Server
• Issue Synchronization Stops due to 'OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104' #792:
  • < x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"South Central US","Slice":"SliceC","Ring":"4","ScaleUnit":"001","RoleInstance":"AGSFE_IN_80"}}
• Issue Download of large files (> 8 GB) fails (most of the time) due to 'OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104' #884:
  • < x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"4","ScaleUnit":"004","RoleInstance":"AGSFE_IN_52"}}

Yes could be. However I have a good setup with fiber and guaranteed 100x100Mbit and my is connected with a cable. I find my internet connection quite stable. Also it runs for approximately the same time before dropping out to the prompt. I would therefore suspect a memory leak or something as one possibility. But I only installed onedrive a couple a days ago and know very little about it.

Similar here. Fiber to the home, 100Mbit circuit, but in AU rather than EU, but never been able to reproduce this issue when testing against any server or different account types.

My only point of suspicion is some sort of mis-configured device within the Microsoft complex servicing OneDrive.

[abraunegg]

@JBrolin90
Closing this issue due to:

• Known issue
• Cause: OpenSSL SSL_read: Connection reset by peer, errno 104

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.