Exception when file is not readable due to local file permissions on external mount point

[tigerjack]

Bug Report Details

Describe the bug
When a file cannot be read by the current user, the synchronization process crash.
It's happening now on wsl2. Sometimes it fails to set the file permission properly, so you can have file showing permission settings like ----------. Even if I think it's a bug on their side, I think that this kind of exception should be caught and managed differently.

Application and Operating System Details:

• Provide your OS & version (CentOS 6.x, Ubuntu 18.x etc) > ArchLinux on wsl2
• Are you using a headless system (no gui) or with a gui installed? > headless
• OneDrive Account Type > Business
• Did you build from source or install from a package? >package
• OneDrive Application Version: Output of onedrive --version > onedrive v2.4.7
• OneDrive Application Configuration: Output of onedrive --display-config > not relevant
• Provide the version of curl you are using: Output of curl --version > curl 7.73.0
• Is your configured 'sync_dir' a local directory or a network mount point? > Local
• What partition format type does your configured 'sync_dir' reside on? > NTFS
• Explain your entire configuration setup - The onedrive folder is shared between a pure ArchLinux OS, a Windows OS and ArchLinux running on wsl.

Complete Verbose Log Output
Application Log Output:

Using 'user' Config Dir: /home/simone/.config/onedrive/polimi/
Using 'system' Config Dir: 
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /mnt/***
Application version: v2.4.8
Account Type: business

Uploading differences of /mnt/***
The file has not changed
Processing ***
The directory has not changed
Processing ***.pdf
The file last modified time has changed
Giving up on sync after three attempts: Cannot open file `***.pdf' in mode `rb' (Permission denied)
std.exception.ErrnoException@std/stdio.d(429): Cannot open file `***.pdf' in mode `rb' (Permission denied)
----------------
??:? @safe void std.exception.bailOut!(std.exception.ErrnoException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x558bdcc10567]
??:? @safe shared(core.stdc.stdio._IO_FILE)* std.exception.enforce!(std.exception.ErrnoException).enforce!(shared(core.stdc.stdio._IO_FILE)*).enforce(shared(core.stdc.stdio._IO_FILE)*, lazy const(char)[], immutable(char)[], ulong) [0x558bdcd55a09]
??:? ref @safe std.stdio.File std.stdio.File.__ctor(immutable(char)[], scope const(char)[]) [0x558bdcd19f5c]
??:? immutable(char)[] util.computeQuickXorHash(immutable(char)[]) [0x558bdccdfba7]
??:? bool sync.testFileHash(const(immutable(char)[]), ref const(itemdb.Item)) [0x558bdccc4dc8]
??:? void sync.SyncEngine.uploadFileDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd1529]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bef]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.uploadDirDifferences(ref const(itemdb.Item), const(immutable(char)[])) [0x558bdccd0d86]
??:? void sync.SyncEngine.uploadDifferences(ref const(itemdb.Item)) [0x558bdccd0bda]
??:? void sync.SyncEngine.scanForDifferences(const(immutable(char)[])) [0x558bdcccf95d]
??:? void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool, long, bool, bool, bool, bool, monitor.Monitor) [0x558bdccb1c1d]
??:? _Dmain [0x558bdccb0404]
std.net.curl.CurlException@std/net/curl.d(4402): A libcurl function was given a bad argument on handle null
----------------
??:? pure @safe void std.exception.bailOut!(std.net.curl.CurlException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x558bdcd55bda]
??:? pure @safe bool std.exception.enforce!(std.net.curl.CurlException).enforce!(bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x558bdcd55b4b]
??:? void std.net.curl.Curl._check(int) [0x558bdcd2baef]
??:? void std.net.curl.Curl.clear(etc.c.curl.CurlOption) [0x558bdcd2be9a]
??:? void std.net.curl.HTTP.clearRequestHeaders() [0x558bdcd2abcf]
??:? void onedrive.OneDriveApi.shutdown() [0x558bdccb5894]
??:? _Dmain [0x558bdccb10f7]

Bug Report Checklist

• Detailed description
• Application and Operating System Details provided in full
• Reproduction steps (if applicable)
• Verbose Log Output from your error
• Debug Log generated and submitted

[abraunegg]

@tigerjack
Please can you provide a full verbose debug log via email, using the latest 'master' please if you can still reproduce this.

Your log / data is inconsistent, and this correlates to the bug which was fixed with #1140

Please rebuild your client to run onedrive v2.4.8-4-g5349340 and provide a full verbose debug log using that version.

[tigerjack]

Hi @abraunegg, is it the version on Arch Linux AUR the latest one? I see this aur/onedrive-abraunegg 2.4.8-1 from my package list. If not, I'll wait just until it get updated upstream and report back in case of errors.

[abraunegg]

@tigerjack
The version on AUR appears correct, but your application logfile is not.

Ensure you are up-to-date and please generate a verbose debug log as per support requirements.

[abraunegg]

@tigerjack
I have spent some time to replicate your environment without the debug file, and I have been able to replicate the issue.

To fix, please can you test the following PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/1179/head:pr1179
git checkout pr1179
./configure; make clean; make;

When running the PR, your version should be: onedrive v2.4.8-8-gfcbf79f

Validation of fix:

Using 'user' Config Dir: /home/alex/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /mnt/OneDrive/
Application version: v2.4.8-8-gfcbf79f
Account Type: personal
Default Drive ID: 66d53be8a5056eca
Default Root ID: 66D53BE8A5056ECA!101
Remaining Free Space: 5368287434
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 66D53BE8A5056ECA!101
Updated Remaining Free Space: 5368287434
Uploading differences of /mnt/OneDrive
Processing .
The directory has not changed
Processing random_files
The directory has not changed
Processing random_images
The directory has not changed
Processing random_videos
The directory has not changed
Processing Dropbox
The directory has not changed
Processing Music
The directory has not changed
Processing Music/audio003.mp3
The file has not changed
Processing Images
The directory has not changed
Processing Images/image003.jpg
The file has not changed
Processing Documents
The directory has not changed
Processing Documents/Uni
The directory has not changed
Processing Documents/Uni/01
The directory has not changed
Processing Documents/Uni/01/asdfasdf.txt

ERROR: The local file system returned an error with the following message:
  Error Message:    Documents/Uni/01/asdfasdf.txt: Permission denied
  Calling Function: readLocalFile()

Skipping processing this file as it cannot be read (file permissions or file corruption): Documents/Uni/01/asdfasdf.txt
Processing Documents/Uni/02
The directory has not changed
Processing Documents/Uni/02/qwerqwer.txt
The file has not changed
Processing Documents/Uni/03
The directory has not changed
Processing Documents/Uni/03/zxcvzxcvzxcv.txt
The file has not changed
Processing Documents/Uni/04
The directory has not changed
Processing Documents/Uni/04/fghjfghjfghj.txt
The file has not changed
Processing Documents/Uni/05
The directory has not changed
Processing Documents/Uni/05/dfghdfghdfgh.txt
The file has not changed
Processing Documents/Music
The directory has not changed
Processing Documents/Music/audio001.mp3
The file has not changed
Processing Documents/Images
The directory has not changed
Processing Documents/Images/image001.jpg
The file has not changed
Processing Documents/SpeedTest
The directory has not changed
Uploading new items of /mnt/OneDrive
Skipping item - has disappeared: ./Documents/SpeedTest/Bandwidth measurement_ Speedtest.net.xlsx
Skipping item - has disappeared: ./Documents/SpeedTest/automated_speedtest.log
Skipping item - has disappeared: ./Documents/SpeedTest/Bandwidth measurement_ Speedtest.net.csv

ERROR: The local file system returned an error with the following message:
  Error Message:    ./Documents/Uni/01/asdfasdf-mint-20-read-only-2.txt: Permission denied
  Calling Function: readLocalFile()

Skipping uploading this file as it cannot be read (file permissions or file corruption): ./Documents/Uni/01/asdfasdf-mint-20-read-only-2.txt

ERROR: The local file system returned an error with the following message:
  Error Message:    ./Documents/Uni/01/asdfasdf-mint-20-read-only-3.txt: Permission denied
  Calling Function: readLocalFile()

Skipping uploading this file as it cannot be read (file permissions or file corruption): ./Documents/Uni/01/asdfasdf-mint-20-read-only-3.txt

ERROR: The local file system returned an error with the following message:
  Error Message:    ./Documents/Uni/01/asdfasdf-mint-20-read-only.txt: Permission denied
  Calling Function: readLocalFile()

Skipping uploading this file as it cannot be read (file permissions or file corruption): ./Documents/Uni/01/asdfasdf-mint-20-read-only.txt
Applying changes of Path ID: 66D53BE8A5056ECA!101
Updated Remaining Free Space: 5368287434

[tigerjack]

It seems to work fine, thanks!!! I'll use the new executable for some time and I'll report back if something strange happens.

[tigerjack]

The only "problem" I see up to now is that I have tons of these files, so it's honestly hard to keep track of what happened in terms of uploading/download. Maybe it could be better to just use a summary reports at the end of the synchronization process?

[abraunegg]

@tigerjack

It seems to work fine, thanks!!! I'll use the new executable for some time and I'll report back if something strange happens.

Thanks for reporting back. Will mark as fixed and merge into 'master'

The only "problem" I see up to now is that I have tons of these files, so it's honestly hard to keep track of what happened in terms of uploading/download.

It sounds like your workflow has an issue. If this was a wider problem, this issue would have surfaced well before now, and be way more widespread / prevalent.

Maybe it could be better to just use a summary reports at the end of the synchronization process?

Not possible as the error is generated when the issue occurs, and the application is written in such a way to output the error and move on. This would require a total re-work of the application.

[tigerjack]

I think the main thing about my workflow is that I'm using onedrive through WSL 2. I think that most people, being already on Windows, would use the official onedrive client for this, but I came from the Linux world and I was already used to the cli application.

Also, another relevant aspect is that most of the file are on a separate partition of the hard disk and WSL is having problems right now to assign the file permissions correctly.

[abraunegg]

Also, another relevant aspect is that most of the file are on a separate partition of the hard disk and WSL is having problems right now to assign the file permissions correctly.

Suggest then you open a new issue with WSL due to the issues you are experiencing.

[tigerjack]

@abraunegg unfortunately, there's an already known issue. But thanks anyway for all your help and your work.

[tigerjack]

Not possible as the error is generated when the issue occurs, and the application is written in such a way to output the error and move on. This would require a total re-work of the application.

Just to make it available a quick shell redirection to show the contents separately that may be used by others to inspect which files were skipped. Basically it redirects the modified files to mods.log and all the errors to err.log.

onedrive --synchronize  1> >(grep -i -E "downloading|uploading|deleting" > mods.log) 2> >(grep -i "permission denied" > err.log)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.