Fix that zero-byte files do not have a hash as reported by the OneDrive API thus should not generate an error message

[abraunegg]

• The OneDrive API does not present a hash for all files, most notably those that are zero byte in size (this may be fixed in the future). Add a wrapper to the existing makeItem function to test the file size before working out if this is a going to be a problem and then print out an error message if the file size is greater than 0

[abraunegg]

Testing Evidence

Using 'user' Config Dir: /home/alex/.config/onedrive-business/
Configuration file successfully loaded
Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveBusiness
Application version: v2.4.24-2-g1d406e0
Account Type: business
Default Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1099499773050
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Initializing the Synchronization Engine ...
Syncing changes and items from OneDrive ...
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 1099499773050
Processing 12 OneDrive items to ensure consistent local state
Skipping item - excluded by skip_dir config: target_excluded
Creating local directory: ./source
Creating local directory: source/folder_a
Downloading file source/folder_a/solaris crash.docx ... done.
Downloading file source/folder_a/Remove Ubuntu Package.docx ... 
Downloading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:01                                                                                                         
done.
Creating local directory: ./Music
Creating local directory: ./Double Folder
WARNING: OneDrive API inconsistency - this zero byte file does not have any hash: ./zero-byte-file
Downloading file ./zero-byte-file ... done.
Performing a database consistency and integrity check on locally stored data ... 
Uploading differences of ~/OneDriveBusiness
Processing .
The directory has not changed
Processing source
The directory has not changed
Processing source/folder_a
The directory has not changed
Processing source/folder_a/solaris crash.docx
The file has not changed
Processing source/folder_a/Remove Ubuntu Package.docx
The file has not changed
Processing Music
The directory has not changed
Processing Double Folder
The directory has not changed
Processing zero-byte-file
The file has not changed
Uploading new items of ~/OneDriveBusiness
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 1099499773050
Processing 12 OneDrive items to ensure consistent local state due to a full scan being triggered by actions on OneDrive
Skipping item - excluded by skip_dir config: target_excluded
WARNING: OneDrive API inconsistency - this zero byte file does not have any hash: ./zero-byte-file
Sync with OneDrive is complete

With a slight tweak locally to actually print a message on a known zero byte file .. this PR is known to be working as intended.

[yapakipe]

Hello @abraunegg,

After updating from 2.4.23 to 2.4.25 (local build), I'm getting LOTS of warnings about files that don't have a hash (warning introduced in commit 268dcf9 for this issue). They look like this:

• WARNING: OneDrive API inconsistency - this file does not have any hash: B2D4923D04704661!1191
• WARNING: OneDrive API inconsistency - this file does not have any hash: Some/Directories/Some File.docx

edit: I'm not sure what the intent of the commit was: To avoid errors that used to happen with empty files that do not have a hash, or to add this warning for non-empty files that do not have a hash. If the latter, then this is expected I guess, but quite scary when you see all of those new warnings :-) If the former, unless there's something special with my files, this might happen to lots of people and might generate a lot of requests for help here, so thought I'd give you a heads-up.

Even if this is a "safe" warning, i.e. nothing bad will happen when it pops up, there are so many of them that it pollutes the output in a way that I'm afraid might hide "real" errors.

Let me know if this is already known or expected - otherwise, here's some more info:

• I have a few hundred warnings (431), out of over 60000 local files (+ many skipped files using a sync_list)
• These warnings are all for non-empty files (as expected - just saying it's not the zero-size check that went wrong)
• A lot of them are OneNote-related files (sections etc.). These are not synced by this onenote client anyway,
• Most of the others are Office files (.docx, .xlsx, etc.)
  • However, not all Office files are affected, only some
  • I also had a .txt file
• In the warning, many of the files are identified by their database item ID instead of file path. Using --verbose --verbose I'm getting a lot more info, including thisItemFullPath which always gives the path. Maybe this could be used to make the warning more informative in these cases?
• Some of the files giving a warning are skipped by a sync_list, some are not, so it's not a skipped-files-only thing.
  • Files that are not skipped are there locally (they were synced previously with 2.4.23 and haven't "disappeared". I haven't tried a sync from scratch to see if they would be synced or not)
• The warnings remain even after a sync (i.e. they're not just one-time warnings that get "fixed" with a sync, they're persistent)
• They happen in monitor mode, synchronize mode, --resync or not, --dry-run or not...
• edit: Some or all of the office files might have been edited through Office Online (Office 365, Microsoft 365, whatever the name). I'm not sure but I could make some tests with new files and see if that's what causes the "no hash" problem.

I'm not attaching a whole log here as it's huge with the extra verbose output. Also I don't know if it contains any sensitive info.

Let me know if you'd like me to do anything else (opening a new issue, sending parts of a log, trying different things like a sync from scratch, trying some changes in the code - I'm not at all familiar with the OneDrive API but can try things if you point me in the right direction)...

(and thanks for your amazing work on this :-))

[abraunegg]

@yapakipe
Please do not raise a new issue as everything is operating as normal - for the zero byte files.

In v2.5.0 the checks have been enhanced further to check the mime type, thus if the mime type is for OneNote items no message is printed.

For the non-zero byte files you must open a bug report with Microsoft - I cannot fix their broken API.

[yapakipe]

Thanks @abraunegg.

Meanwhile I had a look at the code and recent changes and tried a couple of things, and I now understand a bit more - hope you won't mind my spamming this issue a bit more with some info for other people who might be wondering about this:

The sha1 hash being deprecated in the API and replaced with quickXorHash, and MS is encouraging people to make the change prior to the deprecation actually taking place (in July 2023). These recent changes @abraunegg made were exactly that.

In my case, the files giving the warnings have a valid sha1Hash, which is now being ignored by this onedrive client as per MS' recommendation. The problem is they do not (yet?) have a quickXorHash, which as @abraunegg said is a problem on the MS side: If they say you should change your code now, then they need to make sure the data is ready for this updated code.

This doc says "In some cases hash values may not be available. If this is the case, the hash values on an item will be updated after the item is downloaded.", but that doesn't seem to be working (at least for some files) as the warnings keep appearing.

This doesn't seem to affect the proper downloading of the files by this onedrive client - other than maybe re-downloading a file that wouldn't need to be. I might just wait and see if these files get a quickXorHash at some point, and if not, I'll reach out to OneDrive support...

Thanks again @abraunegg!

[abraunegg]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.