Personal vault files generating "WARNING: OneDrive API inconsistency"

[futymax]

Hello!
Today I was testing out the client and got the API Inconsistency error regarding file not having any hash. So, not being really happy with the information about it in the related issues I found here, went directly to trying to find some answers in the logs.
After reading the container log outputted with both double verbose and https was able to find that OneDrive does supply information about files in the Personal Vault in the api (but not fully).

Testing in a freshly downloaded copy of onedrive, single folder, download only and resync. Using default config file except for skip_file which is blank (because there are some files that needed syncing named like the default filter)

# We are being verbose
# Adding --verbose
# We are performing debug output
# Adding --verbose --verbose
# We are performing HTTPS debug output
# Adding --debug-https
# We are performing a --resync
# Adding --resync --resync-auth
# We are synchronizing in download-only mode
# Adding --download-only
# We are printing the application running configuration at application startup
# Adding --display-running-config
# We are synchronizing in single-directory mode
# Adding --single-directory ARG
# Launching onedrive
[DEBUG] homePath: HOME environment variable set
[DEBUG] homePath: /home/onedrive
[DEBUG] configDirName: CLI override to set configDirName to: /onedrive/conf
Using 'user' Config Dir: /onedrive/conf
[DEBUG] refreshTokenFilePath = /onedrive/conf/refresh_token
[DEBUG] deltaLinkFilePath = /onedrive/conf/delta_link
[DEBUG] databaseFilePath = /onedrive/conf/items.sqlite3
[DEBUG] databaseFilePathDryRun = /onedrive/conf/items-dryrun.sqlite3
[DEBUG] uploadStateFilePath = /onedrive/conf/resume_upload
[DEBUG] userConfigFilePath = /onedrive/conf/config
[DEBUG] syncListFilePath = /onedrive/conf/sync_list
[DEBUG] systemConfigFilePath = /config
[DEBUG] businessSharedFolderFilePath = /onedrive/conf/business_shared_folders
Configuration file successfully loaded
[DEBUG] passed in options: ["/usr/local/bin/onedrive", "--single-directory", "\"NasSync\"", 
"--display-running-config", "--download-only", "--resync", "--resync-auth", 
"--debug-https", "--monitor", "--syncdir", "/onedrive/data"]
[DEBUG] note --confdir and --verbose not listed in args
[DEBUG] sync_dir: No SHELL or USER environment variable configuration detected
[DEBUG] sync_dir: Getting syncDir from config value sync_dir
[DEBUG] syncDir: /onedrive/data
--------------- Application Runtime Configuration ---------------
onedrive version                             = v2.4.25
Config path                                  = /onedrive/conf
Config file found in config path             = true
Config option 'sync_dir'                     = /onedrive/data
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = 
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 6
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = true
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = true
Config option 'resync_auth'                  = true
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = true
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 600
Config option 'ip_protocol_version'          = 0
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = false
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false
-----------------------------------------------------------------

First HTTP call to get file information

[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/xxxxxx/items/xxxxxx!246/delta?
select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size

A few calls later, it reaches the offending file

[DEBUG] OneDrive Change: 
{
  "cTag": "xxxxxx",
  "deleted": {
    "state": "hardDeleted"
  },
  "eTag": "xxxxxx",
  "file": {
    "hashes": {
      "sha1Hash": "xxxxxx"
    }
  },
  "id": "xxxxxx!150524",
  "name": "",
  "parentReference": {
    "driveId": "xxxxxx",
    "driveType": "personal",
    "id": "xxxxxx!146271"
  },
  "size": 1320
}

And the corresponding warning just after

WARNING: OneDrive API inconsistency - this file does not have any hash: xxxxxx!150524

And there is some information about the folder 146271

[DEBUG] OneDrive Change: 
{
  "cTag": "xxxxxx",
  "deleted": {
    "state": "hardDeleted"
  },
  "eTag": "xxxxxx",
  "folder": {
    "childCount": 0,
    "view": {
      "sortBy": "name",
      "sortOrder": "ascending",
      "viewType": "thumbnails"
    }
  },
  "id": "xxxxxx!146271",
  "name": "",
  "parentReference": {
    "driveId": "xxxxxx",
    "driveType": "personal",
    "id": "xxxxxx!246"
  },
  "size": 0
}
[DEBUG] isRoot                                               = false
[DEBUG] !hasParentReferenceId(item)                          = false
[DEBUG] isItemDeleted(item)                                  = true
[DEBUG] Handling change as 'root item', has no parent reference or is a deleted item
[DEBUG] Remote deleted item
[DEBUG] The item we are syncing is a folder

And that folder is directly inside the root (246).

After some digging in Google about the meaning of the "hardDeleted" value I came across this post
https://stackoverflow.com/questions/58195061/access-onedrive-personal-vault-through-api
That gave me the idea of checking if there is anything inside my personal vault, and there were exactly 3 files (same quantity as warnings)
Opening the files in the OneDrive web to check the numbers revealed that those files were the ones being returned as HardDeleted.

So... Is there a way of configuring the docker to avoid having OneDrive return information about the vault?

[abraunegg]

@futymax

So... Is there a way of configuring the docker to avoid having OneDrive return information about the vault?

Not with v2.4.x and this will not be investigated.

Please test / try building v2.5.x RC2 as this have been totally rewritten and it would be interesting to understand how this is being handled.

[futymax]

Hi,
I will run some tests during the week and get back with results. I have a 2nd account with an empty drive I can use for testing purposes, as I rather not fiddle much with data I care not to loose.
Thanks in advance!

[abraunegg]

@futymax
So to fully answer this, there is zero issue with v2.5.x code for this problem.

My test account as the following files in the vault:

With v2.4.25 , the following occurs:

Configuration file successfully loaded
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
Initializing the Synchronization Engine ...
Syncing changes and items from OneDrive ...
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!431414
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!431415
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!431416
Performing a database consistency and integrity check on locally stored data ... 
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive
Sync with OneDrive is complete

With v2.5.x (specifically RC2 at the moment) , the following occurs:

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
Fetching items from the OneDrive API for Drive ID: 66d53be8a5056eca ....
Processing 513 applicable changes and items received from Microsoft OneDrive ..
Performing a database consistency and integrity check on locally stored data ..
Scanning the local file system '~/OneDrive' for new data to upload .
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: 66d53be8a5056eca .
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Shutting down Database instance
Shutdown of Database instance is complete

So in v2.4.x, the API data is analysed, and the Personal Vault items kick out a warning that there is no hash

In v2.5.x, the API data is analysed, and items that are hard deleted or in the Vault are correctly handled without a warning message being displayed.

[futymax]

I run some tests today, can confirm v2.5 does not display a warning.
Tested in a 2nd onedrive account with minimal files in a linux environment. Got to the same warning/issue using v2.4. Downloaded, compiled and ran v2.5 with the same configuration and no warnings shown up.

Thank you!

p.s: Is there any time window estimate for v2.5 release?

[abraunegg]

@futymax

p.s: Is there any time window estimate for v2.5 release?

I was hoping to go from RC2 --> Release .. however a few show stoppers have popped up. There will now be an RC3 release before Release ... at least another few months to ensure testing & quality.