Bug: --display-sync-status does not work when OneNote sections (.one files) are in your OneDrive.

[aseanwatson]

Describe the bug

When encountering files with no hash (OneNote .one files are good examples), onedrive --display-sync-status crashes.

Operating System Details

Linux onedrive2 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1

What is your OneDrive Application Configuration

onedrive version                             = v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1
Config path                                  = /home/sean/.config/onedrive
Config file found in config path             = true
Config option 'drive_id'                     =
Config option 'sync_dir'                     = /media/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /home/sean/onedrive_logs/
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     =
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = true
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            =
Config option 'azure_tenant_id'              =
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-alpha-4
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 1048576
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 30
Config option 'data_timeout'                 = 240
Config option 'ip_protocol_version'          = 0

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false
Selective Business Shared Items configured   = false

Config option 'webhook_enabled'              = false
Reading configuration file: /home/sean/.config/onedrive/config
Configuration file successfully loaded

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

/dev/disk/by-uuid/87a7d4d8-fe10-4426-8d96-ea5906729dad on / type btrfs (rw,relatime,idmapped,ssd,space_cache,user_subvol_rm_allowed,subvolid=1246,subvol=/containers/onedrive2)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,uid=1000000,gid=1000000,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,relatime)
udev on /dev/fuse type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/net/tun type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755,inode64)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711,inode64)
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/loadavg type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/slabinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/devices/system/cpu/online type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /var/snap/lxd/common/var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=32717580k,nr_inodes=8179395,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024)
devpts on /dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024)
none on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=1000000,gid=1000000,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=1000000,gid=1000000,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=13110176k,nr_inodes=819200,mode=755,uid=1000000,gid=1000000,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=1000000,gid=1000000,inode64)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
snapfuse on /snap/snapd/20290 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24322 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/2015 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,size=13110176k,nr_inodes=819200,mode=755,uid=1000000,gid=1000000,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
/dev/sdb1 on /media/OneDrive type btrfs (rw,relatime,ssd,space_cache,user_subvol_rm_allowed,subvolid=2230,subvol=/Backup)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=6555084k,nr_inodes=1638771,mode=700,uid=1001001,gid=1001001,inode64)

What are all your local file system partition types

NAME        FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0
loop1
loop3
loop4
loop5
loop6
loop7
loop8
sda
└─sda1
sdb
└─sdb1                               878.9G    76% /media/OneDrive
sdc
└─sdc1
nvme0n1
├─nvme0n1p1
├─nvme0n1p2
└─nvme0n1p3

How do you use 'onedrive'

This is on an lxc container on my "home server".

• I had a Google Pixel 5 which got me free storage on Google Photos. Google canceled that and I just upgraded to a Pixel 8, so now I'm out of space on Google.
• I have a OneDrive personal subscription which gets me 1TiB of storage. I had my phone also upload to OneDrive.
• I used Google Takeout to send all my photos (as tar.gz files) to OneDrive.
• Now I want to download my OneDrive content (including the Google tgz files), delete the redundant photos and push whatever content that was only in Google back to OneDrive.
• I want to periodically sync my photos and index them with PhotoPrisim to get facial recognition, etc. (Maybe someday OneDrive will do facial recognition.)

I'm still in the download everything on OneDrive phase of the project.

Steps to reproduce the behaviour

Steps to reproduce:

1. Make sure you have a OneNote section (.one) file in your OneDrive (personal or Business)
2. Run onedrive --display-sync-status --verbose --verbose

You'll get an error message which ends in this:

DEBUG: Existing Microsoft OneDrive Access Token Expires: 2023-Dec-31 07:21:21.3984928
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/35cab2153779effa/items/35CAB2153779EFFA!168/delta?token=aTE09NjM4Mzk1ODA0MzIwOTM7SUQ9MzVDQUIyMTUzNzc5RUZGQSExNjg7TFI9NjM4Mzk2MDA3MjAzODA7RVA9MjA7U0k9MTI7RExFUD0xO0xJRD0xMjM7RkQ9VHJ1ZTtCVT1UcnVlO1NHPTE7U089MjtQST0z
DEBUG: Setting next deltaLink to (@odata.deltaLink): https://graph.microsoft.com/v1.0/drives/35cab2153779effa/items/35CAB2153779EFFA!168/delta?token=aTE09NjM4Mzk1ODA0MzIwOTM7SUQ9MzVDQUIyMTUzNzc5RUZGQSExNjg7TFI9NjM4Mzk2MDA3MjEyNjA7RVA9MjA7U0k9MTI7RExFUD0xO1NHPTE7U089MjtQST0z
DEBUG:
DEBUG: Failure scope was called
DEBUG: Running performStandardExitProcess due to: failureScope
DEBUG: Shutdown OneDrive API instance
DEBUG: No valid Microsoft OneDrive webhook subscription to delete
DEBUG: Shutdown Sync Engine instance
DEBUG: Shutdown Client Side Filtering instance
DEBUG: Shutdown Application Configuration instance
DEBUG: Shutdown Database instance
DEBUG: Attempting to perform a database vacuum to merge any temporary data
DEBUG: Database vacuum is complete
DEBUG: Setting ALL Class Objects to null due to failure scope
DEBUG: Exit scope was called
DEBUG: Running performStandardExitProcess due to: exitScope
DEBUG: Application exit
std.json.JSONException@std/json.d(688): Key not found: hashes
----------------
/home/sean/dlang/dmd-2.106.0/linux/bin64/../../src/phobos/std/exception.d:521 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x56263616af40]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0x5626362be814]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(return scope immutable(char)[]) [0x5626362695d9]
src/sync.d:7053 void syncEngine.SyncEngine.queryOneDriveForSyncStatus(immutable(char)[]) [0x56263620ea07]
src/main.d:466 _Dmain [0x5626360a7610]

Complete Verbose Log Output

This is full of filenames and such; I don't think you need this to fix the problem.

Screenshots

No response

Other Log Information or Details

No response

Additional context

I made a local fix which gets me past the issue:

@@ -7043,37 +7043,42 @@ class SyncEngine {
                                        // remote drive item
                                        thisItemParentDriveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str;
                                        thisItemId = onedriveJSONItem["id"].str;
                                } else {
                                        // standard drive item
                                        thisItemParentDriveId = onedriveJSONItem["parentReference"]["driveId"].str;
                                        thisItemId = onedriveJSONItem["id"].str;
                                }

                                // Get the file hash
+                               if (hasHashes(onedriveJSONItem)) {
                                        thisItemHash = onedriveJSONItem["file"]["hashes"]["quickXorHash"].str;

                                        // Check if the item has been seen before
                                        Item existingDatabaseItem;
                                        existingDBEntry = itemDB.selectById(thisItemParentDriveId, thisItemId, existingDatabaseItem);

                                        if (existingDBEntry) {
                                                // item exists in database .. do the database details match the JSON record?
                                                if (existingDatabaseItem.quickXorHash != thisItemHash) {
                                                        // file hash is different, will trigger a download event
                                                        downloadSize = downloadSize + onedriveJSONItem["size"].integer;
                                                }
                                        } else {
                                                // item does not exist in the database
                                                // this item has already passed client side filtering rules (skip_dir, skip_file, sync_list)
                                                downloadSize = downloadSize + onedriveJSONItem["size"].integer;
                                        }
+                               } else {
+                                       // file hash data missing
+                                       writeln("Warning: \"" ~ onedriveJSONItem["name"].str ~ "\" lacks a 'hashes' section. Not counting its size.");
+                               }
                        }
                }

                // Was anything detected that would constitute a download?
                if (downloadSize > 0) {
                        // we have something to download
                        if (pathToQueryStatusOn != "/") {
                                writeln("The selected local directory via --single-directory is out of sync with Microsoft OneDrive");
                        } else {
                                writeln("The configured local 'sync_dir' directory is out of sync with Microsoft OneDrive");

[abraunegg]

@aseanwatson
Thanks for filling this issue and potential resolution. I am off tools at the moment due to the holiday period - will look into this when I can.

[abraunegg]

@aseanwatson
Also - for any other issue, please just note it here: #2570 with any details and/or crash or debug output. No need to raise a new issue for 'alpha' code.

[aseanwatson]

No need to raise a new issue for 'alpha' code.

Gotcha. Happy to make the process work for you. I'm going to be on vacation (in Costa Rica) for a week or so, so I'm trying to get this soaking bandwidth while I'm away. But that means little testing/feedback for me.

Happy New Year!

FWIW, I don't think it's guaranteed that a file with hashes will have an quickXorHash. As I recall, you may need to download the file it to force the service to compute it.

[abraunegg]

@aseanwatson

FWIW, I don't think it's guaranteed that a file with hashes will have an quickXorHash. As I recall, you may need to download the file it to force the service to compute it.

Microsoft depreciated sha1hash in the middle of last year - see: #2441 for details.

What this essentially means is that every file online will have a quickXorHash where possible as this is computed online, without having to first download the file.

There are edge cases - 0-byte files + onenote items ... 0-byte files was fixed pretty quickly, but the onenote items are just not supported by OneDrive API ...

[abraunegg]

@aseanwatson
Closing this issue as this is fixed in v2.5.0-alpha-4 GitHub version: v2.4.25-75-g76706f5 or greater

[abraunegg]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.