Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gdrive: raises unexpected error - name: drive version: v2 (again) #7949

Closed
BrettRyland opened this issue Jun 29, 2022 · 19 comments · Fixed by #8142 or #8564
Closed

gdrive: raises unexpected error - name: drive version: v2 (again) #7949

BrettRyland opened this issue Jun 29, 2022 · 19 comments · Fixed by #8142 or #8564
Assignees
@daavoo
Labels
build Issues/features related to building dvc install packages. fs: gdrive Related to the GDrive filesystem p1-important Important, aka current backlog of things to do research

Comments

@BrettRyland
Copy link

Bug Report

Description

As reported here, the latest version (2.11.0) of dvc on Ubuntu 22.04 LTS still has the bug reported in issue #5618.
Downgrading dvc to 2.10.2 allows the push to work properly.

Reproduce

On an existing repository using gdrive, add some files then try to push them.

brett@br-workhorse:~/repos/arctic-ice$ dvc status
models.dvc:                                                                                                                                                                                                                                                                                           
        changed outs:
                modified:           models

brett@br-workhorse:~/repos/arctic-ice$ dvc add models
100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:01,  1.52s/file]

brett@br-workhorse:~/repos/arctic-ice$ dvc push                                                                                                                                                                                                                                                       
ERROR: unexpected error - name: drive  version: v2                                                                                                                                                                                                                                                    

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

With extra verbosity:

brett@br-workhorse:~/repos/arctic-ice$ dvc push -vv
2022-06-29 19:59:18,216 TRACE: Namespace(all_branches=False, all_commits=False, all_tags=False, cd='.', cmd='push', cprofile=False, cprofile_dump=None, func=<class 'dvc.commands.data_sync.CmdDataPush'>, glob=False, instrument=False, instrument_open=False, jobs=None, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False), pdb=False, quiet=0, recursive=False, remote=None, run_cache=False, targets=[], verbose=2, version=None, viztracer=False, viztracer_depth=None, with_deps=False, yappi=False)
2022-06-29 19:59:18,399 TRACE:    16.49 ms in collecting stages from /home/brett/repos/arctic-ice
2022-06-29 19:59:18,405 TRACE:     6.02 ms in collecting stages from /home/brett/repos/arctic-ice/RadarMask
2022-06-29 19:59:18,406 TRACE:    18.37 mks in collecting stages from /home/brett/repos/arctic-ice/RadarMask/RadarMask
2022-06-29 19:59:18,420 DEBUG: Preparing to transfer data from '/home/brett/repos/arctic-ice/.dvc/cache' to '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,420 DEBUG: Preparing to collect status from '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,420 DEBUG: Collecting status from '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,423 DEBUG: Querying 11 hashes via object_exists                                                                                                                                                                                                                                   
2022-06-29 19:59:18,635 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,637 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,638 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,639 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,641 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,642 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,642 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,643 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,644 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,645 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,645 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,647 ERROR: unexpected error - name: drive  version: v2                                                                                                                                                                                                                            
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/cli/__init__.py", line 185, in main
  File "dvc/cli/command.py", line 22, in do_run
  File "dvc/commands/data_sync.py", line 58, in run
  File "dvc/repo/__init__.py", line 49, in wrapper
  File "dvc/repo/push.py", line 68, in push
  File "dvc/data_cloud.py", line 109, in push
  File "dvc/data_cloud.py", line 88, in transfer
  File "dvc_data/transfer.py", line 159, in transfer
  File "dvc_data/status.py", line 179, in compare_status
  File "dvc_data/status.py", line 136, in status
  File "dvc_data/status.py", line 43, in _indexed_dir_hashes
  File "tqdm/std.py", line 1195, in __iter__
  File "dvc_objects/db.py", line 355, in list_hashes_exists
  File "concurrent/futures/_base.py", line 611, in result_iterator
  File "concurrent/futures/_base.py", line 439, in result
  File "concurrent/futures/_base.py", line 388, in __get_result
  File "concurrent/futures/thread.py", line 57, in run
  File "dvc_objects/fs/base.py", line 263, in exists
  File "pydrive2/fs/spec.py", line 225, in exists
  File "pydrive2/fs/spec.py", line 192, in _get_item_id
  File "pydrive2/fs/spec.py", line 174, in _path_to_item_ids
  File "pydrive2/fs/spec.py", line 170, in _get_cached_item_ids
  File "funcy/objects.py", line 50, in __get__
  File "funcy/objects.py", line 28, in __get__
  File "pydrive2/fs/spec.py", line 82, in _ids_cache
  File "pydrive2/fs/spec.py", line 132, in _gdrive_list
  File "funcy/objects.py", line 28, in __get__
  File "pydrive2/fs/spec.py", line 100, in _list_params
  File "funcy/decorators.py", line 45, in wrapper
  File "funcy/flow.py", line 127, in retry
  File "funcy/decorators.py", line 66, in __call__
  File "pydrive2/fs/spec.py", line 121, in _gdrive_shared_drive_id
  File "pydrive2/auth.py", line 64, in _decorated
  File "pydrive2/auth.py", line 671, in Authorize
  File "googleapiclient/_helpers.py", line 130, in positional_wrapper
  File "googleapiclient/discovery.py", line 287, in build
  File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive  version: v2
------------------------------------------------------------
2022-06-29 19:59:19,332 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/cli/__init__.py", line 185, in main
  File "dvc/cli/command.py", line 22, in do_run
  File "dvc/commands/data_sync.py", line 58, in run
  File "dvc/repo/__init__.py", line 49, in wrapper
  File "dvc/repo/push.py", line 68, in push
  File "dvc/data_cloud.py", line 109, in push
  File "dvc/data_cloud.py", line 88, in transfer
  File "dvc_data/transfer.py", line 159, in transfer
  File "dvc_data/status.py", line 179, in compare_status
  File "dvc_data/status.py", line 136, in status
  File "dvc_data/status.py", line 43, in _indexed_dir_hashes
  File "tqdm/std.py", line 1195, in __iter__
  File "dvc_objects/db.py", line 355, in list_hashes_exists
  File "concurrent/futures/_base.py", line 611, in result_iterator
  File "concurrent/futures/_base.py", line 439, in result
  File "concurrent/futures/_base.py", line 388, in __get_result
  File "concurrent/futures/thread.py", line 57, in run
  File "dvc_objects/fs/base.py", line 263, in exists
  File "pydrive2/fs/spec.py", line 225, in exists
  File "pydrive2/fs/spec.py", line 192, in _get_item_id
  File "pydrive2/fs/spec.py", line 174, in _path_to_item_ids
  File "pydrive2/fs/spec.py", line 170, in _get_cached_item_ids
  File "funcy/objects.py", line 50, in __get__
  File "funcy/objects.py", line 28, in __get__
  File "pydrive2/fs/spec.py", line 82, in _ids_cache
  File "pydrive2/fs/spec.py", line 132, in _gdrive_list
  File "funcy/objects.py", line 28, in __get__
  File "pydrive2/fs/spec.py", line 100, in _list_params
  File "funcy/decorators.py", line 45, in wrapper
  File "funcy/flow.py", line 127, in retry
  File "funcy/decorators.py", line 66, in __call__
  File "pydrive2/fs/spec.py", line 121, in _gdrive_shared_drive_id
  File "pydrive2/auth.py", line 64, in _decorated
  File "pydrive2/auth.py", line 671, in Authorize
  File "googleapiclient/_helpers.py", line 130, in positional_wrapper
  File "googleapiclient/discovery.py", line 287, in build
  File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive  version: v2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc_objects/fs/generic.py", line 68, in _try_links
  File "dvc_objects/fs/generic.py", line 28, in _link
  File "dvc_objects/fs/base.py", line 282, in reflink
  File "dvc_objects/fs/implementations/local.py", line 157, in reflink
  File "dvc_objects/fs/system.py", line 115, in reflink
  File "dvc_objects/fs/system.py", line 101, in _reflink_linux
OSError: [Errno 95] Operation not supported

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc_objects/fs/generic.py", line 127, in _test_link
  File "dvc_objects/fs/generic.py", line 76, in _try_links
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/arctic-ice/.dvc/cache/.LAtLtAtBaEGib7LXwXJKVF.tmp'
2022-06-29 19:59:19,337 DEBUG: Version info for developers:
DVC version: 2.11.0 (deb)
---------------------------------
Platform: Python 3.8.3 on Linux-5.15.0-37-generic-x86_64-with-glibc2.14
Supports:
        azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
        gdrive (pydrive2 = 1.10.1),
        gs (gcsfs = 2022.5.0),
        hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
        s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
        ssh (sshfs = 2022.6.0),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-06-29 19:59:19,338 DEBUG: Analytics is enabled.
2022-06-29 19:59:19,339 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp0h_rt9la']'
2022-06-29 19:59:19,341 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp0h_rt9la']'

Expected

The push to succeed without errors.

Environment information

Output of dvc doctor:

brett@br-workhorse:~/repos/arctic-ice$ dvc doctor
DVC version: 2.11.0 (deb)
---------------------------------
Platform: Python 3.8.3 on Linux-5.15.0-37-generic-x86_64-with-glibc2.14
Supports:
        azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
        gdrive (pydrive2 = 1.10.1),
        gs (gcsfs = 2022.5.0),
        hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
        s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
        ssh (sshfs = 2022.6.0),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git

This is running with python 3.10.4 in a virtual environment set up with direnv.

brett@br-workhorse:~/repos/arctic-ice$ python --version
Python 3.10.4

brett@br-workhorse:~/repos/arctic-ice$ cat .envrc 
layout python-venv
brett@br-workhorse:~/repos/arctic-ice$ cat .direnv/python-venv-3.10.4/pyvenv.cfg 
home = /usr/bin
include-system-site-packages = false
version = 3.10.4

brett@br-workhorse:~/repos/arctic-ice$ which dvc
/usr/bin/dvc

brett@br-workhorse:~/repos/arctic-ice$ pip freeze -l
asttokens==2.0.5
backcall==0.2.0
certifi==2021.10.8
charset-normalizer==2.0.12
cycler==0.11.0
decorator==5.1.1
executing==0.8.3
fonttools==4.33.3
idna==3.3
imageio==2.18.0
imgaug==0.4.0
ipython==8.2.0
jedi==0.18.1
kiwisolver==1.4.2
matplotlib==3.5.1
matplotlib-inline==0.1.3
memory-profiler==0.60.0
networkx==2.8
numpy==1.22.3
opencv-contrib-python==4.5.5.64
opencv-python==4.5.5.64
packaging==21.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.0
Pillow-SIMD==8.3.2.post0
prompt-toolkit==3.0.29
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.12.0
pyparsing==3.0.8
python-dateutil==2.8.2
PyWavelets==1.3.0
requests==2.27.1
scikit-image==0.19.2
scipy==1.8.0
Shapely==1.8.1.post1
six==1.16.0
stack-data==0.2.0
tifffile==2022.4.26
torch==1.11.0
torchvision==0.12.0
tqdm==4.64.0
traitlets==5.1.1
typing_extensions==4.2.0
urllib3==1.26.9
wcwidth==0.2.5
@daavoo daavoo added fs: gdrive Related to the GDrive filesystem upstream Issues which need to be resolved in an upstream dependency bug build Issues/features related to building dvc install packages. and removed upstream Issues which need to be resolved in an upstream dependency labels Jun 29, 2022
@daavoo
Copy link
Contributor

daavoo commented Jun 30, 2022

Seems like an issue with pyinstaller hook added in #5619

I the latest deb build the hook-googleapiclient.model.py didn't work as expected, giving this warning:

38027 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...
38070 WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.

However, in the previous build (2.10.2) the hook worked as intended:

36358 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...
36436 INFO: Loading module hook 'hook-dvc.utils.flatten.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...

@daavoo
Copy link
Contributor

daavoo commented Jun 30, 2022

Issue appears to come from latest Pyinstaller release -> pyinstaller/pyinstaller#6789

daavoo added a commit that referenced this issue Jun 30, 2022
@daavoo
Downgrade pyinstaller
3d694e4 
Per #7949 (comment)
@daavoo daavoo mentioned this issue Jun 30, 2022
Merged
efiop pushed a commit that referenced this issue Jun 30, 2022
@daavoo @efiop
Downgrade pyinstaller
de8ab04 
Per #7949 (comment)
@daavoo daavoo mentioned this issue Jul 4, 2022
Closed
@daavoo daavoo mentioned this issue Jul 11, 2022
Merged
@shcheklein
Copy link
Member

shcheklein commented Jul 19, 2022 •
edited
Loading

@daavoo should we merge the bump (5.2) and release the new version?

See also this iterative/PyDrive2#197 (comment) and iterative/PyDrive2#197 (comment)

Those might be relevant, maybe we can do something on the static_discovery=False to not depend on that file? A bit of research is required.

This issue I think is causing this https://stackoverflow.com/questions/73025023/dvc-pull-returns-error-configuration-error-failed-to-authenticate-gdrive-remo problem.

@sorenwacker
Copy link

sorenwacker commented Jul 19, 2022 •
edited
Loading

I wonder, why does it work on a local machine, but not on GitHub actions? Maybe it is possible to create a workaround until this is fixed?

My GitHub actions job runs Python 3.8.10 (python --version) btw. (dvc doctor returns python 3.9)


argcomplete==2.0.0
attrs==[19](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:20).3.0
Automat==0.8.0
blinker==1.4
certifi==[20](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:21)19.11.28
chardet==3.0.4
Click==7.0
cloud-init==22.2
colorama==0.4.3
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
crcmod==1.7
crit==0.0.1
cryptography==2.8
dbus-python==1.2.16
distro==1.4.0
distro-info===0.23ubuntu1
entrypoints==0.3
httplib2==0.14.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.5.0
incremental==16.10.1
Jinja2==2.10.1
jsonpatch==1.22
jsonpointer==2.0
jsonschema==3.2.0
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
MarkupSafe==1.1.0
more-itertools==4.2.0
netifaces==0.10.4
oauthlib==3.1.0
packaging==[21](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:22).3
pexpect==4.6.0
pipx==1.1.0
protobuf==3.6.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
Pygments==2.3.1
PyGObject==3.36.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pyparsing==3.0.9
pyparted==3.11.2
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.7
python-debian===0.1.36ubuntu1
PyYAML==5.3.1
requests==2.[22](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:23).0
requests-unixsocket==0.2.0
SecretStorage==2.3.1
service-identity==18.1.0
simplejson==3.16.0
six==1.14.0
sos==4.3
ssh-import-id==5.10
systemd-python==[23](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:24)4
Twisted==18.9.0
ubuntu-advantage-tools==27.9
ufw==0.36
urllib3==1.[25](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:26).8
userpath==1.8.0
wadllib==1.3.3
WALinuxAgent==2.2.[46](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:47)
zipp==1.0.0
zope.interface==4.7.1

dvc doctor

Platform: Python 3.9.13 on Linux-5.15.0-[101](https://github.com/sorenwacker/dvc-test-project-01/runs/7407189429?check_suite_focus=true#step:4:102)4-azure-x86_64-with-glibc2.31
Supports:
	azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
	gdrive (pydrive2 = 1.10.1),
	gs (gcsfs = 2022.5.0),
	hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
	s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
	ssh (sshfs = 2022.6.0),
	oss (ossfs = 2021.8.0),
	webdav (webdav4 = 0.9.7),
	webdavs (webdav4 = 0.9.7)

@karajan1001 karajan1001 added this to DVC Jul 19, 2022
@karajan1001 karajan1001 moved this to Backlog in DVC Jul 19, 2022
@dberenbaum dberenbaum added the p1-important Important, aka current backlog of things to do label Jul 19, 2022
@efiop efiop self-assigned this Jul 19, 2022
@efiop efiop added the release-blocker Blocks a release label Jul 19, 2022
@efiop efiop moved this from Backlog to In Progress in DVC Jul 19, 2022
@elite3312
Copy link

elite3312 commented Jul 21, 2022 •
edited by dtrifiro
Loading

I'm running on Windows 10, and had the same error message when running dvc pull.
The windows installation for dvc appears to be the cause for me. I removed the windows dvc installation, and used pip to install dvc instead. Had to manually add a dependency though.

py -m pip install 'dvc[gdrive]'
py -m pip install pydrive2

And after this dvc pull worked fine again.

@sorenwacker
Copy link

That definitely changed the error message.

aiohttp==3.8.1
aiohttp-retry==2.5.2
aiosignal==1.2.0
appdirs==1.4.4
argcomplete==2.0.0
async-timeout==4.0.2
asyncssh==2.11.0
atpublic==3.0.1
attrs==19.3.0
Automat==0.8.0
blinker==1.4
cachetools==5.2.0
certifi==2019.11.28
cffi==1.15.1
chardet==3.0.4
charset-normalizer==2.1.0
Click==7.0
cloud-init==22.2
colorama==0.4.3
command-not-found==0.3
commonmark==0.9.1
configobj==5.0.6
constantly==15.1.0
crcmod==1.7
crit==0.0.1
cryptography==37.0.4
dbus-python==1.2.16
dictdiffer==0.9.0
diskcache==5.4.0
distro==1.4.0
distro-info==0.23ubuntu1
dpath==2.0.6
dulwich==0.20.45
dvc==2.13.0
dvc-data==0.0.23
dvc-objects==0.0.23
dvc-render==0.0.6
dvclive==0.9.0
entrypoints==0.3
flatten-dict==0.4.2
flufl.lock==7.0
frozenlist==1.3.0
fsspec==2022.5.0
ftfy==6.1.1
funcy==1.17
future==0.18.2
gitdb==4.0.9
GitPython==3.1.27
google-api-core==2.8.2
google-api-python-client==2.54.0
google-auth==2.9.1
google-auth-httplib2==0.1.0
googleapis-common-protos==1.56.4
grandalf==0.6
httplib2==0.20.4
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.5.0
importlib-resources==5.8.0
incremental==16.10.1
Jinja2==2.10.1
jsonpatch==1.22
jsonpointer==2.0
jsonschema==3.2.0
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
mailchecker==4.1.18
MarkupSafe==1.1.0
more-itertools==4.2.0
multidict==6.0.2
nanotime==0.5.2
netifaces==0.10.4
networkx==2.8.5
oauth2client==4.1.3
oauthlib==3.1.0
packaging==21.3
pathspec==0.9.0
pexpect==4.6.0
phonenumbers==8.12.52
pip==20.0.2
pipx==1.1.0
protobuf==4.21.2
psutil==5.9.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.21
pydot==1.4.2
PyDrive2==1.13.0
pygit2==1.9.2
Pygments==2.12.0
PyGObject==3.36.0
pygtrie==2.5.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyparted==3.11.2
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.7
python-benedict==0.25.2
python-dateutil==2.8.2
python-debian==0.1.36ubuntu1
python-fsutil==0.6.1
python-slugify==6.1.2
PyYAML==6.0
requests==2.22.0
requests-unixsocket==0.2.0
rich==12.5.1
rsa==4.9
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
scmrepo==0.0.25
SecretStorage==2.3.1
service-identity==18.1.0
setuptools==45.2.0
shortuuid==1.0.9
shtab==1.5.5
simplejson==3.16.0
six==1.14.0
smmap==5.0.0
sos==4.3
ssh-import-id==5.10
systemd-python==234
tabulate==0.8.10
text-unidecode==1.3
toml==0.10.2
tqdm==4.64.0
Twisted==18.9.0
typing-extensions==4.3.0
ubuntu-advantage-tools==27.9
ufw==0.36
uritemplate==4.1.1
urllib3==1.25.8
userpath==1.8.0
voluptuous==0.13.1
wadllib==1.3.3
WALinuxAgent==2.2.46
wcwidth==0.2.5
wheel==0.34.2
xmltodict==0.13.0
yarl==1.7.2
zc.lockfile==2.0
zipp==3.8.1
zope.interface==4.7.1
DVC version: 2.13.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.15.0-1014-azure-x86_64-with-glibc2.29
Supports:
	gdrive (pydrive2 = 1.13.0),
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/sda1
Repo: dvc, git
ERROR: unexpected error - Missing required setting client_config
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
Error: Process completed with exit code [255](https://github.com/sorenwacker/dvc-test-project-01/runs/7449367677?check_suite_focus=true#step:4:256).

@efiop
Copy link
Contributor

efiop commented Jul 22, 2022

@sorenwacker Were you able to resolve that new one? Looks like a simple misconfiguration issue.

Upgraded to pyinstaller 5.2 seems to solve this. Releasing 2.14.0 ASAP, please give it a try and let us know if that fixes it for you.

@efiop efiop closed this as completed Jul 22, 2022
Repository owner moved this from In Progress to Done in DVC Jul 22, 2022
@sorenwacker
Copy link

@efiop I am not sure where to look for a misconfiguration. This is my current workflow:

name: auto-testing
on: [push]
jobs:
  run:
    runs-on: [ubuntu-latest]
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-dvc@v1
      - name: Get data
        run: |
          echo '---'
          echo GDRIVE_CREDENTIALS_DATA: $GDRIVE_CREDENTIALS_DATA
          echo '---'
          python --version
          pip uninstall dvc
          pip install dvc[gdrive]
          pip install pydrive2
          pip list --format freeze
          dvc doctor
          dvc remote modify storage --local gdrive_use_service_account true
          dvc remote modify storage --local gdrive_service_account_json_file_path .dvc/gdrive-access.json
          dvc pull
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}   
      - name: Install requirements
        run: |
          pip install -r requirements.txt
      - name: Run tests
        run: python src/test.py

@sorenwacker
Copy link

It seems to be working now.

@sorenwacker
Copy link

Here is a minimal setup that works:

name: auto-testing
on: [push]
jobs:
  run:
    runs-on: [ubuntu-latest]
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-dvc@v1
      - name: Get data
        run: |
          pip install -U dvc[gdrive] pydrive2
          dvc pull
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}   
      - name: Install requirements
        run: |
          pip install -r requirements.txt
      - name: Run tests
        run: python src/test.py

@efiop
Copy link
Contributor

efiop commented Jul 22, 2022

@sorenwacker In your first workflow, I don't see .dvc/gdrive-access.json created anywhere, which I suppose is the issue.

Great to hear it works for you now 🙂 Thanks for the feedback!

@daavoo daavoo moved this from Todo to Backlog in DVC Aug 9, 2022
@efiop efiop moved this from Backlog to Todo in DVC Aug 9, 2022
@NevoMitt
Copy link

Hey
I'm also facing this issue at DVC version: 2.18.1 (deb).
Reached out for daavoo. waiting for a fix now.

daavoo added a commit that referenced this issue Aug 18, 2022
@daavoo
pyinstaller: Restrict to 5.0
eb004ba 
5.2 didn't fix #7949
@daavoo daavoo mentioned this issue Aug 18, 2022
Merged
Repository owner moved this from Todo to Done in DVC Aug 18, 2022
@efiop efiop reopened this Aug 18, 2022
Repository owner moved this from Done to Todo in DVC Aug 18, 2022
Repository owner moved this from Todo to Done in DVC Aug 24, 2022
@jonburdo jonburdo reopened this Aug 26, 2022
Repository owner moved this from Done to Todo in DVC Aug 26, 2022
@efiop
Copy link
Contributor

efiop commented Aug 26, 2022

We reverted to pyinstaller 5.0 (thanks @daavoo ) for now. No time to look into this deeper, so going to close it for now.

@efiop efiop closed this as completed Aug 26, 2022
Repository owner moved this from Todo to Done in DVC Aug 26, 2022
@efiop efiop removed their assignment Aug 26, 2022
@skshetry skshetry removed the release-blocker Blocks a release label Sep 1, 2022
@pmrowla
Copy link
Contributor

pmrowla commented Nov 14, 2022 •
edited
Loading

This appears to be broken again (w/pyinstaller 5.6)

from discord context: https://discord.com/channels/485586884165107732/485596304961962003/1041680983482695751

2022-11-14 11:45:50,298 ERROR: unexpected error - Failed to authenticate GDrive: name: drive  version: v2
------------------------------------------------------------
Traceback (most recent call last):
  File "pydrive2/fs/spec.py", line 70, in _wrap_errors
  File "pydrive2/fs/spec.py", line 150, in _service_auth
  File "pydrive2/auth.py", line 101, in _decorated
  File "pydrive2/auth.py", line 734, in Authorize
  File "googleapiclient/_helpers.py", line 130, in positional_wrapper
  File "googleapiclient/discovery.py", line 287, in build
  File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive  version: v2
...
pydrive2.fs.spec.GDriveAuthError: Failed to authenticate GDrive
------------------------------------------------------------
2022-11-14 11:45:51,251 DEBUG: Version info for developers:
DVC version: 2.34.0 (deb)
---------------------------------
Platform: Python 3.10.8 on Linux-5.15.0-1022-azure-x86_64-with-glibc2.31

We probably need explicit pyinstaller hooks for this library in either DVC or pydrive2 (or we need to contribute an actual fix upstream in pyinstaller's hook)

.deb build logs for 2.34.1 https://github.com/iterative/dvc-s3-repo/actions/runs/3449002686/jobs/5756539961

71580 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/root/.pyenv/versions/3.10.8/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
71635 WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.

@pmrowla pmrowla reopened this Nov 14, 2022
Repository owner moved this from Done to Todo in DVC Nov 14, 2022
@pmrowla pmrowla mentioned this issue Nov 15, 2022
Merged
@pmrowla
Copy link
Contributor

pmrowla commented Nov 15, 2022

this should be resolved by the linked pydrive patch once it is merged (and dvc-gdrive is updated)

@pmrowla pmrowla mentioned this issue Nov 15, 2022
Merged
2 tasks
@pmrowla
Copy link
Contributor

pmrowla commented Nov 15, 2022

Should note that we will probably continue to see the

WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.

warning in the future, but it should no longer matter for us. The warning is generated due to an outdated pyinstaller.contrib hook for googleapiclient.model.

Repository owner moved this from Todo to Done in DVC Nov 15, 2022
@kshitiz305
Copy link

Can I have the requirements.txt file that I can use on my python 3.9 so that I get started with solving an isssue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daavoo daavoo

Labels
build Issues/features related to building dvc install packages. fs: gdrive Related to the GDrive filesystem p1-important Important, aka current backlog of things to do research
Projects
No open projects
DVC
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pyinstaller: Restrict to 5.0 iterative/dvc
pyinstaller: use pydrive2 package hooks pmrowla/dvc