Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible gRPC issue with Arista SSoT #467

Open
qduk opened this issue Jun 10, 2024 · 7 comments
Open

Possible gRPC issue with Arista SSoT #467

qduk opened this issue Jun 10, 2024 · 7 comments
Assignees
Labels
integration: cloudvision Issues/PRs for the Arista CloudVision SSoT integration. type: bug Issues/PRs addressing a bug.

Comments

@qduk
Copy link
Contributor

qduk commented Jun 10, 2024

Environment

  • Python version: Unsure
  • Nautobot version: 2.1.9
  • nautobot-ssot version: 2.5.0

Expected Behavior

Running the Arista SSoT successfully populates data into Nautobot.

Observed Behavior

A gRPC error is raised during the sync.

Steps to Reproduce

  1. Install SSoT app with Arista Integration
  2. Run sync job
@qduk
Copy link
Contributor Author

qduk commented Jun 10, 2024

Traceback:

[2024-05-09 23:53:26,378: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,382: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,386: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,388: WARNING/ForkPoolWorker-30]   File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,394: WARNING/ForkPoolWorker-30] with self._state.condition:
[2[2024-05-09 23:53:26,445: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,447: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,450: WARNING/ForkPoolWorker-30] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,452: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,454: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,457: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,459: WARNING/ForkPoolWorker-30]   File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,463: WARNING/ForkPoolWorker-30]
[2024-05-09 23:53:26,465: WARNING/ForkPoolWorker-30] with self._state.condition:
[2024-05-09 23:53:26,516: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,518: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,521: WARNING/ForkPoolWorker-30] '_Rendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,544: ERROR/ForkPoolWorker-30] Task nautobot.extras.jobs.run_job[a2a6ea55-fc2f-440e-89c1-8c82761c563b] raised unexpected: RpcError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
    result = job(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
    return self.run(*args, **deserialized_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
    super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
  File "/source/nautobot_ssot/jobs/base.py", line 317, in run
    self.sync_data(memory_profiling)
  File "/source/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
    self.source_adapter.load()
  File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 267, in load
    self.load_devices()
  File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 72, in load_devices
    for index, dev in enumerate(cloudvision.get_devices(client=self.conn.comm_channel, import_active=True), start=1):
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/source/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 270, in get_devices
    for resp in responses:
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc.RpcError

This was an error brought to me by @chambersh1129 while at Cisco Live. I need dig into it more but I wanted to get the issue open. @chambersh1129 can you provide the python version you are using?

@chambersh1129
Copy link

Python Version: 3.11.9

PLUGINS_CONFIG Arista SSoT Snippet:

PLUGINS_CONFIG = {
    "nautobot_ssot": {
        "enable_aristacv": True,
        "aristacv_cvp_token": os.getenv("NAUTOBOT_ARISTACV_TOKEN", ""),
        "aristacv_cvaas_url": "www.arista.io:443",
        "aristacv_verify": is_truthy(os.getenv("NAUTOBOT_ARISTACV_VERIFY", True)),
        "aristacv_from_cloudvision_default_site": "cloudvision_imported",
        "aristacv_from_cloudvision_default_device_role": "network",
        "aristacv_from_cloudvision_default_device_role_color": "ff0000",
        "aristacv_delete_devices_on_sync": is_truthy(os.getenv("NAUTOBOT_ARISTACV_DELETE_ON_SYNC", False)),
        "aristacv_apply_import_tag": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_TAG", True)),
        "aristacv_import_active": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_ACTIVE", False)),
        "aristacv_create_controller": is_truthy(os.getenv("NAUTOBOT_ARISTACV_CREATE_CONTROLLER", False)),
        "aristacv_controller_site": os.getenv("NAUTOBOT_ARISTACV_CONTROLLER_SITE", ""),
        "aristacv_hostname_patterns": [],  # scrubbed
        "aristacv_site_mappings": {},  # scrubbed
        "aristacv_role_mappings": {},  # scrubbed
    },
}

@qduk qduk self-assigned this Jun 10, 2024
@jdrew82 jdrew82 added type: bug Issues/PRs addressing a bug. integration: cloudvision Issues/PRs for the Arista CloudVision SSoT integration. labels Jul 18, 2024
@chambersh1129
Copy link

chambersh1129 commented Aug 14, 2024

edit: was building from an older nautobot docker image (2.2.4). versions and logs updated to reflect latest 2.2.9 build.

Following up on this, today I upgraded Nautobot and SSOT plugin:

  • Nautobot: 2.2.9
  • SSOT: 2.7.0
  • Python: 3.11.9

I am getting the same exception. Traceback:

[2024-08-14 19:20:50,391: INFO/MainProcess] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] received
19:20:50.413 DEBUG   nautobot.extras.jobs jobs.py                                run_job() :
  Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,413: DEBUG/ForkPoolWorker-7] Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,417: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Running job
[2024-08-14 19:20:50,440: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading current data from source adapter...
[2024-08-14 19:20:50,472: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading data from CloudVision
[2024-08-14 19:21:02,195: WARNING/ForkPoolWorker-7] Exception ignored in: 
[2024-08-14 19:21:02,201: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,206: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,210: WARNING/ForkPoolWorker-7]   File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,213: WARNING/ForkPoolWorker-7]     
[2024-08-14 19:21:02,217: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,221: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,227: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,231: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,237: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,241: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,245: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,251: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,257: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,263: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,266: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,269: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,272: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,276: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,279: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,283: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,287: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,294: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,299: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,302: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,305: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,308: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,311: WARNING/ForkPoolWorker-7] : 
[2024-08-14 19:21:02,316: WARNING/ForkPoolWorker-7] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,319: WARNING/ForkPoolWorker-7] Exception ignored in: 
[2024-08-14 19:21:02,322: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,325: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,329: WARNING/ForkPoolWorker-7]   File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,332: WARNING/ForkPoolWorker-7]     
[2024-08-14 19:21:02,335: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,338: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,342: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,345: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,349: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,356: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,360: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,363: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,366: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,369: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,374: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,377: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,380: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,383: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,386: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,393: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,396: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,400: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,402: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,405: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,408: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,411: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,415: WARNING/ForkPoolWorker-7] : 
[2024-08-14 19:21:02,418: WARNING/ForkPoolWorker-7] '_Rendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,434: ERROR/ForkPoolWorker-7] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] raised unexpected: RpcError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
    result = job(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
    return self.run(*args, **deserialized_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
    super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 317, in run
    self.sync_data(memory_profiling)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
    self.source_adapter.load()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 269, in load
    self.load_devices()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 95, in load_devices
    self.load_device_tags(device=new_device)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 234, in load_device_tags
    system_tags = cloudvision.get_tags_by_type(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 287, in get_tags_by_type
    for resp in responses:
  File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc.RpcError

@jdrew82
Copy link
Contributor

jdrew82 commented Oct 22, 2024

@chambersh1129 Is this still an issue? Unfortunately, we don't have a CVP instance to test against at this time so we can't validate if this is still occurring or how to fix it.

@jdrew82
Copy link
Contributor

jdrew82 commented Oct 22, 2024

@chambersh1129 Can you see if the PR #580 solves the issue you're experiencing? It should at least allow the code to bypass the RpcError that's being thrown.

@chambersh1129
Copy link

Hi @jdrew82, we shifted course and started using the IPFabric integration which is working for us. I currently don't have the cycles nor a DEV environment to test the Arista CV integration and the IPFabric integration is vendor agnostic and all inclusive.

This can be kept open if you want to use it for tracking/reminder to test in the future. If I'm the only person who has brought up the issue then it can be closed/resolved.

@jdrew82
Copy link
Contributor

jdrew82 commented Oct 29, 2024

@chambersh1129 Appreciate the response. The Arista team has reached out to provide a test instance for us to use. I believe they stated they were able to reproduce the issue so we should hopefully have a fix available in the near future. I'll leave this open for the time being while we work to resolve the bug.

@jdrew82 jdrew82 mentioned this issue Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration: cloudvision Issues/PRs for the Arista CloudVision SSoT integration. type: bug Issues/PRs addressing a bug.
Projects
None yet
Development

No branches or pull requests

3 participants