Update vmware_vm_info.py

[eddiehavila]

SUMMARY

unrolling the return statement for "def get_vm_attributes(self, vm)" results in a ~25 fold speed increase processing a 2,200 VMs inventory. From 2.7 seconds average per VM getting attributes for original code to 0.11 seconds for the new unrolled code

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

vmware_vm_info.py

ADDITIONAL INFORMATION

Comparison using original and new code

    def get_vm_attributes1(self, vm):
        return dict((x.name, v.value) for x in self.custom_field_mgr
                    for v in vm.customValue if x.key == v.key)

    def get_vm_attributes2(self, vm):
      custom_field_values = vm.customValue
      custom_field_mgr = self.custom_field_mgr
      vm_attributes = {}
      for custom_field in custom_field_mgr:
          for custom_value in custom_field_values:
              if custom_field.key == custom_value.key:
                  vm_attributes[custom_field.name] = custom_value.value
      return vm_attributes



    def get_virtual_machines(self):
[.............]
            vm_attributes = dict()
            if self.module.params.get('show_attribute'):
                attributes_start = time.time()
                vm_attributes = self.get_vm_attributes1(vm)
                attributes_end = time.time()
                logging.debug(vm_attributes)
                logging.debug("Time to get VM attributes1: %f seconds", attributes_end - attributes_start)
                attributes_start = time.time()
                vm_attributes = self.get_vm_attributes2(vm)
                attributes_end = time.time()
                logging.debug(vm_attributes)
                logging.debug("Time to get VM attributes2: %f seconds", attributes_end - attributes_start)


2024-10-02 16:41:31,188 - DEBUG - {'BusinessArea': 'b1', 'Project': '199400081', 'CriticalityLevel': '1', 'DataSensitivity': 'L1', 'LicensedProduct': 'N/A', 'OS': 'Windows2019', 'SystemEnvironment': 'prd'}
2024-10-02 16:41:31,189 - DEBUG - Time to get VM attributes1: 2.718272 seconds
2024-10-02 16:41:31,303 - DEBUG - {'BusinessArea': 'b1', 'Project': '199400081', 'CriticalityLevel': '1', 'DataSensitivity': 'L1', 'LicensedProduct': 'N/A', 'OS': 'Windows2019', 'SystemEnvironment': 'prd'}
2024-10-02 16:41:31,304 - DEBUG - Time to get VM attributes2: 0.114557 seconds

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/f581f79e8cea475f9d17c722cea793ed

✔️ ansible-tox-linters SUCCESS in 7m 54s
✔️ build-ansible-collection SUCCESS in 8m 01s
✔️ ansible-test-cloud-integration-vcenter7_only-stable217 SUCCESS in 17m 01s
✔️ ansible-test-cloud-integration-vcenter7_2esxi-stable217 SUCCESS in 15m 10s
✔️ ansible-test-cloud-integration-vcenter7_1esxi-stable217_1_of_2 SUCCESS in 46m 43s
✔️ ansible-test-cloud-integration-vcenter7_1esxi-stable217_2_of_2 SUCCESS in 59m 02s
✔️ ansible-galaxy-importer SUCCESS in 4m 28s

[mariolenz]

Thanks @eddiehavila! I wanted to have a closer look at your PR, but ran into some issues with my test environment that I have to fix first. I'll try to have a closer look at you PR next week!

BTW could you please add a changelog fragment? Maybe something like changelogs/fragments/2194-vmware_vm_info.yml:

minor_changes:
  - vmware_vm_info - Improve performance when gathering tag information
    (https://github.com/ansible-collections/community.vmware/pull/2194).

or something similar.

[eddiehavila]

added the changelog fragment. thanks @mariolenz for your guidance

[softwarefactory-project-zuul]

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/e3d18b0a65b646e1938a4dc7db45e6b1

✔️ ansible-tox-linters SUCCESS in 8m 03s
✔️ build-ansible-collection SUCCESS in 8m 18s
⚠️ ansible-test-cloud-integration-vcenter7_only-stable217 CANCELED
⚠️ ansible-test-cloud-integration-vcenter7_2esxi-stable217 CANCELED
❌ ansible-test-cloud-integration-vcenter7_1esxi-stable217_1_of_2 FAILURE in 9m 05s
⚠️ ansible-test-cloud-integration-vcenter7_1esxi-stable217_2_of_2 CANCELED
✔️ ansible-galaxy-importer SUCCESS in 5m 27s

[alinabuzachis]

recheck

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/8d324f4ae8114c1a9d223863339aa4f1

✔️ ansible-tox-linters SUCCESS in 7m 52s
✔️ build-ansible-collection SUCCESS in 8m 04s
✔️ ansible-test-cloud-integration-vcenter7_only-stable217 SUCCESS in 16m 52s
✔️ ansible-test-cloud-integration-vcenter7_2esxi-stable217 SUCCESS in 13m 53s
✔️ ansible-test-cloud-integration-vcenter7_1esxi-stable217_1_of_2 SUCCESS in 44m 57s
✔️ ansible-test-cloud-integration-vcenter7_1esxi-stable217_2_of_2 SUCCESS in 58m 29s
✔️ ansible-galaxy-importer SUCCESS in 4m 30s

[mariolenz]

I can't reproduce your ~25 fold speed increase, and I wonder why. I've tested with an environment that's not quite as large as yours (you're talking about 2,200 VMs) but coming close. Maybe you just use attributes more heavily than we do?

Still, I've seen an improvement for my very simple test case from a little bit more than 7 to somewhat less than 5 minutes.

Thanks @eddiehavila!

[softwarefactory-project-zuul]

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/a02bbd1c7d5b4002b3c984471e7a7bcb

✔️ ansible-tox-linters SUCCESS in 7m 45s
✔️ build-ansible-collection SUCCESS in 6m 20s
✔️ ansible-galaxy-importer SUCCESS in 4m 26s