Caching parameters doesn't seem to work

[FutureTecSystems]

SUMMARY

Caching parameters doesn't seem to work

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible 2.10.9
  config file = /projects/ansible/ansible.cfg
  configured module search path = ['/projects/ansible/library', '/home/automatix/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /projects/ansible_virtualenv/lib/python3.8/site-packages/ansible
  executable location = /projects/ansible_virtualenv/bin/ansible
  python version = 3.8.3 (default, Aug 18 2020, 08:56:04) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

COLLECTION VERSION

# /projects/ansible/collections/ansible_collections
Collection           Version
-------------------- -------
ansible.netcommon    1.3.0
ansible.posix        1.1.1
community.general    1.2.0
community.kubernetes 1.0.0
google.cloud         1.0.1
theforeman.foreman   2.1.0

KATELLO/FOREMAN VERSION

tfm-rubygem-katello-3.18.2.1-1.el7.noarch
foreman-2.3.5-1.el7.noarch

STEPS TO REPRODUCE

Running playbooks with foreman inventory und the following settings:

# rs.foreman.yml
plugin: theforeman.foreman.foreman
url: URL
user: USER
password: PASSWORD
validate_certs: false
want_params: true
want_facts: true
cache: true

EXPECTED RESULTS

Active caching for subsequent playbook runs

ACTUAL RESULTS

Playbook runs wait several seconds until facts are fetched from foreman

[evgeni]

cc @ares @ezr-ondrej can one of you guys please look into this?

[mdellweg]

Ping, @ares, @ezr-ondrej this has been open for some while now. Do we know anything about this. Can it be closed?

[mpl241]

Caching is a problem, at least with my combination of Ansible (2.9.27) and theforeman.foreman inventory plugin (3.2.0). The contents of the cache is the result of the status call to Foreman/Satellite, not the actual inventory (see below). A couple minutes of looking at plugins/inventory/foreman.py reveals my rudimentary understanding of python and my non-existent understanding of how Ansible inventory plugins work. I hope someone else with more knowledge of both can help with this bug.

Below are the calls made to Foreman/Satellite by theforeman.foreman inventory plugin. The cache contains the response from the first GET. I expected the cache to include the content returned from the last GET.

GET /api/v2/status?per_page=250&page=1 -> Completed 200 OK

POST /ansible/api/v2/ansible_inventories/schedule -> Completed 200 OK

GET /api/v2/report_templates/316-Ansible%20-%20Ansible%20Inventory/report_data/2de7a1b5-ebf1-4e8d-b1d6-95ba598d243d -> Completed 204 No Content

GET /api/v2/report_templates/316-Ansible%20-%20Ansible%20Inventory/report_data/2de7a1b5-ebf1-4e8d-b1d6-95ba598d243d -> Completed 200 OK

$ cat inventory/10-foreman.yml 
plugin: theforeman.foreman.foreman
url: https://satellite.example.com/
user: REDACTED
password: REDACTED
validate_certs: true
want_params: false
cache: true

$ cat ~/.ansible.cfg 
...
[inventory]
cache=True
cache_plugin=jsonfile
cache_connection=~/.ansible/inventory-cache
cache_timeout=3600
...

$ cat ~/.ansible/inventory-cache/ansible_inventory_theforeman.foreman.foreman_fb87es_69536 
{
    "https://satellite.example.com//api/v2/status": {
        "api_version": 2, 
        "result": "ok", 
        "satellite_version": "6.10.2", 
        "status": 200, 
        "version": "2.5.2.18"
    }
}

[jplindquist]

I just started using Foreman, and subsequently this plugin with Ansible, and just noticed the same issue where the cache settings don't work. The inventory cache file is basically just caching the API url and version/statuses as mentioned above. Any thoughts on getting this working? I'm not terribly well versed with the plugin architecture, but can try and take a crack at it if there are suggestions or guidelines somewhere.

[mpl241]

I happened to notice yesterday when using the community.vmware.vmware_host_inventory inventory plugin, the cache populated as expected using the same ansible.cfg settings from my previous post, so the problem appears to be isolated to the theforeman.foreman.foreman inventory plugin.

[evgeni]

Okay, so I investigated this a bit, and it's only not using the cache when you're using the Reports API (which is the default these days). Setting use_reports_api: false makes the cache being used for me, but then you don't benefit from the new stuff.

I'll see that the new API also gets caching implemented.

[evgeni]

I've posted a patch in #1382 and would appreciate if y'all could test it.

[mpl241]

@evgeni, I tested the new version of the plugin, and the plugin populates the cache correctly now. My Satellite system has 87 hosts for reference.

$ grep -i version /usr/share/ansible/collections/ansible_collections/theforeman/foreman/MANIFEST.json 
  "version": "3.4.0-dev",

$ time ansible-inventory -i inventory/10-foreman.yml --graph
...
real	0m12.603s
user	0m0.596s
sys	0m1.178s

$ time ansible-inventory -i inventory/10-foreman.yml --graph
...
real	0m1.830s
user	0m0.508s
sys	0m1.115s

Below is an example of a cached host entry:

        {
            "content_attributes": {
                "applicable_module_stream_count": 0, 
                "applicable_package_count": 0, 
                "content_source": "satellite.example.com", 
                "content_source_id": 1, 
                "content_source_name": "satellite.example.com", 
                "content_view": {
                    "id": 136, 
                    "name": "CentOS Linux 8-stream Standard"
                }, 
                "content_view_id": 136, 
                "content_view_name": "CentOS Linux 8-stream Standard", 
                "errata_counts": {
                    "bugfix": 0, 
                    "enhancement": 0, 
                    "security": 0, 
                    "total": 0
                }, 
                "id": 424, 
                "kickstart_repository": "CentOS Linux 8-stream Release x86_64", 
                "kickstart_repository_id": 63549, 
                "kickstart_repository_name": "63549", 
                "lifecycle_environment": {
                    "id": 25, 
                    "name": "Test"
                }, 
                "lifecycle_environment_id": 25, 
                "lifecycle_environment_name": "Test", 
                "upgradable_module_stream_count": 0, 
                "upgradable_package_count": 0, 
                "uuid": "xxx"
            }, 
            "host_group": "centos-test-base-8stream", 
            "id": 445, 
            "ipv4": "xxx", 
            "ipv6": null, 
            "location": "My Location", 
            "name": "centos8.example.com", 
            "organization": "My Organization", 
            "smart_proxies": [
                "satellite.example.com"
            ], 
            "subnet": "vlan-123", 
            "subnet6": null
        }, 

I still see the result from /api/v2/status in the cache. I'm not sure if this is expected.

    "https://satellite.example.com//api/v2/status": {
        "api_version": 2, 
        "result": "ok", 
        "satellite_version": "6.10.4", 
        "status": 200, 
        "version": "2.5.2.19"
    }

[evgeni]

It is expected, as we cache http requests and their replies.

[jplindquist]

I can confirm this does cache the inventory now as expected as well 🎉

[evgeni]

Awesome, thank y'all for reporting, debugging and testing!

[jplindquist]

@evgeni One more follow-up now that I'm trying to use this more in depth, but while I am getting inventory caching, it's missing some of the additional host attributes like foreman_facts when reading from the cache that were being used when the inventory was causing some composed keys and keyed_groups to fail to load

Sample cached host object

"samplehost": {
                "foreman_facts": {},
                "foreman_id": 10,
                "foreman_ipv4": "1.2.3.4",
                "foreman_subnet": null
            }

[evgeni]

Are you sure it's the cache? The code that is now in the "not in cache" branch shouldn't be doing anything about these.

[jplindquist]

I can try and debug more, but when I deleted the cache file and reran the inventory I do get the composed variables and keyed groups initially, but once the cache is generated they're gone on subsequent runs or inventory lists

[evgeni]

Interesting. I'll see that I reproduce that tomorrow. Thanks!

[evgeni]

Okay, reproduced with want_facts: true, let's see why

[evgeni]

@jplindquist please have a try: #1383

[jplindquist]

That looks like it worked! Thank you so much!