ansible-navigator not able to list roles in a collection

[patrick-tang-redhat]

ISSUE TYPE

• Bug Report

SUMMARY

While we can list roles and check the documents for roles from automation hub, we cannot list roles with cli ansible-navigator ('ansible-navigator collections').

ANSIBLE-NAVIGATOR VERSION

ansible-navigator 1.1.0

CONFIGURATION

LOG FILE

STEPS TO REPRODUCE

1. ansible-navigator collections
2. select a collection with roles, for example 'redhat.insights'. type the number to enter content list of the collection.
3. check if all content of the selected collection shows roles.

EXPECTED RESULTS

expected: the roles in a collection should be shown.

ACTUAL RESULTS

actual: roles are missing from the content of the collection.

ADDITIONAL INFORMATION

As a comparison, please check following link for example collection to show roles in its content:
https://console.redhat.com/ansible/automation-hub/repo/published/redhat/insights/content?showing=all

[cidrblock]

@patrick-tang-redhat Thanks for this!

Role browsing support was added here: #1059 and will be in the 2.0.0 release of ansible-navigator. The upstream release candidate should be out today and the upstream final 2.0.0 next week.

When the RC is out, I'll post a link here so you can give it a run for us and let us know what you think

[patrick-tang-redhat]

Thanks @cidrblock
Good to know this is duplicated. Looking forward the release.
It is not very convenient that not able to get information of roles in cli.

[patrick-tang-redhat]

@cidrblock
I tried to run this pre-release version 'ansible-navigator-2.0.0b0' in rhel8, with python39, but got error (i installed with 'pip install --user'):
FileNotFoundError: [Errno 2] No such file or directory: '/root/.local/share/ansible_navigator/utils/key_value_store.py'

Do you know the fix?

[cidrblock]

@patrick-tang-redhat I see root there, can you try and install it with the --user flag? (if you didn't)

pip install ansible-navigator==2.0.0b0 --user

Still working to get the release candidate out today, keep me posted please

[cidrblock]

Sorry, I see you did, let me look

[cidrblock]

Can you run it with:

ansible-navigator --ll debug --la false and send the resulting log file over?

[patrick-tang-redhat]

@cidrblock
please find below the logs:
`
[root@rhel8 share]# ansible-navigator --ll debug --la false

Traceback (most recent call last):
File "/bin/ansible-navigator", line 11, in
load_entry_point('ansible-navigator==1.1.0', 'console_scripts', 'ansible-navigator')()
File "/usr/lib/python3.8/site-packages/ansible_navigator/cli.py", line 102, in main
new_messages, new_exit_messages = parse_and_update(sys.argv[1:], args=args, initial=True)
File "/usr/lib/python3.8/site-packages/ansible_navigator/initialization.py", line 194, in parse_and_update
new_messages, new_exit_messages, cache = get_and_check_collection_doc_cache(
File "/usr/lib/python3.8/site-packages/ansible_navigator/initialization.py", line 113, in get_and_check_collection_doc_cache
collection_cache = _get_kvs(share_directory, collection_doc_cache_path)
File "/usr/lib/python3.8/site-packages/ansible_navigator/initialization.py", line 139, in _get_kvs
spec.loader.exec_module(mod)
File "", line 779, in exec_module
File "", line 915, in get_code
File "", line 972, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/root/.local/share/ansible_navigator/utils/key_value_store.py'
`

[patrick-tang-redhat]

@cidrblock
seems the key_value_store.py is not in the place:
`
[root@rhel8 share]# ls -l
total 0
drwxr-xr-x. 5 root root 48 Apr 7 13:30 ansible_navigator
-rw-r--r--. 1 root root 0 Apr 7 13:31 ansible-navigator.log
drwxr-xr-x. 3 root root 19 Apr 7 11:35 ansible-runner
[root@rhel8 share]# ls ansible_navigator/
grammar themes utils
[root@rhel8 share]# ls ansible_navigator/utils
catalog_collections.py image_introspect.py init.py pycache

`

After I manually copied the file 'key_value_store.py' to ~/.local/share/ansible_navigator/utils folder. I'm able to launch ansible-navigator command.
However when I tried to check information of one collection, the application crashed with following error:
`
Traceback (most recent call last):
File "/bin/ansible-navigator", line 11, in
load_entry_point('ansible-navigator==1.1.0', 'console_scripts', 'ansible-navigator')()
File "/usr/lib/python3.8/site-packages/ansible_navigator/cli.py", line 141, in main
return_code = run(args)
File "/usr/lib/python3.8/site-packages/ansible_navigator/cli.py", line 82, in run
wrapper(ActionRunner(args=args).run)
File "/usr/lib64/python3.8/curses/init.py", line 105, in wrapper
return func(stdscr, *args, **kwds)
File "/usr/lib/python3.8/site-packages/ansible_navigator/action_runner.py", line 64, in run
self._run_app(interaction)
File "/usr/lib/python3.8/site-packages/ansible_navigator/action_runner.py", line 73, in _run_app
interaction = run_action(self.steps.current.name, self.app, self.steps.current)
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/_actions.py", line 97, in run_interactive
return action_cls(app.args).run(app=app, interaction=interaction)
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/collections.py", line 138, in run
self._take_step()
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/collections.py", line 153, in _take_step
result = run_action(self.steps.current.name, self.app, self.steps.current)
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/_actions.py", line 97, in run_interactive
return action_cls(app.args).run(app=app, interaction=interaction)
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/select.py", line 37, in run
app.steps.append(app.steps.current.select_func()) # add next
File "/usr/lib/python3.8/site-packages/ansible_navigator/actions/collections.py", line 199, in _build_plugin_menu
for plugin_chksum, details in selected_collection["plugin_chksums"].items():
KeyError: 'plugin_chksums'

`

[cidrblock]

Somehow I think your installation is odd, your first message has a mention of _get_kvs which is not in the current code base, and also mentions load_entry_point('ansible-navigator==1.1.0', 'console_scripts', 'ansible-navigator') which is the old version

[cidrblock]

Can you try with an install in a virtual environment? I can provide the steps, this might be related to an RPM install + a pip installation?

Or uninstall the pip version, uninstall the RPM and then install the pip version

[patrick-tang-redhat]

@cidrblock could you provide the steps for virtual environment? I'm not familiar with it.
In the meantime, let me clean up the environment to put back.

I installed python39 as module, so everything goes together

[cidrblock]

$ mkdir navigator-test
$ cd navigator-test 
$ python3.9 -m venv venv 
$ source venv/bin/activate
$ pip install ansible-navigator==2.0.0b0
$ ansible-navigator --version
ansible-navigator 2.0.0b0

This will build a virtual environment and install navigator in it, when you are done, you can delete the directory and nothing will be left over, clean up is easy. Good for testing

[patrick-tang-redhat]

@cidrblock
it might have something to do with the ee environment.
I have tried to use venv, and 2.0.0b0 worked for me.
However, I noticed there is a big difference:
with venv, it tried to pull down a new ee image: quay.io/ansible/creator-ee:v0.4.0.
While i was using non virtual env, it was using: registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest

The old error log:
`
[root@rhel8 tmp]# ansible-navigator collections

Execution environment image and pull policy overview

Execution environment image name: registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest
Execution environment image tag: latest
Execution environment pull policy: tag
Execution environment pull needed: True

Updating the execution environment

Trying to pull registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 510abfcdf6bc skipped: already exists
Copying blob 4aa6faa2acb0 skipped: already exists
Copying blob 916e0bda4eea skipped: already exists
Copying blob 4a3604715398 skipped: already exists
Copying blob 6ccfcef17d4b skipped: already exists
Copying config 43cfa4a7f9 done
Writing manifest to image destination
Storing signatures
43cfa4a7f9f7176c797dfc05a211cc754e0cb29e1ffd0d2a1ad954a8dd690b81
Traceback (most recent call last):
File "/bin/ansible-navigator", line 11, in
load_entry_point('ansible-navigator==1.1.0', 'console_scripts', 'ansible-navigator')()
File "/usr/lib/python3.8/site-packages/ansible_navigator/cli.py", line 141, in main
return_code = run(args)
File "/usr/lib/python3.8/site-packages/ansible_navigator/cli.py", line 82, in run
wrapper(ActionRunner(args=args).run)
File "/usr/lib64/python3.8/curses/init.py", line 105, in wrapper
return func(stdscr, *args, **kwds)
File "/usr/lib/python3.8/site-packages/ansible_navigator/action_runner.py", line 58, in run
self.initialize_ui(DEFAULT_REFRESH)
File "/usr/lib/python3.8/site-packages/ansible_navigator/action_runner.py", line 48, in initialize_ui
self._ui = UserInterface(
File "/usr/lib/python3.8/site-packages/ansible_navigator/ui_framework/ui.py", line 123, in init
super().init(ui_config=ui_config)
File "/usr/lib/python3.8/site-packages/ansible_navigator/ui_framework/curses_window.py", line 63, in init
self._set_colors()
File "/usr/lib/python3.8/site-packages/ansible_navigator/ui_framework/curses_window.py", line 189, in _set_colors
with open(self._ui_config.terminal_colors_path) as data_file:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.local/share/ansible_navigator/themes/terminal_colors.json'

`

[cidrblock]

You can use the supported ee by supplying it on the command line:

ansible-navigator --eei registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest

Or putting it in a ansible-navigator settings file.

The upstream releases will point to the creator-ee. I need to see what is being done during the downstream RPM build, they might be using a settings file or default being changed.

[cidrblock]

BTW, sorry you are having trouble here... we'll get this sorted out

[patrick-tang-redhat]

@cidrblock I just tried with --eei to specify ee-supported image, but it just ran fine in venv. So not sure why i got issue in non-venv .
like below:

(venv) [root@rhel8 navigator-test]# ansible-navigator collections --eei registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest

Execution environment image and pull policy overview

Execution environment image name: registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest
Execution environment image tag: latest
Execution environment pull arguments: None
Execution environment pull policy: tag
Execution environment pull needed: True

Updating the execution environment

Running the command: podman pull registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest
Trying to pull registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4aa6faa2acb0 skipped: already exists
Copying blob 4a3604715398 skipped: already exists
Copying blob 6ccfcef17d4b skipped: already exists
Copying blob 916e0bda4eea skipped: already exists
Copying blob 510abfcdf6bc skipped: already exists
Copying config 43cfa4a7f9 done
Writing manifest to image destination
Storing signatures
43cfa4a7f9f7176c797dfc05a211cc754e0cb29e1ffd0d2a1ad954a8dd690b81
(venv) [root@rhel8 navigator-test]# python3 --version
Python 3.9.6
(venv) [root@rhel8 navigator-test]# ansible-navigator --version
ansible-navigator 2.0.0b0

[cidrblock]

Something about the way it was installed outside the virtual environment...

If you want to use it without the virtual environment, you can

deactivate the virtual environment
$ deactivate

and then remove the RPM installed ansible-navigator
and remove the previously installed pip package
$ pip uninstall ansible-navigator

and try to install if fresh:

$ pip install ansible-navigator==2.0.0b0 --user

this might work, I haven't tried a pip install on top of an RPM...

[cidrblock]

@patrick-tang-redhat JSYK:

pip install ansible-navigator==2.0.0rc1

Just released.

Let me know if it's ok to close this issue now :)

[patrick-tang-redhat]

@cidrblock
sorry. was caught in customer work.
I uninstalled the rpm, and used pip to install 2.0.0rc1 ('pip3 install ansible-navigator==2.0.0rc1'),
but got error:

`
[root@rhel8 ~]# /usr/local/bin/ansible-navigator --version
ansible-navigator 2.0.0rc1

[root@rhel8 ~]# /usr/local/bin/ansible-navigator collections --eei ee-supported-rhel8

Execution environment image and pull policy overview

Execution environment image name: ee-supported-rhel8:latest
Execution environment image tag: latest
Execution environment pull arguments: None
Execution environment pull policy: tag
Execution environment pull needed: True

Updating the execution environment

Running the command: podman pull ee-supported-rhel8:latest
Trying to pull registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4a3604715398 skipped: already exists
Copying blob 4aa6faa2acb0 skipped: already exists
Copying blob 6ccfcef17d4b skipped: already exists
Copying blob 916e0bda4eea skipped: already exists
Copying blob 510abfcdf6bc skipped: already exists
Copying config 43cfa4a7f9 done
Writing manifest to image destination
Storing signatures
43cfa4a7f9f7176c797dfc05a211cc754e0cb29e1ffd0d2a1ad954a8dd690b81
Traceback (most recent call last):
File "/usr/local/bin/ansible-navigator", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/cli.py", line 159, in main
run_return = run(args)
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/cli.py", line 97, in run
wrapper(ActionRunner(args=args).run)
File "/usr/lib64/python3.9/curses/init.py", line 94, in wrapper
return func(stdscr, *args, **kwds)
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/action_runner.py", line 90, in run
self.initialize_ui(DEFAULT_REFRESH)
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/action_runner.py", line 73, in initialize_ui
self._ui = UserInterface(
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/ui_framework/ui.py", line 154, in init
super().init(ui_config=ui_config)
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/ui_framework/curses_window.py", line 64, in init
self._set_colors()
File "/usr/local/lib/python3.9/site-packages/ansible_navigator/ui_framework/curses_window.py", line 198, in _set_colors
with open(self._ui_config.terminal_colors_path, encoding="utf-8") as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/ansible_navigator/themes/terminal_colors.json'

`

I guess I just wait for the new release.

[cidrblock]

hmmm I'm wondering if the RPM uninstall didn't remove some of the directories

try deleting the /usr/share/ansible_navigator directory

[patrick-tang-redhat]

I uninstall rpm ansible-navigator then install 2.0.0rc1 with pip. So yum should not remove any rc1 directories.
When I was uninstalling, yum removed some python38 dependencies together:

`
Removing:
ansible-navigator noarch 1.1.0-1.el8ap @ansible-automation-platform 1.1 M
Removing unused dependencies:
python38-importlib-resources noarch 1.1.0-2.el8ap @ansible-automation-platform 109 k
python38-onigurumacffi x86_64 1.1.0-1.el8ap @ansible-automation-platform 45 k

Transaction Summary

`

[patrick-tang-redhat]

@cidrblock
After removing /usr/share/ansible_navigator directory, I'm able to run new rc1 successfully to see roles and its related docs.
What is explanation about the removed directory? and similar '~/.local/share/ansible_navigator' directory when using 'pip install --user'.

[cidrblock]

It seems the RPM uninstall doesn't remove the directory it created, so when the pip installed version runs it finds that directory, even though it is empty, and tries to load from it. We are trying to get rid of the share directory entirely, which is the real long term fix, given that I'm not sure there is a reason to do anything right now. I think it would be rare for someone to be switching from RPM to pip, unless they are testing or developing...

[patrick-tang-redhat]

It seems the RPM uninstall doesn't remove the directory it created, so when the pip installed version runs it finds that directory, even though it is empty, and tries to load from it. We are trying to get rid of the share directory entirely, which is the real long term fix, given that I'm not sure there is a reason to do anything right now. I think it would be rare for someone to be switching from RPM to pip, unless they are testing or developing...

@cidrblock
Thanks for the explanation.

[patrick-tang-redhat]

@cidrblock
Since the change is planned and on the way, I think this can be closed.

[shatakshiiii]

Resolved by #1059