Error when system pip packages don't match Ansible-DK pip packages

[rpufky]

Ran into this error when running some ansible code that uses the ansible elasticache module (http://docs.ansible.com/ansible/elasticache_module.html)
fatal: [elc-m]: FAILED! => {"changed": false, "failed": true, "msg": "boto required for this module"}

Ansible-DK comes with boto 2.36.0 pre-installed at /opt/ansible-dk/embedded/lib/python2.7/site-packages/boto/, and is seen via Ansible-DK's pip:

(ADK) smee:applications rpufky$ pip list
ansible (2.0.0.2)
ansible-lint (2.3.3)
awscli (1.10.1)
boto (2.36.0)
botocore (1.3.23)
click (5.1)
colorama (0.3.3)
docopt (0.6.2)
docutils (0.12)
ecdsa (0.13)
Jinja2 (2.8)
jmespath (0.9.0)
MarkupSafe (0.23)
molecule (1.3.0)
paramiko (1.16.0)
pbr (1.8.1)
pexpect (4.0.1)
pip (8.0.3)
prettytable (0.7.2)
ptyprocess (0.5.1)
py (1.4.31)
pyasn1 (0.1.9)
pycrypto (2.6.1)
pytest (2.9.0)
python-dateutil (2.5.1)
python-vagrant (0.5.10)
PyYAML (3.11)
rsa (3.3)
setuptools (18.5)
sh (1.11)
six (1.10.0)
testinfra (1.0.2)
You are using pip version 8.0.3, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I have not installed pip modules to system python, trying to leave that as clean as possible, system python does not have boto installed:

smee:~ rpufky$ pip list
altgraph (0.10.2)
awscli (1.10.8)
bdist-mpkg (0.5.0)
bonjour-py (0.3)
botocore (1.3.30)
colorama (0.3.3)
docutils (0.12)
futures (3.0.5)
jmespath (0.9.0)
macholib (1.5.1)
matplotlib (1.3.1)
modulegraph (0.10.4)
numpy (1.8.0rc1)
pip (8.0.3)
py2app (0.7.3)
pyasn1 (0.1.9)
pyobjc-core (2.5.1)
pyobjc-framework-Accounts (2.5.1)
pyobjc-framework-AddressBook (2.5.1)
pyobjc-framework-AppleScriptKit (2.5.1)
pyobjc-framework-AppleScriptObjC (2.5.1)
pyobjc-framework-Automator (2.5.1)
pyobjc-framework-CFNetwork (2.5.1)
pyobjc-framework-Cocoa (2.5.1)
pyobjc-framework-Collaboration (2.5.1)
pyobjc-framework-CoreData (2.5.1)
pyobjc-framework-CoreLocation (2.5.1)
pyobjc-framework-CoreText (2.5.1)
pyobjc-framework-DictionaryServices (2.5.1)
pyobjc-framework-EventKit (2.5.1)
pyobjc-framework-ExceptionHandling (2.5.1)
pyobjc-framework-FSEvents (2.5.1)
pyobjc-framework-InputMethodKit (2.5.1)
pyobjc-framework-InstallerPlugins (2.5.1)
pyobjc-framework-InstantMessage (2.5.1)
pyobjc-framework-LatentSemanticMapping (2.5.1)
pyobjc-framework-LaunchServices (2.5.1)
pyobjc-framework-Message (2.5.1)
pyobjc-framework-OpenDirectory (2.5.1)
pyobjc-framework-PreferencePanes (2.5.1)
pyobjc-framework-PubSub (2.5.1)
pyobjc-framework-QTKit (2.5.1)
pyobjc-framework-Quartz (2.5.1)
pyobjc-framework-ScreenSaver (2.5.1)
pyobjc-framework-ScriptingBridge (2.5.1)
pyobjc-framework-SearchKit (2.5.1)
pyobjc-framework-ServiceManagement (2.5.1)
pyobjc-framework-Social (2.5.1)
pyobjc-framework-SyncServices (2.5.1)
pyobjc-framework-SystemConfiguration (2.5.1)
pyobjc-framework-WebKit (2.5.1)
pyOpenSSL (0.13.1)
pyparsing (2.0.1)
python-dateutil (1.5)
pytz (2013.7)
rsa (3.3)
s3transfer (0.0.1)
scipy (0.13.0b1)
setuptools (1.1.6)
six (1.4.1)
vboxapi (1.0)
xattr (0.6.4)
zope.interface (4.1.1)
You are using pip version 8.0.3, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

To work around this, I added an environment variable to the set returned by eval "$(ansible-dk shell-init bash)":

export PYTHONPATH="/opt/ansible-dk/embedded/lib/python2.7/site-packages"

This let my Ansible-DK installation find the boto installation included with the embedded python.

[jblancett]

This is strange. Without that env set from python shell if you run sys.path, '/opt/ansible-dk/embedded/lib/python2.7/site-package' is in there so this shouldn't be necessary. Yet, I can reproduce the issue.

[jblancett]

adding it to the path with ENV does move it further up the path list, so maybe that makes the difference?

[rpufky]

Probably. What does the rest of your sys.path look like? Without using Ansible-DK, I get

['', '/Library/Python/2.7/site-packages/pip-8.0.3-py2.7.egg', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages']

Using Ansible-DK, I get:

['', '/opt/ansible-dk/embedded/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg', '/opt/ansible-dk/embedded/lib/python2.7/site-packages/pip-8.0.3-py2.7.egg', '/opt/ansible-dk/embedded/lib/python2.7/site-packages', '/opt/ansible-dk/embedded/lib/python27.zip', '/opt/ansible-dk/embedded/lib/python2.7', '/opt/ansible-dk/embedded/lib/python2.7/plat-darwin', '/opt/ansible-dk/embedded/lib/python2.7/plat-mac', '/opt/ansible-dk/embedded/lib/python2.7/plat-mac/lib-scriptpackages', '/opt/ansible-dk/embedded/lib/python2.7/lib-tk', '/opt/ansible-dk/embedded/lib/python2.7/lib-old', '/opt/ansible-dk/embedded/lib/python2.7/lib-dynload']

I would think that the location of the site-packages in the second one is high enough in the order -- there are no system locations with higher priority.

This has to be some system interplay with ansible -- possibly ansible is invoking some shell command somewhere specifically using /usr/bin/python instead of using whichever is resolved by path first.

[jblancett]

I made a quick little ansible module to check the pythonpath in the context of ansible. Here's my output without PYTHONPATH set:

$ ansible localhost -m pythonpath
localhost | SUCCESS => {
"changed": false,
"pythonpath": [
"/home/jblancett/.ansible/tmp/ansible-tmp-1460647933.49-271788190270997",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages/pip-8.0.3-py2.7.egg",
"/opt/ansible-dk/embedded/lib/python27.zip",
"/opt/ansible-dk/embedded/lib/python2.7",
"/opt/ansible-dk/embedded/lib/python2.7/plat-linux2",
"/opt/ansible-dk/embedded/lib/python2.7/lib-tk",
"/opt/ansible-dk/embedded/lib/python2.7/lib-old",
"/opt/ansible-dk/embedded/lib/python2.7/lib-dynload",
"/home/jblancett/.ansible-dk/python/lib/python2.7/site-packages",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages"
]
}

And here's with PYTHONPATH set to /home/jblancett/.ansible-dk/python/lib/python2.7/site-packages:/opt/ansible-dk/embedded/lib/python2.7/site-packages:

$ ansible localhost -m pythonpath
localhost | SUCCESS => {
"changed": false,
"pythonpath": [
"/home/jblancett/.ansible/tmp/ansible-tmp-1460647905.82-64899776042685",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages/pip-8.0.3-py2.7.egg",
"/home/jblancett/.ansible-dk/python/lib/python2.7/site-packages",
"/opt/ansible-dk/embedded/lib/python2.7/site-packages",
"/opt/ansible-dk/embedded/lib/python27.zip",
"/opt/ansible-dk/embedded/lib/python2.7",
"/opt/ansible-dk/embedded/lib/python2.7/plat-linux2",
"/opt/ansible-dk/embedded/lib/python2.7/lib-tk",
"/opt/ansible-dk/embedded/lib/python2.7/lib-old",
"/opt/ansible-dk/embedded/lib/python2.7/lib-dynload"
]
}

Same paths but the two I specify get moved higher up the list

[jblancett]

bingo. https://github.com/ansible/ansible-modules-core/blob/devel/cloud/amazon/elasticache.py#L1
I used #!/usr/bin/env python in my module, but the core modules do not.

[jblancett]

when I set my module to use the system python I still got the same output above. I'm thinking go ahead and set the pythonpath as it doesn't seem to negatively impact anything else and does seem to solve this issue