offline installation fails on linux

[immon]

Rally version (get with esrally --version): 2.0.0

Invoked command: ./install.sh

JVM version: 1.8.0_171

OS version: Linux Ubuntu 18.04

Description of the problem including expected versus actual behavior:

Offline installation does not work on linux. Following steps from the docs gives the following error.

(venv) imo@lake:~/tmp/rally/esrally-dist-2.0.0$ ./install.sh 
Installing Rally 2.0.0...
Looking in links: file:///home/imo/tmp/rally/esrally-dist-2.0.0/bin
Collecting esrally==2.0.0
Collecting certifi (from esrally==2.0.0)
Collecting thespian==3.10.0 (from esrally==2.0.0)
Collecting elasticsearch==7.6.0 (from esrally==2.0.0)
Collecting elasticsearch-async==6.2.0 (from esrally==2.0.0)
Collecting Jinja2==2.10.3 (from esrally==2.0.0)
Collecting py-cpuinfo==3.2.0 (from esrally==2.0.0)
Collecting boto3==1.10.32 (from esrally==2.0.0)
Collecting psutil==5.7.0 (from esrally==2.0.0)
Collecting ijson==2.6.1 (from esrally==2.0.0)
Collecting jsonschema==3.1.1 (from esrally==2.0.0)
Collecting tabulate==0.8.5 (from esrally==2.0.0)
Collecting yappi==1.2.3 (from esrally==2.0.0)
Collecting urllib3>=1.21.1 (from elasticsearch==7.6.0->esrally==2.0.0)
Collecting aiohttp (from elasticsearch-async==6.2.0->esrally==2.0.0)
Collecting async_timeout (from elasticsearch-async==6.2.0->esrally==2.0.0)
Collecting MarkupSafe>=0.23 (from Jinja2==2.10.3->esrally==2.0.0)
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3==1.10.32->esrally==2.0.0)
Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.10.32->esrally==2.0.0)
Collecting botocore<1.14.0,>=1.13.32 (from boto3==1.10.32->esrally==2.0.0)
Requirement already satisfied: six>=1.11.0 in /home/imo/.pyenv/versions/3.8.0/envs/venv/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally==2.0.0) (1.15.0)
Collecting importlib-metadata (from jsonschema==3.1.1->esrally==2.0.0)
Collecting pyrsistent>=0.14.0 (from jsonschema==3.1.1->esrally==2.0.0)
Collecting attrs>=17.4.0 (from jsonschema==3.1.1->esrally==2.0.0)
Requirement already satisfied: setuptools in /home/imo/.pyenv/versions/3.8.0/envs/venv/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally==2.0.0) (41.2.0)
Collecting multidict<5.0,>=4.5 (from aiohttp->elasticsearch-async==6.2.0->esrally==2.0.0)
  ERROR: Could not find a version that satisfies the requirement multidict<5.0,>=4.5 (from aiohttp->elasticsearch-async==6.2.0->esrally==2.0.0) (from versions: none)
ERROR: No matching distribution found for multidict<5.0,>=4.5 (from aiohttp->elasticsearch-async==6.2.0->esrally==2.0.0)

This happens because macos specific packages are included in the bundle:

imo@lake:~/Downloads/esrally-dist-2.0.0$ ls bin/*macos*
bin/multidict-4.7.5-cp38-cp38-macosx_10_13_x86_64.whl
bin/yarl-1.4.2-cp38-cp38-macosx_10_13_x86_64.whl

To workaround I downloaded multidict-4.7.5-cp38-cp38-manylinux1_x86_64.whl and yarl-1.4.2-cp38-cp38-manylinux1_x86_64.whl from https://pypi.org/project/multidict/4.7.5/#files and https://pypi.org/project/yarl/#files and moved them to esrally-dist-2.0.0/bin which solved the issue.

I tried to update offline-install.sh script and change here to use --no-binary=MarkupSafe,multidict,yarl but multidict packet installation failed with:

Collecting multidict<5.0,>=4.5 (from aiohttp->elasticsearch-async==6.2.0->esrally==2.0.0)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/imo/.pyenv/versions/3.8.0/envs/env2/bin/python /home/imo/.pyenv/versions/3.8.0/envs/env2/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7ze9g3ug/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///home/imo/tmp/rally/esrally-dist-2.0.0/bin -- 'pip>=18' 'setuptools>=40' wheel
       cwd: None
  Complete output (4 lines):
  Looking in links: file:///home/imo/tmp/rally/esrally-dist-2.0.0/bin
  Collecting pip>=18
    ERROR: Could not find a version that satisfies the requirement pip>=18 (from versions: none)
  ERROR: No matching distribution found for pip>=18
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/imo/.pyenv/versions/3.8.0/envs/env2/bin/python /home/imo/.pyenv/versions/3.8.0/envs/env2/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7ze9g3ug/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///home/imo/tmp/rally/esrally-dist-2.0.0/bin -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.

Steps to reproduce:

1. follow offline installation steps on linux
2. it fails to complete due to the lack of linux specific python packages

[dliappis]

Thanks for reporting this issue. This started happening after aiohttp became a transitive dependency due to the async Python client.

After an offline discussion, we decided to have platform specific (for now, Linux only) offline installation packages which will be done via #1043.

[dliappis]

I'll close this once we release Rally 2.0.1 which will have a platform specific offline artifact.

[dliappis]

With Rally 2.0.1 released, there is now a Linux specific offline installation package.