[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

fnothaft · 2018-01-01T05:46:32Z

Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full, self-contained pip install-able Spark by finding the JARs and bin scripts and packaging them up as packages which are deployed to pip. We then needed to modify the bin scripts to find the pip installed JARs.

To test this out, from inside of a virtualenv, do:

pip install pyspark
pip install --index-url https://test.pypi.org/simple/ bdgenomics.adam==0.23.0rc18

You should be able to run adam-shell, adam-submit, etc. You can see the release docs here.

As you can tell, this took a bit of trial and error...

Resolves bigdatagenomics#1847. Cribs heavily from PySpark's script flow for supporting a full, self-contained pip install-able Spark by finding the JARs and bin scripts and packaging them up as packages which are deployed to pip. We then needed to modify the bin scripts to find the pip installed JARs.

AmplabJenkins · 2018-01-01T06:06:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2533/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 0c5c82b # timeout=10Checking out Revision 0c5c82b (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0c5c82bf0c49ab5bdb11c84e4ca569153278db14First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2018-01-01T06:12:40Z

adam-python/setup.py

+
+# Provide guidance about how to use setup.py
+incorrect_invocation_message = """
+If you are installing PyADAM from ADAM's source, you must first build ADAM and


We don't actually call this PyADAM anywhere, right? Rather bdgenomics.adam, or Python support for ADAM, or somesuch.

heuermh · 2018-01-01T06:13:22Z

adam-python/setup.py

+if len(JARS_PATH) == 1:
+    JARS_PATH = JARS_PATH[0]
+elif len(JARS_PATH) > 1:
+    print("Assembly jars exist for multiple scalas ({0}), please cleanup assembly/target".format(


scalas → Scala versions

heuermh · 2018-01-01T06:14:28Z

adam-python/setup.py

+ADAM_HOME = os.path.abspath("../")
+TEMP_PATH = "deps"
+
+JARS_PATH = glob.glob(os.path.join(ADAM_HOME, "adam-assembly/target/adam-assembly*.jar"))


Do we need the same regex mess to avoid -sources and -javadoc jars as in find-adam-assembly.sh here?

heuermh · 2018-01-01T06:15:38Z

adam-python/setup.py

+        import pypandoc
+        long_description = pypandoc.convert('README.md', 'rst')
+    except ImportError:
+        print("Could not import pypandoc - required to package PySpark", file=sys.stderr)


heuermh · 2018-01-01T06:17:20Z

adam-python/version.py

@@ -16,7 +16,7 @@
 # limitations under the License.
 #

-version = '0.23.0-SNAPSHOT'
+version = '0.23.0rc18'


Evidence of trial and error noted. ;) Was this meant to be committed?

heuermh · 2018-01-01T06:19:14Z

bin/find-adam-home

+   exit 0
+elif [ ! -f "$FIND_ADAM_HOME_PYTHON_SCRIPT" ]; then
+  # If we are not in the same directory as find_adam_home.py we are not pip installed so we don't
+  # need to search the different Python directories for a Adam installation.


for a Adam → for an ADAM

fnothaft · 2018-01-01T21:45:07Z

Thanks for the quick review @heuermh! I've addressed your comments, and the commits are OK to squash down. Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

heuermh · 2018-01-01T21:50:35Z

scripts/release/release.sh

+rm -rf release-venv
+virtualenv release-venv
+. release-venv/bin/activate
+pip install pyspark


I don't have pip installed by default, does that come in the virtualenv?

Correct; virtualenvs include their own pip.

heuermh · 2018-01-01T21:53:25Z

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server? How are those credentials handled by the release script?

fnothaft · 2018-01-01T21:56:38Z

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server?

Yup; you'll need to set up an account following these directions.

How are those credentials handled by the release script?

They can be stored in a plaintext file (urgh), but the twine command in the release script prompts for credentials before beginning the upload.

AmplabJenkins · 2018-01-01T22:07:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2534/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains bb54419 # timeout=10Checking out Revision bb54419 (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bb54419315a3505d657508a61346769225bbddc4First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2018-01-02T22:39:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2536/
Test PASSed.

fnothaft · 2018-01-03T06:00:34Z

Closing in favor of this merging in as a part of #1849.

fnothaft added this to the 0.23.0 milestone Jan 1, 2018

fnothaft requested review from heuermh and akmorrow13 January 1, 2018 05:46

heuermh requested changes Jan 1, 2018

View reviewed changes

fnothaft added 2 commits January 1, 2018 13:43

Addressing reviewer feedback.

fe48a9e

Integrating PyPI release flow into our release scripts.

e42aee3

heuermh reviewed Jan 1, 2018

View reviewed changes

fnothaft mentioned this pull request Jan 2, 2018

[ADAM-1783] Resolve check issues that block pushing to CRAN. #1849

Merged

1 task

Exclude source/javadoc jars.

0735ecd

fnothaft removed the request for review from akmorrow13 January 3, 2018 05:43

fnothaft closed this Jan 3, 2018

fnothaft deleted the issues/1847-pip-release branch January 3, 2018 06:00

heuermh mentioned this pull request Feb 26, 2019

TypeError: 'JavaPackage' object is not callable #2122

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

fnothaft commented Jan 1, 2018

AmplabJenkins commented Jan 1, 2018

heuermh Jan 1, 2018

heuermh Jan 1, 2018

heuermh Jan 1, 2018

heuermh Jan 1, 2018

heuermh Jan 1, 2018

heuermh Jan 1, 2018

fnothaft commented Jan 1, 2018

heuermh Jan 1, 2018

fnothaft Jan 1, 2018

heuermh commented Jan 1, 2018

fnothaft commented Jan 1, 2018

AmplabJenkins commented Jan 1, 2018

AmplabJenkins commented Jan 2, 2018

fnothaft commented Jan 3, 2018

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

Conversation

fnothaft commented Jan 1, 2018

AmplabJenkins commented Jan 1, 2018

Build result: FAILURE

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnothaft commented Jan 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heuermh commented Jan 1, 2018

fnothaft commented Jan 1, 2018

AmplabJenkins commented Jan 1, 2018

Build result: FAILURE

AmplabJenkins commented Jan 2, 2018

fnothaft commented Jan 3, 2018