Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relocate python sources into py/ #2249

Merged
merged 10 commits into from
Apr 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ public class GeneratePyV2FigureAPI {
private static final String INDENT = " ";
private static final String JCLASS = "io.deephaven.plot.Figure";
private static final String PYPREAMBLE = "/Generators/src/main/java/io/deephaven/pythonPreambles/plotV2.py";
private static final String PYMODUlE = "/pyintegration/deephaven/plot/figure.py";
private static final String PYMODUlE = "/py/server/deephaven/plot/figure.py";
private static String figureWrapperPreamble;
private static String devroot;
private static boolean assertNoChange;
Expand Down
2 changes: 1 addition & 1 deletion Integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@
or manually,

```shell
docker build --build-arg DEEPHAVEN_VERSION=x.y.z --tag deephaven/deephaven-wheel --target build Integrations/python
docker build --build-arg DEEPHAVEN_VERSION=x.y.z --tag deephaven/deephaven-legacy-wheel --target build Integrations/python
```
4 changes: 2 additions & 2 deletions Integrations/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,8 @@ Closure composeConfig = { task ->

def pyTestLegacy = runInDocker("test-py-37-legacy", 'python', ['python3', '-m', 'xmlrunner', 'discover', '-v', '-o', '/out/report'], composeConfig)
def pyFunctionalTestLegacy = runInDocker('py-functional-test-legacy', 'python', ['/bin/bash', 'run-functional-tests.sh'])
def pyTest = runInDocker('test-py-deephaven', '../pyintegration', ['python3', '-m', 'xmlrunner', 'discover', '-s', 'tests', '-t', '.', '-v', '-o', '/out/report'], composeConfig)
def pyTestIntegration = runInDocker('test-py-deephaven-integration', '../pyintegration', ['/bin/bash', '-c', 'PYTHONPATH=/python integration-tests/run-tests.sh'], composeConfig)
def pyTest = runInDocker('test-py-deephaven', '../py/server', ['python3', '-m', 'xmlrunner', 'discover', '-s', 'tests', '-t', '.', '-v', '-o', '/out/report'], composeConfig)
def pyTestIntegration = runInDocker('test-py-deephaven-integration', '../py/server', ['/bin/bash', '-c', 'PYTHONPATH=/python integration-tests/run-tests.sh'], composeConfig)
pyTestLegacy.configure({
onlyIf { TestTools.shouldRunTests(project) }
})
Expand Down
1 change: 1 addition & 0 deletions buildSrc/src/main/groovy/Docker.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,7 @@ class Docker {
return registerDockerTask(project, taskName) { DockerTaskConfig config ->
config.copyIn { Sync sync ->
sync.from(sourcePath) { CopySpec copySpec ->
copySpec.exclude 'build', 'dist'
copySpec.into 'src'
}
}
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/src/main/groovy/io.deephaven.project.root.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ def verifyAllProjectsRegistered = project.tasks.register('verifyAllProjectsRegis
task.doLast {
project.allprojects { Project p ->
if (!ProjectType.isRegistered(p)) {
throw new IllegalStateException("Project '${project.name}' has not registered. Please apply the plugin 'io.deephaven.project.register'.")
throw new IllegalStateException("Project '${p.name}' has not registered. Please apply the plugin 'io.deephaven.project.register'.")
}
}
}
Expand Down
15 changes: 15 additions & 0 deletions buildSrc/src/main/groovy/io.deephaven.python-wheel.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
plugins {
id 'com.bmuschko.docker-remote-api'
id 'io.deephaven.project.register'
}

configurations {
pythonWheel
getByName(Dependency.DEFAULT_CONFIGURATION).extendsFrom(pythonWheel)
}

def buildWheel = Docker.buildPyWheel(project, 'buildWheel', "deephaven/${project.name}", project.projectDir.toString())

artifacts {
pythonWheel buildWheel
}
33 changes: 13 additions & 20 deletions docker/runtime-base/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -4,30 +4,23 @@ plugins {
}

evaluationDependsOn Docker.registryProject('server-base')
evaluationDependsOn ':deephaven-jpy'
evaluationDependsOn ':deephaven-wheel'
evaluationDependsOn ':deephaven2-wheel'

def dockerContext = project.layout.buildDirectory.dir('context')
configurations {
pythonWheel
}

def prepareDocker = project.tasks.register('prepareDocker', Sync) {
// deephaven-jpy.whl
def deephavenJpyWheel = project(':deephaven-jpy').tasks.getByName('buildWheel')
dependencies {
pythonWheel project(path: ':deephaven-jpy', targetConfiguration: 'pythonWheel')
pythonWheel project(':deephaven-legacy-wheel')
pythonWheel project(':py-server')
}

// deephaven.whl
def deephavenWheel = project(':deephaven-wheel').tasks.getByName('buildWheel')
def dockerContext = project.layout.buildDirectory.dir('context')

// deephaven2.whl
def deephaven2Wheel = project(':deephaven2-wheel').tasks.getByName('buildWheel')
def prepareDocker = project.tasks.register('prepareDocker', Sync) {

from (deephavenJpyWheel.outputs.files) {
into 'deephaven-jpy-wheel'
}
from (deephavenWheel.outputs.files) {
into 'deephaven-wheel'
}
from (deephaven2Wheel.outputs.files) {
into 'deephaven2-wheel'
from (configurations.pythonWheel) {
into 'wheels/'
}

from 'src/main/docker'
Expand All @@ -44,4 +37,4 @@ def buildDocker = Docker.registerDockerImage(project, 'buildDocker') {
images.add('deephaven/runtime-base:local-build')
}

assemble.dependsOn buildDocker
assemble.dependsOn buildDocker
16 changes: 3 additions & 13 deletions docker/runtime-base/src/main/docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,7 @@ FROM deephaven/server-base:local-build
# already satisfied via the base image. If that is not the case, we want the install to error out, and we'll need to
# update the base image with the extra requirements before proceeding here.

COPY deephaven-jpy-wheel/ /deephaven-jpy-wheel
COPY wheels/ /wheels
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven-jpy-wheel/*.whl; \
rm -r /deephaven-jpy-wheel

COPY deephaven-wheel/ /deephaven-wheel
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven-wheel/*.whl; \
rm -r /deephaven-wheel

COPY deephaven2-wheel/ /deephaven2-wheel
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven2-wheel/*.whl; \
rm -r /deephaven2-wheel
python -m pip install -q --no-index --no-cache-dir /wheels/*.whl; \
rm -r /wheels
28 changes: 8 additions & 20 deletions docker/server/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -26,31 +26,23 @@ baseMap.keySet().each {base ->
evaluationDependsOn Docker.registryProject(base)
}

evaluationDependsOn ':deephaven-jpy'
evaluationDependsOn ':deephaven-wheel'
evaluationDependsOn ':deephaven2-wheel'

configurations {
serverApplicationDist
pythonWheel
}

dependencies {
serverApplicationDist project(path: ':server-netty-app', configuration: 'applicationDist')

pythonWheel project(':deephaven-legacy-wheel')
pythonWheel project(':py-server')
}

def dockerLicenses = License.createFrom(project).syncDockerLicense().get()

def context= project.layout.buildDirectory.dir('context')

def prepareDocker = project.tasks.register('prepareDocker', Sync) {
// deephaven-jpy.whl
def deephavenJpyWheel = project(':deephaven-jpy').tasks.getByName('buildWheel')

// deephaven.whl
def deephavenWheel = project(':deephaven-wheel').tasks.getByName('buildWheel')

// deephaven2.whl
def deephaven2Wheel = project(':deephaven2-wheel').tasks.getByName('buildWheel')

from 'src/main/docker'
baseMap.values().each { server ->
Expand All @@ -61,15 +53,11 @@ def prepareDocker = project.tasks.register('prepareDocker', Sync) {
from ('src/main/configure') {
into 'configure'
}
from (deephavenJpyWheel.outputs.files) {
into 'deephaven-jpy-wheel'
}
from (deephavenWheel.outputs.files) {
into 'deephaven-wheel'
}
from (deephaven2Wheel.outputs.files) {
into 'deephaven2-wheel'

from(configurations.pythonWheel) {
into 'wheels'
}

from(configurations.serverApplicationDist) {
into 'serverApplicationDist'
}
Expand Down
16 changes: 3 additions & 13 deletions docker/server/src/main/docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,10 @@ RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir -r requirements.txt; \
rm requirements.txt

COPY deephaven-jpy-wheel/ /deephaven-jpy-wheel
COPY wheels/ /wheels
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven-jpy-wheel/*.whl; \
rm -r /deephaven-jpy-wheel

COPY deephaven-wheel/ /deephaven-wheel
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven-wheel/*.whl; \
rm -r /deephaven-wheel

COPY deephaven2-wheel/ /deephaven2-wheel
RUN set -eux; \
python -m pip install -q --no-index --no-cache-dir /deephaven2-wheel/*.whl; \
rm -r /deephaven2-wheel
python -m pip install -q --no-index --no-cache-dir /wheels/*.whl; \
rm -r /wheels
Comment on lines +16 to +19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are "losing" a bit of docker cachability here, but I doubt it's of much consequence.


ARG DEEPHAVEN_VERSION
ADD serverApplicationDist/server-${DEEPHAVEN_VERSION}.tar /opt/deephaven
Expand Down
File renamed without changes.
33 changes: 1 addition & 32 deletions pyclient/build.gradle → py/client/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,40 +7,9 @@ import io.deephaven.tools.docker.WaitForHealthyContainer

plugins {
id 'com.bmuschko.docker-remote-api'
id 'io.deephaven.project.register'
id 'io.deephaven.python-wheel'
}

evaluationDependsOn Docker.registryProject('python')

// Build the pydeephaven wheel from sources in a vanilla docker container
tasks.getByName('build').dependsOn(Docker.registerDockerTask(project, 'buildPyClient') {
copyIn {
from('requirements.txt') {
into 'project'
}
from('setup.py') {
into 'project'
}
from('README.md') {
into 'project'
}
from('pydeephaven') {
into 'project/pydeephaven'
}
}
dockerfile {
from('deephaven/python:local-build')
copyFile('project', '/project')
workingDir('/project')
runCommand('python3 setup.py bdist_wheel')
}
parentContainers = [ Docker.registryTask(project, 'python') ]
containerOutPath = '/project/dist'
copyOut {
into 'build/wheel'
}
})

// Start up a docker container for the grpc server, then run pydeephaven test
evaluationDependsOn(':docker-server')
String randomSuffix = UUID.randomUUID().toString();
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
17 changes: 17 additions & 0 deletions py/deephaven-legacy-wheel/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
plugins {
id 'com.bmuschko.docker-remote-api'
id 'io.deephaven.project.register'
}

def buildWheel = Docker.buildPyWheel(project, 'buildWheel', 'deephaven/deephaven-legacy-wheel', "${project(':Integrations').projectDir}/python")
configurations {
pythonWheel
getByName(Dependency.DEFAULT_CONFIGURATION).extendsFrom(pythonWheel)
}
dependencies {
pythonWheel project(path: ':deephaven-jpy', targetConfiguration: 'pythonWheel')
}

artifacts {
pythonWheel buildWheel
}
8 changes: 0 additions & 8 deletions py/deephaven-wheel/build.gradle

This file was deleted.

8 changes: 0 additions & 8 deletions py/deephaven2-wheel/build.gradle

This file was deleted.

12 changes: 10 additions & 2 deletions py/jpy/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ plugins {

evaluationDependsOn Docker.registryProject('python-dev-base')

configurations {
pythonWheel
}

test {
onlyIf { false } // don't run tests, it depends on python assets, which we aren't building ATM
exclude "**/*" // make sure intellij doesn't try to use this disabled test class
Expand All @@ -17,7 +21,7 @@ dependencies {
// For DH Community, we need to currently move fast, and will prefer to make edits directly in py/jpy.
// These can be backported to jpy proper in the future if necessary.

Docker.registerDockerTask(project, 'buildWheel') {
def buildWheel = Docker.registerDockerTask(project, 'buildWheel') {
copyIn {
from(project.projectDir) {
exclude 'build'
Expand All @@ -30,4 +34,8 @@ Docker.registerDockerTask(project, 'buildWheel') {
copyOut {
into 'build/deephaven-jpy-wheel'
}
}
}

artifacts {
pythonWheel buildWheel
}
File renamed without changes.
7 changes: 7 additions & 0 deletions py/server/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
plugins {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to other reviewers, this replaces py/deephaven2-wheel/build.gradle

id 'io.deephaven.python-wheel'
}

dependencies {
pythonWheel project(path: ':deephaven-jpy', targetConfiguration: 'pythonWheel')
}
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading