Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[gradle] Support for multi-project builds #815

Closed
anuraaga opened this issue Aug 9, 2018 · 8 comments
Closed

[gradle] Support for multi-project builds #815

anuraaga opened this issue Aug 9, 2018 · 8 comments
Milestone
v0.9.10

Comments

@anuraaga
Copy link
Contributor

anuraaga commented Aug 9, 2018 •
edited
Loading

I think a part of this would be also thinking about the layering - I guess project dependencies should go in a separate layer on top of upstream dependencies, or maybe even together with the app.

Description of the issue:

Repro: https://github.com/anuraaga/gradle-jib-multiproject-example

When trying to run jibExportDockerContext for a gradle project with multiple subprojects that depend on each other, I found that the project dependency only has compileJava run, not jar. Because of this, jibExportDockerContext fails when trying to copy the jar with

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':server:jibExportDockerContext'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
        at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:273)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:258)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.api.GradleException: Export Docker context failed, perhaps you should check if the command-line option `--jibTargetDir` is set correctly
        at com.google.cloud.tools.jib.gradle.DockerContextTask.generateDockerContext(DockerContextTask.java:144)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:786)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:753)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:131)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:120)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99)
        ... 31 more
Caused by: java.nio.file.NoSuchFileException: C:\tools\msys64\home\Anuraag\git\gradle-jib-multiproject-example\lib\build\libs\lib.jar
        at com.google.cloud.tools.jib.filesystem.FileOperations.lambda$copy$0(FileOperations.java:43)
        at com.google.cloud.tools.jib.filesystem.FileOperations.copy(FileOperations.java:50)
        at com.google.cloud.tools.jib.docker.DockerContextGenerator.generate(DockerContextGenerator.java:209)
        at com.google.cloud.tools.jib.gradle.DockerContextTask.generateDockerContext(DockerContextTask.java:128)
        ... 48 more

Expected behavior:

jar task should be run for project dependencies

Steps to reproduce:

In a multi-project build, where server depends on lib, run

$ git clone https://github.com/anuraaga/gradle-jib-multiproject-example
$ cd gradle-jib-multiproject-example
$ ./gradlew :server:jibExportDockerContext --stacktrace --info

There will be a NoSuchFileException for lib.jar

Environment:

Windows 10+ Msys2, Gradle 4.9, Java 10

jib-gradle-plugin Configuration:

Adapted:

jib {
  from {
    image = 'openjdk:10-jre-slim'
  }
  to {
    image = 'gcr.io/test-project/test-image'
  }
  container {
    mainClass = 'Main'
  }
}

Log output:

Additional Information:

@anuraaga anuraaga changed the title Support for multi-project builds [gradle] Support for multi-project builds Aug 9, 2018
@coollog
Copy link
Contributor

coollog commented Aug 9, 2018

Hi @anuraaga , thanks for reporting this.

As for separating project dependencies from other dependencies, that is related to #403 - feel free to chime in there.

As for the issue described here, Jib, by default, only depends on the classes task (of compileJava) since it does not require JAR packaging for normal Java applications. However, here, since your server project build depends on lib project's jar task, you can have jib explicitly depend on that:

server/build.gradle

...
tasks.jib.dependsOn project(':lib').jar
...

@anuraaga
Copy link
Contributor Author

anuraaga commented Aug 9, 2018

Thanks for the pointer to that issue, glad multi layering is already on the radar.

For the jar issue, I agree that workaround works. But I disagree that Jib only depends on classes. I am running a Jib task, jibExportDockerContext, which is clearly failing in it's own code. It seems like Jib needs to handle the task wiring automatically since it has a dependency on the jar for multi-project builds. FWIW, gradle-docker-plugin works correctly when gathering jars so I guess it'd be a simple fix.

@loosebazooka
Copy link
Member

Yeah we might have to do something to force resolution of depenedencies.

@loosebazooka loosebazooka mentioned this issue Aug 9, 2018
Merged
@coollog coollog added this to the v0.9.9 milestone Aug 10, 2018
@coollog
Copy link
Contributor

coollog commented Aug 22, 2018

Hi @anuraaga , we have released version 0.9.9 with the change to build project dependency JARs. Let us know if this resolves your issue.

@anuraaga
Copy link
Contributor Author

anuraaga commented Aug 24, 2018 •
edited
Loading

Thanks - I updated to 0.9.9 in a complicated project and Gradle configuration fails with this error preventing anything from happening. Will try to repro in my simple project (but wondering if missing an afterEvaluate somewhere)

        at com.sun.proxy.$Proxy32.afterEvaluate(Unknown Source)
        at org.gradle.configuration.project.LifecycleProjectEvaluator$NotifyAfterEvaluate$1.execute(LifecycleProjectEvaluator.java:187)
        at org.gradle.configuration.project.LifecycleProjectEvaluator$NotifyAfterEvaluate$1.execute(LifecycleProjectEvaluator.java:184)
        at org.gradle.api.internal.project.DefaultProject.stepEvaluationListener(DefaultProject.java:1398)
        at org.gradle.configuration.project.LifecycleProjectEvaluator$NotifyAfterEvaluate.run(LifecycleProjectEvaluator.java:193)
        ... 91 more
Caused by: org.gradle.api.UnknownTaskException: Task with path 'assemble' not found in project ':common:server:framework'.
        at org.gradle.api.internal.tasks.DefaultTaskContainer.getByPath(DefaultTaskContainer.java:414)
        at com.google.cloud.tools.jib.gradle.JibPlugin.lambda$getProjectDependencyAssembleTasks$0(JibPlugin.java:62)
        at com.google.cloud.tools.jib.gradle.JibPlugin.getProjectDependencyAssembleTasks(JibPlugin.java:63)
        at com.google.cloud.tools.jib.gradle.JibPlugin.lambda$apply$1(JibPlugin.java:112)
        ... 107 more

@coollog coollog reopened this Aug 24, 2018
@coollog coollog modified the milestones: v0.9.9, v0.9.10 Aug 24, 2018
@anuraaga
Copy link
Contributor Author

By the way for context, the project here ':common:server:framework' does have the Java plugin applied and has the task assemble. Haven't gotten a chance to play with my simple repro yet and will try to but just to add more data.

@anuraaga anuraaga mentioned this issue Aug 26, 2018
Merged
@anuraaga
Copy link
Contributor Author

I found that the problem happens when the jib project is evaluated before a dependency project, which is allowed by Gradle as it evaluates projects in an alphabetical way, not a dependency graph one (since it has no dependency graph available during evaluation).

Sent #880 to fix and add a regression test (my repro repository in the original post is also updated to exercise this behavior)

@coollog
Copy link
Contributor

coollog commented Aug 27, 2018

@anuraaga Thanks for filing the fix!

@loosebazooka loosebazooka mentioned this issue Dec 4, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.9.10
Development

No branches or pull requests
3 participants
@anuraaga @loosebazooka @coollog