Link git outputs between tasks

- Implementation details
Pipelinerun creates pvc for the lifetime for object and uses that pvc as
scratch space to transfer git resources between them. This information
is passed to taskrun via resource paths. Paths are array of strings and
incase of inouts these paths will be considered as new source of
pipeline resource. In the case of outputs paths will be considered as
new destination directory.
- Update docs to include examples of paths

Partially fixes #148

docs/Concepts.md

@@ -33,18 +33,48 @@ Below diagram lists the main custom resources created by Pipeline CRDs:
 ### Task
 
 A Task is a collection of sequential steps you would want to run as part of your continous integration flow.
-
 A task will run inside a container on your cluster. A Task declares:
 
-1. Inputs the task needs.
-1. Outputs the task will produce.
-1. Sequence of steps to execute. Each step is [a container image](./using.md#image-contract).
+### Inputs:
+
+Declare the inputs the task needs. Every task input resource should provide name and type (like git, image). It can also provide optionally `targetPath` to initialize resource in specific directory. If `targetPath` is set then resource will be initialized under `/workspace/targetPath`. If `targetPath` is not specified then resource will be initialized under `/workspace`. Following example demonstrates how git input repository could be initialized in GOPATH to run tests.
+
+```yaml
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: Task
+metadata:
+  name: task-with-input
+  namespace: default
+spec:
+  inputs:
+    resources:
+    - name: workspace
+      type: git
+      targetPath: /go/src/github.com/knative/build-pipeline
+  steps:
+  - name: unit-tests
+    image: golang
+    command: ['go']
+    args:
+    - 'test'
+    - './...'
+    workingDir: "/workspace/go/src/github.com/knative/build-pipeline"
+    env:
+    - name: GOPATH
+      value: /workspace/go
+```
+
+### Outputs:
+Declare the outputs task will produce.
+
+### Steps:
+Sequence of steps to execute. Each step is [a container image](./using.md#image-contract).
 
 Here is an example simple Task definition which echoes "hello world". The `hello-world` task does not define any inputs or outputs.
 
 It only has one step named `echo`. The step uses the builder image `busybox` whose entrypoint set to `\bin\sh`.
 
-```shell
+```yaml
 apiVersion: pipeline.knative.dev/v1alpha1
 kind: Task
 metadata:
@@ -169,6 +199,66 @@ Creating a `TaskRun` will invoke a [Task](#task), running all of the steps until
 completion or failure. Creating a `TaskRun` will require satisfying all of the input
 requirements of the `Task`.
 
+`TaskRun` definition includes `inputs`, `outputs` for `Task` referred in spec.
+
+Input resource includes name and reference to pipeline resource and optionally `paths`. `paths` will be used by `TaskRun` as the resource's new source paths i.e., copy the resource from specified list of paths. `TaskRun` expects the folder and contents to be already present in specified paths. `paths` feature could be used to provide extra files or altered version of existing resource before execution of steps.
+
+Output resource includes name and reference to pipeline resource and optionally `paths`. `paths` will be used by `TaskRun` as the resource's new destination paths i.e., copy the resource entirely to specified paths. `TaskRun` will be responsible for creating required directories and copying contents over. `paths` feature could be used to inspect the results of taskrun after execution of steps.
+
+`paths` feature for input and output resource is heavily used to pass same version of resources across tasks in context of pipelinerun.
+
+In the following example, task and taskrun are defined with input resource, output resource and step which builds war artifact. After execution of taskrun(`volume-taskrun`), `custom` volume will have entire resource `java-git-resource`(including the war artifact) copied to the destination path `/custom/workspace/`.
+
+```yaml
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: Task
+metadata:
+  name: volume-task
+  namespace: default
+spec:
+  generation: 1
+  inputs:
+    resources:
+    - name: workspace
+      type: git
+  steps:
+  - name: build-war
+    image: objectuser/run-java-jar #https://hub.docker.com/r/objectuser/run-java-jar/
+    command: jar
+    args: ['-cvf', 'projectname.war', '*']
+    volumeMounts:
+    - name: custom-volume
+      mountPath: /custom
+```
+
+```yaml
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: volume-taskrun
+  namespace: default
+spec:
+  taskRef:
+    name: volume-task
+  inputs:
+    resources:
+    - name: workspace
+      resourceRef:
+        name: java-git-resource
+  outputs:
+    resources:
+    - name: workspace
+      paths:
+      - /custom/workspace/
+      resourceRef:
+        name: java-git-resource
+  volumes:
+  - name: custom-volume
+    emptyDir: {}
+```
+
+
+
 `TaskRuns` can be created directly by a user or by a [PipelineRun](#pipelinerun).
 
 #### PipelineRun

docs/using.md

@@ -106,6 +106,8 @@ steps:
     value: 'world'
 ```
 
+Pipeline Tasks are allowed to pass resources from previous tasks via `providedBy` field. This feature is implemented using Persistent Volume Claim under the hood but however has an implication that tasks cannot have any volume mounted under path `/pvc`.
+
 ### Conventions
 
 * `/workspace/<resource-name>`: [`PipelineResources` are made available in this mounted dir](#creating-resources)

examples/invocations/kritis-pvc-run.yaml

@@ -0,0 +1,25 @@
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: PipelineRun
+metadata:
+  name: kritis-pipeline-pvc
+  namespace: default
+spec:
+  pipelineRef:
+    name: kritis-pipeline-pvc
+  triggerRef:
+    type: manual
+  resources:
+  - name: create-file-kritis
+    inputs:
+    - name: workspace
+      resourceRef:
+        name: kritis-resources-git
+    outputs:
+    - name: workspace
+      resourceRef:
+        name: kritis-resources-git
+  - name: check-kritis
+    inputs:
+    - name: workspace
+      resourceRef:
+        name: kritis-resources-git

examples/kritis-pvc-tasks.yaml

@@ -0,0 +1,42 @@
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: Task
+metadata:
+  name: create-file
+  namespace: default
+spec:
+  inputs:
+    resources:
+    - name: workspace
+      type: git
+      targetPath: /damnworkspace
+  outputs:
+    resources:
+    - name: workspace
+      type: git
+  steps:
+  - name: read-docs-old
+    image: ubuntu
+    command: ["/bin/bash"]
+    args: ['-c', 'ls -la /workspace/damnworkspace/docs/install.md'] # tests that targetpath works
+  - name: write-new-stuff
+    image: ubuntu
+    command: ['bash']
+    args: ['-c', 'echo some stuff > /workspace/damnworkspace/stuff']
+
+---
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: Task
+metadata:
+  name: check-stuff-file-exists
+  namespace: default
+spec:
+  inputs:
+    resources:
+    - name: workspace
+      type: git
+      targetPath: /newworkspace
+  steps:
+  - name: read
+    image: ubuntu
+    command: ["/bin/bash"]
+    args: ['-c', 'cat /workspace/newworkspace/stuff'] # tests that new targetpath and previous task output is dumped

examples/pipelines/kritis-pvc-pipeline.yaml

@@ -0,0 +1,16 @@
+apiVersion: pipeline.knative.dev/v1alpha1
+kind: Pipeline
+metadata:
+  name: kritis-pipeline-pvc
+  namespace: default
+spec:
+  tasks:
+  - name: create-file-kritis          # 1. create file
+    taskRef:
+      name: create-file
+  - name: check-kritis               # 2.  check file exists
+    taskRef:
+      name: check-stuff-file-exists
+    resources:
+      - name: workspace
+        providedBy: [create-file-kritis]

examples/pipelines/kritis-resources.yaml

@@ -37,4 +37,4 @@ spec:
   - name: token
     value: eyJhbGciOiJ.....
   - name: username
-    value: admin    
+    value: admin

pkg/apis/pipeline/v1alpha1/pipelinerun_types.go

@@ -17,10 +17,23 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"fmt"
+
 	duckv1alpha1 "github.com/knative/pkg/apis/duck/v1alpha1"
 	"github.com/knative/pkg/webhook"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+)
+
+var (
+	pipelineRunControllerName = "PipelineRun"
+	groupVersionKind          = schema.GroupVersionKind{
+		Group:   SchemeGroupVersion.Group,
+		Version: SchemeGroupVersion.Version,
+		Kind:    pipelineRunControllerName,
+	}
 )
 
 // Assert that TaskRun implements the GenericCRD interface.
@@ -178,3 +191,38 @@ func (pr *PipelineRun) GetTaskRunRef() corev1.ObjectReference {
 
 // SetDefaults for pipelinerun
 func (pr *PipelineRun) SetDefaults() {}
+
+// GetPVC gets PVC for
+func (pr *PipelineRun) GetPVC() *corev1.PersistentVolumeClaim {
+	var pvcSizeBytes int64
+	// TODO(shashwathi): make this value configurable
+	pvcSizeBytes = 5 * 1024 * 1024 * 1024 // 5 GBs
+	return &corev1.PersistentVolumeClaim{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace:       pr.Namespace,
+			Name:            pr.GetPVCName(),
+			OwnerReferences: pr.GetOwnerReference(),
+		},
+		Spec: corev1.PersistentVolumeClaimSpec{
+			// Multiple tasks should be allowed to read
+			AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
+			Resources: corev1.ResourceRequirements{
+				Requests: map[corev1.ResourceName]resource.Quantity{
+					corev1.ResourceStorage: *resource.NewQuantity(pvcSizeBytes, resource.BinarySI),
+				},
+			},
+		},
+	}
+}
+
+// GetOwnerReference gets the pipeline run as owner reference for any related objects
+func (pr *PipelineRun) GetOwnerReference() []metav1.OwnerReference {
+	return []metav1.OwnerReference{
+		*metav1.NewControllerRef(pr, groupVersionKind),
+	}
+}
+
+// GetPVCName provides name of PVC for corresponding PR
+func (pr *PipelineRun) GetPVCName() string {
+	return fmt.Sprintf("%s-pvc", pr.Name)
+}

pkg/apis/pipeline/v1alpha1/pipelinerun_types_test.go

@@ -83,3 +83,24 @@ func TestInitializeConditions(t *testing.T) {
 		t.Fatalf("PipelineRun status getting reset")
 	}
 }
+
+func Test_GetPVC(t *testing.T) {
+	p := &PipelineRun{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "test-name",
+			Namespace: "test-ns",
+		},
+	}
+	expectedPVC := corev1.PersistentVolumeClaim{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "test-ns",
+			Name:      "test-name-pvc", // prname-pvc
+			OwnerReferences: []metav1.OwnerReference{
+				*metav1.NewControllerRef(p, groupVersionKind),
+			},
+		},
+	}
+	if d := cmp.Diff(p.GetPVC().ObjectMeta, expectedPVC.ObjectMeta); d != "" {
+		t.Fatalf("GetPVC mismatch; want %v got %v; diff %s", expectedPVC.ObjectMeta, p.GetPVC().ObjectMeta, d)
+	}
+}

pkg/apis/pipeline/v1alpha1/resource_types.go

@@ -93,6 +93,9 @@ var _ webhook.GenericCRD = (*PipelineResource)(nil)
 type TaskResource struct {
 	Name string               `json:"name"`
 	Type PipelineResourceType `json:"type"`
+	// +optional
+	// TargetPath is the path in workspace directory where the task resource will be copied.
+	TargetPath string `json:"targetPath"`
 }
 
 // +genclient
@@ -118,6 +121,8 @@ type PipelineResource struct {
 type TaskRunResource struct {
 	Name        string              `json:"name"`
 	ResourceRef PipelineResourceRef `json:"resourceRef"`
+	// +optional
+	Paths []string `json:"paths"`
 }
 
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

pkg/apis/pipeline/v1alpha1/taskrun_types.go

@@ -17,6 +17,8 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"fmt"
+
 	"github.com/knative/pkg/apis"
 	duckv1alpha1 "github.com/knative/pkg/apis/duck/v1alpha1"
 	"github.com/knative/pkg/webhook"
@@ -43,7 +45,7 @@ type TaskRunSpec struct {
 	// +optional
 	Generation int64 `json:"generation,omitempty"`
 	// +optional
-	ServiceAccount string `json:"serviceAccount"`
+	ServiceAccount string `json:"serviceAccount,omitempty"`
 }
 
 // TaskRunInputs holds the input values that this task was invoked with.
@@ -174,3 +176,13 @@ func (tr *TaskRun) GetBuildRef() corev1.ObjectReference {
 		Name:       tr.Name,
 	}
 }
+
+// GetPipelineRunPVCName for taskrun gets pipelinerun
+func (tr *TaskRun) GetPipelineRunPVCName() string {
+	for _, ref := range tr.GetOwnerReferences() {
+		if ref.Kind == pipelineRunControllerName {
+			return fmt.Sprintf("%s-pvc", ref.Name)
+		}
+	}
+	return ""
+}