Oci support rebased (kptdev#2621)

* design document template (kptdev#2563)

* updated maintainers

* update to the front page

* installation paragraph

* design doc template

* additional process detail

* slight updates

* update kyaml and cli-utils to latest (kptdev#2523)

* Do not error while adding merge comment (kptdev#2564)

* Remove check if repo is checked into git for update (kptdev#2569)

* Remove check if repo is checked into git for update

* Update tests

* Update should accept empty path with version (kptdev#2572)

* Update roadmap for Q4 (kptdev#2571)

* Update roadmap for Q4

* Suggested changes

* Update roadmap (kptdev#2574)

* Update roadmap

* Suggested changes

* Update with suggested format

* Organize better

* update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)

* Update kpt book with depends-on annotation (kptdev#2565)

* Update kpt book with depends-on annotation

* Addressed comments

* Add snippet for resource dependencies in chapter overview

* fix: use KptFileKind variable instead of KptFileName for GVR (kptdev#2580)

When specifying the Kptfile resource kind, some code were using the
KptFileName variable instead of KptFileKind. Both variables currently
contain the same value: "Kptfile", but this will be an issue in the future
should the KptFileName and KptFileKind variables have different values.

* Disable unwrapping to read fnconfig (kptdev#2582)

* Disable wrapping to read fnconfig

* Update index annotation

* Update licenses (kptdev#2583)

- Use go install with explicit tag to avoid modifying go mod
- Add licenses to yaml files (new addlicense feature)
- Add updated license-check to make all

* Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)" (kptdev#2584)

This reverts commit 98f0299.

* add tests to exercise modify path annotation (kptdev#2586)

* Update design doc review process (kptdev#2594)

* Update beta 9 (kptdev#2597)

* Add missing json annotation to KptFile (kptdev#2598)

* Implement metrics annotation (kptdev#2588)

* Implement metrics annotation

* Changes

* Suggested changes

* Suggested changes

* Attributor

* Design proposal: kpt live -- Inventory to ResourceGroup (kptdev#2576)

* DD for kpt live apply from STDIN

* resolve comments

* Update doc with rg fn interaction

Co-authored-by: Phani Teja Marupaka <pmarupaka@google.com>

* Add cnrm annotation to merge3 (kptdev#2605)

* docs: Specify origin in git push command in 3.8 (kptdev#2608)

Issues: kptdev#2596

* upgrade to kyaml 0.13.0+ (kptdev#2603)

* upgrade to kyaml 0.13.0+

* use the latest kyaml

* --image flag typo fix (kptdev#2601)

Fix typo in `kpt fn eval`'s --image flag description

* Simplify and speed up "kpt live" e2e tests (kptdev#2613)

* Simplify and speed up live e2e tests

* Addressed comments

* Adding OCI to Kptfile schema

Adding OCI to error utils

Adding OCI argument parsing to kpt pkg get

Adding get/fetch validation

Adding pull and untar to support kpt get

Making change in md file

go fmt and fixing tests

* Adds `oci://` to disambiguate `kpt pkg get` argument

Factoring upstream fetcher out of switch cases

go fmt fixes

Ignore scopelint in table tests

Adding support for `kpt pkg update`

Fix fmt and tests

Path must be added to returned absPath

Removing git repo error message check

* this startup check probably doesn't make sense any more? the pkg
update logic doesn't rely on the state of the local git repo to merge

Adding OCI support to kpt pkg diff

* Mainly involves using upstream.Fetcher methods

Adding prototype kpt pkg pull command

Based on design and latest conversations

Adding kpt pkg push command

Minor fixes

* Fixing a direct reference to Git field
* Adding output to pkg push command

Rename upstream pkg to remote

Refactoring remote names

* Rename Fether to Upstream
* Split Upstream and Origin implementations

Removing redundant Origin suffix from names

Adding compile-time interface assertion

go fix formatting

Adding missing err checks

Fixing package reference case

Fixing case sensitive package

* Fixing go.sum after rebase

* Adjusting merge of update

* Adding support for directory in oci subpackage upstream

* Fix formatting

* Remove check if repo is checked into git for update (kptdev#2569)

* Remove check if repo is checked into git for update

* Update tests

* Updating go.sum

* Adding previous round of PR feedback

* Updating scheme

* Adding TODO(oci-support) marker to kpt site content

Co-authored-by: Mike Borozdin <mikebz@google.com>
Co-authored-by: Natasha Sarkar <natashasarkar@google.com>
Co-authored-by: phani <phanitejamarupaka@gmail.com>
Co-authored-by: Mengqi Yu <mengqiy@google.com>
Co-authored-by: Morten Torkildsen <mortent@google.com>
Co-authored-by: Ramon Quitales <quitalesramon+github@gmail.com>
Co-authored-by: Karl Isenberg <karlisenberg@google.com>
Co-authored-by: Martin Maly <mmaly@google.com>
Co-authored-by: Yuwen Ma <yuwenma@google.com>
Co-authored-by: Phani Teja Marupaka <pmarupaka@google.com>
Co-authored-by: sdowell <sdowell@google.com>
Co-authored-by: Ben Congdon <bencongdon96@gmail.com>
Co-authored-by: Louis DeJardin <dejardin@google.com>

commands/pkgcmd.go

@@ -20,6 +20,8 @@ import (
 	"github.com/GoogleContainerTools/kpt/internal/cmddiff"
 	"github.com/GoogleContainerTools/kpt/internal/cmdget"
 	"github.com/GoogleContainerTools/kpt/internal/cmdinit"
+	"github.com/GoogleContainerTools/kpt/internal/cmdpull"
+	"github.com/GoogleContainerTools/kpt/internal/cmdpush"
 	"github.com/GoogleContainerTools/kpt/internal/cmdupdate"
 	"github.com/GoogleContainerTools/kpt/internal/docs/generated/pkgdocs"
 	"github.com/GoogleContainerTools/kpt/thirdparty/cmdconfig/commands/cmdtree"
@@ -47,7 +49,8 @@ func GetPkgCommand(ctx context.Context, name string) *cobra.Command {
 	pkg.AddCommand(
 		cmdget.NewCommand(ctx, name), cmdinit.NewCommand(ctx, name),
 		cmdupdate.NewCommand(ctx, name), cmddiff.NewCommand(ctx, name),
-		cmdtree.NewCommand(ctx, name),
+		cmdtree.NewCommand(ctx, name), cmdpull.NewCommand(ctx, name),
+		cmdpush.NewCommand(ctx, name),
 	)
 	return pkg
 }

docs/design-docs/00-template.md

@@ -79,6 +79,7 @@ Please name your documents with the design document number.  Pick n+1 from what'
 renaming might need to happen if multiple design docs get accepted simultaneously.
 Please keep the name of the design doc in lower case with dashes in between words.
 
+=======
 Reviewers are auto-assigned to review the PR. Optionally, doc owner can mention any
 specific reviewers on the PR description. The turn around time for each review cycle
 on the PR from kpt maintainers is 1-2 days. After maintainers add comments to the PR,

docs/design-docs/01-live_Inventory-to-ResourceGroup.md

@@ -0,0 +1,297 @@
+# Title
+
+* Author(s): Yuwen Ma <yuwenma@google.com>
+* Contributor(s):
+    - Sean Sullivan <seans@google.com>
+    - Sunil Arora <sunilarora@google.com>
+    - Mortent Torkildsen <mortent@google.com>
+    - Mengqi Yu <mengqiy@google.com>
+    - Natasha Sarkar <natashasarkar@google.com>
+
+* Approver: Sunil Arora <sunilarora@google.com>
+
+## Why
+
+This design is aimed at making `kpt live` be hydration-tool-neutral so that `kpt live`
+applicable use cases can be expanded to `kustomize build`, `helm templates` and raw
+kubernetes manifests.
+
+### Non-goal
+- This doc is not aimed at designing a unified hydration between kpt and kustomize
+- This doc is not aimed at changing the kpt in-place hydration models.
+
+## Design
+
+We propose to make `kpt live` support reading configurations from standard input,
+detach the Inventory from Kptfile and simplify the Inventory-ResourceGroup mechanism
+to use  ResourceGroup only.
+
+### Read standard input resources
+
+We will change the existing `kpt live apply` from STDIN to not require the existence of Kptfile (
+to be more specific, the `inventory` file from Kptfile). As long as the STDIN contains
+one and only one valid ResourceGroup, `kpt live apply` should be able to create/match the
+ResourceGroup in cluster.
+
+### Initialize a `ResourceGroup` object
+
+`kpt live init` will create a ResourceGroup CR in resourcegroup.yaml. 
+
+By default, the ResoureGroup will have name and namespace assigned as below. And users can
+override via existing flags "--name" and "--namespace".
+
+- metadata.name: A client-provided valid RFC 1123 DNS subdomain name. Default value has prefix “inventory-” with
+  8 digit numbers. E.g. "inventory-02014045"
+- metadata.namespace: a valid namespace. default to value "default"
+
+If users want to reuse an existing inventory (from Kptfile) or ResourceGroup (which has been deployed to the cluster),
+they shall provide the value of the inventory's inventory-id or the ResourceGroup's "metadata.labels[0].cli-utils.sigs.k8s.io/inventory-id"
+via "--inventory-id" flag.
+
+### Convert Inventory to ResourceGroup
+
+The client-side Inventory is considered as the identifier of the cluster-side
+ResourceGroup CR. Right now, the Inventory is stored in the Kptfile to guarantee
+the ResourceGroup is unique. This requires Kptfile to exist to use `kpt live apply`.
+
+To split the Inventory from Kptfile to resourcegroup.yaml and convert the Inventory to
+ResourceGroup, `kpt live migrate` should be extended to map the inventory.name,
+inventory.namespace, inventory.inventoryID to ResourceGroup CR metadata.name,
+metadata.namespace, metadata.labels.cli-utils.sigs.k8s.io/inventory-id correspondingly.
+
+Inventory in Kptfile
+```yaml
+apiVersion: kpt.dev/v1
+kind: Kptfile
+inventory:
+  name: <INVENTORY_NAME>
+  namespace: <INVENTORY_NAMESPACE>
+  inventoryID: <INVENTORY_ID>
+```
+resoucegroup.yaml
+```yaml
+apiVersion: kpt.dev/v1alpha1
+kind: ResourceGroup
+metadata  
+  name: <INVENTORY_NAME>
+  namespace: <INVENTORY_NAMESPACE>
+  labels:
+    cli-utils.sigs.k8s.io/inventory-id: <INVENTORY_ID>
+```
+
+### Simplify the Inventory
+
+Current inventory contains inventory-id which is required to match the label `cli-utils.sigs.k8s.io/inventory-id`.
+
+For new users, they should no longer need to be exposed to the inventory-id, but kpt will
+build one composed by `name-namespace` on the fly. 
+
+For existing users, the inventory-id is still
+required in the standalone ResourceGroup file to guarantee the adoption matches, unless they use "--inventory-policy=adopt"
+to override the label. This flag is only required as a one-off via `kpt live apply -` to override the label to `name-namespace`.
+
+### ResourceGroup as a singleton object
+
+`kpt live apply [--rg-file] -` from STDIN accepts and only accepts a single
+ResourceGroup, including the ResourceGroup provided by the flag. It detects
+1. If more than one ResourceGroup is found, raise errors and display all the ResourceGroup objects.
+2. If no ResourceGroup is found in STDIN and and Kptfile inventory does not exists, raise errors and suggest users to
+   run kpt live init
+3. If no ResourceGroup is found in STDIN and Kptfile inventory exists, raise errors and
+   suggest users to run kpt live migrate
+
+### New flags
+
+#### `--rg-file` 
+
+- description: The file path to the ResourceGroup CR, default to `resourcegroup.yaml`
+- short form `--rg`
+- This flag will be added to `kpt live init`, `kpt live migrate` and `kpt live apply`
+
+#### `--name` for inventory
+
+- description: The name for the ResourceGroup
+- This flag will continue to be used by `kpt live init`. Rather than overriding the
+  inventory.name in Kptfile, it will override the default metadata.name in the standalone ResourceGroup file.
+
+#### `--namespace` for inventory
+
+- description: The namespace for the ResourceGroup
+- This flag will continue to be used by `kpt live init`, Rather than overriding the
+  inventory.namespace in Kptfile, it will override the default metadata.namespace in the standalone ResourceGroup file.
+
+#### `--inventory-id` for inventory
+
+- description: Inventory identifier. This is used to detect overlap between
+  two sets of ResourceGroup managed resources that might use the same name and namespace.
+- This flag will continue to be accepted by `kpt live init` for backward compatibility reasons. 
+  If given, ResourceGroup will store the inventory-id value in "metadata.labels[0].cli-utils.sigs.k8s.io/inventory-id"
+  of the ResourceGroup. 
+  If not given, the ResourceGroup labels will be empty and the value of "<name>-<namespace>" will be
+  used as the "cli-utils.sigs.k8s.io/inventory-id" label in `kpt live apply` from STD.
+
+### `resoucegroup.yaml` interaction with `kpt fn` commands
+
+All `kpt fn` commands should treat `resoucegroup.yaml` as a special resource 
+and not modify it in any scenario. It should not be considered as a meta resource and 
+`include-meta-resources` flag should not include the `ResourceGroup` resource in 
+the `ResourceList`. In the future, we can add a new flag `--include-rg` if there are
+valid use-cases to modify or include `ResourceGroup` resource in `ResourceList`.
+
+## User Guide
+
+### To hydrate via kustomize and deploy via kpt
+
+#### Day 1
+
+##### For new users to start from scratch (no Kptfile)
+User can run `kpt live init [--rg-file=resourcegroup.yaml]` to create a
+ResourceGroup object and store it in a resourcegroup.yaml file.
+Users can customize the file path with the flag “--rg-file”.
+
+##### For existing kpt users to migrate from Kptfile
+Users run `kpt live migrate [--rg-file=resourcegroup.yaml]` to convert the
+Inventory object from Kptfile to a standalone resourcegroup.yaml file.
+Users can customize the file path with the flag “--rg-file”.
+
+##### [optional]: Add shareable ResourceGroup to kustomize resources
+If the ResourceGroup is expected to be shared in the Gitops workflow, users can add
+the resourcegroup.yaml  file path to the .resources field in kustomization.yaml.
+This simplifies the Day N deployment by omitting the “–rg-file“ flag.
+
+#### Day N
+
+- Users can configure the hydration rules in kustomization.yaml
+- Users can run `kustomize build <DIR> | kpt live apply -` to hydrate and deploy
+  the kustomize-managed configurations. If resourcegroup.yaml is not added to
+  kustomize .resources field, users should provide the “–rg-file“ flag.  
+  `kustomize build <DIR> | kpt live apply –rg-file <resourcegroup.yaml> -`
+
+### To hydrate via helm and deploy via kpt
+
+#### Day 1
+
+##### For new users to start from scratch (no Kptfile)
+User can run `kpt live init --rg-file=<DIR>/resourcegroup.yaml` to create
+a ResourceGroup object and store it in the helm template <DIR>.
+
+##### For existing kpt users to migrate from Kptfile
+Users run `kpt live migrate --rg-file=<DIR>/resourcegroup.yaml` to convert
+the Inventory object from Kptfile to  a standalone resourcegroup.yaml file.
+
+#### Day N
+
+Users can run `helm templates <DIR> | kpt live apply -` to hydrate and deploy the
+helm-managed resources.
+
+See kpt issue #2399 for expected Inventory usage in package publisher/consumer.
+
+## FAQ
+
+### Will Inventory be deprecated from Kptfile?
+
+Kpt still supports inventory in Kptfile and it is not required to migrate to the
+standalone rg-path. In fact, users who do not use STDIN in `kpt live apply`
+will still have inventory read from Kptfile by default unless the –inventory-path flag
+is given.
+
+### Why not kpt live apply -k?
+
+We originally propose to use a single command kpt live apply -k (similar idea as
+kubectl apply -k ) to kustomize-hydrate and kpt-deploy the resources, storing the
+inventory in the annotation fields of the kustomization.yaml (See below).
+The inventory can be auto added to kustomization.yaml by kpt live init
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+metadata:
+  annotations:
+     kpt-inventory-id:
+     kpt-inventory-name:
+     kpt-inventory-namespace: 
+```
+#### Pros
+
+- Extreme simple user steps. Kustomize users only need to install kpt and run kpt live
+  init. Then they can start using kpt live apply to deploy their kustomize resources.
+  No manual changes to any configurations.
+- Easy to understand. The command is similar to  kubectl apply -k which makes it easier
+  to be accepted and remembered.
+
+There are two main reasons to abandon this approach:
+- Making Kpt a hydration-neutral deployment tool helps it better hook up with different
+  hydration tools.
+- kpt is not planning to treat kustomize differently and thus it does not make sense to
+  provide kustomize a shortcut.
+
+## Alternatives Considered
+
+### Use Inventory metadata to store ResourceGroup info
+
+Store the Kptfile inventory to a standalone Inventory object. Since the inventory is
+the metadata of the cluster side ResourceGroup,  hiding the ResourceGroup behind
+inventory provides users simpler syntax and mitigates the probability of overriding
+the ResourceGroup unintentionally (e.g. via kubectl apply).
+
+inventory.yaml
+```yaml
+apiVersion: kpt.dev/v1
+kind: Inventory
+metadata:
+  name: <INVENTORY_NAME>
+  namespace: <INVENTORY_NAMESPACE>
+inventoryID: <INVENTORY_ID>
+```
+
+#### Pros
+
+- Simpler resource syntax
+- Mitigate unintentional resourceGroup override.
+
+#### Cons
+- The client-side inventory requires users to manage.
+  This new resource increases the overall complexity of the kpt actuation config.
+  We frequently receive user confusions about the difference between ResourceGroup
+  and Inventory and why they cannot configure inventory like other kubernetes resource
+  (adoption mismatch errors).
+
+### Use Inventory spec to store ResourceGroup info
+
+
+inventory.yaml
+```yaml
+apiVersion: kpt.dev/v1
+kind: Inventory
+spec:
+  resourceGroup:
+    name: <INVENTORY_NAME>
+    Namespace: <INVENTORY_NAMESPACE>
+    labels:
+      cli-utils.sigs.k8s.io/inventory-id: <INVENTORY_ID>
+```
+#### Pros
+
+- Obey the kubernetes convention and make it clearer that users are expected to provide the resourceGroup kind with name, namespace and labels.
+- Mitigate unintentional resourceGroup override.
+
+#### Cons
+More complex syntax than ResourceGroup or Inventory metadata solution
+
+### Add a controller for ResourceGroup pruning; No client-side Inventory
+
+Inventory is a [cli-utils object](https://github.com/kubernetes-sigs/cli-utils/blob/52b000849deb2b233f0a58e9be16ca2725a4c9cf/pkg/inventory/inventory.go#L30) used to store the [ResourceGroup metadata](https://docs.google.com/document/d/1x_02xGKZH6YGuc3K-KNeY4KmADPAwzrlQKkvTNXYekc/edit#heading=h.265kfx5ku27).
+It is stored in Kptfile .Inventory and required by `kpt live apply`. In lieu of a user configuration, it is more of a data for kpt to identify the corresponding ResourceGroup for pruning. This adds non-trivial overheads for users to understand, maintain and retrieve the inventory.
+
+The alternative proposal is to remove the inventory but use a new controller to prune the resource.
+The controller will be installed together with ResourceGroup CRD (in each `kpt live apply`).
+It will find the previous ResourceGroup CR from current resources, three-way merging and deploying
+the new resources and update the ResourceGroup.
+
+#### Pros
+- Improve the user experience on using kpt live and minimize the confusions around inventory and ResourceGroup
+- Eliminate the pain points on retrieving the missing Inventory (list all ResourceGroup, check the .spec.resources and try one by one until not encounter the “missing adoption” error).
+
+#### Cons
+The controller needs multiple requests to find out the previous ResourceGroup CR. This causes heavy traffic in a production cluster.  
+The controller cannot 100% guarantee the resourceGroup reverse lookup are accurate.
+Since the pruning happens on the cluster side, kpt can no longer provide real-time status updates.

e2e/testdata/live-apply/crd-and-cr/config.yaml

@@ -26,6 +26,7 @@ stdOut: |
   custom.kpt.dev/cr reconcile pending
   custom.kpt.dev/cr reconciled
   2 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out
+
 inventory:
   - group: apiextensions.k8s.io
     kind: CustomResourceDefinition

e2e/testdata/live-apply/json-output/resources/fourth.yaml

@@ -0,0 +1,37 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: fourth-nginx
+  namespace: json-output
+  annotations:
+    config.k8s.io/owning-inventory: json-output
+    config.kubernetes.io/depends-on: apps/namespaces/json-output/Deployment/third-nginx
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: fourth-nginx
+  template:
+    metadata:
+      labels:
+        app: fourth-nginx
+    spec:
+      containers:
+        - name: nginx
+          image: nginx:1.14.2
+          ports:
+            - containerPort: 80