Skip to content

Commit

Permalink
E2E tests now run in multiple clouds in addition to KIND (vmware-tanz…
Browse files Browse the repository at this point in the history
…u#3286)

Split plug-in provider into cloud provider/object provider
Moved velero install/uninstall for tests into velero_utils
Added remove of CRDs to test v elero uninstall
Added remove of cluster role binding to test velero uninstall
Added dump of velero describe and logs on error
Added velero namespace argument to velero_utils functions
Modified api group versions e2e tests to use VeleroInstall
Added velero logs dumps for api group versions e2e testing
Added DeleteNamespace to test/e2e/common.go
Fixed VeleroInstall to use the image specified
Changed enable_api_group_versions_test to use veleroNamespace instead of hardcoded "velero"

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
  • Loading branch information
dsu-igeek authored and ywk253100 committed Jun 29, 2021
1 parent 5257b6e commit 7321917
Show file tree
Hide file tree
Showing 9 changed files with 282 additions and 123 deletions.
12 changes: 8 additions & 4 deletions test/e2e/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,14 @@ OUTPUT_DIR := _output/$(GOOS)/$(GOARCH)/bin
GINKGO_FOCUS ?=
VELERO_CLI ?=$$(pwd)/../../_output/bin/$(GOOS)/$(GOARCH)/velero
VELERO_IMAGE ?= velero/velero:main
VELERO_NAMESPACE ?=
CREDS_FILE ?=
BSL_BUCKET ?=
BSL_PREFIX ?=
BSL_CONFIG ?=
VSL_CONFIG ?=
PLUGIN_PROVIDER ?=
CLOUD_PROVIDER ?=
OBJECT_STORE_PROVIDER ?=

.PHONY:ginkgo
ginkgo: # Make sure ginkgo is in $GOPATH/bin
Expand All @@ -65,16 +67,18 @@ run: ginkgo
( echo "A credentials file is required to run E2E tests, please re-run the make target with CREDS_FILE=<PathToCredentialsFile>"; exit 1 )
@[ "${BSL_BUCKET}" ] && echo "Using bucket ${BSL_BUCKET} to store backups from E2E tests" || \
(echo "Bucket to store the backups from E2E tests is required, please re-run with BSL_BUCKET=<BucketName>"; exit 1 )
@[ "${PLUGIN_PROVIDER}" ] && echo "Using plugin provider ${PLUGIN_PROVIDER} for object storage and volume snaphotting" || \
(echo "Plugin provider to store the backups from E2E tests is required, please re-run with PLUGIN_PROVIDER=<pluginProviderName>"; exit 1 )
@[ "${CLOUD_PROVIDER}" ] && echo "Using cloud provider ${CLOUD_PROVIDER}" || \
(echo "Cloud provider for target cloud/plug-in provider is required, please rerun with CLOUD_PROVIDER=<aws,azure,kind,vsphere>"; exit 1)
@$(GINKGO) -v -focus="$(GINKGO_FOCUS)" . -- -velerocli=$(VELERO_CLI) \
-velero-image=$(VELERO_IMAGE) \
-velero-namespace=$(VELERO_NAMESPACE) \
-credentials-file=$(CREDS_FILE) \
-bucket=$(BSL_BUCKET) \
-prefix=$(BSL_PREFIX) \
-bsl-config=$(BSL_CONFIG) \
-vsl-config=$(VSL_CONFIG) \
-plugin-provider=$(PLUGIN_PROVIDER)
-cloud-provider=$(CLOUD_PROVIDER) \
-object-store-provider="$(OBJECT_STORE_PROVIDER)"

build: ginkgo
mkdir -p $(OUTPUT_DIR)
Expand Down
25 changes: 18 additions & 7 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,21 @@ These configuration parameters are expected as values to the following command l

1. `-credentials-file`: File containing credentials for backup and volume provider. Required.
1. `-bucket`: Name of the object storage bucket where backups from e2e tests should be stored. Required.
1. `-plugin-provider`: Provider of object store and volume snapshotter plugins. Required.
1. `-cloud-provider`: The cloud the tests will be run in. Appropriate plug-ins will be installed except for kind which requires
the object-store-provider to be specified.
1. `-object-store-provider`: Object store provider to use. Required when kind is the cloud provider.
1. `-velerocli`: Path to the velero application to use. Optional, by default uses `velero` in the `$PATH`
1. `-velero-image`: Image for the velero server to be tested. Optional, by default uses `velero/velero:main`
1. `-bsl-config`: Configuration to use for the backup storage location. Format is key1=value1,key2=value2. Optional.
1. `-prefix`: Prefix in the `bucket`, under which all Velero data should be stored within the bucket. Optional.
1. `-vsl-config`: Configuration to use for the volume snapshot location. Format is key1=value1,key2=value2. Optional.
1. `-velero-namespace`: Namespace to install velero in. Optional, defaults to "velero".
1. `-install-velero`: Specifies whether to install/uninstall velero for the tests. Optional, defaults to "true".

These configurations or parameters are used to generate install options for Velero for each test suite.

Tests can be run with the Kubernetes cluster hosted in various cloud providers or in a _kind_ cluster with storage in
a specified object store type. Currently supported cloud provider types are _aws_, _azure_, _vsphere_ and _kind_.
## Running tests locally

### Running using `make`
Expand All @@ -49,22 +55,27 @@ E2E tests can be run from the Velero repository root by running `make test-e2e`.
Below is a mapping between `make` variables to E2E configuration flags.
1. `CREDS_FILE`: `-credentials-file`. Required.
1. `BSL_BUCKET`: `-bucket`. Required.
1. `PLUGIN_PROVIDER`: `-plugin-provider`. Required.
1. `CLOUD_PROVIDER`: `-cloud-provider`. Required
1. `OBJECT_STORE_PROVIDER`: `-object-store-provider`. Required when kind is the cloud provider.
1. `VELERO_CLI`: the `-velerocli`. Optional.
1. `VELERO_IMAGE`: the `-velero-image`. Optional.
1. `BSL_PREFIX`: `-prefix`. Optional.
1. `BSL_CONFIG`: `-bsl-config`. Optional.
1. `VSL_CONFIG`: `-vsl-config`. Optional.

For example, E2E tests can be run from Velero repository roots using the below command:
For example, E2E tests can be run from Velero repository roots using the commands below:

1. Run Velero tests using AWS as the storage provider:
1. Run Velero tests in a kind cluster with AWS (or Minio) as the storage provider:
```bash
BSL_PREFIX=<PREFIX_UNDER_BUCKET> BSL_BUCKET=<BUCKET_FOR_E2E_TEST_BACKUP> CREDS_FILE=/path/to/aws-creds PLUGIN_PROVIDER=aws make test-e2e
BSL_PREFIX=<PREFIX_UNDER_BUCKET> BSL_BUCKET=<BUCKET_FOR_E2E_TEST_BACKUP> CREDS_FILE=/path/to/aws-creds CLOUD_PROVIDER=kind OBJECT_STORE_PROVIDER=aws make test-e2e
```
1. Run Velero tests using Microsoft Azure as the storage provider:
1. Run Velero tests in an AWS cluster:
```bash
BSL_CONFIG="resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID" BSL_BUCKET=velero CREDS_FILE=~/bin/velero-dev/aks-creds PLUGIN_PROVIDER=azure make test-e2e
BSL_PREFIX=<PREFIX_UNDER_BUCKET> BSL_BUCKET=<BUCKET_FOR_E2E_TEST_BACKUP> CREDS_FILE=/path/to/aws-creds CLOUD_PROVIDER=aws make test-e2e
```
1. Run Velero tests in a Microsoft Azure cluster:
```bash
BSL_CONFIG="resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID" BSL_BUCKET=<BUCKET_FOR_E2E_TEST_BACKUP> CREDS_FILE=/path/to/azure-creds CLOUD_PROVIDER=azure make test-e2e
```
Please refer to `velero-plugin-for-microsoft-azure` documentation for instruction to [set up permissions for Velero](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#set-permissions-for-velero) and to [set up azure storage account and blob container](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#setup-azure-storage-account-and-blob-container)
1. Run Ginko-focused Restore Multi-API Groups tests using an image built for PR #3133 and Minio as the backup storage location:
Expand Down
57 changes: 27 additions & 30 deletions test/e2e/backup_test.go
Original file line number Diff line number Diff line change
@@ -1,63 +1,60 @@
package e2e

import (
"context"
"flag"
"fmt"
"time"

"github.com/google/uuid"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"golang.org/x/net/context"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
apiextensionsclientset "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
"k8s.io/client-go/kubernetes"

"github.com/vmware-tanzu/velero/pkg/cmd/cli/install"
)

var (
uuidgen uuid.UUID
veleroInstallOptions *install.InstallOptions
uuidgen uuid.UUID
)

func veleroInstall(pluginProvider string, useRestic bool) {
var err error
flag.Parse()
Expect(EnsureClusterExists(context.TODO())).To(Succeed(), "Failed to ensure kubernetes cluster exists")
uuidgen, err = uuid.NewRandom()
Expect(err).To(Succeed())
veleroInstallOptions, err = GetProviderVeleroInstallOptions(pluginProvider, cloudCredentialsFile, bslBucket, bslPrefix, bslConfig, vslConfig, getProviderPlugins(pluginProvider))
Expect(err).To(Succeed(), fmt.Sprintf("Failed to get Velero InstallOptions for plugin provider %s", pluginProvider))
veleroInstallOptions.UseRestic = useRestic
Expect(InstallVeleroServer(veleroInstallOptions)).To(Succeed(), "Failed to install Velero on KinD cluster")
}

// Test backup and restore of Kibishi using restic
var _ = Describe("[Restic] [KinD] Velero tests on KinD cluster using the plugin provider for object storage and Restic for volume backups", func() {
var _ = Describe("[Restic] Velero tests on cluster using the plugin provider for object storage and Restic for volume backups", func() {
var (
client *kubernetes.Clientset
backupName string
restoreName string
client *kubernetes.Clientset
extensionsClient *apiextensionsclientset.Clientset
backupName string
restoreName string
)

BeforeEach(func() {
var err error
veleroInstall(pluginProvider, true)
client, err = GetClusterClient()
flag.Parse()
uuidgen, err = uuid.NewRandom()
Expect(err).To(Succeed())
if installVelero {
VeleroInstall(context.Background(), veleroImage, veleroNamespace, cloudProvider, objectStoreProvider, useVolumeSnapshots,
cloudCredentialsFile, bslBucket, bslPrefix, bslConfig, vslConfig, "")
}
client, extensionsClient, err = GetClusterClient()
Expect(err).To(Succeed(), "Failed to instantiate cluster client")
})

AfterEach(func() {
timeoutCTX, _ := context.WithTimeout(context.Background(), time.Minute)
err := client.CoreV1().Namespaces().Delete(timeoutCTX, veleroInstallOptions.Namespace, metav1.DeleteOptions{})
Expect(err).To(Succeed())
if installVelero {
timeoutCTX, _ := context.WithTimeout(context.Background(), time.Minute)
err := VeleroUninstall(timeoutCTX, client, extensionsClient, veleroNamespace)
Expect(err).To(Succeed())
}

})

Context("When kibishii is the sample workload", func() {
It("should be successfully backed up and restored", func() {
backupName = "backup-" + uuidgen.String()
restoreName = "restore-" + uuidgen.String()
// Even though we are using Velero's CloudProvider plugin for object storage, the kubernetes cluster is running on
// KinD. So use the kind installation for Kibishii.
Expect(RunKibishiiTests(client, "kind", veleroCLI, backupName, restoreName)).To(Succeed(), "Failed to successfully backup and restore Kibishii namespace")
Expect(RunKibishiiTests(client, cloudProvider, veleroCLI, veleroNamespace, backupName, restoreName)).To(Succeed(),
"Failed to successfully backup and restore Kibishii namespace")
})
})
})
34 changes: 30 additions & 4 deletions test/e2e/common.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
package e2e

import (
"fmt"
"os/exec"
"time"

"k8s.io/apimachinery/pkg/util/wait"

"github.com/pkg/errors"
"golang.org/x/net/context"
apiextensionsclientset "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
Expand All @@ -19,20 +24,26 @@ func EnsureClusterExists(ctx context.Context) error {
}

// GetClusterClient instantiates and returns a client for the cluster.
func GetClusterClient() (*kubernetes.Clientset, error) {
func GetClusterClient() (*kubernetes.Clientset, *apiextensionsclientset.Clientset, error) {
loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
configOverrides := &clientcmd.ConfigOverrides{}
kubeConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, configOverrides)
clientConfig, err := kubeConfig.ClientConfig()
if err != nil {
return nil, errors.WithStack(err)
return nil, nil, errors.WithStack(err)
}

client, err := kubernetes.NewForConfig(clientConfig)
if err != nil {
return nil, errors.WithStack(err)
return nil, nil, errors.WithStack(err)
}
return client, nil

extensionClientSet, err := apiextensionsclientset.NewForConfig(clientConfig)
if err != nil {
return nil, nil, errors.WithStack(err)
}

return client, extensionClientSet, nil
}

// CreateNamespace creates a kubernetes namespace
Expand All @@ -44,3 +55,18 @@ func CreateNamespace(ctx context.Context, client *kubernetes.Clientset, namespac
}
return err
}

func DeleteNamespace(ctx context.Context, client *kubernetes.Clientset, namespace string) error {
err := client.CoreV1().Namespaces().Delete(ctx, namespace, metav1.DeleteOptions{})
if err != nil {
return errors.WithMessagef(err, "Delete namespace failed removing namespace %s", namespace)
}
return wait.Poll(1*time.Second, 3*time.Minute, func() (bool, error) {
_, err := client.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{})
if err != nil {
fmt.Printf("Namespaces.Get after delete return err %v\n", err)
return true, nil // Assume any error means the delete was successful
}
return false, nil
})
}
9 changes: 7 additions & 2 deletions test/e2e/e2e_suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,23 @@ import (
)

var (
veleroCLI, veleroImage, cloudCredentialsFile, bslConfig, bslBucket, bslPrefix, vslConfig, pluginProvider string
veleroCLI, veleroImage, cloudCredentialsFile, bslConfig, bslBucket, bslPrefix, vslConfig, cloudProvider, objectStoreProvider, veleroNamespace string
installVelero, useVolumeSnapshots bool
)

func init() {
flag.StringVar(&pluginProvider, "plugin-provider", "", "Provider of object store and volume snapshotter plugins. Required.")
flag.StringVar(&cloudProvider, "cloud-provider", "", "Cloud that Velero will be installed into. Required.")
flag.StringVar(&objectStoreProvider, "object-store-provider", "", "Provider of object store plugin. Required if cloud-provider is kind, otherwise ignored.")
flag.StringVar(&bslBucket, "bucket", "", "name of the object storage bucket where backups from e2e tests should be stored. Required.")
flag.StringVar(&cloudCredentialsFile, "credentials-file", "", "file containing credentials for backup and volume provider. Required.")
flag.StringVar(&veleroCLI, "velerocli", "velero", "path to the velero application to use.")
flag.StringVar(&veleroImage, "velero-image", "velero/velero:main", "image for the velero server to be tested.")
flag.StringVar(&bslConfig, "bsl-config", "", "configuration to use for the backup storage location. Format is key1=value1,key2=value2")
flag.StringVar(&bslPrefix, "prefix", "", "prefix under which all Velero data should be stored within the bucket. Optional.")
flag.StringVar(&vslConfig, "vsl-config", "", "configuration to use for the volume snapshot location. Format is key1=value1,key2=value2")
flag.StringVar(&veleroNamespace, "velero-namespace", "velero", "Namespace to install Velero into")
flag.BoolVar(&installVelero, "install-velero", true, "Install/uninstall velero during the test. Optional.")
flag.BoolVar(&useVolumeSnapshots, "use-volume-snapshots", false, "Use volume-snapshotter plugin for volume backup. Optional")
}

func TestE2e(t *testing.T) {
Expand Down
Loading

0 comments on commit 7321917

Please sign in to comment.