backend: add Azure Blob Storage (ABS) backup support #1237

vdice · 2017-06-29T20:11:20Z

This PR adds Azure Blob Storage (ABS) backup support.

Accomplishes part of #1110

[skip ci]

hongchaodeng · 2017-06-29T21:09:06Z

Thanks for the contribution.
Can you discuss about your design (api, spec change) first in #1110 ?

vdice · 2017-07-03T17:35:41Z

@hongchaodeng design document added in latest push; please see doc/design/abs_backup.md. Thank you!

xiang90 · 2017-07-05T02:49:40Z

doc/design/abs_backup.md

+
+When starting etcd operator, we need to provide flags in order to access the ABS storage account:
+```bash
+$ etcd-operator --backup-abs-secret ${secret_name} --backup-abs-container ${container_name} ...


if both s3 and abs options are given, users can choose from s3 or abs? they are not mutual-exclusive, right?

xiang90 · 2017-07-05T02:59:14Z

pkg/backup/abs.go

+	"io/ioutil"
+
+	"github.com/Sirupsen/logrus"
+	"github.com/coreos/etcd-operator/pkg/backup/abs"


separate internal and external imports

xiang90 · 2017-07-05T03:00:31Z

pkg/backup/abs.go

+func (ab *absBackend) getBlobSize(key string) (int64, error) {
+	rc, err := ab.open(key)
+
+	b, err := ioutil.ReadAll(rc)


do you need to close rc on error?

I've updated to close rc on error and otherwise return ioutil.Close(rc) as the error returned from the function -- how does this sound?

xiang90 · 2017-07-05T03:01:22Z

pkg/backup/abs/abs.go

+	containerExists, err := containerRef.Exists()
+	if err != nil {
+		return nil, err
+	} else if !containerExists {


change this to if !containerExists

else is not needed.

xiang90 · 2017-07-05T03:01:55Z

pkg/backup/abs/abs.go

+	opts := &storage.DeleteBlobOptions{}
+	err := blob.Delete(opts)
+
+	return err


return blob.Delete(&storage.DeleteBlobOptions{})

xiang90 · 2017-07-05T03:02:11Z

pkg/backup/abs/abs.go

+	blob := w.container.GetBlobReference(blobName)
+
+	opts := &storage.GetBlobOptions{}
+	resp, err := blob.Get(opts)


return blob.Get(&storage.GetBlobOptions{})

xiang90 · 2017-07-05T03:03:11Z

pkg/backup/abs_backend_test.go

+		t.Fatal(err)
+	}
+	if err.Error() != "missing required environment variable of AZURE_STORAGE_ACCOUNT" {
+		t.Fatal(err)


t.Error("explain the problem here")

xiang90 · 2017-07-05T03:04:49Z

@vdice Looks good in general. @hongchaodeng needs to approve this.

ultimateboy · 2017-07-05T15:44:31Z

pkg/backup/abs/abs.go

+		return fmt.Errorf("create block blob from reader failed: %v", err)
+	}
+
+	return err


this should probably just be return nil

hongchaodeng · 2017-07-05T16:17:24Z

Code freeze right now. K8s 1.7 is released and it breaks a lot of our tests. We need to fix failures and make tests stable before merging any feature.

vdice · 2017-07-14T22:55:19Z

PR updated/rebased from master; also added changelog addition. PTAL @hongchaodeng

hongchaodeng · 2017-07-15T17:06:33Z

We won't merge any feature for a while.
First, we need to make CRD migration happen. TPR will be removed in k8s 1.8 . For survival, we need to put all our efforts into migration and won't have much time on other stuff.
Second, for longer term, we will move to more extensible backup storage. We will eventually move all stuff except PV over there too due to security concerns and easier management, e.g. separate persistent disk to save the credentials and config.

vdice · 2017-08-23T01:07:16Z

@hongchaodeng rebased off master and tested to success on a (minikube) k8s 1.7.0 cluster -- please let me know if anything looks awry, when convenient. Thank you!

xiang90 · 2017-08-24T18:58:14Z

example/secret-abs-credentials.yaml.template

+type: Opaque
+stringData:
+  storage-account: <storage-account-name>  
+  storage-key: <storage-key>


xiang90 · 2017-08-24T19:00:42Z

example/example-etcd-cluster-with-abs-backup.yaml

+  ## the section below can be added to restore from a pre-existing cluster backup
+  restore:
+    backupClusterName: "cluster-a"
+    storageType: "ABS"


xiang90 · 2017-08-24T19:03:05Z

@hongchaodeng

I feel this is OK to merge. I am OK with adding Azure/AWS/GCP support in tree initially. Then we need to figure out a way to add out of tree support for backup storage.

I give this pr a quick look. It seems OK. defer to @hongchaodeng, @hasbro17 and @fanminshi for a detailed review.

hongchaodeng · 2017-08-24T19:18:35Z

Yeah. I agree that we should merge this and move forward.
After we finished the CRD migration, our focus has moved a bit off etcd operator and doesn't have much time making the extensible design for short term. Let's merge this for easier cloud support at this moment.

hongchaodeng · 2017-08-24T19:19:50Z

cmd/operator/main.go

@@ -76,6 +79,9 @@ func init() {
 	flag.StringVar(&awsConfig, "backup-aws-config", "",
 		"The name of the kube configmap object that stores the AWS config file. The file name must be 'config'.")
 	flag.StringVar(&s3Bucket, "backup-s3-bucket", "", "The name of the AWS S3 bucket to store backups in.")
+	flag.StringVar(&absSecret, "backup-abs-secret", "",


We will deprecate the the operator wide flag for backup storage options.
Each cluster's backup storage option should sit in its own cluster spec.

hongchaodeng · 2017-08-24T19:24:27Z

example/example-etcd-cluster-with-abs-backup.yaml

@@ -0,0 +1,19 @@
+apiVersion: "etcd.database.coreos.com/v1beta2"


The example in spec_examples.md is good enough.
To keep it consistent, no duplicate example here again.

hongchaodeng · 2017-08-24T19:26:02Z

doc/user/backup_config.md

+
+See the [ABS backup with cluster specific configuration](spec_examples.md#abs-backup-and-cluster-specific-abs-configuration) spec to see what the cluster's `spec.backup` field should be configured as to set a cluster specific ABS backup configuration. The following additional fields need to be set under the cluster spec's `spec.backup.abs` field:
+- `absContainer`: The name of the ABS container to store backups in.
+- `absSecret`: The secret object name which should contain two key value pairs for the ABS account name and key.


Please mention the file layout of the secret:

storage-account: storage-key:

hongchaodeng · 2017-08-24T19:26:20Z

example/secret-abs-credentials.yaml.template

@@ -0,0 +1,8 @@
+apiVersion: v1


put this under example/azure/ ?

hongchaodeng · 2017-08-24T19:26:46Z

glide.yaml

@@ -25,3 +25,5 @@ import:
 - package: github.com/jpillora/go-ogle-analytics
 - package: golang.org/x/net
 - package: golang.org/x/time
+- package: github.com/Azure/azure-sdk-for-go/storage


Put glide.yaml, glide.lock change in separate commit ?

It is no different to vendor the entire repo or specific dir under the repo.
Just

package: github.com/Azure/azure-sdk-for-go version ...

BTW, it's v10.3.0-beta now :)

hongchaodeng · 2017-08-24T19:42:47Z

@vdice
I would like to move forward this PR. Here's some thoughts:

First, I would like this PR to be separated into four smaller PRs to better review it
1. Separate the design docs, user docs, spec changes into a PR and submit first
2. Separate the backup sidecar (mainly pkg/backup/backup.go) related changes into a PR and submit.
3. Finish the operator code.
4. Add testing. Backup sidecar should be testable since it is just an HTTP endpoint. See s3_backend_test.go . I will help setup azurite container running in jenkins.
The operator flag is not recommended. Just add the options in cluster spec. That's the way to go. We have moved storage class into cluster spec too. All backup storage related flags will be deprecated.

vdice · 2017-08-24T21:59:04Z

@hongchaodeng Great news! I'll get to work addressing code comments and breaking this up into separate PRs as requested. Will have an update (and new PRs) by tomorrow.

vdice · 2017-08-25T19:53:59Z

@hongchaodeng pushed an additional commit that hopefully addresses all feedback. Wanted to make sure I'm on the right track prior to breaking these changes out.

In particular, hoping for confirmation that I fully removed operator flag/backup customization in anticipation of the deprecation you've mentioned. Thanks!

hongchaodeng

The direction looks good. Just some implementation detail issues. But we can fix them along the way in individual PRs.

Thanks @vdice for the fast work.

hongchaodeng · 2017-08-25T19:58:05Z

doc/design/abs_backup.md

+
+When starting etcd operator, we need to provide flags in order to access the ABS storage account:
+```bash
+$ etcd-operator --backup-abs-secret ${secret_name} --backup-abs-container ${container_name} ...


remove the flag?

hongchaodeng · 2017-08-25T20:01:43Z

pkg/cluster/backupstorage/abs.go

+		return err
+	}
+
+	os.Setenv(env.ABSStorageAccount, string(se.Data[spec.ABSStorageAccount]))


Don't set env. This will affect other clusters.

hongchaodeng · 2017-08-25T20:04:56Z

doc/user/backup_config.md

@@ -4,6 +4,7 @@ In etcd operator, we provide the following options to save cluster backups to:
 - Persistent Volume (PV) on GCE or AWS
 - Persistent Volume (PV) with custom StorageClasses
 - S3 bucket on AWS
+- ABS container on Azure


Azure Blob Storage (ABS) container

vdice · 2017-08-30T00:58:05Z

(work split up into PRs linked above -- follow those for progress)

vdice force-pushed the feat/add-abs-support branch 2 times, most recently from 901f1b5 to b6313c1 Compare June 29, 2017 20:29

vdice changed the title ~~backend: add Azure Blob Storage (abs) backup support~~ backend: add Azure Blob Storage (ABS) backup support Jun 29, 2017

vdice force-pushed the feat/add-abs-support branch from b6313c1 to 3af6935 Compare July 3, 2017 17:34

hongchaodeng mentioned this pull request Jul 3, 2017

Extensible backup storage interface #1241

Closed

xiang90 reviewed Jul 5, 2017

View reviewed changes

ultimateboy reviewed Jul 5, 2017

View reviewed changes

vdice force-pushed the feat/add-abs-support branch 2 times, most recently from 2c8a0c2 to ebdb5e1 Compare July 5, 2017 17:04

vdice force-pushed the feat/add-abs-support branch from ebdb5e1 to 3d081d3 Compare July 14, 2017 22:54

vdice force-pushed the feat/add-abs-support branch from 3d081d3 to f5e5227 Compare August 23, 2017 00:48

backend: add Azure Blob Storage (abs) backup support

ce02710

vdice force-pushed the feat/add-abs-support branch from f5e5227 to ce02710 Compare August 23, 2017 01:04

xiang90 reviewed Aug 24, 2017

View reviewed changes

hongchaodeng reviewed Aug 24, 2017

View reviewed changes

vdice force-pushed the feat/add-abs-support branch from a5c48b3 to 05c91cf Compare August 25, 2017 20:03

hongchaodeng reviewed Aug 25, 2017

View reviewed changes

address feedback

6296c5b

vdice force-pushed the feat/add-abs-support branch from 05c91cf to 6296c5b Compare August 25, 2017 21:57

This was referenced Aug 25, 2017

feat(abs): add docs, design, spec #1367

Merged

feat(abs): add backup sidecar logic #1368

Merged

feat(abs): add operator logic #1369

Merged

feat(abs): add tests #1370

Merged

vdice closed this Aug 30, 2017

		@@ -0,0 +1,19 @@
		apiVersion: "etcd.database.coreos.com/v1beta2"

backend: add Azure Blob Storage (ABS) backup support #1237

backend: add Azure Blob Storage (ABS) backup support #1237

Conversation

vdice commented Jun 29, 2017 • edited by hongchaodeng Loading

hongchaodeng commented Jun 29, 2017

vdice commented Jul 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Jul 5, 2017

Choose a reason for hiding this comment

hongchaodeng commented Jul 5, 2017

vdice commented Jul 14, 2017

hongchaodeng commented Jul 15, 2017

vdice commented Aug 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Aug 24, 2017

hongchaodeng commented Aug 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongchaodeng commented Aug 24, 2017 • edited Loading

vdice commented Aug 24, 2017

vdice commented Aug 25, 2017

hongchaodeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdice commented Aug 30, 2017

vdice commented Jun 29, 2017 •

edited by hongchaodeng

Loading

vdice commented Aug 23, 2017 •

edited

Loading

hongchaodeng commented Aug 24, 2017 •

edited

Loading

hongchaodeng commented Aug 24, 2017 •

edited

Loading