Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow GCP buckets in other regions #1570

Merged
merged 3 commits into from
Jul 27, 2022
Merged

Conversation

KastenMike
Copy link
Contributor

@KastenMike KastenMike commented Jul 27, 2022

This PR lets you create buckets in specific regions. Currently default args are used (StorageClass: "MULTI_REGIONAL", Location: "US"); Location default is mentioned here, StorageClass through testing. Settings are passed via config.Region like S3 does in s3Config func.

Example test (placed inside a new dir+file mike/mike_test.go in my case):

func TestRegionalBucket(t *testing.T) {
	fmt.Println("TestRegionalBucket......")
	projectId := "projectId goes here"
	configJSON := "configJSON goes here"
	bucket := "mimckayd-euro"
	location := "europe-west8"
	providerType := objectstore.ProviderTypeGCS

	ctx := context.Background()
	pc := objectstore.ProviderConfig{
		Type:     providerType,
		Region:   location,
	}
	secret := &objectstore.Secret{}
	secret.Type = objectstore.SecretTypeGcpServiceAccountKey
	secret.Gcp = &objectstore.SecretGcp{
		ProjectID:  projectId,
		ServiceKey: configJSON,
	}
	p, err := objectstore.NewProvider(ctx, pc, secret)
	if err != nil {
		t.Errorf("NO NewProvider! %v", err)
	}

	b, err := p.CreateBucket(ctx, bucket)
	if err != nil {
		t.Errorf("NO BUCKET! %v", err)
	}

	fmt.Println("bucket created...", b.String())
}

I wanted to add some kind of test to objectstore_test like this:

func (s *ObjectStoreProviderSuite) TestGCSRegions(c *C) {
	if s.osType != ProviderTypeGCS {
		c.Skip("Test only applicable to GCS")
	}
	gcsRegions := []string{"us-west1", "europe-west8", "asia-south2"}
	ctx := context.Background()
	for _, region := range gcsRegions {
		s.initProvider(c, region)
		bucketName := s.createBucketName(c)
		_, err := s.provider.CreateBucket(ctx, bucketName)
		c.Check(err, IsNil)
		// try to find region from d~
		s.provider.DeleteBucket(ctx, bucketName)
	}
}

But at this point it isn't possible; it requires another update to stow to support it (exporting the real region), in particular: kastenhq/stow#27

Can be added in a future PR after that goes in

@pchistikov pchistikov requested a review from pavannd1 July 27, 2022 09:43
if secret != nil {
if secret.Type != SecretTypeGcpServiceAccountKey {
return "", nil, errors.Errorf("invalid secret type %s", secret.Type)
}
configJSON = secret.Gcp.ServiceKey
projectID = secret.Gcp.ProjectID
if config.Region != "" {
cm[stowgcs.ConfigLocation] = config.Region
cm[stowgcs.ConfigStorageClass] = REGIONAL
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively config.StorageClass could be added but it would only be relevant for GCS

Copy link
Contributor

@ankitjain235 ankitjain235 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@mergify mergify bot merged commit c3a90d5 into kanisterio:master Jul 27, 2022
@KastenMike KastenMike deleted the gcp-regions branch July 27, 2022 11:00
KastenMike added a commit to KastenMike/kanister that referenced this pull request Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants