Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rework cache package - add gcs cache - add cache purge command #750

Merged
merged 13 commits into from
Nov 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -595,19 +595,29 @@ _Adding a remote cache_

* AWS S3
* _As a prerequisite `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are required as environmental variables._
* Configuration, ``` k8sgpt cache add --region <aws region> --bucket <name> ```
* Configuration, ``` k8sgpt cache add s3 --region <aws region> --bucket <name> ```
* K8sGPT will create the bucket if it does not exist
* Azure Storage
* We support a number of [techniques](https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=bash#2-authenticate-with-azure) to authenticate against Azure
* Configuration, ``` k8sgpt cache add --storageacc <storage account name> --container <container name> ```
* Configuration, ``` k8sgpt cache add azure --storageacc <storage account name> --container <container name> ```
* K8sGPT assumes that the storage account already exist and it will create the container if it does not exist
* It's **users'** responsibility have to grant specific permissions to their identity in order to be able to upload blob files and create SA containers (e.g Storage Blob Data Contributor)
* It is the **user** responsibility have to grant specific permissions to their identity in order to be able to upload blob files and create SA containers (e.g Storage Blob Data Contributor)
* Google Cloud Storage
* _As a prerequisite `GOOGLE_APPLICATION_CREDENTIALS` are required as environmental variables._
* Configuration, ``` k8sgpt cache add gcs --region <gcp region> --bucket <name> --projectid <project id>```
* K8sGPT will create the bucket if it does not exist

_Listing cache items_
```
k8sgpt cache list
```

_Purging an object from the cache_
Note: purging an object using this command will delete upstream files, so it requires appropriate permissions.
```
k8sgpt cache purge $OBJECT_NAME
```

_Removing the remote cache_
Note: this will not delete the upstream S3 bucket or Azure storage container
```
Expand Down
22 changes: 18 additions & 4 deletions cmd/cache/add.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,20 +28,31 @@ var (
bucketName string
storageAccount string
containerName string
projectId string
)

// addCmd represents the add command
var addCmd = &cobra.Command{
Use: "add",
Use: "add [cache type]",
Short: "Add a remote cache",
Long: `This command allows you to add a remote cache to store the results of an analysis.
The supported cache types are:
- Azure Blob storage
- Google Cloud storage
- S3`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
color.Red("Error: Please provide a value for cache types. Run k8sgpt cache add --help")
os.Exit(1)
}
fmt.Println(color.YellowString("Adding remote based cache"))
remoteCache := cache.NewCacheProvider(bucketname, region, storageAccount, containerName)
err := cache.AddRemoteCache(remoteCache)
cacheType := args[0]
remoteCache, err := cache.NewCacheProvider(cacheType, bucketname, region, storageAccount, containerName, projectId)
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
}
err = cache.AddRemoteCache(remoteCache)
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
Expand All @@ -51,9 +62,10 @@ var addCmd = &cobra.Command{

func init() {
CacheCmd.AddCommand(addCmd)
addCmd.Flags().StringVarP(&region, "region", "r", "", "The region to use for the AWS S3 cache")
addCmd.Flags().StringVarP(&region, "region", "r", "", "The region to use for the AWS S3 or GCS cache")
addCmd.Flags().StringVarP(&bucketname, "bucket", "b", "", "The name of the AWS S3 bucket to use for the cache")
addCmd.MarkFlagsRequiredTogether("region", "bucket")
addCmd.Flags().StringVarP(&projectId, "projectid", "p", "", "The GCP project ID")
addCmd.Flags().StringVarP(&storageAccount, "storageacc", "s", "", "The Azure storage account name of the container")
addCmd.Flags().StringVarP(&containerName, "container", "c", "", "The Azure container name to use for the cache")
addCmd.MarkFlagsRequiredTogether("storageacc", "container")
matthisholleville marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -62,4 +74,6 @@ func init() {
addCmd.MarkFlagsMutuallyExclusive("region", "container")
addCmd.MarkFlagsMutuallyExclusive("bucket", "storageacc")
addCmd.MarkFlagsMutuallyExclusive("bucket", "container")
addCmd.MarkFlagsMutuallyExclusive("projectid", "storageacc")
addCmd.MarkFlagsMutuallyExclusive("projectid", "container")
}
22 changes: 17 additions & 5 deletions cmd/cache/list.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ package cache

import (
"os"
"reflect"

"github.com/fatih/color"
"github.com/k8sgpt-ai/k8sgpt/pkg/cache"
"github.com/olekukonko/tablewriter"
"github.com/spf13/cobra"
)

Expand All @@ -30,22 +32,32 @@ var listCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {

// load remote cache if it is configured
remoteCacheEnabled, err := cache.RemoteCacheEnabled()
c, err := cache.GetCacheConfiguration()
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
}
c := cache.New(false, remoteCacheEnabled)
// list the contents of the cache
names, err := c.List()
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
}

for _, name := range names {
println(name)
var headers []string
obj := cache.CacheObjectDetails{}
objType := reflect.TypeOf(obj)
for i := 0; i < objType.NumField(); i++ {
field := objType.Field(i)
headers = append(headers, field.Name)
}

table := tablewriter.NewWriter(os.Stdout)
table.SetHeader(headers)

for _, v := range names {
table.Append([]string{v.Name, v.UpdatedAt.String()})
}
table.Render()
},
}

Expand Down
54 changes: 54 additions & 0 deletions cmd/cache/purge.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/*
Copyright 2023 The K8sGPT Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cache

import (
"fmt"
"os"

"github.com/fatih/color"
"github.com/k8sgpt-ai/k8sgpt/pkg/cache"
"github.com/spf13/cobra"
)

var purgeCmd = &cobra.Command{
Use: "purge [object name]",
Short: "Purge a remote cache",
Long: "This command allows you to delete/purge one object from the cache",
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
color.Red("Error: Please provide a value for object name. Run k8sgpt cache purge --help")
os.Exit(1)
}
objectKey := args[0]
fmt.Println(color.YellowString("Purging a remote cache."))
c, err := cache.GetCacheConfiguration()
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
}

err = c.Remove(objectKey)
if err != nil {
color.Red("Error: %v", err)
os.Exit(1)
}
fmt.Println(color.GreenString("Object deleted."))
},
}

func init() {
CacheCmd.AddCommand(purgeCmd)
}
18 changes: 16 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,22 @@ require (
require github.com/adrg/xdg v0.4.0

require (
buf.build/gen/go/k8sgpt-ai/k8sgpt/grpc/go v1.3.0-20231002095256-194bc640518b.1
buf.build/gen/go/k8sgpt-ai/k8sgpt/protocolbuffers/go v1.31.0-20231002095256-194bc640518b.1
buf.build/gen/go/k8sgpt-ai/k8sgpt/grpc/go v1.3.0-20231116211251-9f5041346631.2
buf.build/gen/go/k8sgpt-ai/k8sgpt/protocolbuffers/go v1.28.1-20231116211251-9f5041346631.4
cloud.google.com/go/storage v1.30.1
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.2.0
github.com/aws/aws-sdk-go v1.48.0
github.com/cohere-ai/cohere-go v0.2.0
github.com/olekukonko/tablewriter v0.0.5
google.golang.org/api v0.143.0
)

require (
cloud.google.com/go v0.110.7 // indirect
cloud.google.com/go/compute v1.23.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.8.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.1.1 // indirect
Expand All @@ -41,13 +48,20 @@ require (
github.com/cohere-ai/tokenizer v1.1.1 // indirect
github.com/dlclark/regexp2 v1.4.0 // indirect
github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/gnostic-models v0.6.9-0.20230804172637-c7be7c783f49 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.1 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/sagikazarmark/locafero v0.3.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
go.opencensus.io v0.24.0 // indirect
google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230920204549-e6e6cdab5c13 // indirect
)

Expand Down
Loading