Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Seamless K8s Cluster Provisioning Support #1582

Merged
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions assets/k8sclusterinfo.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Configuration for Cluster(Kubernetes) of Cloud Service Providers (CSPs)
# This file is used to define the feature of clusters

# The file is in YAML format and contains the following fields:
# k8scluster: Top level key
# <csp>: Name of the CSP
# nodegroupsWithCluster:
# version:
# - region: [region1, region2, all(special keyword)]

k8scluster:
nhncloud:
nodegroupsWithCluster: true
version:
- region: [kr1, kr2]
available:
- name: 1.28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: 버전 별로 가용 이미지가 변하지는 않나요? (예를 들어, 1.24.6-aliyun.1은 aliyun_3_9_x64_20G_alibase_20231219.vhd 를 사용할 수 없다든지)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AWS, GCP, Azure, Alibaba, Tencent, NHNCloud, NCPVPC의 경우 버전별로 가용 이미지가 다르진 않는 것으로 파악되었고, IBM은 불명확하나 제약이 있는 것으로 보이진 않습니다.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sykim-etri 확인 감사합니다. 그럼 Spec, Image 와는 별도로 관리하면 좋을 것 같습니다.
사용자가 API로 가용 version 리스트 확인이 가능하게 하고, 클러스터 생성/노드 추가시에 해당 값을 사용할 수 있도록 해두면 좋을 것 같습니다. :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

사용자가 API로 가용 version 리스트 확인이 가능하게 하고,

k8sclusterinfo.yaml에 저장된 내용을 확인할 수 있는 API는 가능한데, 어떤 시나리오에서 활용이 되는 걸까요?

CSP에서 가용 Version 목록 조회가 가능한 API가 있는 경우도 있는데, 현재 CB-SP에서는 지원하지는 않고 있습니다만 추후 지원시 k8sclusterinfo.yaml 대신 CB-SP를 통해 해당 정보를 전달할 수 있을 것 같습니다.

클러스터 생성/노드 추가시에 해당 값을 사용할 수 있도록 해두면 좋을 것 같습니다. :)

참고로 버전 정보는 클러스터 생성시에만 적용 가능한 상황입니다.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecommendK8sClusterSpec 방식이 아닌 노드 스펙 필터링 후 이용 가능한 버전 등을 제공하는 방식으로 진행하기로 함.

id: 1.28.9-aliyun.1
- name: 1.26
id: 1.26.15-aliyun.1
- name: 1.24
id: 1.24.6-aliyun.1
- region: [jp1]
nodeImage:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Q: Image와 다르게 Spec 명세는 구성파일에 없는 것 같은데, Spec의 경우 기존 VM 의 Spec 관리 체계를 따르도록 처리 예정이신가요?
  • Q: Image도 향후에 이슈가 없으면, 기존 VM의 Image 관리 체계를 사용할 수 있지않을까요?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Q: Image와 다르게 Spec 명세는 구성파일에 없는 것 같은데, Spec의 경우 기존 VM 의 Spec 관리 체계를 따르도록 처리 예정이신가요?

일부 CSP(ex. Alibaba)의 경우 콘솔에서 확인되는 가용 Node Spec이 VM Spec과는 다른 경우가 왕왕 보이긴 하지만 assets/cloudspec.csv에 등록된 VM의 Spec들과 크게 차이가 나지는 않는 것으로 보여서 기존 VM의 Spec 관리 체계를 활용해볼 예정입니다.

  • Q: Image도 향후에 이슈가 없으면, 기존 VM의 Image 관리 체계를 사용할 수 있지않을까요?

K8s용 이미지들도 기존 VM의 Image 관리 체계(assets/cloudimage.csv)에 별도 필드(container runtime, gpu, ...)를 추가하여 통합할 방법을 고려해보겠습니다.

참고로 K8s 이미지의 경우 CSP별로 자체 최적화한 이미지(자동 업그레이드 기능 연동 등) 사용을 추천하고 있습니다.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecommendK8sClusterSpec 방식이 아닌 노드 스펙 필터링 후 이용 가능한 버전 등을 제공하는 방식으로 진행하기로 함.

- region: [kr1]
available:
- name: default
id: default
- name: aliyun 3.9
id: aliyun_3_9_x64_20G_alibase_20231219.vhd
- region: [kr2]
available:
- name: default
id: default
- name: aliyun 3.9
id: aliyun_3_9_x64_20G_alibase_20231219.vhd
- region: [jp1]
rootDisk:
- region: [all]
type:
- name: default
id: default
- name: basic
id: CLOUD_BASIC
size:
min: 10
max: 40
alibaba:
nodegroupsWithCluster: true
version:
- region: [all]
available:
- name: 1.28
id: 1.28.9-aliyun.1
- name: 1.26
id: 1.26.15-aliyun.1
- name: 1.24
id: 1.24.6-aliyun.1
nodeImage:
- region: [eu-west-1]
#configurable: true
available:
- name: default
id: default
- name: aliyun 3.9
id: aliyun_3_9_x64_20G_alibase_20231219.vhd
- region: [eu-west-2,kr2]
rootDisk:
- region: [eu-west-1]
type:
#configurable: true
- name: default
id: default
- name: basic
id: CLOUD_BASIC
size:
#configurable: true
min: 10
max: 40

48 changes: 48 additions & 0 deletions src/api/rest/docs/docs.go
Original file line number Diff line number Diff line change
Expand Up @@ -8079,6 +8079,54 @@ const docTemplate = `{
}
}
},
"/recommendK8sClusterSpec": {
"post": {
"description": "Recommend K8s Cluster plan (filter and priority) Find details from https://github.com/cloud-barista/cb-tumblebug/discussions/1234",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"[Infra service] Cluster Provisioning Management"
],
"summary": "Recommend K8s Cluster plan (filter and priority)",
"parameters": [
{
"description": "Recommend K8s Cluster plan (filter and priority)",
"name": "deploymentPlan",
"in": "body",
"schema": {
"$ref": "#/definitions/mcis.DeploymentPlan"
}
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/mcir.TbSpecInfo"
}
}
},
"404": {
"description": "Not Found",
"schema": {
"$ref": "#/definitions/common.SimpleMsg"
}
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/common.SimpleMsg"
}
}
}
}
},
"/region": {
"get": {
"description": "List all registered regions",
Expand Down
48 changes: 48 additions & 0 deletions src/api/rest/docs/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -8072,6 +8072,54 @@
}
}
},
"/recommendK8sClusterSpec": {
"post": {
"description": "Recommend K8s Cluster plan (filter and priority) Find details from https://github.com/cloud-barista/cb-tumblebug/discussions/1234",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"[Infra service] Cluster Provisioning Management"
],
"summary": "Recommend K8s Cluster plan (filter and priority)",
"parameters": [
{
"description": "Recommend K8s Cluster plan (filter and priority)",
"name": "deploymentPlan",
"in": "body",
"schema": {
"$ref": "#/definitions/mcis.DeploymentPlan"
}
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/mcir.TbSpecInfo"
}
}
},
"404": {
"description": "Not Found",
"schema": {
"$ref": "#/definitions/common.SimpleMsg"
}
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/common.SimpleMsg"
}
}
}
}
},
"/region": {
"get": {
"description": "List all registered regions",
Expand Down
32 changes: 32 additions & 0 deletions src/api/rest/docs/swagger.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8408,6 +8408,38 @@ paths:
summary: Check Tumblebug is ready
tags:
- '[Admin] System management'
/recommendK8sClusterSpec:
post:
consumes:
- application/json
description: Recommend K8s Cluster plan (filter and priority) Find details from
https://github.com/cloud-barista/cb-tumblebug/discussions/1234
parameters:
- description: Recommend K8s Cluster plan (filter and priority)
in: body
name: deploymentPlan
schema:
$ref: '#/definitions/mcis.DeploymentPlan'
produces:
- application/json
responses:
"200":
description: OK
schema:
items:
$ref: '#/definitions/mcir.TbSpecInfo'
type: array
"404":
description: Not Found
schema:
$ref: '#/definitions/common.SimpleMsg'
"500":
description: Internal Server Error
schema:
$ref: '#/definitions/common.SimpleMsg'
summary: Recommend K8s Cluster plan (filter and priority)
tags:
- '[Infra service] Cluster Provisioning Management'
/region:
get:
consumes:
Expand Down
27 changes: 27 additions & 0 deletions src/api/rest/server/mcis/recommendation.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,33 @@ func RestRecommendVm(c echo.Context) error {
return common.EndRequestWithLog(c, reqID, err, content)
}

// RestRecommendK8sClusterSpec godoc
// @Summary Recommend K8s Cluster plan (filter and priority)
// @Description Recommend K8s Cluster plan (filter and priority) Find details from https://github.com/cloud-barista/cb-tumblebug/discussions/1234
// @Tags [Infra service] Cluster Provisioning Management
// @Accept json
// @Produce json
// @Param deploymentPlan body mcis.DeploymentPlan false "Recommend K8s Cluster plan (filter and priority)"
// @Success 200 {object} []mcir.TbSpecInfo
// @Failure 404 {object} common.SimpleMsg
// @Failure 500 {object} common.SimpleMsg
// @Router /recommendK8sClusterSpec [post]
func RestRecommendK8sClusterSpec(c echo.Context) error {
reqID, idErr := common.StartRequestWithLog(c)
if idErr != nil {
return c.JSON(http.StatusBadRequest, map[string]string{"message": idErr.Error()})
}
nsId := common.SystemCommonNs

u := &mcis.DeploymentPlan{}
if err := c.Bind(u); err != nil {
return common.EndRequestWithLog(c, reqID, err, nil)
}

content, err := mcis.RecommendK8sClusterSpec(nsId, *u)
return common.EndRequestWithLog(c, reqID, err, content)
}

type RestPostMcisRecommendResponse struct {
//VmReq []TbVmRecommendReq `json:"vmReq"`
VmRecommend []mcis.TbVmRecommendInfo `json:"vmRecommend"`
Expand Down
3 changes: 3 additions & 0 deletions src/api/rest/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,9 @@ func RunServer(port string) {
g.POST("/:nsId/mcisDynamic", rest_mcis.RestPostMcisDynamic)
g.POST("/:nsId/mcis/:mcisId/vmDynamic", rest_mcis.RestPostMcisVmDynamic)

// @Tags [Infra] Cluster Provisioning Management
e.POST("/tumblebug/recommendK8sClusterSpec", rest_mcis.RestRecommendK8sClusterSpec)

//g.GET("/:nsId/mcis/:mcisId", rest_mcis.RestGetMcis, middleware.TimeoutWithConfig(middleware.TimeoutConfig{Timeout: 20 * time.Second}), middleware.RateLimiter(middleware.NewRateLimiterMemoryStore(1)))
//g.GET("/:nsId/mcis", rest_mcis.RestGetAllMcis, middleware.TimeoutWithConfig(middleware.TimeoutConfig{Timeout: 20 * time.Second}), middleware.RateLimiter(middleware.NewRateLimiterMemoryStore(1)))
// path specific timeout and ratelimit
Expand Down
59 changes: 59 additions & 0 deletions src/core/common/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,65 @@ type Credential struct {

var RuntimeCredential = Credential{}

// K8sClusterInfo is structure for kubernetes cluster information
type K8sClusterInfo struct {
CSPs map[string]K8sClusterDetail `mapstructure:"k8scluster" json:"k8s_cluster"`
}

// K8sClusterDetail is structure for kubernetes cluster detail information
type K8sClusterDetail struct {
NodeGroupsWithCluster bool `mapstructure:"nodegroupsWithCluster" json:"nodegroups_with_cluster"`
Version []VersionDetail `mapstructure:"version" json:"versions"`
NodeImage []NodeImageDetail `mapstructure:"nodeImage" json:"node_images"`
RootDisk []RootDiskDetail `mapstructure:"rootDisk" json:"root_disks"`
}

// VersionDetail is structure for kubernetes cluster version detail information
type VersionDetail struct {
Region []string `mapstructure:"region" json:"region"`
Available []VersionDetailAvailable `mapstructure:"available" json:"availables"`
}

// VersionDetailAvailable is structure for kubernetes cluster version detail's available information
type VersionDetailAvailable struct {
Name string `mapstructure:"name" json:"name"`
Id string `mapstructure:"id" json:"id"`
}

// NodeImageDetail is structure for kubernetes cluster node image detail information
type NodeImageDetail struct {
Region []string `mapstructure:"region" json:"region"`
Available []NodeImageDetailAvailable `mapstructure:"available" json:"availables"`
}

// NodeImageDetailAvailable is structure for kubernetes cluster node image detail's available information
type NodeImageDetailAvailable struct {
Name string `mapstructure:"name" json:"name"`
Id string `mapstructure:"id" json:"id"`
}

// RootDiskDetail is structure for kubernetes cluster root disk detail information
type RootDiskDetail struct {
Region []string `mapstructure:"region" json:"region"`
Type []RootDiskDetailType `mapstructure:"type" json:"type"`
Size RootDiskDetailSize `mapstructure:"size" json:"size"`
}

// RootDiskDetailType is structure for kubernetes cluster root disk detail's type information
type RootDiskDetailType struct {
Name string `mapstructure:"name" json:"name"`
Id string `mapstructure:"id" json:"id"`
}

// RootDiskDetailSize is structure for kubernetes cluster root disk detail's size information
type RootDiskDetailSize struct {
Min uint `mapstructure:"min" json:"min"`
Max uint `mapstructure:"max" json:"max"`
}

// RuntimeK8sClusterInfo is global variable for K8sClusterInfo
var RuntimeK8sClusterInfo = K8sClusterInfo{}

// AdjustKeysToLowercase adjusts the keys of nested maps to lowercase.
func AdjustKeysToLowercase(cloudInfo *CloudInfo) {
newCSPs := make(map[string]CSPDetail)
Expand Down
5 changes: 5 additions & 0 deletions src/core/mcir/spec.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,11 @@ type FilterSpecsByRangeRequest struct {
EvaluationScore10 Range `json:"evaluationScore10"`
}

type FilterK8sClusterByRangeRequest struct {
FilterSpecsByRangeRequest
Version string `json:"version"`
}

// ConvertSpiderSpecToTumblebugSpec accepts an Spider spec object, converts to and returns an TB spec object
func ConvertSpiderSpecToTumblebugSpec(spiderSpec SpiderSpecInfo) (TbSpecInfo, error) {
if spiderSpec.Name == "" {
Expand Down
Loading