feat:add higress automatic https #854

2456868764 · 2024-03-01T12:33:33Z

Ⅰ. Describe what this PR did

在Higress控制面集成支持域名静态配置和支持secret对接lets encrypt自动续签，在higress-system namesapce 新增一个 configmap ,其配置格式如下

apiVersion: v1
kind: ConfigMap
metadata:
  name: higress-https
  namespace: higress-system
data:
  cert: |
    automaticHttps: true
    renewBeforeDays: 30
    credentialConfig:
    - domains:
      - example.com
      tlsIssuer: letsencrypt
      tlsSecret: default-example-com-certificate
    - domains: 
      - sub.example.net
      tlsIssuer: aliyunssl
      tlsSecret: sub-example-net-tls-certificate
    - domains:  
       - statica.example.org
       - staticb.example.org
      tlsSecret: static-example-org-certificate
      cacertSecret: my-ca-cert
    - domains:  
       - "*"
      tlsSecret: default-certificate
    acmeIssuer:
    - email: your77658@yours69316.com
      name: letsencrypt
    - email: your77658@yours69316.com
      name:  aliyunssl
      sk: ""
      ak: ""
    version: "20240413090849"

在这个ConfigMap中，data字段中 cert 配置如下：

automaticHttps: 设置为true，表示启用自动letsencrypt等自动签发功能。
renewBeforeDays: 设置为30，表示在证书到期前30天自动续订证书。
credentialConfig: 是一个列表，包含多个TLS证书的配置项。
- domains: 指定了证书应该为哪些域名提供服务。
- tlsIssuer: 指定了证书颁发机构（Certificate Authority，CA），如letsencrypt或aliyunssl。如何没有指定issuer,表示是静态配置
- tlsSecret: 指定了存储TLS证书和密钥的Kubernetes Secret对象的名称。
- cacertSecret: 指定了自定义CA证书的Kubernetes Secret对象的名称，用于当您使用自定义CA时。
acmeIssuer
- name: 这是ACME颁发机构的名称。有两个颁发机构：letsencrypt和aliyunssl, 目前只支持letsencrypt
- email: 这是与ACME服务器注册账户时使用的电子邮件地址
- ak: ACME访问密钥
- sk: ACME账户私钥

Ⅱ. Does this pull request fix one issue?

Higress控制面集成能力
控制Ingress最终生效
支持域名上静态secret配置
支持secret对接lets encrypt自动续签

Ⅲ. Why don't you add test cases (unit test/integration test)?

fixes #841
fixes #64

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

codecov-commenter · 2024-03-03T07:45:22Z

Codecov Report

Attention: Patch coverage is 18.06854% with 789 lines in your changes are missing coverage. Please review.

Project coverage is 36.24%. Comparing base (6c7b175) to head (6c4a4d9).
Report is 15 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #854      +/-   ##
==========================================
- Coverage   38.01%   36.24%   -1.78%     
==========================================
  Files          61       69       +8     
  Lines       10474    11464     +990     
==========================================
+ Hits         3982     4155     +173     
- Misses       6192     6993     +801     
- Partials      300      316      +16

Files	Coverage Δ
pkg/cmd/server.go	`85.88% <100.00%> (+0.88%)`	⬆️
pkg/ingress/kube/ingress/controller.go	`62.28% <63.63%> (-0.07%)`	⬇️
pkg/bootstrap/server.go	`57.45% <60.00%> (-6.09%)`	⬇️
pkg/ingress/kube/ingressv1/controller.go	`5.36% <0.00%> (-0.06%)`	⬇️
pkg/cert/util.go	`8.16% <8.16%> (ø)`
pkg/cert/secret.go	`0.00% <0.00%> (ø)`
pkg/cert/server.go	`0.00% <0.00%> (ø)`
pkg/cert/ingress.go	`0.00% <0.00%> (ø)`
pkg/cert/controller.go	`0.00% <0.00%> (ø)`
pkg/cert/storage.go	`55.65% <55.65%> (ø)`
... and 2 more

cmd/cert/main.go

pkg/cert/config.go

2456868764 · 2024-04-12T12:48:06Z

已经重新提交，麻烦再看一下

zzjin · 2024-04-15T03:44:22Z

这个configmap,有可能考虑改成CRD的模式吗,配置在一个configmap里面,不同的域名的RBAC会比较难控制

johnlanni · 2024-04-16T02:13:00Z

@zzjin 这个configmap我们不希望放到用户命名空间下，而是放在系统命名空间下（如:higress-system)，统一交由集群SRE来管理

johnlanni · 2024-04-16T02:17:46Z

@2456868764 这个配置里的cert字段是否应该是higress的子字段比较合理

2456868764 · 2024-04-16T04:10:23Z

higress-https

@2456868764 这个配置里的cert字段是否应该是higress的子字段比较合理

现在 cert 是保存新 higress-https configmap 里，是要保存到 higress-config configmap 里 higress 属性下？

2456868764 · 2024-04-16T04:12:15Z

higress-https

@2456868764 这个配置里的cert字段是否应该是higress的子字段比较合理

现在 cert 是保存新 higress-https configmap 里，是要保存到 higress-config configmap 里 higress 属性下？

johnlanni · 2024-04-16T05:39:28Z

higress-https

@2456868764 这个配置里的cert字段是否应该是higress的子字段比较合理

现在 cert 是保存新 higress-https configmap 里，是要保存到 higress-config configmap 里 higress 属性下？

哦，独立出来也有好处，可以用hgctl简化运维

johnlanni

似乎没有看到在将ingress转换为istio gateway资源时，根据此份configmap配置credential的逻辑？

helm/core/values.yaml

johnlanni · 2024-04-16T05:52:29Z

似乎没有看到在将ingress转换为istio gateway资源时，根据此份configmap配置credential的逻辑？

higress/pkg/ingress/kube/ingressv1/controller.go

Lines 411 to 442 in 9c112a0

    
           secretName := extractTLSSecretName(rule.Host, ingressV1.TLS) 
        
           if secretName == "" { 
        
           	// There no matching secret, so just skip. 
        
           	continue 
        
           } 
        
           domainBuilder.Protocol = common.HTTPS 
        
           domainBuilder.SecretName = path.Join(c.options.ClusterId, cfg.Namespace, secretName) 
        
           // There is a matching secret and the gateway has already a tls secret. 
        
           // We should report the duplicated tls secret event. 
        
           if wrapperGateway.IsHTTPS() { 
        
           	domainBuilder.Event = common.DuplicatedTls 
        
           	domainBuilder.PreIngress = preDomainBuilder.Ingress 
        
           	convertOptions.IngressDomainCache.Invalid = append(convertOptions.IngressDomainCache.Invalid, 
        
           		domainBuilder.Build()) 
        
           	continue 
        
           } 
        
           // Append https server 
        
           wrapperGateway.Gateway.Servers = append(wrapperGateway.Gateway.Servers, &networking.Server{ 
        
           	Port: &networking.Port{ 
        
           		Number:   uint32(c.options.GatewayHttpsPort), 
        
           		Protocol: string(protocol.HTTPS), 
        
           		Name:     common.CreateConvertedName("https-"+strconv.FormatUint(uint64(c.options.GatewayHttpsPort), 10)+"-ingress", c.options.ClusterId), 
        
           	}, 
        
           	Hosts: []string{rule.Host}, 
        
           	Tls: &networking.ServerTLSSettings{ 
        
           		Mode:           networking.ServerTLSSettings_SIMPLE, 
        
           		CredentialName: credentials.ToKubernetesIngressResource(c.options.RawClusterId, cfg.Namespace, secretName), 
        
           	}, 
        
           })

@2456868764 即这段逻辑需要调整适配

2456868764 · 2024-04-16T08:13:06Z

似乎没有看到在将ingress转换为istio gateway资源时，根据此份configmap配置credential的逻辑？

higress/pkg/ingress/kube/ingressv1/controller.go

Lines 411 to 442 in 9c112a0

secretName := extractTLSSecretName(rule.Host, ingressV1.TLS)

if secretName == "" {

// There no matching secret, so just skip.

continue

}

domainBuilder.Protocol = common.HTTPS

domainBuilder.SecretName = path.Join(c.options.ClusterId, cfg.Namespace, secretName)

// There is a matching secret and the gateway has already a tls secret.

// We should report the duplicated tls secret event.

if wrapperGateway.IsHTTPS() {

domainBuilder.Event = common.DuplicatedTls

domainBuilder.PreIngress = preDomainBuilder.Ingress

convertOptions.IngressDomainCache.Invalid = append(convertOptions.IngressDomainCache.Invalid,

domainBuilder.Build())

continue

}

// Append https server

wrapperGateway.Gateway.Servers = append(wrapperGateway.Gateway.Servers, &networking.Server{

Port: &networking.Port{

Number: uint32(c.options.GatewayHttpsPort),

Protocol: string(protocol.HTTPS),

Name: common.CreateConvertedName("https-"+strconv.FormatUint(uint64(c.options.GatewayHttpsPort), 10)+"-ingress", c.options.ClusterId),

},

Hosts: []string{rule.Host},

Tls: &networking.ServerTLSSettings{

Mode: networking.ServerTLSSettings_SIMPLE,

CredentialName: credentials.ToKubernetesIngressResource(c.options.RawClusterId, cfg.Namespace, secretName),

},

})

@2456868764 即这段逻辑需要调整适配

好的，我修改一下

…eat-autohttps

2456868764 · 2024-04-21T01:52:57Z

似乎没有看到在将ingress转换为istio gateway资源时，根据此份configmap配置credential的逻辑？

higress/pkg/ingress/kube/ingressv1/controller.go

Lines 411 to 442 in 9c112a0

secretName := extractTLSSecretName(rule.Host, ingressV1.TLS)

if secretName == "" {

// There no matching secret, so just skip.

continue

}

domainBuilder.Protocol = common.HTTPS

domainBuilder.SecretName = path.Join(c.options.ClusterId, cfg.Namespace, secretName)

// There is a matching secret and the gateway has already a tls secret.

// We should report the duplicated tls secret event.

if wrapperGateway.IsHTTPS() {

domainBuilder.Event = common.DuplicatedTls

domainBuilder.PreIngress = preDomainBuilder.Ingress

convertOptions.IngressDomainCache.Invalid = append(convertOptions.IngressDomainCache.Invalid,

domainBuilder.Build())

continue

}

// Append https server

wrapperGateway.Gateway.Servers = append(wrapperGateway.Gateway.Servers, &networking.Server{

Port: &networking.Port{

Number: uint32(c.options.GatewayHttpsPort),

Protocol: string(protocol.HTTPS),

Name: common.CreateConvertedName("https-"+strconv.FormatUint(uint64(c.options.GatewayHttpsPort), 10)+"-ingress", c.options.ClusterId),

},

Hosts: []string{rule.Host},

Tls: &networking.ServerTLSSettings{

Mode: networking.ServerTLSSettings_SIMPLE,

CredentialName: credentials.ToKubernetesIngressResource(c.options.RawClusterId, cfg.Namespace, secretName),

},

})

@2456868764 即这段逻辑需要调整适配

@johnlanni 已经适配，麻烦看一下

johnlanni

能否添加一个e2e test，可以只验证static secret的方式，校验服务端返回的证书与配置的是否一致

pkg/cert/config_test.go

johnlanni

现在区分了ingressv1和ingress两个目录对应不同的ingress版本，除了 ingressv1， ingress目录下的controller也需要调整一下

2456868764 · 2024-04-25T12:38:38Z

现在区分了ingressv1和ingress两个目录对应不同的ingress版本，除了 ingressv1， ingress目录下的controller也需要调整一下

ingress & ingressv1 都修改，增加了 e2e test , 另外配置在 higress-https 里 secret 应该在 higress-system namespace 下？

johnlanni

LGTM

optimize domain match

add higress automatic https

4b50355

2456868764 requested review from SpecialYang, johnlanni and CH3CHO as code owners March 1, 2024 12:33

add go mod and configmap storage sharding and unit test

9a3da17

johnlanni reviewed Mar 5, 2024

View reviewed changes

cmd/cert/main.go Outdated Show resolved Hide resolved

johnlanni requested changes Mar 5, 2024

View reviewed changes

pkg/cert/config.go Show resolved Hide resolved

Merge branch 'main' into feat-autohttps

3dc7202

2456868764 force-pushed the feat-autohttps branch 2 times, most recently from 4c8e4b5 to a14dc76 Compare April 12, 2024 12:34

2456868764 force-pushed the feat-autohttps branch from a14dc76 to fba6b6e Compare April 12, 2024 12:56

adjust automatic https service to higress controller

25a9286

2456868764 force-pushed the feat-autohttps branch from fba6b6e to 25a9286 Compare April 12, 2024 12:59

Merge branch 'main' into feat-autohttps

4cae172

johnlanni requested changes Apr 16, 2024

View reviewed changes

helm/core/values.yaml Outdated Show resolved Hide resolved

2456868764 added 3 commits April 21, 2024 09:42

get secretname from configmap in ingress controller

53c3cda

Merge branch 'feat-autohttps' of github.com:2456868764/higress into f…

2071c15

…eat-autohttps

fix import

3ad6431

johnlanni requested changes Apr 22, 2024

View reviewed changes

pkg/cert/config_test.go Show resolved Hide resolved

add license

855a050

johnlanni requested changes Apr 23, 2024

View reviewed changes

2456868764 added 2 commits April 23, 2024 11:31

add getting secertName from configmap if there is no matching found

a69955f

fix secret namespace bug and add e2e test for https configmap

6c4a4d9

2456868764 requested a review from Xunzhuo as a code owner April 25, 2024 12:14

restore consul & eureka e2e test

a8ce261

2456868764 added 2 commits April 25, 2024 20:42

fix ca secret name

58b4835

Merge branch 'main' into feat-autohttps

acaa9fa

johnlanni approved these changes Apr 28, 2024

View reviewed changes

johnlanni and others added 3 commits April 28, 2024 10:10

optimize domain match

1d3fa41

Merge pull request #3 from johnlanni/optimize-domain-match

79e0fff

optimize domain match

Merge branch 'main' into feat-autohttps

f0e43d1

johnlanni merged commit 03d2f01 into alibaba:main May 9, 2024
11 checks passed

zzjin mentioned this pull request May 20, 2024

希望higress-https中的全局https证书配置可以用于ingress中secret不存在时的兜底 #978

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat:add higress automatic https #854

feat:add higress automatic https #854

2456868764 commented Mar 1, 2024 •

edited

Loading

codecov-commenter commented Mar 3, 2024 •

edited

Loading

2456868764 commented Apr 12, 2024

zzjin commented Apr 15, 2024

johnlanni commented Apr 16, 2024

johnlanni commented Apr 16, 2024

2456868764 commented Apr 16, 2024

2456868764 commented Apr 16, 2024

johnlanni commented Apr 16, 2024

johnlanni left a comment

johnlanni commented Apr 16, 2024 •

edited

Loading

2456868764 commented Apr 16, 2024

2456868764 commented Apr 21, 2024

johnlanni left a comment

johnlanni left a comment

2456868764 commented Apr 25, 2024

johnlanni left a comment

feat:add higress automatic https #854

feat:add higress automatic https #854

Conversation

2456868764 commented Mar 1, 2024 • edited Loading

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

codecov-commenter commented Mar 3, 2024 • edited Loading

Codecov Report

2456868764 commented Apr 12, 2024

zzjin commented Apr 15, 2024

johnlanni commented Apr 16, 2024

johnlanni commented Apr 16, 2024

2456868764 commented Apr 16, 2024

2456868764 commented Apr 16, 2024

johnlanni commented Apr 16, 2024

johnlanni left a comment

Choose a reason for hiding this comment

johnlanni commented Apr 16, 2024 • edited Loading

2456868764 commented Apr 16, 2024

2456868764 commented Apr 21, 2024

johnlanni left a comment

Choose a reason for hiding this comment

johnlanni left a comment

Choose a reason for hiding this comment

2456868764 commented Apr 25, 2024

johnlanni left a comment

Choose a reason for hiding this comment

2456868764 commented Mar 1, 2024 •

edited

Loading

codecov-commenter commented Mar 3, 2024 •

edited

Loading

johnlanni commented Apr 16, 2024 •

edited

Loading