Support multiple log security configs in Ingresses #1513

rafwegv · 2021-04-06T09:51:00Z

Proposed changes

Will be introduced in NAP 3.2
Support multiple log destinations and formats. Since we split log config into two annotations, allowing admins to setup multiple log destination is implemented as a coma separated value to the existing log conf annotations, where a config corresponds to the same index of the destination annotation.
for example:
appprotect.f5.com/app-protect-security-log: "default/logconf,default/logconf_splunk"
appprotect.f5.com/app-protect-security-log-destination: "syslog:server=127.0.0.1:514,syslog:server=192.168.1.1:514"

Both lists need to be the same length, if one of the referenced configs is ivalid/nonexistent no logconfs are aplied.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING doc
I have added tests that prove my fix is effective or that my feature works
I have checked that all unit tests pass after adding my changes
I have updated necessary documentation
I have rebased my branch onto master
I will ensure my PR is targeting the master branch and pulling from my branch from my own fork

Dean-Coakley · 2021-04-06T14:23:43Z

internal/k8s/controller.go

@@ -2050,7 +2050,7 @@ func (lbc *LoadBalancerController) createIngressEx(ing *networking.Ingress, vali
 			if apLogConfAntn, exists := ingEx.Ingress.Annotations[configs.AppProtectLogConfAnnotation]; exists {
 				logConf, logDst, err := lbc.getAppProtectLogConfAndDst(ing)
 				if err != nil {
-					glog.Warningf("Error Getting App Protect policy %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)
+					glog.Warningf("Error Getting App Protect Log Comnfig %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)


Suggested change

glog.Warningf("Error Getting App Protect Log Comnfig %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)

glog.Warningf("Error Getting App Protect Log Config %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)

Dean-Coakley · 2021-04-06T14:27:52Z

internal/k8s/appprotect/app_protect_resources.go

@@ -131,6 +131,15 @@ func ParseResourceReferenceAnnotation(ns, antn string) string {
 	return antn
 }

+// ParseResourceReferenceAnnotationList returns a slice of ns/names strings
+func ParseResourceReferenceAnnotationList(ns, antns string) []string {


Optional: Maybe this should be just called annotations? For a moment I understand antns to mean annotationNamespace

Suggested change

func ParseResourceReferenceAnnotationList(ns, antns string) []string {

func ParseResourceReferenceAnnotationList(ns, annotations string) []string {

Suggested change

func ParseResourceReferenceAnnotationList(ns, antns string) []string {

func ParseResourceReferenceAnnotationList(ns, antnRefs string) []string {

Feel free to ignore if you think it's clear.

soneillf5 · 2021-06-02T15:37:20Z

internal/k8s/controller.go

+	logDsts = strings.Split(ing.Annotations[configs.AppProtectLogConfDstAnnotation], ",")
+	logConfNsNs := appprotect.ParseResourceReferenceAnnotationList(ing.Namespace, ing.Annotations[configs.AppProtectLogConfAnnotation])
+	if len(logDsts) != len(logConfNsNs) {
+		return nil, []string{""}, fmt.Errorf("Error Validating App Protect Destination and config for Ingress %v: LogConf and LogDestination must be equal length", ing.Name)


why do we have an empty string in this return value ?

+1 to @soneillf5 why not return nil?

soneillf5 · 2021-06-02T15:43:12Z

internal/configs/configurator.go

+		for i, logConf := range ingEx.AppProtectLogConf {
+			logConfFileName := appProtectLogConfFileNameFromUnstruct(logConf)
+			logConfContent := generateApResourceFileContent(logConf)
+			cnf.nginxManager.CreateAppProtectResourceFile(logConfFileName, logConfContent)


I just looked into this function, I'm surprised it doesn't return an error. @pleshakov is there any reason the manager doesn't return errors from most of it's functions?

@soneillf5 those errors are file IO related and treated as unrecoverable to prevent NGINX from diverging from the desired configuration.

soneillf5 · 2021-06-02T15:48:46Z

internal/k8s/reference_checkers.go

-		if pol == namespace+"/"+name || (namespace == ing.Namespace && pol == name) {
-			return true
+	if resName, exists := ing.Annotations[rc.annotation]; exists {
+		resNames := strings.Split(resName, ",")


do we need to split and iterate? Would a string.Contains(resName, namespace+"/"+name) work?

I think the second case when the namespace is assumed to be the namespace of the Ingress resource makes things difficult cause it could be "name1,name2" or "ns1/name1,ns2/name2" or "name1,ns2/name2"

Could you add additional test cases for the new code in TestAppProtectResourceIsReferencedByIngresses ?

pleshakov

@rafwegv please see my comments and suggestions.

Additionally, could we update the docs about changed annotations https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#app-protect ?

Also, do we need to update WAF policy as well https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/#waf ? Perhaps in a different PR? I suggest sharing the changes to the fields before starting implementing so that we all agree on the new configuration.

internal/configs/version1/nginx-plus.ingress.tmpl

internal/k8s/appprotect/app_protect_resources.go

pleshakov · 2021-06-03T22:27:58Z

internal/k8s/controller.go

+	logDsts = strings.Split(ing.Annotations[configs.AppProtectLogConfDstAnnotation], ",")
+	logConfNsNs := appprotect.ParseResourceReferenceAnnotationList(ing.Namespace, ing.Annotations[configs.AppProtectLogConfAnnotation])
+	if len(logDsts) != len(logConfNsNs) {
+		return nil, []string{""}, fmt.Errorf("Error Validating App Protect Destination and config for Ingress %v: LogConf and LogDestination must be equal length", ing.Name)


+1 to @soneillf5 why not return nil?

pleshakov · 2021-06-03T22:37:43Z

internal/k8s/reference_checkers.go

-		if pol == namespace+"/"+name || (namespace == ing.Namespace && pol == name) {
-			return true
+	if resName, exists := ing.Annotations[rc.annotation]; exists {
+		resNames := strings.Split(resName, ",")


I think the second case when the namespace is assumed to be the namespace of the Ingress resource makes things difficult cause it could be "name1,name2" or "ns1/name1,ns2/name2" or "name1,ns2/name2"

Could you add additional test cases for the new code in TestAppProtectResourceIsReferencedByIngresses ?

internal/configs/ingress.go

internal/configs/configurator.go

pleshakov · 2021-06-03T23:02:07Z

tests/suite/test_app_protect_integration.py

+            # both lists need to be the same length, if one of the referenced configs is invalid/non-existent then no logconfs are applied.
+            doc["metadata"]["annotations"][
+                "appprotect.f5.com/app-protect-security-log"
+            ] = f"{test_namespace}/logconf,{test_namespace}/logconf"


@vepatel Perhaps we should also test that implicit namespace works?

Suggested change

] = f"{test_namespace}/logconf,{test_namespace}/logconf"

] = f"{test_namespace}/logconf,logconf"

but I've deployed test resources in a separate namespace here so need to reference by the same as nothing related to test goes to default ns.

pleshakov

Hi @rafwegv

the changes look good! just a few small suggestions.

Additionally, could we update the docs about changed annotations https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#app-protect ?

are you planning to update docs in a separate PR?

pleshakov · 2021-06-04T23:20:21Z

internal/configs/ingress.go

+	AppProtectLogconfs []string
+}
+
+type AppProtectLog struct {


neat: could you add a doc string for consistency? since the type is exported

pleshakov · 2021-06-04T23:27:23Z

internal/k8s/controller.go

@@ -2160,27 +2159,38 @@ func (lbc *LoadBalancerController) createIngressEx(ing *networking.Ingress, vali
 	return ingEx
 }

-func (lbc *LoadBalancerController) getAppProtectLogConfAndDst(ing *networking.Ingress) (logConf *unstructured.Unstructured, logDst string, err error) {
-	logConfNsN := appprotect.ParseResourceReferenceAnnotation(ing.Namespace, ing.Annotations[configs.AppProtectLogConfAnnotation])
+func (lbc *LoadBalancerController) getAppProtectLogConfAndDst(ing *networking.Ingress) (apLog []configs.AppProtectLog, err error) {


because we use err variable throughout the code in the method body and without any intention to return its value, I suggest dropping the names in (apLog []configs.AppProtectLog, err error) -> ([]configs.AppProtectLog, error) and inside the body, simply declare var apLog []configs.AppProtectLog . And a small suggestion, because we have a slice, better to use plural apLogs.

pleshakov

🚀

rafwegv added the enhancement Pull requests for new features/feature enhancements label Apr 6, 2021

rafwegv requested review from pleshakov and Dean-Coakley April 6, 2021 09:51

rafwegv self-assigned this Apr 6, 2021

Dean-Coakley approved these changes Apr 6, 2021

View reviewed changes

rafwegv force-pushed the ap-multi-log-dst branch from da75b10 to 79cab81 Compare June 2, 2021 11:03

rafwegv changed the title ~~WIP: AP: add support for multiple log configs~~ AP: add support for multiple log configs Jun 2, 2021

rafwegv force-pushed the ap-multi-log-dst branch 2 times, most recently from 36609d1 to 0bf0a0a Compare June 2, 2021 14:17

soneillf5 reviewed Jun 2, 2021

View reviewed changes

soneillf5 approved these changes Jun 3, 2021

View reviewed changes

pleshakov reviewed Jun 3, 2021

View reviewed changes

rafwegv force-pushed the ap-multi-log-dst branch from d0f21a2 to 5c8069e Compare June 4, 2021 11:38

pleshakov reviewed Jun 4, 2021

View reviewed changes

rafwegv force-pushed the ap-multi-log-dst branch from 8765318 to 531fb18 Compare June 7, 2021 08:19

github-actions bot added the documentation Pull requests/issues for documentation label Jun 7, 2021

AP: add support for multiple log configs

d20db07

rafwegv force-pushed the ap-multi-log-dst branch from 531fb18 to d20db07 Compare June 7, 2021 08:24

pleshakov approved these changes Jun 7, 2021

View reviewed changes

vepatel added 8 commits June 8, 2021 13:03

Add wait and asset for ts tests

5c4ea59

Add get_service_endpoints

417642e

add conditional wait

f89fe01

remove unused imports

1b3dfe9

Fix delays and indentations

cc0b353

Increase wait for sec logs

974efae

Add retry on AP security logs

0f7bbb9

change order due to prolonged uds ingress del.

e27a433

vepatel merged commit 6c02b96 into master Jun 9, 2021

vepatel deleted the ap-multi-log-dst branch June 9, 2021 14:52

pleshakov changed the title ~~AP: add support for multiple log configs~~ Support multiple log security configs in Ingresses Jun 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multiple log security configs in Ingresses #1513

Support multiple log security configs in Ingresses #1513

rafwegv commented Apr 6, 2021

Dean-Coakley Apr 6, 2021

Dean-Coakley Apr 6, 2021 •

edited

Loading

soneillf5 Jun 2, 2021

pleshakov Jun 3, 2021

soneillf5 Jun 2, 2021

pleshakov Jun 3, 2021

soneillf5 Jun 2, 2021

pleshakov Jun 3, 2021

pleshakov left a comment

pleshakov Jun 3, 2021

pleshakov Jun 3, 2021

pleshakov Jun 3, 2021

vepatel Jun 4, 2021 •

edited

Loading

pleshakov left a comment •

edited

Loading

pleshakov Jun 4, 2021

pleshakov Jun 4, 2021

pleshakov left a comment

	glog.Warningf("Error Getting App Protect Log Comnfig %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)
	glog.Warningf("Error Getting App Protect Log Config %v for Ingress %v/%v: %v", apLogConfAntn, ing.Namespace, ing.Name, err)

	func ParseResourceReferenceAnnotationList(ns, antns string) []string {
	func ParseResourceReferenceAnnotationList(ns, annotations string) []string {

	] = f"{test_namespace}/logconf,{test_namespace}/logconf"
	] = f"{test_namespace}/logconf,logconf"

Support multiple log security configs in Ingresses #1513

Support multiple log security configs in Ingresses #1513

Conversation

rafwegv commented Apr 6, 2021

Proposed changes

Checklist

Choose a reason for hiding this comment

Dean-Coakley Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pleshakov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vepatel Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

pleshakov left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pleshakov left a comment

Choose a reason for hiding this comment

Dean-Coakley Apr 6, 2021 •

edited

Loading

vepatel Jun 4, 2021 •

edited

Loading

pleshakov left a comment •

edited

Loading