VirtualService fails to create due to Istio 1.4's regex engine check if ksvc has a little longer domain #6058

nak3 · 2019-11-18T01:59:11Z

In what area(s)?

/area networking
/area test-and-release
/kind spec

What version of Knative?

Istio 1.4.x or later.

Actual Behavior

As I mentioned #6039 (comment), Istio 1.4 introduced Regex EngineChanges / istio/istio#18539 so most of E2E tests fail due to exceed the size of regex (=only 100 bytes) in VirtualService.

e.g:

    match:
    - authority:
        regex: ^service-to-service-call-via-activator-both-disabled-rlzfmjaj\.serving-tests\.example\.com(?::\d{1,5})?$

Then, we will get following error and VirtualService is not created.
{"level":"error","ts":"2019-11-16T08:13:54.839Z","logger":"istiocontroller.ingress-controller","caller":"controller/controller.go:368","msg":"Reconcile error","commit":"f961c64","knative.dev/controller":"ingress-controller","error":"failed to create VirtualService: admission webhook \"pilot.validation.istio.io\" denied the request: configuration is invalid: regex match '^service-to-service-call-via-activator-both-disabled-rlzfmjaj\\.serving-tests(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$' cannot be greater than 100 bytes","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/knakayam/.go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:368\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/knakayam/.go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:354\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/knakayam/.go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:302"}

Steps to Reproduce the Problem

Install Istio 1.4.x
Run E2E test or deploy ksvc with long name.

The text was updated successfully, but these errors were encountered:

nak3 · 2019-11-18T02:09:32Z

It looks like we have a few options.

A. Modify ObjectNameForTest() to create shorter ksvc name for tests.
B. Use PILOT_ENABLE_UNSAFE_REGEX=true for Istio Pilot.

I am thinking that we need optionA rather than B, but is there any other idea?

cc @tcnghia @mattmoor

vagababov · 2019-11-18T06:09:09Z

Also we can use pkg/kmeta/ChildName. I wonder how does that surface in the ksvc/revision status? I.e. if users actually use this long names, how can they understand that this _is_ the cause?

…

On Sun, Nov 17, 2019 at 6:09 PM Kenjiro Nakayama ***@***.***> wrote: It looks like we have a few options. A. Modify ObjectNameForTest() <https://github.com/knative/serving/blob/e552aaa9d7142a242b83ff4729a7157730262e92/vendor/knative.dev/pkg/test/helpers/name.go#L49-L52> to create shorter ksvc name for tests. B. Use PILOT_ENABLE_UNSAFE_REGEX=true for Istio Pilot. I am thinking that we need optionA rather than B, but is there any other idea? cc @tcnghia <https://github.com/tcnghia> @mattmoor <https://github.com/mattmoor> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6058?email_source=notifications&email_token=AAF2WX4OTMPPCYGGJ3H2J3TQUH2N5A5CNFSM4JONAY22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEI6IQY#issuecomment-554820675>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF2WX6QDGQPVNMVOLN7ITTQUH2N5ANCNFSM4JONAY2Q> .

nak3 · 2019-11-18T06:18:26Z

Also we can use pkg/kmeta/ChildName.

Oh, I see. Thank you.

I wonder how does that surface in the ksvc/revision status?

For now, users can notice it by Ingress's status field (if #6048 is merged) and event log (current event log is already reveals this error).

c.f. ingress's status field by adding #6048.

  conditions:
  - lastTransitionTime: "2019-11-16T11:41:46Z"
    message: 'failed to create VirtualService: admission webhook "pilot.validation.istio.io"
      denied the request: configuration is invalid: regex match ''^service-to-service-call-via-activator-both-disabled-vktmmxev\.serving-tests(\.svc(\.cluster\.local)?)?(?::\d{1,5})?$''
      cannot be greater than 100 bytes'
    reason: ReconcileVirtualServiceFailed
    status: "False"

ksvc will just say IngressNotConfigured or something, though.

nak3 · 2019-11-20T04:44:23Z

One note, this error does not happen with istio-learn for now. istio/istio#18539 implements for Galley validation so following setting is needed:

(e.g) 1.3.3 does not have the validation, but just as a config sample.

serving/third_party/istio-1.3.3/istio.yaml

Line 196 in 6c8bbf0

- name: pilot.validation.istio.io

This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ```

ZhiminXiang · 2019-11-24T00:13:44Z

It looks like we have a few options.

A. Modify ObjectNameForTest() to create shorter ksvc name for tests.

This option seems like hiding the real issue in our E2E tests as users could create ksvc with long name.

This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ```

nak3 · 2019-11-24T04:23:08Z

@ZhiminXiang Thanks. Yes, I realized that it is not good option and so filed #6088 to use prefix instead of regex. (prefix does not have the limit.)

duglin · 2019-11-25T17:58:28Z

should using

B. Use PILOT_ENABLE_UNSAFE_REGEX=true for Istio Pilot.

work around it? It doesn't seem to work for me

) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in #6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

nak3 · 2019-11-25T22:40:35Z

@duglin Hmm... If it does not work, I think that it is an Istio's bug. The config is mentioned in the Istio's release note.

https://istio.io/news/releases/1.4.x/announcing-1.4/upgrade-notes/#regex-engine-changes

If you depend on specific behavior of the old regex engine, you can opt out of this change by adding the environment variable PILOT_ENABLE_UNSAFE_REGEX=true to the Pilot deployment. Note: this will be removed in future releases.

Anyway, #6088 is merged so Knative does not need it.

duglin · 2019-11-26T01:07:24Z

I'll try it again. Was just hoping it would work then I could apply that patch to our v0.10 stuff :-(

nak3 · 2019-11-26T02:48:30Z

I am feeling that Istio needs https://github.com/istio/istio/pull/19089/files (it is NOT included in Istio 1.4 branch?!) But anyway, it would be an Istio topic rather than Knative.

duglin · 2019-11-26T13:37:43Z

I'm guessing that PR isn't part of Istio 1.4
Bummer

…ative#6088) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

) (#6183) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in #6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

…ative#6088) (knative#6183) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

* squash (knative#6174) (knative#6175) * Use prefix instead of regex for authority match in virtualservice (knative#6088) (knative#6183) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

…088) * Use prefix instead of regex for authority match in virtualservice This patch changes to use prefix instead of regex for authority match in virtualservice. As described in knative/serving#6058, Istio 1.4 introduced 100 bytes limitation for the regex. So, Knative service which has long service name or domain name, it hits the limit easily. To fix it, this patch uses `prefix` and stop using `regex`. Current regex in VirtualService should be able to replaced with Prefix. CURRENT: ``` regex: ^hello-example\.default\.example\.com(?::\d{1,5})?$ ``` AFTER: ``` prefix: hello-example.default.example.com ``` * Trim cluster local domain to match local

nak3 added the kind/bug Categorizes issue or PR as related to a bug. label Nov 18, 2019

knative-prow-robot added area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/spec Discussion of how a feature should be exposed to customers. labels Nov 18, 2019

nak3 mentioned this issue Nov 18, 2019

Update to Istio 1.4 and retire 1.2 #6039

Closed

nak3 mentioned this issue Nov 19, 2019

[WIP] PoC for pull/6088 on Istio 1.4 mesh #6065

Closed

nak3 mentioned this issue Nov 22, 2019

Use prefix instead of regex for authority match in virtualservice #6088

Merged

knative-prow-robot closed this as completed in #6088 Nov 25, 2019

JRBANCEL mentioned this issue Dec 2, 2019

Update installing-istio.md knative/docs#2002

Closed

tcnghia mentioned this issue Dec 9, 2019

[RELEASE-0.10] Cherry-pick https://github.com/knative/serving/pull/6088 #6183

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VirtualService fails to create due to Istio 1.4's regex engine check if ksvc has a little longer domain #6058

VirtualService fails to create due to Istio 1.4's regex engine check if ksvc has a little longer domain #6058

nak3 commented Nov 18, 2019 •

edited

Loading

nak3 commented Nov 18, 2019

vagababov commented Nov 18, 2019 via email

nak3 commented Nov 18, 2019 •

edited

Loading

nak3 commented Nov 20, 2019

ZhiminXiang commented Nov 24, 2019

nak3 commented Nov 24, 2019

duglin commented Nov 25, 2019

nak3 commented Nov 25, 2019

duglin commented Nov 26, 2019

nak3 commented Nov 26, 2019

duglin commented Nov 26, 2019

VirtualService fails to create due to Istio 1.4's regex engine check if ksvc has a little longer domain #6058

VirtualService fails to create due to Istio 1.4's regex engine check if ksvc has a little longer domain #6058

Comments

nak3 commented Nov 18, 2019 • edited Loading

In what area(s)?

What version of Knative?

Actual Behavior

Steps to Reproduce the Problem

nak3 commented Nov 18, 2019

vagababov commented Nov 18, 2019 via email

nak3 commented Nov 18, 2019 • edited Loading

nak3 commented Nov 20, 2019

ZhiminXiang commented Nov 24, 2019

nak3 commented Nov 24, 2019

duglin commented Nov 25, 2019

nak3 commented Nov 25, 2019

duglin commented Nov 26, 2019

nak3 commented Nov 26, 2019

duglin commented Nov 26, 2019

nak3 commented Nov 18, 2019 •

edited

Loading

nak3 commented Nov 18, 2019 •

edited

Loading