Traffic shadowing not working as expected

[aggarwala3]

Describe the bug
We're looking to replicate the traffic from original jenkins to another parallel running jenkins. We have added zalando.org/skipper-filter in original jenkins with tee predicate to replicate the traffic to another-jenkins.company.org but as soon as we add this annotation, jenkins API calls stop working and result in timeout.
Annotation added

    zalando.org/skipper-filter: tee("https://another-jenkins.com.org")

To Reproduce
Add context, for example:

1. route or ingress definition
2. http request with headers (truncate your secrets)
3. response with headers

Expected behavior
We'd expect same traffic to reach another-jenkins instance as well

Observed behavior
But we only see timeout in original jenkins post this addition of annotation on skipper ingress.

[szuecs]

Please show the logs that show the timeouts.
Does the request have a body?
Does another Jenkins see the http request headers?

I am pretty sure that the problem is that the target application is not able to read the body fast enough and therefore the other call hangs in the I/O pipe. That's expected behavior enforced by the current efficient implementation which we likely won't change. It's likely a side effect of being efficient.

[aggarwala3]

Here are the logs

│ [APP]time="2024-12-02T12:57:20Z" level=warning msg="tee: error while tee request Post \"https://another-jenkins.com.org/github-webhook/\": dial tcp x.x.x.x:443: connect: connection timed o │
│ [APP]time="2024-12-02T12:57:20Z" level=error msg="tee: error while tee requestio: read/write on closed pipe"                                                                                                                     │

I doubt if its about reading body fast enough, since both applications are in same network and there shouldnt be much of delay.

Plus, adding this annotation on ingress meant for original jenkins breaks the original jenkins itself and all webhook calls meant for "original" jenkins are broken post that with above logs.