ignore_localhost doesn't work as expected

[aviramha]

user has ignore_localhost but fails to connect to local instance, while having specific filter like

{
  "feature": {
    "network": {
      "outgoing": {
        "filter": {"local": [":11211"]}
      }
    }
  }
}

works.
maybe related to #2820

[linear]

MBE-463 `ignore_localhost` doesn't work as expected

[aviramha]

Managed to reproduce it on macOS using this repo:
https://github.com/aviramha/springflux/tree/main
Changed MyController to use

        String externalServiceUrl = "http://127.0.0.1:1337/health"; // Replace with your actual service URL

run a simple http server on port 1337 I used:

python3 -m http.server 1337                                                                         
Serving HTTP on :: port 1337 (http://[::]:1337/) ...

then run it using IntelliJ Idea, send curl from cluster to 8080 (probably localhost works too)
with the following mirrord.json

{
    "target": "deployment/ip-visit-counter",
    "feature": {
        "network": {
            "incoming": "steal",
            "outgoing": {
                "ignore_localhost": true
            }
        },
        "fs": "read",
        "env": true
    }
}

	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:172) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:480) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:436) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:205) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:583) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.Operators.error(Operators.java:198) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.MonoError.subscribe(MonoError.java:53) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4576) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.6.10.jar:3.6.10]
		at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:576) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.netty.transport.TransportConnector$MonoChannelPromise.setFailure(TransportConnector.java:522) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at reactor.netty.transport.TransportConnector.lambda$doConnect$7(TransportConnector.java:261) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
		at reactor.netty.transport.TransportConnector.lambda$doConnect$8(TransportConnector.java:246) ~[reactor-netty-core-1.1.22.jar:1.1.22]
		at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
		at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Unknown error: 111: /127.0.0.1:1337
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoError] :
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:576)
Error has been observed at the following site(s):
	*__FluxOnErrorResume$ResumeSubscriber.onError ⇢ at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:576)
	*__________________________Mono.onErrorResume ⇢ at reactor.netty.transport.TransportConnector.lambda$connect$6(TransportConnector.java:166)
	*________________________________Mono.flatMap ⇢ at reactor.netty.transport.TransportConnector.connect(TransportConnector.java:165)
	*_________________________________Mono.create ⇢ at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator.connectChannel(DefaultPooledConnectionProvider.java:535)
	*___________________________Mono.contextWrite ⇢ at reactor.netty.resources.PooledConnectionProvider.lambda$acquire$3(PooledConnectionProvider.java:174)
	*_________________________________Mono.create ⇢ at reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:126)
	*_________________________________Mono.create ⇢ at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:210)
	|_                             Mono.retryWhen ⇢ at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:275)
	*______________________________Flux.concatMap ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:360)
	|_                           Flux.onErrorStop ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:379)
	*________________________Flux.deferContextual ⇢ at reactor.util.retry.RetrySpec.generateCompanion(RetrySpec.java:357)
	*____________________________Mono.flatMapMany ⇢ at reactor.netty.http.client.HttpClientFinalizer.responseConnection(HttpClientFinalizer.java:102)
	|_                                  Flux.next ⇢ at org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(ReactorClientHttpConnector.java:165)
	|_                            Mono.doOnCancel ⇢ at org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(ReactorClientHttpConnector.java:166)
	|_                           Mono.doOnRequest ⇢ at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:103)
	|_                            Mono.doOnCancel ⇢ at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:104)
Original Stack Trace:
Caused by: java.net.SocketException: Unknown error: 111
	at java.base/sun.nio.ch.Net.connect0(Native Method) ~[na:na]
	at java.base/sun.nio.ch.Net.connect(Net.java:579) ~[na:na]
	at java.base/sun.nio.ch.Net.connect(Net.java:586) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853) ~[na:na]
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) ~[na:na]
	at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:321) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:250) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1289) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:634) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:657) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:634) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:618) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:545) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:891) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:228) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at reactor.netty.transport.TransportConnector.lambda$doConnect$8(TransportConnector.java:239) ~[reactor-netty-core-1.1.22.jar:1.1.22]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

[aviramha]

When using this config it works (it errors but on 404):

{
    "target": "deployment/ip-visit-counter",
    "feature": {
        "network": {
            "incoming": "steal",
            "outgoing": {
                "filter": {
                    "local": [":1337"]
                }
            }
        },
        "fs": "read",
        "env": true
    }
}

[aviramha]

Relevant logs when this happens:

2024-10-11T18:08:22.108602Z TRACE ThreadId(14) socket: mirrord_layer::socket::ops: new domain=30 type_=1 protocol=0 pid=25778
2024-10-11T18:08:22.108640Z TRACE ThreadId(14) socket: mirrord_layer::socket::ops: return=Success(101) domain=30 type_=1 protocol=0 pid=25778
2024-10-11T18:08:22.108655Z TRACE ThreadId(14) socket: mirrord_layer::socket::ops: close time.busy=39.9µs time.idle=13.1µs domain=30 type_=1 protocol=0 pid=25778
2024-10-11T18:08:22.108899Z TRACE ThreadId(14) fcntl: mirrord_layer::socket::ops: new orig_fd=101 cmd=3 fcntl_fd=2
2024-10-11T18:08:22.108916Z TRACE ThreadId(14) fcntl: mirrord_layer::socket::ops: close time.busy=1.50µs time.idle=15.0µs orig_fd=101 cmd=3 fcntl_fd=2
2024-10-11T18:08:22.108936Z TRACE ThreadId(14) fcntl: mirrord_layer::socket::ops: new orig_fd=101 cmd=4 fcntl_fd=0
2024-10-11T18:08:22.108948Z TRACE ThreadId(14) fcntl: mirrord_layer::socket::ops: close time.busy=1.38µs time.idle=10.4µs orig_fd=101 cmd=4 fcntl_fd=0
2024-10-11T18:08:22.615800Z TRACE ThreadId(14) connect: mirrord_layer::socket::ops: new sockfd=101 address_length=28
2024-10-11T18:08:22.615812Z TRACE ThreadId(14) connect: mirrord_layer::socket::ops: in connect LazyLock(
    Mutex {
        data: {
            101: UserSocket {
                domain: 30,
                type_: 1,
                protocol: 0,
                state: Initialized,
                kind: Tcp(
                    1,
                ),
            },
            100: UserSocket {
                domain: 30,
                type_: 1,
                protocol: 0,
                state: Connected(
                    Connected {
                        remote_address: Ip(
                            127.0.0.1:48012,
                        ),
                        local_address: Ip(
                            127.0.0.1:8080,
                        ),
                        layer_address: None,
                    },
                ),
                kind: Tcp(
                    1,
                ),
            },
            99: UserSocket {
                domain: 30,
                type_: 1,
                protocol: 0,
                state: Listening(
                    Bound {
                        requested_address: [::]:8080,
                        address: [::]:8080,
                    },
                ),
                kind: Tcp(
                    1,
                ),
            },
            102: UserSocket {
                domain: 30,
                type_: 2,
                protocol: 0,
                state: Bound(
                    Bound {
                        requested_address: [::]:0,
                        address: [::]:49239,
                    },
                ),
                kind: Udp(
                    2,
                ),
            },
        },
        poisoned: false,
        ..
    },
) sockfd=101 address_length=28
2024-10-11T18:08:22.615879Z TRACE ThreadId(14) connect:connect_outgoing: mirrord_layer::socket::ops: new sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream
2024-10-11T18:08:22.616102Z TRACE ThreadId(14) connect:connect_outgoing:get_connection_through: mirrord_layer::socket: new sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream self=Unfiltered address=[::ffff:127.0.0.1]:1337 protocol=Stream
2024-10-11T18:08:22.616121Z TRACE ThreadId(14) connect:connect_outgoing:get_connection_through: mirrord_layer::socket: return=Ok(Remote([::ffff:127.0.0.1]:1337)) sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream self=Unfiltered address=[::ffff:127.0.0.1]:1337 protocol=Stream
2024-10-11T18:08:22.616140Z TRACE ThreadId(14) connect:connect_outgoing:get_connection_through: mirrord_layer::socket: close time.busy=19.5µs time.idle=19.0µs sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream self=Unfiltered address=[::ffff:127.0.0.1]:1337 protocol=Stream
2024-10-11T18:08:22.616306Z TRACE ThreadId(14) connect:connect_outgoing:make_request_with_response: mirrord_layer::proxy_connection: new sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream request=OutgoingConnectRequest { remote_address: Ip([::ffff:127.0.0.1]:1337), protocol: Stream }
2024-10-11T18:08:23.021912Z TRACE ThreadId(14) connect:connect_outgoing:make_request_with_response: mirrord_layer::proxy_connection: return=Ok(Err(RemoteIO(RemoteIOError { raw_os_error: Some(111), kind: ConnectionRefused }))) sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream request=OutgoingConnectRequest { remote_address: Ip([::ffff:127.0.0.1]:1337), protocol: Stream }
2024-10-11T18:08:23.021972Z TRACE ThreadId(14) connect:connect_outgoing:make_request_with_response: mirrord_layer::proxy_connection: close time.busy=406ms time.idle=20.3µs sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream request=OutgoingConnectRequest { remote_address: Ip([::ffff:127.0.0.1]:1337), protocol: Stream }
2024-10-11T18:08:23.022004Z TRACE ThreadId(14) connect:connect_outgoing: mirrord_layer::socket::ops: return=Error(ResponseError(RemoteIO(RemoteIOError { raw_os_error: Some(111), kind: ConnectionRefused }))) sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream
2024-10-11T18:08:23.022022Z TRACE ThreadId(14) connect:connect_outgoing: mirrord_layer::socket::ops: close time.busy=406ms time.idle=13.5µs sockfd=101 address_length=28 sockfd=101 remote_address=SockAddr { ss_len: 0, ss_family: 30, len: 28 } user_socket_info=UserSocket { domain: 30, type_: 1, protocol: 0, state: Initialized, kind: Tcp(1) } protocol=Stream
2024-10-11T18:08:23.022038Z TRACE ThreadId(14) connect: mirrord_layer::socket::ops: return=Error(ResponseError(RemoteIO(RemoteIOError { raw_os_error: Some(111), kind: ConnectionRefused }))) sockfd=101 address_length=28
2024-10-11T18:08:23.022052Z TRACE ThreadId(14) connect: mirrord_layer::socket::ops: close time.busy=406ms time.idle=12.2µs sockfd=101 address_length=28
2024-10-11T18:08:23.022065Z  INFO ThreadId(14) mirrord_layer::error: libc error (doesn't indicate a problem) >> ResponseError(
    RemoteIO(
        RemoteIOError {
            raw_os_error: Some(
                111,
            ),
            kind: ConnectionRefused,
        },
    ),
)