java.net.BindException: Can't assign requested address when fetching metrics from client in dashboard

[Force-King]

最近公司准备用Sentinel做限流降级，我本地引入搭建好环境，采用注解@SentinelResource的方式，wiki 中提到的jar 包都引用了，控制台显示正常。
使用压测工具进行压测时，控制台报错如下：

2019-04-12 15:44:08.983 ERROR 2152 --- [pool-2-thread-1] c.a.c.s.dashboard.metric.MetricFetcher : fetch metric http://172.30.4.87:8720/metric?startTime=1555055040000&endTime=1555055046000&refetch=false error

java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_191]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_191]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_191]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_191]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:273) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:139) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194) [httpasyncclient-4.1.3.jar:4.1.3]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) [httpasyncclient-4.1.3.jar:4.1.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

环境：
mac os
jdk 1.8
使用工具 idea
web 项目使用spring boot

请问是什么原因？
@jasonjoo2010

[sentinel-bot]

Hi @Force-King, we detect non-English characters in the issue. This comment is an auto translation from @sentinel-bot to help other users to understand this issue.
We encourage you to describe your issue in English which is more friendly to other users.

Pressure test console error Can't assign requested address

Recently, the company is going to use Sentinel to do the flow-reduction downgrade. I introduced the local environment and adopted the @SentinelResource annotation. The jar packages mentioned in the wiki are all referenced, and the console display is normal.
When using a pressure test tool for pressure measurement, the console reports this:

2019-04-12 15:44:08.983 ERROR 2152 --- [pool-2-thread-1] c.a.c.s.dashboard.metric.MetricFetcher : fetch metric http://172.30.4.87:8720/metric?startTime=1555055040000&endTime=1555055046000&refetch=false error

java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_191]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_191]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_191]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_191]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:273) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:139) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) [httpcore-nio-4.4.6.jar:4.4.6]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194) [httpasyncclient-4.1.3.jar:4.1.3]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) [httpasyncclient-4.1.3.jar:4.1.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

surroundings:
mac os
jdk 1.8
Use tool idea
Web project using spring boot

May I ask what is the reason?
@jasonjoo2010

[sczyh30]

You may need to check whether your network config is correct.

[Force-King]

@sczyh30 网络没有问题，压测 10个线程 循环 100次没有问题，循环 10000次就会报这个错了。 压测量大了以后报的错，所以应该和网络配置没关系。

[jasonjoo2010]

检查 TIME_WAIT连接是不是满了

[jasonjoo2010]

这个错误一般是在connect的时候，需要随机分配本机上一个空闲的端口用于建立tcp连接，而找不到可用的端口时发生。

When you want create a connection with remote address OS will fetch one spare local port on interface first(TCP is a 4-element protocol: src.addr/src.port/dst.addr/dst.port).

In linux you can:

# sysctl -a | grep ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 1024 30000

That means you OS will pick an unused port in range from 1024 to 30000. If OS can't find one the invocation will fail.

You can try:

1. Extend the range to 1024 - 65535
2. If you don't have so much connection concurrently actually you may suffer TIME_WAIT problem.

TIME_WAIT

TCP conns have several state like: TIME_WAIT, CLOSE_WAIT, FIN_WAIT, FIN2_WAIT due to different closing orders. Some periods of them can be adjust through sysctl but TIME_WAIT. So you must avoid to turn connections into TIME_WAIT (close it correctly) or reduce the period from 60 seconds(hard coding in kernel header) to 1 seconds by recompiling the kernel.

[Force-King]

@jasonjoo2010 谢谢，我查看了，TIME_WAIT=16304 的时候开始报错。
网上查看文档：Linux动态端口号默认范围是32768-65535，也就是说，作为客户端连接同一个IP和同一个端口号，最多只能建立30000多个连接，而Mac默认只能建立16000个左右的连接。
确实是连接太多，导致没法创建了，可以设置socket 连接配置参数，参考：https://blog.csdn.net/ys5773477/article/details/73614220

[jasonjoo2010]

@jasonjoo2010 谢谢，我查看了，TIME_WAIT 还没有达到上限，持续增长的过程中就开始报错了，我在研究一下。。

有数个影响因素，backlog, time_wait, port range等

看了下你的报错，是dashboard端的报错，并且极其可能是可用端口用尽，你可以考虑在另外的节点（非压测节点）上来部署运行Dashboard，另外压测节点要注意调整参数和压测方式，看情况应该多半是压测没用连接池导致大量http短连接或是并发连接数过高

[Force-King]

@jasonjoo2010 谢谢，我查看了，TIME_WAIT 还没有达到上限，持续增长的过程中就开始报错了，我在研究一下。。

有数个影响因素，backlog, time_wait, port range等

看了下你的报错，是dashboard端的报错，并且极其可能是可用端口用尽，你可以考虑在另外的节点（非压测节点）上来部署运行Dashboard，另外压测节点要注意调整参数和压测方式，看情况应该多半是压测没用连接池导致大量http短连接或是并发连接数过高

压测用的是 Jmeter, Threads =10 ,Loop Count =20000

[Force-King]

@jasonjoo2010 Your answer is right. Thank you very much