Graphs in Reporting: Traffic causes rapid drop in FW throughput

[drobnymichal]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

We recently tried deploying an instance of the OPNsense firewall in our infrastructure. Since the beginning, we've experienced a few issues with firewall throughput. One of them is most likely caused by data loading for the graphs in the Reporting section.

We have experienced a significant drop in throughput when a user looks at the graphs in the Reporting: section of the Traffic application via the WEB GUI. This suggests that the problem is caused by data loading for the graph that displays Top hosts in/out (bps) information, because once this graph is inactive, the firewall throughput reaches the expected values.

This behavior was also noticed in the previous version 24.7.

To Reproduce

1. Go to Reporting: Traffix section in the WEB GUI.
2. Generate network traffic that passes through the Firewall.

For testing we used 10 virtual machines in Proxmox and the iperf3 tool.
The VMs are on a network that is on an FW interface - VLAN361 (opt1).
Server side (5 VMs)

iperf3 -s

Client side (5 VMs)

iperf3 -c <server-node-ip> -P 10 -t 55

3. Watch the traffic that is displayed in the graphs (Reporting: Traffic WEB GUI section).

In the page you can see the data regarding the actual traffic that goes through the FW and data about Top hosts. (interface VLAN361)

At this point we were achieving throughput of around 8.5 Gbps. I got a more detailed view of this data using the command vnstat -i vlan0361 -l

4. Block data download for Top hosts in/out (bps) graphs in the browser.

Data for graphs named Top hosts in/out (bps) are retrieved using an HTTP GET request to the URL https://<opnsense-gui-url>/api/diagnostics/traffic/top/<interface-identifier>. In our case, the URL is: https://<masked-opnsense-gui-url>/api/diagnostics/traffic/top/opt1.

When this URL is blocked (I used browser URL blocking feature), the amount of data that is transmitted via the FW increases rapidly. The increase can be seen in the following screenshot:

If I run the iperf3 test again, I get the following results:

At this point we were achieving throughput of around 64.5 Gbps. Again, I got a more detailed view of this data using the command vnstat -i vlan0361 -l

The https://<opnsense-gui-url>/api/diagnostics/traffic/top/<interface-identifier> HTTP request calls a Python script called /usr/local/opnsense/scripts/interfaces/traffic_top.py in the background. The script calls the iftop command on line 45, which causes the behavior described above.

When I try to run the /usr/local/sbin/iftop -nNb -i vlan0361 -s 2 -t command directly in CLI, I observe the same drop in throughput as when loading data for Top hosts in/out (bps) graphs.

Expected behavior

Displaying the traffic report should not affect firewall performance.

Describe alternatives you considered

I would consider the following options:

• Deleting the Top hosts in/out (bps) graph.
• Finding a new way to get data for that graph. (Do not call the /usr/local/sbin/iftop script with the mentioned parameters)
• Hide the graph and make it available only after warning about the possible impact on firewall performance.

Environment

Software version:

OPNsense 25.1-amd64
FreeBSD 14.2-RELEASE
OpenSSL 3.0.15

Hardware:

CPU: AMD EPYC 9254 24-Core Processor
RAM: 98 GB
NIC: Ethernet Controller E810-C for QSFP (100GB)