[Feature Request] / [Feedback] Extension of the NMAP functionality.

[accessiblepixel]

Hi!

Many thanks for your excellent fork, I'm very impressed with it so far.

I've encountered a few issues so far, but nothing so serious that I couldn't work around it.

My network is separated into segmented VLANs so the default arp scanning can only access the segment pialert is assigned to. To work around this, and because pihole and pialert are hosted in a Proxmox LXC, I added interfaces for each VLAN without an IP address, for example eth0 is my default interface that all the other network communication happens through, and eth5 is for VLAN 5 is a segmented vlan.

Because I didn't want to actually have the lxc accessible on each of these VLANs via an IP address I didn't add one to any other VLAN interfaces, but I found in the documentation for arpscan I can specify the reply address as an IP somewhere on the VLAN - As far as I am aware this will allow devices that check if an arp request is on the same subnet to reply - without specifying this it uses the (invalid) source address 0.0.0.0 (which some devices take issue with)

So in the pialert.conf file I have
SCAN_SUBNETS = ['192.168.0.1/24 --interface=eth0','192.168.5.0/24 --interface=eth5 --arpspa=192.168.5.253']

This has now shown up most devices on my network :)

Now to my question - The settings for Speedtest. The option for SPEEDTEST_TASK_HOUR - what does this specify, the hour in which a speedtest will be run?

Also, does the speedtest-cli need to be installed from ookla to the system or is the one included already with the pi.alert distribution ( pialert/back/speedtest-cli) enough?

Many thanks for the amazing work you've put into maintaining this fork so far.

If I've put this in the wrong place, please feel free to move it to where is more appropriate - I'm not particularly used to github practices yet (and they can vary per repo)

Hope you're having a great day.
Kind regards!

Edit: I was also wondering if there was some way to schedule automatic nmap scans and have the results stored in the DB?

[leiweibau]

Because I didn't want to actually have the lxc accessible on each of these VLANs via an IP address I didn't add one to any other VLAN interfaces, but I found in the documentation for arpscan I can specify the reply address as an IP somewhere on the VLAN - As far as I am aware this will allow devices that check if an arp request is on the same subnet to reply - without specifying this it uses the (invalid) source address 0.0.0.0 (which some devices take issue with)

Nice trick. As this is probably a slightly more unusual configuration, I'm still undecided whether to include this in the documentation so as not to cause any additional confusion.

The settings for Speedtest. The option for SPEEDTEST_TASK_HOUR - what does this specify, the hour in which a speedtest will be run?

Yes, the full hour at which the speed test is performed is specified here. 9 = 9am, 14 = 2pm (14 o'clock). However, the Ookla Speedtest must be "installed" for this (https://raw.githubusercontent.com/leiweibau/Pi.Alert/assets/screen_dev_internet_01.png)

Also, does the speedtest-cli need to be installed from ookla to the system or is the one included already with the pi.alert distribution ( pialert/back/speedtest-cli) enough?

The speedtest-cli will be removed sooner or later. The background is that there was once an issue with this client (Starlink user). However, the official binary from Ookla worked correctly. The last update should also fix the bug that you have to re-download the Ookla client after every Pi.Alert update.

Thank you for the praise and kind words

Addendum:

I was also wondering if there was some way to schedule automatic nmap scans and have the results stored in the DB?

I had thought about it once, but rejected it for the time being, because depending on the configuration, a nmap scan can take quite a while and I don't want to have a race condition on the database.
My last thoughts in this direction were to write the results of a manual scan to the database and, if possible, to compare them with an already existing scan. With this commit (9945dc7), among others, I have already laid the foundation for further processing of the nmap output. I have not yet decided when and, above all, how I will implement it. What is the point of having all kinds of data if it is presented in a confusing way? I'm not a UI designer, but I try to make my fork as organized as possible.

[Tuxtom007]

Thanks for the tip with "arpspa", but unfortunately it did not solve my problem that the VLANs are scanned.

However, an "arp-scan 10.0.10.0/24 --interface=eth1" to the corresponding VLAN still works without any problems, which also worked before with my various configuration attempts, just not from the PiAlert

[accessiblepixel]

@leiweibau 'unusual network configuration' - yes please! :D

I imagine my configuration won't be anywhere near the standard. I just ended up segmenting everything out as VLANs, mainly for organisation in my head- Oh, IP is 192.168.5.x I know what "group" it's in, VLAN5 and it's level of trust. I'm not sure if LXC can have one network interface that has multiple tagged VLANs to it (read docs for a while... it says maybe), so I just kinda "brute forced" it by adding an interface for each VLAN I used!

I didn't wanna have a bunch of random IPs on each one though, since if you've got 10 VLANs, that's 10 extra IPs - which would probably appear in arpscan too lol

Thanks for the hints about speedtest - I installed the speedtest apt repo and the client, and now things seem to be working. I also figured you can specify multiple hours - at first I thought it was just one. Although might it be a good idea to randomise the minute/second that the automated scan is done? - I vaguely remember as a young sysadmin being taught that automation on the hour for other people's systems is a bad thing 🤣

RE: Nmap - My thinking with it was maybe a scan could be done over a predefined period, not that often and even if it's just as the text output is now it can display the "cached" nmap results when you click on to it... although guess that won't do anything for "monitoring changes"... 🤷‍♀️

Though I do like the idea of doing the scan, getting the results and once they've been "got" adding to the database - sounds like a sensible way to do it to me! :) Won't have to worry about locks too much then.

@Tuxtom007 I'm not sure if arpspa is required or not, or if it just needs a network interface on each VLAN. I guess I could remove arpspa and see if I lose any devices but I'm hesitant to fix what ain't broke for me. 🤪

Hope you're both doing well. :)

[leiweibau]

I vaguely remember as a young sysadmin being taught that automation on the hour for other people's systems is a bad thing 🤣

Since the users of my fork are a bit more internationally dispersed and Ookla certainly doesn't just have the one speedtest server, that should still be okay 😉 Randomizing the minutes in the relevant hour is not as trivial to implement as it might seem at first. I'll take it up as an idea, but without the guarantee that I will implement it.

My thinking with it was maybe a scan could be done over a predefined period, not that often

There are actually 2 things to address here:

1. a cyclic nmap scan of the network as a addition to the ARP scan:
   Depending on the size of the network, there will inevitably be a race condition here, as the scan is "cron based", i.e. it starts every 5 min.
2. a cyclic scan on one or more devices:
   Even with the objective of doing this only for selected devices, there will be the one user who activates the function for all, or at least for many devices and then gets frustrated that nothing works anymore and Pi.Alert generates lots of error messages. I don't really need that 😉

For these reasons I have decided against automatic nmap scans

[accessiblepixel]

@leiweibau

I understand the reasoning, although one thing to say about things that run with cron of undetermined length, a flag set "is_running" could be set so if it's called again while one is in progress... I have do that on microprocessors all the time :) Again there's the issue if that gets set and then things don't work and I can see why you don't want the added support overhead...

Would you consider adding caching of last nmap result? Since that's probably enough for my own personal needs.

Anyway, thanks for your replies and work so far. My suggestions are mearly that, any you think are valuable to use going forward, please do so, any that make no sense ... you don't gotta use haha :)

[leiweibau]

a flag set "is_running" could be set so if it's called again while one is in progress

good point 🤔 and so obviously

[leiweibau]

Would you consider adding caching of last nmap result?

I think that something will happen here, as I have already done the groundwork. I just have to think about how to implement it properly.

[Tuxtom007]

@accessiblepixel

I have the same constellation, Pialtert runs on a LXC in Proxmox, I use 12 VLANs in my network and have given the PiAlert-LXC an additional network card in the first 3 VLANs as a test (each also with a static IP in the VLAN via DHCP reservation).
An arpscan on the LXC directly with the network and the Inetrface works perfectly, I get all clients displayed as the 3 VLAN respectively.

But no matter how I enter it in pialert.conf, I don't get any clients from the other VLANs. I have already tried all possible combinations from the documentation: IP network + interface + VLAN, then without VLAN, only IP network + VLAN, etc.

I used to run PiAlert in the same way some time ago, it was a real pain to get all VLANs scanned, but unfortunately I can't remember how I did it.

Yesterday I switched to another fork, let's see which is better, the WebUI at least looks different and there is also the option to get the client from the Unifi controller.

[leiweibau]

This is the current status of my work.
If there is already a saved "nmap scan", this is displayed below the buttons

If a new scan has been performed, the two results are displayed next to each other. As soon as the page has been reloaded, only the most recent scan is displayed again. In the case that the database is currently being locked, there is currently no fallback solution as with the journal.

A cleanup is also run over this table, i.e. old entries are deleted regularly. I plan to add a button in the "Tools" tab to save all available nmap scans for a device

[leiweibau]

Update released with b3da81c

[leiweibau]

The manual Nmap scan function has been significantly expanded. Other aspects of the request will not be implemented for the time being, as such changes would have far-reaching consequences that need to be weighed up.
For these reasons, this request is now closed.

[accessiblepixel]

@leiweibau - Looks great! I'll pull in the new version and have a play. Thanks for the additions :)

[accessiblepixel]

@Tuxtom007

@accessiblepixel

I have the same constellation, Pialtert runs on a LXC in Proxmox, I use 12 VLANs in my network and have given the PiAlert-LXC an additional network card in the first 3 VLANs as a test (each also with a static IP in the VLAN via DHCP reservation). An arpscan on the LXC directly with the network and the Inetrface works perfectly, I get all clients displayed as the 3 VLAN respectively.

But no matter how I enter it in pialert.conf, I don't get any clients from the other VLANs. I have already tried all possible combinations from the documentation: IP network + interface + VLAN, then without VLAN, only IP network + VLAN, etc.

I used to run PiAlert in the same way some time ago, it was a real pain to get all VLANs scanned, but unfortunately I can't remember how I did it.

Yesterday I switched to another fork, let's see which is better, the WebUI at least looks different and there is also the option to get the client from the Unifi controller.

Translated with DeepL.com (free version)

I've just had an issue that seems similar to what yours was. It was because the devices became undefined for the VLAN interfaces within the /etc/network/interfaces file.

Readding to that file and restartng the LXC got my arp scans working for the VLANs again.

...
auto eth5
iface eth5 inet manual
auto eth10
iface eth10 inet manual

Hope that helps!

[Tuxtom007]

@Tuxtom007
I've just had an issue that seems similar to what yours was. It was because the devices became undefined for the VLAN interfaces within the /etc/network/interfaces file.

Readding to that file and restartng the LXC got my arp scans working for the VLANs again.

...
auto eth5
iface eth5 inet manual
auto eth10
iface eth10 inet manual

Hope that helps!

Thanks.
I have found a solution that has been running for some time without any problems:

• one NIC per VLAN in the LXC
• only the main NIC has an IP, the other VLANs have no IP address

pialert.conf:

SCAN_SUBNETS = ['--localnet --interface=eth0','10.0.10.0/24 --interface=eth0.10 --arpspa=10.0.10.253','10.0.20.0/24 --interface=eth0.20 --arpspa=10.0.20.253','aso.....]

[accessiblepixel]

@Tuxtom007 - yep that's exactly what I'm doing :> Glad you got it working.