WARNING: USE A VPN. Genmon exposed to the internet was remotely started

[wotseurba]

For the past 2 days, my generator has started up (not transferred) for no reason (no outage). The first time I didn't know what was happening, reviewed my Genmon status to see only that the engine was running, with the log stating "remote start" which I didn't do. It took me a few minutes to assess the situation, and then shut down the generator under the maintenance tab.

Last night, same thing happened. Except, my site name has changed to: "Hacked by Nariozon"

So, someone has figured out how to log into my genmon, and is doing this. What should I do?

I'd like to change the IP port to something other than 8000, so I can close off the port forwarding of 8000.

Has anyone else experienced this?

Logs submitted

Your Environment

• Generator Model: 20kw Evolution
• Genmon Logs: Submit via the About page
• Genmon Version: V1.18.10

[skipfire]

Genmon's port can be changed in /etc/genmon/genmon.conf. There is a setting for http_port. If you are wanting to expose genmon to the internet on any port you should probably switch to SSL at a minimum, though you may want to see if your router supports inbound VPN so you can VPN in instead of exposing genmon at all. And of course, I would suggest changing both the username and password you use for genmon.

[djrallard]

I’d recommend a new SD card and rebuild your GenMon install. You can’t know what they might have installed or done to your installation.
Then, use a strong password, MFA, and change your workflow to use a VPN for access to your internal network instead of exposing devices directly to the internet.

[wotseurba]

Thank you both for the comments and recommendations. I've powered-down the Genmon Pi and closed up the router port forwarding 8000. I will make an assessment of my router's capability of inbound VPN before moving forward to rebuild of my Genmon. I find it strangely interesting (and disturbing) that someone actually knows the workings of Genmon, and gains satisfaction my turning on my generator remotely. And, changing my site name, thus exposing themselves as doing this. When my generator started up at 3am, with no power outage, and not scheduled exercise, it had me wondering if my Evolution controller had failed, or Genmon has some glitch. But after it happened again last evening, and the hacker changed my site name, I now know the cause. I thought it would be a heads-up to other users and jgyates might offer a security alert and advice for users to use VPN, and-or using something other than port 8000.

[jgyates]

I agree that this is a good reminder for everyone exposing genmon to the internet to everyone.

Here are all of the times VPN was advised in the issues:

https://github.com/jgyates/genmon/issues?q=is%3Aissue+vpn

And the front page of the wiki also advises against this in the Connectivity section:

https://github.com/jgyates/genmon/wiki#connectivity

From an intruders point of view, changing the port and using SSL has little to no effect to a hacker. The path they are likely taking is probing your router for exposed ports. This is easily done by scanning consecutive IP addresses probing all portss until they find something open. Your exposed port will eventually be hit. If they find your open port they try to look for a web interface on that port. This brings up the genmon logon page. From there they just google genmon and they will find the github project with 100% of the source code since this is an open source project. They don't even need the source code to use one of hundreds of publicly available password cracker programs. A 7 character (or less) password can be cracked by a computer in less than a second. Once they have your password cracked then what they do is up to them. Best case scenario they let you know you are exposed, like your hacker appeared to do to you. Worst case they try to gain access to your pi (not just he web interface) and then launch attacks on your local network, possibly gaining access to other computers.

In addition, the web server used by genmon is not intended for production use. It is the web server built into the flask libraries and the developers of that software do not recommend using it in a production environment as the web server was not built with security in mind. It would not be resilient to attacks like a denial of service or other non password cracking hacks.

In short, don't expose genmon to the internet. If you must access genmon from outside your network, use a VPN as it is much harder to hack and always use strong passwords.

I am editing the title of this thread to better warn everyone to use a VPN.

[jgyates]

Also, as mentioned by @skipfire, enabling genmon MFA would go a long way to defeating people trying to do a simple password crack.

[wotseurba]

Thank you

[jgyates]

Two notes on this thread to close it out:

1. There is a new version of genmon today (V1.18.11) that implements a temporary lockout after the maximum number of failed logins after a period of time. This feature has two parameters (max failed login attempts and lockout duration in seconds) both of which are on the Advanced settings page. The defaults are 5 and 300 seconds (5 min). For example with the defaults if there are 5 failed logins within 5 minutes then the login is locked out for 5 minutes. This should deter most brute force password cracking attempts for reasonable lengths passwords.

2. I examined the logs for the hack in question. In this instance the port was exposed but no username and login were used so it looks like someone just did a port scan and found an open port and pointed a browser at it. No password cracking.

[wotseurba]

Thank you for the efforts you expended on this.  I sent you a (modest) donation yesterday.   I also did a fresh install on a new SD card yesterday, and also updated to V1.18.11.  I've never used a login for Genmon, but might consider it.   Also, the new install was extremely easy to accomplish with both the latest OS, and the Genmon installation/configuration script.  I remember doing my first install many years ago, and having to manually configure things like serial port, etc.  Much easier and automatic now.   Thanks again Brian Ruestow     Sent: Tuesday, April 19, 2022 at 7:12 PM From: "jgyates" ***@***.***> To: "jgyates/genmon" ***@***.***> Cc: "wotseurba" ***@***.***>, "Author" ***@***.***> Subject: Re: [jgyates/genmon] WARNING: USE A VPN. Genmon exposed to the internet was hacked. Someone is remotely starting my generator (Issue #702)   Two notes on this thread to close it out: There is a new version of genmon today (V1.18.11) that implements a temporary lockout after the maximum number of failed logins after a period of time. This feature has two parameters (max failed login attempts and lockout duration in seconds) both of which are on the Advanced settings page. The defaults are 5 and 300 seconds (5 min). For example with the defaults if there are 5 failed logins within 5 minutes then the login is locked out for 5 minutes. This should deter most brute force password cracking attempts for reasonable lengths passwords. I examined the logs for the hack in question. In this instance the port was exposed but no username and login were used so it looks like someone just did a port scan and found an open port and pointed a browser at it. No password cracking. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[mattm55]

Tonight I just had my Generac start in the middle of the nite. Same thing, hacked. Settings were changed. Time to lock things down here. Rebuilding PI & Genmon tomorrow.

[jgyates]

Thanks for the info.

[jgyates]

I have updated the wiki, referencing this thread and pinned this thread in hopes of getting more visibility.

[mattm55]

Great and Thanks. I love "genmon" so it is something I need to completely reinstall, with the proper security.

I woke up to the gen running at midnight, determined it wasn't my utility and shutdown thru genmon. 10 minutes later it started again. Before I dropped the Pi after shutting down the 2nd time was to see all my settings were changed (excercise time, site name, config (24v over 12v, email, etc...) Grrrr.... can only blame myself.

[diverbill]

If you are looking for a VPN solution one i have switched to and really like is the Cloudflare Zero Trust service. https://developers.cloudflare.com/cloudflare-one/ With a little tweaking i have an always on vpn that routes 90% of my DNS connections back to my house for
A) dns security with pi-hole & local DNS
B) access to all my local service GENMON Security, cameras, NAS Shares
C) no open internet ports for someone to access
The one requirement is you must own a domain

[lmamakos]

I've also had great success using tailscale.com as a VPN between your devices. There's clients for mobile devices, windows, mac and Linux. I run the tailscale client on a few Raspberry Pi systems, including the one used for genmon. It's free for a whole bunch of devices, and the paid version is really only required for multi-user enterprise sorts of applications.