The Water Utility Hacks Weren't Hacks. They Were Undocumented Cell Modems.

This compromise is interesting to watch because even though it was billed as a hack, it's more about a lack of proper infrastructure management and a lack of maintaining a security posture that is essential in this day and age. This wasn't so much about someone hacking into a system. This was about someone taking advantage of a compromise that is a known compromise that has been reported over time, and using it to do something very simple that had a large lasting impact.

CBS News reports that malicious cyber activity hit water systems in at least seven states, with more than thirty Minnesota community water systems affected. CISA's advisory that same week called out something the SCADA press mostly skipped past: cellular modems installed by operators, vendors, or system integrators that may not be documented or included in routine attack surface scans. That line is my territory, and it deserves more than a footnote.

So what actually happened here?

What they did is they found publicly available IP addresses that were on the WAN side of a device that had PLC controllers behind it. They were able to access that WAN IP address because it was on a device connected to the 4G or 5G network. As it was connected to a carrier network, it had a public IP associated with it.

When you scan that public IP, it gives you a number of information pieces about what was connected to it. When you run the scan and see what comes back, either in the banner or whatever else was reported, it can reveal that a piece of equipment is exposed and may be vulnerable. This isn't a new class of problem. Internet-exposed operational technology has been a known risk for years.

For this activity, CISA specifically directed owners, operators, and integrators of Rockwell Automation MicroLogix 1400 PLCs to Rockwell's guidance for restoring access when the controller password is unknown. The immediate fix list is about reducing exposure: disconnect the PLC from the public internet, route remote access through a VPN or gateway, change default passwords, allowlist trusted IPs, and maintain a known clean backup of the PLC image.

So essentially, a group found the public IP address, port scanned it, saw what was available, and the banner turned around something with a known ability to be compromised. They went in and they changed passwords and changed IP addresses and blocked access for people to be able to control and manage their infrastructure.

Why do I keep saying this wasn't really a hack?

The way that it's being talked about as a hack, I think, is good and bad. It's good that it's being talked about as a hack because it's raising awareness, but it's not necessarily a true hack. It's just someone taking advantage of an open system with a public facing IP address.

This isn't necessarily a hack. This is laziness.

If people are looking for ways to see if those are available on their subnet or on their IP addresses, there are tools out there like Shodan and Censys that allow you to look yourself up and exclude those IPs. In Censys, for example, you can look up what your IP block is, and you can put in an exclusion request to make sure people don't get into that. But with ZoomEye and FOFA and all these other tools that are out there, people can punch in an IP address or blocks of IPv4 addresses and see what is publicly reporting and what's available.

It's interesting because no one's reporting this to an attacker. It's not like a piece of equipment is being narced on by somebody. The device announces itself online, and people can scan that through any of those registries and find out that they've got access to a compromised piece of equipment. When you have a device that's connected with a public IP address, that means it's open to the entire world.

How does a public works director find every radio nobody wrote down?

The first thing they have to do on Monday morning when they get in there is get that inventory going. Get an inventory from the carrier, because that's not running on your network. Get the SIM list with assigned IPs out of the carrier portal, and find out who holds that account. That's frequently the system integrator. In this case, not even the utilities. So the utility doesn't know, it's whoever did the integration for them.

Find out who's using that, then cross reference against the accounting. Is there a recurring static IP add-on line item? Devices you can't see are devices nobody has patched. So think about it like that.

Number two, eliminate public reachability. Move to a private APN with an IPsec tunnel, or CGNAT-only SIMs. It's low cost, and it's deployed by the integrator that's already on site. Where a public IP is unavoidable, the PLC goes behind a firewall or secure gateway and is never directly exposed. Use ACLs to restrict communication to authorized control systems.

Disable unused radios on the gateway. Many field cellular routers still ship with Wi-Fi enabled, configured at commissioning, and those have been up for years. Make sure remote support runs via a broker or jump host and not just an open port.

Then validate your logic and credentials. Assume the controller is lying. Compare running project files against known good baselines. Check ladder logic and add-on instructions for unauthorized changes. Rotate all PLC and HMI passwords, and eliminate the defaults so no defaults are being used anywhere. Validate backups before restoring, log cellular modem connections, and verify that manual fallback procedures work. Those are really, really key to do.

What breaks when you pull the towers off the cellular network?

Everything operationally goes back to those manual processes. In this case, what they were able to do is roll back into manual mode. In Braham, a city of about 1,700 people, the well supplying the water tower malfunctioned. The staff isolated the system and restored the backup, and they restarted in about 90 minutes. The tower held two days' worth of storage, so there was no service loss.

In South St. Paul, the issue was identified Monday and public works transitioned to manual operations. Drinking water treatment, quality, pressure, and delivery all went unaffected. In Plymouth, automated controls were affected, so they rolled back to manual operations too. In Maple Plain, the mayor declared a local state of emergency, but again, everything went back to manual operations.

What happens then is you rely on that human asset. You need a human to go out there and manually change things. You manually change valves and manually open or close different flow controls. You need a person to go out there and do that.

That's important because a lot of people think you can just have AI running your systems for you, opening valves and closing valves and running these operations. AI may be good at things like transcribing phone calls or making really cool images of cats, but AI might not be the best solution for running machine operations like this, because at the end of the day you're still going to need humans to go out there in the event of a failure and conduct those manual processes.

Why doesn't "get the PLCs off the internet" actually work?

The problem is that the larger the city, the more PLCs there are. So you can't just take everything off the internet. That sounds like an easy out. What you can do is secure it. You can make sure that you're operating a secure network from start to finish. Run your security audits, make sure that your security posture is strong, and make sure that you're doing things the way they're supposed to be done.

A lot of this happens when people cut corners and they don't want to spend money on security. They don't want to spend money on audits. They don't want to spend money on making sure that this critical infrastructure is secured. Unfortunately, what we're seeing is that all of those places where people cut corners and didn't make those investments are the first ones that are going to be the most susceptible to these attacks, and the first ones that are going to go down.

So it's not necessarily that you've got outside actors attacking all the infrastructure across the United States. They've been able to find vulnerabilities from the people that cut corners, and attack those. It might seem like it's a random attack on small town America, but it really isn't. It's an attack on someone that didn't take the time to secure their infrastructure, and it left them wide open.

This is happening in small communities around the United States. It's not happening in these larger communities, because they might have a different way that they've set it up and they might be more security conscious. It's important for those smaller communities to look at this.

Have I really found radios nobody documented?

In just about every large installation that you go into, you can find some piece of equipment that's unauthorized and not supposed to be there. With wireless intrusion detection and intrusion prevention now being installed almost across the board on equipment, that really gives you the ability to see what's going on the network.

You always used to find some router that was plugged into a switch, or some access point that was plugged into a switch because someone wanted to bring in their own Wi-Fi. We've done industrial installations in cities and municipalities where you walk in and you see someone's brought their own home based router system and plugged it in, because they wanted Wi-Fi in the office and they didn't think it worked well enough, or it had a password they didn't know, or they wanted to get around it. So you find all these instances where equipment was just used to solve a problem or fix a gap in communications, and then it was left there.

It reminds me of this old image that you'd see on the internet where it's got this old eight or ten port hub sitting there, covered in cobwebs, with all this infrastructure connected to it. That was supposed to be a temporary fix.

What's the real warning in all of this?

I know it gets scary when there's a lot of politics being thrown around about who is attacking who, and whether it's really an attack from that person, or a false flag where it shows as being attacked from one country but it's actually another country, or it's internal to the United States launching the attack. There's a lot of politics involved in this. You really have to look at where those vulnerabilities are and what you're setting yourself up for.

Because if it's a true attack, a true attack on something isn't going to just change a password and change IP addresses. An attack is going to decimate the infrastructure. It's going to make it completely unusable. It's going to knock it out and make sure something terrible happens, and this is just the warning sign of that.

This could be some script kiddie out there running a script to compromise something that's a known vulnerability through a CVE. The press reporting it as a hack again may not be truthful, and they might try to wrap some politics all around it, but it is a huge shot across the bow. People should look at this, because if you can use a simple attack to take down infrastructure in a community, imagine what the real threat is. Imagine what a true bad actor can do and the damage that they can cause.

Attacks like this are a shot across the bow for communities across the United States. Something simple like this means they can be compromised in a matter of minutes on a public network. Imagine what a real bad actor could do and the destruction they could cause with a concerted, focused effort.

Do this

• Pull the carrier inventory first, not the network scan. Get the SIM list with assigned IPs out of the carrier portal, and find out who holds the account, because it's frequently the system integrator and not the utility.
• Cross reference that SIM list against accounting records and look for recurring static IP add-on line items, since devices you can't see are devices nobody has patched.
• Eliminate public reachability by moving to a private APN with an IPsec tunnel or CGNAT-only SIMs, and where a public IP is unavoidable, put the PLC behind a firewall or secure gateway so it is never directly exposed.
• Disable every unused radio on the field cellular gateway, because many of those routers still ship with Wi-Fi enabled and configured at commissioning, and those have been up for years.
• Assume the controller is lying, and compare running project files, ladder logic, and add-on instructions against known good baselines before you trust anything.
• Rotate all PLC and HMI passwords, eliminate defaults everywhere, log cellular modem connections, and verify that your manual fallback procedures actually work with the staff you have on shift.

Questions I keep getting

What should MicroLogix operators do first? CISA recommends disconnecting PLCs from direct internet exposure, routing remote access through a VPN or gateway, changing default passwords, allowlisting trusted IP addresses, and keeping a known clean backup of the PLC image. MicroLogix 1400 operators should also follow Rockwell's password-recovery guidance linked from the CISA advisory.

How do I know if my equipment is already advertising itself? Tools like Shodan and Censys let you look yourself up. In Censys you can look up your IP block and put in an exclusion request. Other tools like ZoomEye and FOFA let anybody punch in blocks of IPv4 addresses and see what's publicly reporting.

Can't we just take the controllers off the internet? The larger the city, the more PLCs there are, so you can't just take everything off the internet. That sounds like an easy out. Secure it end to end instead, run the audits, and keep the security posture strong.

Was this Iran? Nobody has attributed this week's activity, and there's a lot of politics being thrown around about who is attacking who. My point is that the attribution fight doesn't change the fix list on your desk Monday morning.

Want this kind of thing weekly, in audio? That's the show.