It Must Be Something In The Water

An (very) informal retrospective analysis of the Attack on the Water Treatment Facility of the City of Oldsmar Florida.

DISCLAIMER: IN ABSOLUTELY NO WAY, SHAPE, OR FORM, AM I INVOLVED WITH ANY CURRENT OR ONGOING INVESTIGATION. I’m just a Cyberphysical System Integration Security (CSIS) Subject Matter Expert giving his professional take & opinion based on what can be arguably considered as conjecture. So please take this all with a grain of salt. But do know this: This is the kind of shit I live for, to solve problems like this in an effort to protect people from any sort of cyberphysical harm. I also never ever approach anything under the mindset that I’m 100% correct no matter what. So please feel free to eviscerate my analysis. Also, nothing in this article shall be misconstrued as any sort of attack or downplay on the City of Oldsmar. They are simply the victim in this scenario, so any part of this analysis MUST be read in an objective light. I am simply seeking either to educate or spark interest in this specific case and perhaps the broader topic of OT security as a whole. There are many cities in the US that are just like Oldsmar, and we need to ensure all areas of our nation’s infrastructure is secure. With all of this said, under absolutely NO circumstance does any part of my analysis seek to blame this city in ANY way at all.

In response to the attack on the Water Treatment Facility of the City of Oldsmar Florida, I figured I’d read a couple of the popular articles from Bleeping Computer and The Hacker News, and give my analysis of the prevailing narratives with my own knowledge and experience as a Cybersecurity Professional with an expertise in Cyberphysical Systems (any technical system with a physical impact).

A recap of events (based on facts presented in aforementioned articles):

  • Attack occurred at the City of Oldsmar’s Water Treatment Facility, in Oldsmar Florida (Pinellas County).
  • Attack occurred twice on Friday, February 5th (2021).
    • Once In The Morning: between 8:00 – 8:05 p.m. EST. (estimated based on details provided in Bleeping Computer article).
    • Again In The Afternoon: between 1:30 – 1:35 p.m. EST. (estimated based on details provided in Bleeping Computer article).
  • The morning occurrence increased the amount of sodium hydroxide in the water supply from 100 parts-per-million to 11,100 parts-per-million.
  • The afternoon occurrence entailed a second remote user opening/checking functions in the system that control the amount of sodium hydroxide in the water.
  • TeamViewer was leveraged to remotely access target system.
  • A plant operator intervened and immediately reduced the level back, doing so early enough so that there was no significant adverse effect to the city’s water supply.

Operational Assumptions (based on inferences from aforementioned articles, and my own personal artistic license) :

  • The SCADA software was compromised via an attacker interfacing with it via an adjacent Level 3 asset (Perdue Model), allowing attacker to control the amount of sodium hydroxide present by accessing said software.
  • The attack originated from outside of the facility itself.
  • Multiple TeamViewer user accounts were compromised (at least 2).
  • The attack wasn’t made apparent until it observed the second time (1:30-1:35 p.m. EST), when plant operator visually tracked/observed attacker accessing certain control functions in target system via on-screen monitoring visually.

Now that we have the playing field set, let’s think and put on our detective-caps on see if we can better understand the narrative here, and even maybe even improve upon it.

Who – Whodunnit?!

Due to the seemingly unsophisticated nature of the attack, anyone’s first thought might be to just write this off as someone with too much time on their hands, trolling Shodan for TeamViewer ports open and exposed on the internet. This very well might be the case, but it’s always good to be more thorough than the semblance of “Good Enough” many lazily subscribe to. I (could be VERY wrong, but I) believe the nature of the attack to be a bit more targeted than the general reader of such popular articles might be led to believe. Though there isn’t much specifically stated in these articles that would lead you to this conclusion, I often take the approach of reading between the lines on stuff this. So let’s examine a couple of things here:

  1. The Attacker’s Expediency in the environment is indicated to me by how much time it took to enter, discover the functions necessary to control sodium hydroxide amounts, and enact those changes, and then get out. It took 3-5 minutes. If I’m working a blackbox assessment on any kind (for a client), it’s taking me longer than 3-5 minutes to find my objective, or anything interesting. Granted, this is an anecdotal point after all. But my sense is that the attacker knew the environment he was attacking, or at least the SCADA software that sat on the other side of the remote-access session.
  2. Leveraging TeamViewer as the attack-vector initially leads me to one of 2 conclusions here. First being that it’s possibly an Advanced Persistent Threat (APT) group. So if we entertain this to it’s logical conclusion, one can surmise that the Tactics, Techniques & Procedures (TTPs) present in this attack can be identified in our handy MITRE ATT&CK Navigator (tuned to ICS of course) and correlated to any number of potential APT groups. When I input (what I believe to be) the apparent TTPs into the navigator, it looks something like this (though this isn’t quite 100%, as we can argue the semantics of what constitutes some of these items of course).

In this exercise I was only able to correlate (some of) the observed TTPs to a few relevant APT groups (listed below) that are known to target Industrial Control Systems (ICS). However nothing matched exactly. But below is a list of what I consider to be relatively the most relevant groups. But I implore you check the MITRE ATT&CK resources regarding the differing APT groups so you can make the analysis yourself, and possible produce some differing results.

So I am not saying with a large sense of certainty that this was the work of a known APT Group, in fact my initial analysis says it likely isn’t. However, as I do NOT have 100% of the information (only what the rest of the public knows), I won’t assert anything to that level/degree. But if I were to compare with known APT groups, the above 3 is where I’d end up (before further investigation).

Now, the relation between TeamViewer and any known APT group is slim. I was only able to find limited information regarding FireEye’s Soiree with APT41, even though this was an attack on TeamViewer itself, and not the end-user directly (but end user compromise was the goal. So essentially a supply-chain attack…sorta).

Even thinking outside of trying to map this to an APT group, known exploits for TeamViewer aren’t plentiful. TeamViewer has only 4 CVEs in total to date, and they are not exactly a walk in the park to implement. So there might be a higher technical acumen present in this attack.

If I had to characterize who might’ve performed the attack, I’d state the following, based on what I know:

  • Unknown APT Group: I’m unsure how likely this is, but let’s consider it a catch-all category for now.
  • Insider Attack: Someone with access to multiple TeamViewer user accounts and a basic knowledge of the target’s OS & SCADA software can achieve this in the time it apparently took to execute the action on objective.
  • Some jackass: Anyone who’s a jackass. I suppose this is the category that’s hard to define, but is likely someone who wanted to be a dick to a city of 15,000.

I know this isn’t quite conclusive, however in regards to “Whodunnit”, I don’t think there will be a concrete answer to this. Keep in mind too that there were reportedly 2 user accounts observed in these attacks, so you can look at it like there are 2 different attackers using separately compromised Teamviewer user accounts, OR look at it like 1 attacker with 2 different TeamViewer accounts. But likely someone will scrounge up some artifacts like a DNS query or an IP Address which hails from somewhere like Russia, or China, and some knucklehead is gonna forget VPNs and proxies are things that exist and rush to tell media that our enemies are closing in on us. Idk, just use your heads on this one, and take nothing at face value.

What – What the phuck just happened?

2 Separate TeamViewer User Accounts were used to compromise the safety of a water treatment facility, by chemically altering the potable water to a dangerous degree, making it extremely harmful for human consumption. Below is a high-level diagram of what had reportedly occurred.

The impact of this attack was potentially poisoning the water supply for a population of approximately 15,000 citizens. However, in multiple articles and YouTube press release video it was stated that the City of Oldsmar had safety controls in place to ensure that the hazardous water would never make it to the resident’s homes. I am inclined to believe these statements. Though municipalities might not all have Cybersecurity expertise, however one thing these asset owners understand extremely well is safety controls. And I would take that to the bank. Also, the plant operator and related staff possessed a vigil that allowed them to spot and quash this treat almost immediately.

When – When are we?

In the section titled “A recap of events” I stated that the time frames of this attack occurred between the times of 8:00-8:05 a.m. EST and 1:30-1:35 EST on February 5th (2021). In the aforementioned articles I referenced in the earlier portion of this piece state that it took the attacker(s) 3-5 minutes to complete the attack(s), so I applied that same timeline to both of the attacks that occurred in the morning and the afternoon. This might not be exact, but I still believe this paints a relatively decent picture of when this occurred.

Also, I am unaware of any reasoning or significance behind the particular date of the attack. I am ignorant to any relevance this would have to the Oldsmar’s Water Treatment Facility unfortunately. Perhaps the attacker(s) was hoping that staff would be apt to leave in preparation to SuperBowl weekend, and be lax in their daily duties, but this is pure speculation. I have no idea regarding why the attack might’ve taken place on this date and time.

Where – Where is the beef?

The attack took place at the Water Treatment Plant in Oldsmar, Florida (Pinellas County).

I’m unsure if this facility was targeted for any reason, or if it was simply low-hanging fruit and this was just a crime of opportunity. It doesn’t quite strike me as the most appetizing target. Though this is not a major metropolitan area (that I know of), it is a target that’d have a high enough impact. But as to the choice of city, perhaps there is possibly some nuanced history I am unaware of. I don’t have all that much info to add in here. I leave this to the experts of this location to fill in any blanks.

Why – Tell me why; Ain’t nothin’ but a Heartache

Why the Oldsmar Water Treatment Plant was chosen as target of this attack remains a mystery to me as of now. However a few arguments might include:

  • Too easy a target to avoid. a.k.a crime of opportunity.
  • An effort by a larger entity to show that “no target or city is too small to target” regarding dangers to our infrastructure.
  • Perhaps an insider attack, or someone with intimate knowledge of the facility’s operations.

I don’t think this one will be as easy to pin down without more information. So when any more information is released, this section is subject to change.

How – How Will I Know

All articles I’ve read so far point to TeamViewer as the attack vector in this case. When it comes to remote access of ICS, typically there are very stringent policies in place to govern the time frames in which maintenance personnel, integrators, and third-party technicians can access target assets, and what changes they can actually affect in their remote-access sessions. From the information gathered from varying articles, it’d stand to reason that these practices weren’t in place at the time of the attack. In fact I’m even more apt to believe this due to the Pinnellas County Sherriff in the YouTube Press Release video that the employee who spotted this at first “didn’t think much of it” since his supervisors and others would remotely access the workstation (assuming this is the same workstation that is compromised).

So a lack of remote access policy combined with 2 separate TeamViewer user sessions seems to be how attacker(s) “got in”. Now regarding how attacker(s) compromised the TeamViewer sessions seems to be a trickier topic. I had mentioned earlier that it’s no walk in the park and referenced the 4 CVEs present for TeamViewer. Occam’s razor (or the Law of Parsimony) dictates that the simplest solution is often the right one, and in this case I’d postulate the possibility of the attacker having compromised the actual machines of the 2 TeamViewer users who’s accounts were used in both the morning and afternoon attack occurrences. In the past, attackers have performed supply-chain style attacks where TeamViewer agent downloads were maliciously altered to allow attacker(s) to access a myriad of remote access sessions. So perhaps this could’ve been the result of a bad copy of TeamViewer, perhaps disseminated via a successful phishing attack on the users in question.

Much of this is theory, serving to tickle the parts of your brain that identifies with a Cyber-Sherlock Holmes of sorts. So again, take this section (and all others present) as what it is, theory.

Recommendations:

Some of my general recommendations to the City of Oldsmar.

  1. Get a Gap Assessment done ASAP. This event should serve as a call to action to do this, as it’s evidently paramount to grasp knowledge in the areas that need shoring up.
  2. Implement granular policies on your remote access. I.e. strict timelines on when outside personnel can access target assets, and what exactly they can perform on the target system.
  3. Monitoring of network choke points and remote access points. Mainly to ascertain validity of user access relative to their respective location, etc.
  4. Deploy Host Intrusion Detection System agents in each employee computer that requires access to remote assets via TeamViewer. Ideally these machines/laptops/PCs would be owned by the city.
  5. Go Back to Basics, and review the Perdue Model as a guideline to effectively model the industrial network.
  6. Bring on security personnel, or bring in a 3rd party security partner that is experienced in Operation Technology (OT) to ensure your security integrity is above-board at all times, and that will supplement your mission of safety & uptime 100% of the time.
  7. Stay away from silver bullet products. Any security product or offering claiming to be your knight in shining armor is likely after your wallet. Look for purpose built solutions that don’t hinder your ability to operate, and that can evolve with you and your operational needs.
  8. Read the Intro to CSIS.

Final Thoughts

As our world becomes more connected, so the possibility increases that this connectivity will be leveraged against innocent civilians with a sense ill-will, in effort to do wrong. It’s important that our efforts to secure infrastructure are done with a mindset of safety and security first, and we work together to actualize this goal. Too often these efforts are vendor and solution driven when the situation calls for a much more objective vantage point and a holistic touch.

Make Python go Vrooooom!

I’ve found myself in scenarios recently where I need to write portable apps & modules in Python, and disseminate them to varying devices. Through some very well crafted and quite elegant Google search queries, I came upon an interesting StackOverflow answer regarding Cython. The aforementioned answer covers how to (in Ubuntu) convert your Python file to a C file, and then compile it with gcc. While I am championing this method as a cool way to create efficient & portable modules to send off to your device fleet, there is some nuance to consider. In this article we’ll cover that so it’s very easy for you to manage expectations here.

Converting & Compiling

I typically like to use docker containers as miniature forges of sorts for building C code and compiling for specific environments. In my opinion it beats standing up VMs all the time, working strictly within the constraints of your bare-metal server or workstation and having to reconfigure your environment multiple times for varying compiles.

Below is a high level view of the conversion and compilation flow.

High-level flow of conversion and compilation of your Python file

Below screenshot video of me running the commands laid out in the aforementioned StackOverflow answer.

Screenshot video of converting and compiling from Python to C, and testing compiled binary file.

Requirements to Run

Requirements to run compiled code outside of compiled environment (remember, we’re talking about code that’s to be distributed):

  1. Ubuntu – This is a loose requirement, but all of my personal testing and has taken place with the newer versions of Ubuntu ( >= 16.04).
  2. libpythonX.X – This should reflect the version of python your compiling device uses. So if you’re compiled this with Python3.8 in Ubuntu you can install this with sudo apt-get install -y libpython3.8.
  3. All Python3 libraries that were imported in the original file. You can obtain these via pip install <package-name>, or building them from source.

Once all is said and done, I’d likely coalesce everything into a .deb file for easy install. But I need to stop the article here, as you can go so many places from here. So hopefully this was of some help to whomever may be reading. If you have any questions, feel free to hit up my contact page.

Go Fuzz Yourself! Where do I even begin?!

For those who are unfamiliar with the term or practice of Fuzz Testing, it’s essentially the combination of testing input validation coupled with stress testing, which is performed by spraying inputs of a target device, application, or system with random and crafted input at high velocity and observing the effects on the target. Use cases can involve testing input forms on a website to ensure it doesn’t accept foreign or script characters or ensuring that IIoT devices are programmed to be functionally robust.

I’ve performed fuzz testing for many web & mobile applications, and even industrial networks and devices. There are common denominators amongst fuzz tests of different technology or hardware stacks that map to higher level processes that translate well across the board and can be universally applied. Many fuzzing frameworks claim to be unified, but are only such when employed in their respective technology silo(s). As someone who works simultaneously in very low and high levels of varying engagements, I tend never to work, research, or experiment in a vacuum. And so the thought of having to find and apply fuzzing frameworks a la carte aggravates the living shit out of me, as It’s an inefficient and misapplied expenditure of effort (i.e. a waste of time).

I wanted to provide something of value that others who need to perform fuzz testing of anything can use from front-to-end, and even build off + flesh out as seen fit by those planning and/or testing. I just wanted to make your life easier. Enjoy!!