Why did the pipeline get shut down if the hackers only attacked the business end? 

For each column in this series, rAVe writer Paul Konikowski takes a deeper dive into a recent security event or data breach, shedding light on supply chain vulnerabilities, infrastructure and cyber-physical security.

By now, most people heard about the Colonial Pipeline ransomware attack. Many readers (and North Carolina-based rAVe [PUBS] employees) were directly affected by the resulting gasoline shortage.

On May 7, 2021, the DarkSide ransomware locked up the “business end” of the pipeline, shutting down office computers, file servers and phone systems. According to statements put out by Colonial, the attack did not directly impact the pipeline operations, but the company decided to shut down the pipeline. Colonial likely did this for two reasons: One, as a precaution, to make sure the ransomware didn’t spread any further; two, without the business office computer system, they could not bill customers for the oil flowing through the pipeline. The company quickly paid a ransom of almost five million dollars to get the systems unlocked a few hours after the attack but took almost a week to get the pipeline fully operational. This raised a lot of questions and resulted in the White House issuing an Executive Order.

It is standard practice for a utility or energy provider to keep its business networks (Information Technology, or IT) mostly segregated from its operations networks (often dubbed Operations Technology, or OT), and with good reason. Front-end office workers surf the web, exchange emails with third parties, bring in their own devices (BYOD) and sometimes work from home or a coffee shop. Their computer programs are designed for ease of use; security, privacy and resource access is based on the end user. Business IT networks are used for videoconferencing, faxes, phone calls, emails and collaboration. The IT networks also enable accounts payable, warehouse receiving, payroll and human resources. It can be a nuisance if a user needs to reboot their computer or if the IT team needs to perform server maintenance. But the impact is still relatively minor. People reboot stuff all the time.

Conversely, operations technology (OT) networks and computers are typically purpose-built or configured only for the task they perform. They are designed for safety, efficiency and to keep operations up and running with minimal downtime. Programmable Logic Controllers (PLCs) often run one program over and over, 24/7/365. SCADA systems collect industrial data. Other computers are set up for safety and work as detection systems. They keep an eye on the system status and alert the operations workers if there is a problem, and sometimes they shut down the operation and/or bring it to a safe state so that no one gets injured and/or killed. A reboot of any system could take days and cost a company millions of dollars.

This separation of IT and OT is analogous to an integrated audiovisual system that includes a control subnet, Dante or an AV-over-IP network. Those networks are usually implemented using multiple Ethernet switches or VLANs. Add in some old-school RS-232 serial connections (or proprietary bus networks like Crestnet or AXLINK), and the analogy becomes even closer to industrial control systems and utility networks.

Most industrial control networks were built to be efficient first and secure second.  Any safety measures that are in place are not cybersecurity controls. Do not mistake “safety” for “security” — they are not the same! It is like vehicle safety devices; seatbelts and airbags save lives, but they don’t prevent carjacking!

The traditional approach has been to keep the business networks and OT networks segregated. Over time, that separation has blurred because the business folks want to know the operations numbers, and the operations people need to check their emails and do their timecards on the business networks. The firewalls between these networks are good, but not perfect, especially when they are not configured correctly. So, when Colonial decided to shut down the pipeline, stopping the data traffic from flowing between the OT network and the business IT networks, that was a smart move in this writer’s opinion. Instead of being mad at them for shutting down the pipeline, we really should be applauding them for it.

Similarly, I still believe that AV networks and devices should be completely separate from local or wide area networks, but that only goes so far. Videoconferencing and remote monitoring typically require internet access. Also, don’t just assume that if it is segregated, it is secure. One of the tenets of the emerging Zero Trust approach is to assume that all networks, including AV, have been breached.

There’s a lot more to Zero Trust. What I suggest is to do your best to keep the networks segregated, not just using VLANs. Manufacturers and integrators should learn to use the principles of “Least Privilege (or Use)” and “Least Route.” “Least Privilege” means that a certain user or device should only be given the minimum access required to do their job. “Least Route” addresses the actual wire connections.

Audiovisual devices (and people) should only receive the minimum level of network access required to do their jobs.  The network switches and routers should be configured using access control lists (ACLs) so that only specific, authorized devices can send traffic inside or outside of the network. VLANs are not an effective security measure [See Figures 1 and 2 for a simple example of implementing Least Route].

Stay tuned for Part 2 of my deep dive into the Colonial Pipeline cyberattack, where I discuss what motivates cyberattackers. Spoiler alert: It’s usually money, but we’ll talk about other ways that cybercriminals are motivated including politics, activism, vengeance, mental illness or even blackmail.