The ICS Perimeter – A Line in the Sand

Hey all,

This is going to be a non-technical post, so if you’re looking for some sort of tool or walkthrough this post probably isn’t for you. I’m going to make an argument that the perimeter for Industrial Control Systems (ICS) is one of if not the most important security control. I understand that some might see this as antiquated way of thinking and are probably calling me a dinosaur. My response is to read below and that I think of myself more as a crustacean from the Proterozoic era looking for about tree-fiddy.

I have the opportunity to do offensive and defensive work for Critical Infrastructure. Critical Infrastructure (Utilities, Generation, Transmission, O&G, etc.) is an interesting industry from a cybersecurity standpoint. In the Enterprise space, cybersecurity is often viewed as risk mitigation. For example, our organization makes $1B dollars, the consequences of a cyber-incident is ABC, our threats are XYZ; therefore, we’re going to spend $1M dollars. Just like in the Enterprise space, Critical Infrastructure sites vastly vary in size and revenue. I’ve seen sites with one person handling all IT administration and cybersecurity to sites with large fully financed security teams. Regardless of size, staffing, and solutions implemented; Critical Infrastructure usually has one thing in common – the extreme consequence of a security incident can result in loss of life.

What is my point? Resources for cybersecurity at Critical Infrastructure can be limited while the consequences are severe. Why is that important? Well I imagine it could be overwhelming trying to secure Critical Infrastructure. Also, the next thing I’m going to say might ruffle some feathers so I wanted to give a bit of context.

The Industrial Control Systems (ICS) perimeter is one of the most important security controls when it comes to Critical Infrastructure … I can hear your collective moan from here. You say – “Nearly every security professional has adopted the ‘assume breach’ mantra, you’re taking us back years!”, my reply is to finish this paragraph. I understand why one needs to assume breach in the Enterprise space. The internet is a cesspool with exploits flying around, nasty emails, C2 traffic, etc. Enterprise infosec has to deal with thousands of end users reaching out to the internet downloading junk and having junk land in their inboxes each with an itchy trigger finger. Your perimeter will be breached and it will be breached often. However, what if I told you, that in order to breach your perimeter you had to come through one of five endpoints, welcome to Critical Infrastructure networks.

When performing Critical Infrastructure pentests, we often spend a great deal of time staging our perimeter breach on the Enterprise network. However, once we successfully breach the ICS perimeter we often go from domain user to domain admin in a very short amount of time. Additionally, this will often land us in the ICS DMZ which is good-ish, but usually we have everything we need in the DMZ to gain access to the most critical networks. DMZs have a tendency to be a security administration zone for Critical Infrastructure networks instead of semi-trusted zone. We’ll often see domain spanning and trusts, NMS, AV management, and virtual management in the DMZ. Long story short, an engagement or attack moves very quickly once the ICS perimeter is breached.

What is the problem? Well often at sites I speak to qualified infosec professionals that are tasked with applying security policies and standards to the lowest, most critical tiers of the network – vuln scanning controllers, endpoint hardening, ensuring SSH is enabled on all network equipment, etc. I sometimes semi-joke “I don’t care if your switch uses telnet, as long as it cannot route to other networks spend your security hours elsewhere.” I know, it’s an exaggeration, encrypting traffic is important; however, if an attacker is able to communicate with the non-routable telnet enabled switch, consider how far he/she has come. They have established a foothold on the Enterprise network, exploited and stalked ICS users on the Enterprise network, breached the ICS perimeter, worked their way through the ICS DMZ, and finally escaped and landed on the most critical networks. I guarantee that an attacker has everything he/she needs to authenticate to that switch without snooping on traffic in-transit.

You say – “I hear a lot of complaining and not a lot of suggestions.” If the time and resources are there to assume breach, do it. However, if you’re a site with restricted resources start with focusing on the perimeter. Here are some perimeter themed security controls that come to mind:

• Inventory the required dataflows between Enterprise and ICS DMZ networks. If you can reduce these by having users establish an interactive session before connecting to remote service, do so. For example, have users authenticate and connect to an ICS jumphost before connecting to the ICS DMZ web service.
• Ensure all required dataflows match firewall rules – layer 4 is better than layer 3, layer 7 filtering if possible. Egress filtering is nearly as important as ingress filtering. If an attacker hijacks an interactive session he/she will be most likely looking to establish a reverse shell session to get away from the hijacked session. Egress filtering will make this difficult and often little to none egress ports are required for normal operation. Additionally, any attempted egress dataflow to the internet needs to be restricted and fully investigated, this could be an attacker slipping up.
• If any of the dataflows have implemented some sort of shared authentication (domain spanning, domain trusts, local authentication w/ shared passwords) with the Enterprise network, remove these immediately.
• Multifactor authentication at the perimeter significantly increases the security posture as it reduces the accessibility of the credential theft avenue. Take it to next level and use a hard-token to establish a VPN session which has split-tunnel disabled.
• Understand each of the Enterprise to ICS dataflows. Document and distinguish interactive from non-interactive dataflows. Deep dive attack scenarios on each type. Session hijacking, credential theft, and token theft for interactive sessions. Application vulnerabilities for non-interactive sessions. Assess the privilege level of hypothetical exploitation and try to claw that back if possible. For example, restrict local administrators or privileged domain accounts from authenticating to the jumphost.
• Detection. Given enough time banging on the perimeter an attacker will eventually breach. However, considering that the scope is so small, this is probably our best chance at detecting a remote attacker. An IDS at the perimeter is a must. Decryption and inspection of all egress dataflows is also a must … very little in the way of privacy concerns at the ICS perimeter. Session tracking is very possible, modest sized sites might only have 100 or so authentications per day.
• Incident Response. Whatever it looks like, I recommend including a logical and a physical kill switch. Logical meaning a super restrictive ACL that restricts pretty well everything. Physical meaning pulling the plug to the Enterprise network.
• Administration of ICS endpoints accessible to the Enterprise should originate from the ICS environment. Treat each session into the ICS environment as potentially hostile. This includes agent and management applications. Do NOT manage your ICS network equipment from the Enterprise network. Do NOT use security solutions on the Enterprise network to manage agents on ICS systems.
• Application Whitelisting. Deploying an enforced and secure application whitelisting configuration to an entire environment can be difficult. Deploying it to Enterprise accessible ICS systems is much easier and a must in my opinion. Consider that each of those ICS systems rarely change and are traditionally servers not workstations … a tailored made environment for AWL.
• HIDS. Similar to AWL, these systems rarely change. If a service or scheduled task is created, if an account is created, if system files are being modified; you need to know about it.
• Advanced Endpoint Solutions. Perhaps management is shying away from deploying advanced endpoint solutions due to some of those legacy systems on the lower tiers of the network. That’s fine, roll it out to the Enterprise accessible systems only.
• Change Control. Any change to the perimeter or Enterprise accessible ICS systems need to be thoroughly vetted for accidental security holes.
• And finally assess the perimeter with periodic vulnerability scanning and penetration testing.

At the end of the day, I’m not advocating that you neglect the rest of your ICS network including the most critical sections. My opinion is that if you’re a person or group of persons with restricted resources, I think you’re best starting at the perimeter.