Proactive Threat Hunting – no longer a whim

Originally posted in 2018.

We are undoubtedly in the era of huge security alert fatigue. This has been caused by the vast number of false positive alerts generated every day by countless security products that organizations put in place to improve their defences. Because of this, it’s hard to justify resources who would essentially focus on… Producing even more alerts instead of reacting to the current ones. Those who do, however, want to invest in a second layer of more fine-grained detection often steer their organization towards proactive threat hunting activities. The ultimate goal of the threat hunting process is to find malicious actors already present in the environment who have the intent, capability and opportunity to cause harm.

This is accomplished in parallel to and in cooperation with other detection systems and methods like Anti-Virus (AV), Host Intrusion Prevention System (HIPS), Endpoint Detection and Response (EDR), Intrusion Prevention System (IPS), Security, Security Information and Event management (SIEM), Advanced Threat Defense (ATD), etc. Successful threat hunting activity should provide at least three visible effects:

1. Security Incidents being reported only for identified and properly scoped intrusions. Reducing false positives.
2. High quality threat intelligence combined with Indicators of Compromise that can be utilized by detection and remediation tools being created. This intelligence can’t be fully replaced by third party feeds. Security savvy organizations in general do not share intelligence about what they believe are the active Advanced Persistent Threats attackers (at least not as soon as they have it). Doing so would mean losing the leverage they’ve gained.
3. Gradually improving the automation of hunting and detection capabilities.

“Manual” threat hunting is also available to supplement and utilize existing security monitoring mechanisms and not to replace them. Two good use cases are:

1. Finding previously unseen threats (including not only malware but also Tactics, Techniques and Procedures used by adversaries).
2. Better scoping and understanding of events that manifest themselves in some manner, like IPS or AV detection, but in many cases are ignored and considered as successfully remediated when in fact they could be some tiny crumbs dropped by a much more serious intrusion.

What is needed

In short – visibility.

The common issue organizations face with security incident detection and response is that they focus on the Interruption step (the numerous alerts) without assuring mature Monitoring processes (to ensure all alerts are necessary). Large sums of money are invested in tooling but there is no resource allocated to operating and using these tools beyond responding to automatically detected events – real or unreal. This results in a fractional understanding of any given threat’s real scope and leaves organizations chasing their own tails when it comes to fighting off Advanced Persistent Threats.

Figure 1: Visibility is crucial to early response. It allows an organization to transform from a reactive to hunting approach. The later in the kill chain the bigger the risk, costs and effort.

Threat hunting is not a recent invention. It’s been there in various forms for many years. What is new is the mindset and approach that needs to be applied. Setting up a successful threat hunting process requires:

1. Dedicating full time people to this in the same way as is done in the case of Security Operation Center analysts, responders etc. These people should not be responsible for Incident Response (IR) process.
2. Integrating with existing detection methods.
3. Some additional tools that are rarely deployed will be needed like threat intelligence exchange framework, data analytics solutions, network traffic recording and analysis capability, enterprise scale endpoint visibility.
4. Baselines need to be created and constantly updated to understand ‘the normal’.
5. Building a solid understanding about the protected environment.
6. Defining realistic goals. Threat hunters can’t be required to find X intrusions in a month or they will focus on trivial things.

A possible simplified process could look like the following:

Figure 2:  A simplified  threat hunting process

1. Analytics is used to help automate finding of known indicators by applying Threat Intelligence (TI) feeds on enterprise wide gathered data. A TI platform can be implemented for example with use of MISP.
2. Threat hunter works with analytics to deep dive on automated findings but also to look for events that look out of ordinary to them.
3. Threat hunter uses TI platform to get more context on TI recorded there and to submit new TI information coming from their findings.
4. Findings that match the criteria of a security incident are forwarded to the Responder for IR process.
5. Responder applies additional context to findings and performs IR process.
6. Additional findings gathered during the IR process get recorded in the TI Platform to close the loop.

Practical Use Cases

Don’t ignore your Anti-Virus (AV) logs

Many treat their AV as an install and forget kind of solution – the simplest mechanism out there to get rid of the known bad. That is a mistake. While it’s hard to expect fireworks from traditional AV when it comes to stopping targeted attacks, it’s important to understand that even state sponsored actors make mistakes or get lazy. It isn’t uncommon to see, especially in the later stages of attack, known malicious tools being dropped on compromised systems alongside the more customized ones. By analyzing those dropped crumbs it is sometimes possible to find a clue to a much more significant compromise.

So, what should organizations focus on if they face hundreds of blocked infections daily? Here are some good places to start:

• Seeing malware removed from directories like C:\Windows\* or C:\ root catalog is never a good sign. It means that a process with local admin or system privileges tried to put it there which usually means that the system was successfully exploited.
• Look for threat names or create a list of threat categories that are more interesting or rare. Each AV product maintains some taxonomy that describes threats matched by their signatures. While looking at coin miner scripts blocked in temporary internet files might be very time-consuming, looking through things classified as HackTools, PWStealers, Trojans may give better and faster results.
• Differentiate between type of scan blocking the threat. Threat blocked by an on access scanner in a user’s temporary directory will often mean the file was blocked before infection could be successful while a threat being removed by a scheduled scan will often mean that the infection could have been active for some time before it started to be recognized by the AV and hence the impact is different.

Monitor PowerShell usage

Over the years PowerShell evolved to become a very powerful and easy to use scripting language used by administrators to manage Windows environments. Attackers realized that all they need to accomplish their goals is already provided by this native, built-in Windows tool. It can’t be blocked by AV and it can be used to manage remote computers. Without proper restrictions and logging defined it also stays below the radar as essentially no files need to be created to run PowerShell commands. This means that even if attackers’ techniques are identified it can be very challenging to identify all impacted systems. Generally, a good approach is to first create filters over a few days from known good activity in the monitored environment. After this, what is left can be reviewed more closely. It is always a good idea to have a look at PowerShell invocations that are:

• Executed with the hidden parameter like powershell.exe -nop -w hidden
• With base64 encoded command or/and obfuscated like:

powershell -NoP -E W3N0cmluZ10kbWU9R2V0LUNvbnRlbnRDOlxXaW5kb3dzXHRlbXBcZnNmc2Rhcy50bXANCg0KSW52b2tlLUV4cHJlc3Npb24kbWUNCg0KJG09Y29udnl5cXRkNjdxd2RnZw0KDQpJbnZva2UtRXhwcmVzc2lvbiRt

• Referencing environmental variables or registry entries to get the code from and hence facilitate file-less attack like:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NonI -W hidden -c "IEX ([Text.Encoding]::UNICODE.GetString([Convert]::FromBase64String((gp HKLM:\Software\Microsoft\Bing devenv).devenv)))"

PowerShell commands can be monitored even using the native Windows event logging with proper auditing options configured. If collected centrally, a good data analytics tool can prove to be a great working base for threat hunters. Good article on using PowerShell by Blue Teams can be found for reference here.

Obviously, tools exist for example on the Endpoint Detection & Response market that will focus on detecting suspicious PowerShell actions directly on the endpoints further automating detection and prevention capabilities.

Set up traps

There is nothing more ‘rewarding’ than seeing an attempt to use a domain admin account that was planted as a decoy. Or seeing someone trying to use stolen credentials to authenticate to this decoy file server named FINANCE_OLNY. One possible way to create decoy credentials can be found in this article. When it comes to setting up honey traps, the sky is the limit here. There are some free open source as well as commercial solutions on the market. Alternatively some may find it interesting to define a percentage of their infrastructure on which to deploy high visibility monitoring on the network and endpoints level. While for many organizations this is too expensive or not practical to deploy everywhere, then smartly divided locations can provide indications that something is wrong for a fraction of the cost and effort. The good use case here is to look for lateral movement attempts targeted at the monitored assets.

Finally, there is nothing wrong in grabbing a set of known IoC (Indicators of Compromise) from public sources for example utilizing MISP as a central repository and automating checks for selected sets in your environment.

Starting with the known bad sooner or later ends with finding the new and unique threats in the monitored environment and turns the most mature organizations from Threat Intelligence consumers to Threat Intelligence producers.