Hunting for Chains

In my last blog post I touched on something that I want to talk more about.  I think far too often, when hunting, we can focus on single indications for signs of something being bad.  Intrusions are far more than a single event though, so why spend all of your time hunting that way?  Often the actions that an attacker will take will be in short time bursts.  These bursts may only last for a few seconds to a few minutes.  Looking for multiple events within these windows of time, I feel, can be a very effective way to spend your time.

One of the things I try and do with my blog is to share how I think about the work that I do.  I have gained so much knowledge from others over the years and feel strongly about giving back.  Lately I have been thinking a lot about , what I like to call, behavior chains.  Basically these are the actions that an attacker would likely take to accomplish their goal.  Each action is a link and the sum of actions make up the chain.  A link can be something as simple as a process name like net.exe or cmd.exe (these are probably 2 of the most commonly spawned processes and attackers definitely take advantage of these).  A link can also be something as complex as a correlation search that surfaces a specific behavior (see my last blog post for some of these correlations).  So why do I care about looking at hunting this way? There are multiple reasons, but a few of them are to model how attackers may operate, data reduction and higher fidelity output.

Lets say that we want to look for odd network logon behavior because we know that when an attacker is active in our environment they will generate anomalous 4624 type 3 events.  The problem with this is that there is likely thousands upon thousands, if not millions, of these events per day.  We could try and reduce this data by looking for rare occurrences of source machine, dest machine, user or a combination of the three. This will likely produce anomalies, but will also take time to investigate each one.  Now take that time and multiply it by the number of days in a week.  How much time are you spending looking at these events and are the results beneficial?

I think the question to ask is what are the things an attacker would do that would generate these logon events.  We know that just by the meaning of the event that it was the result of an action taken by a remote machine and we probably care more about the action than the logon itself.  We can also assume that we would be trying to identify KC7 type of activity so lets break that down into different categories that may likely generate these types of events.  I’ll also include a couple of sample commands for each one.

1. Recon / Enumeration
2. 1. net group “domain admins” /domain
   2. net view /domain
3. Command Execution
4. 1. psexec -accepteula \\192.168.56.10 ipconfig
   2. wmic /node:192.168.56.10 /user:administrator /password:supersecret process call create "cmd.exe /c ipconfig" 
5. Data Movement
6. 1. copy backdoor.exe \\192.168.56.10\c$\windows\temp\backdoor.exe
   2. net use x: \\192.168.56.10\c$ /user:administrator “superscecret”

Seeing the above commands should obviously raise some alarms, but often it can be difficult to distinguish bad from legitimate activity.  I’ve personally chased many admins over the years.  The difficulty can be raised by how you craft your queries, the amount of data that’s returned and your ability to identify anomalies.  I think we can also limit ourselves based on truly understanding what we are looking for, the log sources we have available and what we perceive as our limitations to meld the two together.  I should say that I don’t have command line process audit logging enabled, but am still able to accurately identify the actions of the above commands based on the logs generated on the destination machine or the domain controller.

When I began looking into the best way to identify these chains I knew that there would be numerous actions that I would want to be bale to identify and that I would want to have some freedom in how I was able to search the data.  Being able to correlate based on source user, dest user, source machine, dest machine, process or any other pivot point that the data provided was very important.  I also wanted to be able to look across different windows of time and to easily incorporate multiple data sources.

I’ll explain what was accomplished using Splunk.  If you want to explore this and you are not a Splunk user, my hope is that you will be able to use the thought behind the technology and incorporate it into a solution that you have available.  

1. Scheduled queries for specific behaviors are sent to a summary index to house only interesting data.
2. Custom event types are created for broader queries (logon types, process names…).
3. Normalized field names for pivoting through data.  This can be important when searching or displaying fields from multiple log sources.

The screen shots below show 2 different dates.  The first clusters of events on 2/21 are the results of running the following commands on my domain controller (95N).

    net view \\192.168.56.10

    net group “domain admins” /domain

    net use x: \\192.168.56.10\c$ /user:administrator “password”

The second cluster of events on 2/23 are the results of copying a batch script to a member server and executing it.  The contents of the batch script were:

    net group “domain admins” /domain >> data.txt

    netstat -an >> data.out

    ipconfig >> data.out

By ComputerName

By Source IP

I know that this may be an overly simplistic view, but I believe that it shows the value of trying to create chains of events.  If someone was looking at each action individually they may be easily glossed over, but when combining them you may begin to see clusters of anomalous behavior.  The more of these individual actions you can include in your data collection the greater chance you have of surfacing these anomalies.

The power of this is not the technology, but the effort you put into your methods and analysis.  Also, I’m not at all saying that you should stop looking for single events that point to malicious activity, but think of this as simply another arrow you can put in your quiver.

As always, I would love to hear your thoughts.  Feel free to use the comment section below or reach out to me on twitter @jackcr.