Thursday, February 23, 2017

Hunting for Chains

In my last blog post I touched on something that I want to talk more about.  I think far too often, when hunting, we can focus on single indications for signs of something being bad.  Intrusions are far more than a single event though, so why spend all of your time hunting that way?  Often the actions that an attacker will take will be in short time bursts.  These bursts may only last for a few seconds to a few minutes.  Looking for multiple events within these windows of time, I feel, can be a very effective way to spend your time.

One of the things I try and do with my blog is to share how I think about the work that I do.  I have gained so much knowledge from others over the years and feel strongly about giving back.  Lately I have been thinking a lot about , what I like to call, behavior chains.  Basically these are the actions that an attacker would likely take to accomplish their goal.  Each action is a link and the sum of actions make up the chain.  A link can be something as simple as a process name like net.exe or cmd.exe (these are probably 2 of the most commonly spawned processes and attackers definitely take advantage of these).  A link can also be something as complex as a correlation search that surfaces a specific behavior (see my last blog post for some of these correlations).  So why do I care about looking at hunting this way? There are multiple reasons, but a few of them are to model how attackers may operate, data reduction and higher fidelity output.

Lets say that we want to look for odd network logon behavior because we know that when an attacker is active in our environment they will generate anomalous 4624 type 3 events.  The problem with this is that there is likely thousands upon thousands, if not millions, of these events per day.  We could try and reduce this data by looking for rare occurrences of source machine, dest machine, user or a combination of the three. This will likely produce anomalies, but will also take time to investigate each one.  Now take that time and multiply it by the number of days in a week.  How much time are you spending looking at these events and are the results beneficial?

I think the question to ask is what are the things an attacker would do that would generate these logon events.  We know that just by the meaning of the event that it was the result of an action taken by a remote machine and we probably care more about the action than the logon itself.  We can also assume that we would be trying to identify KC7 type of activity so lets break that down into different categories that may likely generate these types of events.  I’ll also include a couple of sample commands for each one.
  1. Recon / Enumeration
    1. net group “domain admins” /domain
    2. net view /domain
  2. Command Execution
    1. psexec -accepteula \\192.168.56.10 ipconfig
    2. wmic /node:192.168.56.10 /user:administrator /password:supersecret process call create "cmd.exe /c ipconfig" 
  3. Data Movement
    1. copy backdoor.exe \\192.168.56.10\c$\windows\temp\backdoor.exe
    2. net use x: \\192.168.56.10\c$ /user:administrator “superscecret”

Seeing the above commands should obviously raise some alarms, but often it can be difficult to distinguish bad from legitimate activity.  I’ve personally chased many admins over the years.  The difficulty can be raised by how you craft your queries, the amount of data that’s returned and your ability to identify anomalies.  I think we can also limit ourselves based on truly understanding what we are looking for, the log sources we have available and what we perceive as our limitations to meld the two together.  I should say that I don’t have command line process audit logging enabled, but am still able to accurately identify the actions of the above commands based on the logs generated on the destination machine or the domain controller.

When I began looking into the best way to identify these chains I knew that there would be numerous actions that I would want to be bale to identify and that I would want to have some freedom in how I was able to search the data.  Being able to correlate based on source user, dest user, source machine, dest machine, process or any other pivot point that the data provided was very important.  I also wanted to be able to look across different windows of time and to easily incorporate multiple data sources.

I’ll explain what was accomplished using Splunk.  If you want to explore this and you are not a Splunk user, my hope is that you will be able to use the thought behind the technology and incorporate it into a solution that you have available.  

  1. Scheduled queries for specific behaviors are sent to a summary index to house only interesting data.
  2. Custom event types are created for broader queries (logon types, process names…).
  3. Normalized field names for pivoting through data.  This can be important when searching or displaying fields from multiple log sources.

The screen shots below show 2 different dates.  The first clusters of events on 2/21 are the results of running the following commands on my domain controller (95N).
    net view \\192.168.56.10
    net group “domain admins” /domain
    net use x: \\192.168.56.10\c$ /user:administrator “password”

The second cluster of events on 2/23 are the results of copying a batch script to a member server and executing it.  The contents of the batch script were:
    net group “domain admins” /domain >> data.txt
    netstat -an >> data.out
    ipconfig >> data.out

By ComputerName

By Source IP


I know that this may be an overly simplistic view, but I believe that it shows the value of trying to create chains of events.  If someone was looking at each action individually they may be easily glossed over, but when combining them you may begin to see clusters of anomalous behavior.  The more of these individual actions you can include in your data collection the greater chance you have of surfacing these anomalies.

The power of this is not the technology, but the effort you put into your methods and analysis.  Also, I’m not at all saying that you should stop looking for single events that point to malicious activity, but think of this as simply another arrow you can put in your quiver.

As always, I would love to hear your thoughts.  Feel free to use the comment section below or reach out to me on twitter @jackcr. 

Saturday, February 11, 2017

Patterns of Behavior

If you have been in security for any length of time then you have probably seen David Bianco’s Pyramid of Pain.  One of the thoughts behind the pyramid is that the higher up you go the more cost to the adversary you bring.  The top of the pyramid is where we really want to be as it’s much harder for an adversary to change how they operate and we, as defenders, can make it very difficult for them to carry out their operations if we can begin to find them at these upper levels.. 

I said yesterday in a tweet that a goal of mine is to be able to track an entire intrusion simply by the alerts that I generate.  While that tweet may be a little far fetched, the sentiment behind the tweet is not.  I want to know as much as I can about attacker activity so that I can effectively build a web of detection.  The bigger and tighter the web, the more chances I will have to identify patterns that would be indicative of attacker activity.  By studying past attacks, public reporting and any peer sharing I can shape the things that I should be going after.

Lets say that we have a fictitious actor that we are tracking that we’re calling FridayFun.  Lets also say that we have been tracking FridayFun since 2012 when we initially responded to them HERE (Note: This is a link to the download of a memory forensics challenge I created in 2012).  During the course of time we have identified some of the following regarding techniques used for Kill Chain 7 activity.
  1. The use of net commands for enumeration and lateral movement.
  2. The scheduling of remote tasks.
  3. The use of psexec for tool execution.
  4. Copying of files via command line.

Lets take a look at each of the above and how we may be able to find it in the Windows security events.  Note that these methods require additional windows logging.  I’m also including Splunk searches that can be used to find this behavior.  If you aren’t a Splunk user, my hope is that you can take these queries and incorporate them into a solution or method you have available.

We have identified FridayFun often copying files to $ shares via the command line as in the example below.
copy thisisbad.bat \\192.168.56.10\c$\windows\thisisbad.bat

When a file is copied via the command the logs produced are different than if you were to copy them via Windows Explorer.  Both methods produce multiple file share access events, but the Access Masks are different depending on method. From the command line, theses are the unique values:
0x100180
0x80
0x130197

The following query will identify the 3 unique access masks being created within the same second on a single machine.  

sourcetype=wineventlog:security EventCode=5145 Object_Type=File Share_Name=*$ (Access_Mask=0x100180 OR Access_Mask=0x80 OR Access_Mask=0x130197) |bucket span=1s _time |rex "(?<thingtype>(0x100180|0x80|0x130197))" |stats values(Relative_Target_Name) AS Relative_Target_Name, values(Account_Name) AS Account_Name, values(Source_Address) AS Source_Address, dc(thingtype) AS distinct_things by ComputerName, _time |search distinct_things=3

For the scheduling of remote tasks, lets say that we have identified a pattern of behavior each time FridayFun has utilized this tactic.
  1. Query the remote machine for the local time (think about machines in different time zones).
  2. The remote task is created.
  3. The source machine queries the scheduled task on the remote machine.
This command sequence may look like:
  1. net time \\192.168.56.10
  2. at \\192.168.56.10 03:25 c:\windows\thisisbad.bat
  3. at \\192.168.56.10

The net time command and remote At query will both a produce a 5145 file share event where the IPC$ is the share being accessed.  The Access Mask for these 2 events will be 0x12019f.  The Relative Target Name will be different.  A srvsvc named pipe will be created for the net time command while an atsvc named pipe will be created for the At query.  For the creation of the scheduled task we are looking for 4698 events and specifically looking for At jobs in the Task Name.

The following query will identify the above 3 events occurring within an hour time frame on a single machine.

sourcetype=wineventlog:security (EventCode=4698 Task_Name=\\At*) OR (EventCode=5145 Object_Type=File Share_Name=*IPC$ Access_Mask=0x12019f (Relative_Target_Name=srvsvc OR Relative_Target_Name=atsvc)) |bucket span=1h _time |rex "(?<thingtype>(\\\\At.|srvsvc|atsvc))" |stats values(Relative_Target_Name) AS Relative_Target_Name, values(Account_Name) AS Account_Name, values(Task_Name) AS Task_Name, dc(thingtype) AS distinct_things by ComputerName, _time |search distinct_things=3

Friday fun often utilizes psexec to aid in execution on remote machines.  We have identified them using commands such as the following:

psexec -accepteula \\192.168.56.10 -c gsecdump.exe

When looking for psexec being executed I have found that it may be more beneficial to look for indications on the destination side.  Psexec can often be renamed and may be harder to identify on the source if you aren’t able to log the FileDescription metadata field of the binary when it executes.  On the destination side a 5145 event will be created where the share name is ADMIN$ and the Relative Target Name is PSEXESVC.EXE.  We are also looking for any new process creation within that same second to identify what psexec executed on the destination machine (Note: when psexec copies a file and executes it , it will be spawned from the IPC$ share).

This behavior can be identified by the following query.

sourcetype=wineventlog:security ((EventCode=4688 New_Process_Name!=*\\PSEXESVC.EXE New_Process_Name!=*\\conhost.exe New_Process_Name!=*\\dllhost.exe) OR (EventCode=5145 Share_Name=*ADMIN$ Relative_Target_Name=PSEXESVC.EXE)) |bucket span=1s _time |rex "(?<thingtype>(4688|PSEXESVC))" |stats values(Relative_Target_Name) AS Relative_Target_Name, values(Account_Name) AS Account_Name, values(Source_Address) AS Source_Address, values(New_Process_Name) AS New_Process_Name, dc(thingtype) AS distinct_things by ComputerName, _time |search distinct_things=2

Copying files from the destination to the source machine will produce different logs than if I were to copy files from the source to the destination.  FridayFun has used commands such as the following.

copy \\192.168.56.20\c$\windows\system32\1.txt .\1.txt

This method is much like the first query with the exception of the Access Masks.  The following query can be used to identify this behavior.

sourcetype=wineventlog:security EventCode=5145 Object_Type=File Share_Name=*$ (Access_Mask=0x100081 OR Access_Mask=0x80 OR Access_Mask=0x120089)  |bucket span=1s _time |rex "(?<thingtype>(0x100081|0x80|0x120089))" |stats values(Relative_Target_Name) AS Relative_Target_Name, values(Account_Name) AS Account_Name, values(Source_Address) AS Source_Address, dc(thingtype) AS distinct_things by ComputerName, _time |search distinct_things=3

For those that like dashboards, I created the following to show what the output would look like.



As a defender, i think far too often we are focused on singular events that will give us that "ah ha" moment.  Sure, there are some that may be a dead giveaway of a malicious actor operating in our environment, but I think often singular alerts are valued by their true to false positive ratio.  With the method I showed above I can look for specific behavior, regardless of noise, and bucket events into previously identified attacker activity.  When the number of unique events meets a certain value based on source, destination or user I can then raise an alarm.  The more behavior that I can identify and query for, the larger my web becomes.  I don’t know that I will ever be able to piece together wan entire intrusion by the alerts I generate, but I at least want to make it extremely difficult for any adversary to be able to operate freely in my environment. 


I would love to hear any comments.  You can find me on twitter under @jackcr.