Sunday, July 31, 2016

My Thoughts on Threat Hunting

As some of you may have noticed, I have taken down  I’ve created this new blog to post my thoughts on incident response and finding bad guys in your environment.  This is my first post in quite some time and has been something I’ve been meaning to do sooner, but has always seemed to get put on the back burner.  I hope this blog can have as much success as my last.

There has been a lot of talk about the need for hunting.  Many vendors say their product has it baked in or they provide it as a service, but there is not a lot of discussion about how to do it.  David Bianco has been trying to change this with his newly created site and has a lot of great content there.  If you haven’t checked it out you really should.  What I want to talk about though is how do we get to the point that we can begin to hunt effectively.  I see a lot of people simply take indicators, search for those and call it good.  That may be one method, but is probably not the best use of your time as you can take the majority of those, write signatures for them and be alerted when there is a positive match.  I like to look at the task a different way.  What steps must an adversary take to accomplish their goal and can I look for indications of those steps.  I see the following steps as:
  1. Enter and persist in your environment
  2. Establish and maintain C2
  3. Elevate privileges
  4. Recon (system and network)
  5. Locate data for exfil
  6. Move laterally to access that data or to gain a great foothold
  7. Package data for exfiltration
  8. Exfil data out of your environment
Bucketing your hunting in different steps I think helps focus your ideas.  You can also track your progress much easier and know what ground you’re covering. 

Once you know what you want to look for, you need to know what it looks like and if you can find it in the data you have available.  I would suggest setting up a couple of vm’s, and perform the actions that you want to look for.  An example would be looking for the execution of password dumpers.  Execute mimikakz, gsecdump, wce, pwdump (or others you may be concerned about) and identify what logs are generated (OS, AV, hips…), and how it behaves on a machine.  Is there any type of log generated that I may be able to use as a trigger to look for something else?  Or can I generate logs based on the way it executes that may help increase my chances of finding it?

The next thing that I find very helpful is to understand how things may look if done in a malicious way.  We can use hands on keyboard recon as an example. Assume we are interested in:
If I were to look at the above for indications of recon I may be looking for a single cmd.exe process spawning some of the above processes.  These processes would likely be spread out over a couple of minutes as it takes longer to type the commands on a keyboard and have the data returned than it would if I were execute the same commands from a script.  A script would also likely spawn multiple cmd.exe processes, each to run one of the above.  Some other things I would be interested in are the parent process of cmd.exe, the user that owns these processes and the time of day that this occurred.

Taking the 8 steps above, how can I use these to help find a webshell?  Listing out what I know about them I can say the following.
  1. They beacon only when being used.
  2. They facilitate the execution of commands.
  3. Many facilitate the upload of additional tools.
  4. The webshell will be in a web accessible directory.
  5. Processes will be spawned by the owner of the web server process.

Taking this information I can start planning my hunting strategy for them?

Look at AV hits across your web server environment for any hits with a web file extension and is in a path that would be accessible from the internet.  From my experience these are typically pretty high fidelity alerts and any positive hits should be investigated.  Other items of interest would be any alerts for hacking tools.
Note: These are good for alerting, but have included them here for those starting out.

Log Source: AV Logs
File Extension: .jsp, .war, .asp, .aspx, .php, .cmf
Step: Enter and persist in your environment, establish and maintain C2

Look for inbound POST requests that have no referrer.  When these are identified, look at the number of posts.  Typically if I’m executing commands I’m not just going to run a single command.  There will be multiple commands executed in a fairly short amount of time.  These commands will vary in size as well as the data being returned.  

Log Source: Webserver logs
Method: POST
Referrer: None
Count > 10
Duration: 30 Min
Byte counts vary in size
Single or multiple src ip's
Step: Establish and maintain C2

Process Execution:
When a webshell gets placed on a machine, the commands executed will run under the context of the user owning the web server process and a child process of the web server process.  So if I execute a command from a webshell on a web server running IIS, it will likely spawn a cmd.exe process that will be owned by that user and will be a child process of w3wp.exe.  Again, we want to look for frequency with these as the times they execute will be sporadic, but when they execute there will likely be multiple processes executing (cmd.exe -> net.exe, cmd.exe -> ipconfig.exe...).

Log Source: Windows security event logs (Event code 4688 or 592)
Child process of web server process (IIS, Tomcat, Apache, ColdFusion)
Owner of web server process
Count > 20
Duration: 30 Min
Step: Lateral movement

When a web server gets compromised there are some typical things that will likely occur (Though not in every case). 
  1. A new file will be placed on the web server with a web file extension and in a web accessible directory.
  2. Commands will be executed from the webshell. (cmd.exe, powershell.exe…)
  3. Additional tools will be uploaded to the web server. (Look for newly created files with extensions of .zip, .rar, .7z, .exe…)

Looking for at least 2 of these occurrences from the above in a 15 minute window can help identify suspicious behavior that may need investigating.  Multiple log sources may need to be incorporated for this, but the work in doing so can be well worth the effort.

Step: Enter your environment, recon, lateral movement

With each of the above searches it is important to review this on a regular basis.  I would recommend setting up a schedule to collect this data and format it so that it’s easy to review.  It’s a little bit of work to setup these searches, but once done, the time needed to review them is pretty minimal.

So to sum things up.  These are the steps I see that are essential when you begin to hunt for bad guys.

      1. Define what you are looking for.
  1. Know what it looks like.
  2. Identify the data you have available to find it.
  3. Create and schedule searches.
  4. Regular review of data.

Questions or comments are welcome.  You can use the comment section or hit me up on twitter @jackcr

Happy Hunting 


  1. Jack,

    Great first post, and a great way to get started.

    "What steps must an adversary take to accomplish their goal..."

    In my experience as a consultant, having done DFIR work, moved into hunting, and now doing both, I see a common thread when working with clients, as well as analysts who are new to hunting; that is, what constitutes an adversary's goals?

    When we base our belief in what the adversary's goals are on what we can see, and our own experiences, we have to ask ourselves (and be hones with the answers), how much of what we believe to be those goals is based on lack of visibility, and our own cognitive bias?

    When doing just DFIR work, one of the things I used to hear analysts tell clients was, "...if I were the bad guy, I'd do X...". But the point is, you're not the bad guy, and without adequate instrumentation, you're basing your assessment on their activities and goals on entirely too little information.

    I know that there are a lot of folks out there talking about this, but perhaps too few who are listening.

  2. "Once you know what you want to look for, you need to know what it looks like and if you can find it in the data you have available. I would suggest setting up a couple of vm’s, and perform the actions that you want to look for."

    I would recommend adding Sysmon to the mix, and from there, looking at various activities and exploring how you could increase your instrumentation and visibility, without overwhelming yourself with data.

  3. Thanks Harlan,

    Agree, you may never know what an adversary's true goal is, but if you focus on hunting for them in different stages of an intrusion you will have a much greater chance to find them.

    Cognitive bias is a hard one. Someone may see winword.exe executed 10 times within a few minutes on a machine and think nothing of it. Someone else may say that executing winword.exe 10 times at 2:00 AM on a server, using an administrative account is odd, dig further and see that it is the parent process of net.exe, ipconfig.exe, ping.exe... I think that's why a lot of hunting comes down to experience. It's definitely a hard one to teach, but if you know how please share ;)

    "...if I were the bad guy, I'd do X...".

    My pet peeve. Even if you have a red team background, you probably aren't red teaming like a real adversary /rant

  4. Jack,

    " probably aren't red teaming like a real adversary..."

    As someone who's seen adversaries and red teams at a process level...uh...yeah. There're big differences between what you see an adversary do, and what you see a red team do.

    1. So what are adversaries doing so differently?

    2. One of the biggest things I see is a heavy use of publicly available tools to facilitate C2 and lateral movement.

  5. Great first post Jack, to me I think that hunting efforts must be correlated with Business Risks (Something most of Analyst and technical folks lacks) as well as sensitive and critical information flow (rest/use/transit), from there analyst can define with order of priority the scope of monitoring (IP/Hostnames/Credentials/Mailboxes). for mailboxes access auditing I suggest to have a look to this: Cheers.


Note: Only a member of this blog may post a comment.