My Thoughts on Threat Hunting

As some of you may have noticed, I have taken down handlerdiaries.com.  I’ve created this new blog to post my thoughts on incident response and finding bad guys in your environment.  This is my first post in quite some time and has been something I’ve been meaning to do sooner, but has always seemed to get put on the back burner.  I hope this blog can have as much success as my last.

There has been a lot of talk about the need for hunting.  Many vendors say their product has it baked in or they provide it as a service, but there is not a lot of discussion about how to do it.  David Bianco has been trying to change this with his newly created site www.threathunting.net and has a lot of great content there.  If you haven’t checked it out you really should.  What I want to talk about though is how do we get to the point that we can begin to hunt effectively.  I see a lot of people simply take indicators, search for those and call it good.  That may be one method, but is probably not the best use of your time as you can take the majority of those, write signatures for them and be alerted when there is a positive match.  I like to look at the task a different way.  What steps must an adversary take to accomplish their goal and can I look for indications of those steps.  I see the following steps as:

1. Enter and persist in your environment
2. Establish and maintain C2
3. Elevate privileges
4. Recon (system and network)
5. Locate data for exfil
6. Move laterally to access that data or to gain a great foothold
7. Package data for exfiltration
8. Exfil data out of your environment

Bucketing your hunting in different steps I think helps focus your ideas.  You can also track your progress much easier and know what ground you’re covering. 

Once you know what you want to look for, you need to know what it looks like and if you can find it in the data you have available.  I would suggest setting up a couple of vm’s, and perform the actions that you want to look for.  An example would be looking for the execution of password dumpers.  Execute mimikakz, gsecdump, wce, pwdump (or others you may be concerned about) and identify what logs are generated (OS, AV, hips…), and how it behaves on a machine.  Is there any type of log generated that I may be able to use as a trigger to look for something else?  Or can I generate logs based on the way it executes that may help increase my chances of finding it?

The next thing that I find very helpful is to understand how things may look if done in a malicious way.  We can use hands on keyboard recon as an example. Assume we are interested in:

net.exe

ping.exe

netstat.exe

whoami.exe

nbtstat.exe

If I were to look at the above for indications of recon I may be looking for a single cmd.exe process spawning some of the above processes.  These processes would likely be spread out over a couple of minutes as it takes longer to type the commands on a keyboard and have the data returned than it would if I were execute the same commands from a script.  A script would also likely spawn multiple cmd.exe processes, each to run one of the above.  Some other things I would be interested in are the parent process of cmd.exe, the user that owns these processes and the time of day that this occurred.

Taking the 8 steps above, how can I use these to help find a webshell?  Listing out what I know about them I can say the following.

1. They beacon only when being used.
2. They facilitate the execution of commands.
3. Many facilitate the upload of additional tools.
4. The webshell will be in a web accessible directory.
5. Processes will be spawned by the owner of the web server process.

Taking this information I can start planning my hunting strategy for them?

AV:

Look at AV hits across your web server environment for any hits with a web file extension and is in a path that would be accessible from the internet.  From my experience these are typically pretty high fidelity alerts and any positive hits should be investigated.  Other items of interest would be any alerts for hacking tools.

Note: These are good for alerting, but have included them here for those starting out.

Log Source: AV Logs

File Extension: .jsp, .war, .asp, .aspx, .php, .cmf

Step: Enter and persist in your environment, establish and maintain C2

HTTP POST:

Look for inbound POST requests that have no referrer.  When these are identified, look at the number of posts.  Typically if I’m executing commands I’m not just going to run a single command.  There will be multiple commands executed in a fairly short amount of time.  These commands will vary in size as well as the data being returned.  

Log Source: Webserver logs

Method: POST

Referrer: None

Count > 10

Duration: 30 Min

Byte counts vary in size

Single or multiple src ip's

Step: Establish and maintain C2

Process Execution:

When a webshell gets placed on a machine, the commands executed will run under the context of the user owning the web server process and a child process of the web server process.  So if I execute a command from a webshell on a web server running IIS, it will likely spawn a cmd.exe process that will be owned by that user and will be a child process of w3wp.exe.  Again, we want to look for frequency with these as the times they execute will be sporadic, but when they execute there will likely be multiple processes executing (cmd.exe -> net.exe, cmd.exe -> ipconfig.exe...).

Log Source: Windows security event logs (Event code 4688 or 592)

Child process of web server process (IIS, Tomcat, Apache, ColdFusion)

Owner of web server process

Count > 20

Duration: 30 Min

Step: Lateral movement

Behavior:

When a web server gets compromised there are some typical things that will likely occur (Though not in every case). 

1. A new file will be placed on the web server with a web file extension and in a web accessible directory.
2. Commands will be executed from the webshell. (cmd.exe, powershell.exe…)
3. Additional tools will be uploaded to the web server. (Look for newly created files with extensions of .zip, .rar, .7z, .exe…)

Looking for at least 2 of these occurrences from the above in a 15 minute window can help identify suspicious behavior that may need investigating.  Multiple log sources may need to be incorporated for this, but the work in doing so can be well worth the effort.

Step: Enter your environment, recon, lateral movement

With each of the above searches it is important to review this on a regular basis.  I would recommend setting up a schedule to collect this data and format it so that it’s easy to review.  It’s a little bit of work to setup these searches, but once done, the time needed to review them is pretty minimal.

So to sum things up.  These are the steps I see that are essential when you begin to hunt for bad guys.

      1. Define what you are looking for.

2. Know what it looks like.
3. Identify the data you have available to find it.
4. Create and schedule searches.
5. Regular review of data.

Questions or comments are welcome.  You can use the comment section or hit me up on twitter @jackcr

Happy Hunting