Thoughts on Incident Response Teams

With all of the breach notifications that seem to be flying around daily over the past few months I can’t help but wonder how their IR teams operate.  I won’t speculate or cast any blames or failures as I simply don’t know.  I do have definite opinions on how I think teams can operate and grow though.  These are only my opinions and I’m sure other’s have different ones.  If you have differing opinions I would love to hear about them.

First, you can’t have a team without having some type of structure that defines roles and responsibilities.  This structure is by no means new and my opinion is “if it’s not broke then why try and fix it”.

Incident Handlers

The incident handler is your subject matter expert.  He/she should be a highly technical person that is also cognizant of risk and business impact.  The IH needs to understand the threats that your company faces and be able to direct efforts based on these threats and data being relayed by others performing analysis.  The IH should have the ability and freedom to say “contain this device based on these facts” (of course, good or bad, business needs can always trump these decisions).  All aspects of response should flow through this person so that they can delegate duties appropriately. The IH should also be the go to person for any information/explanation related to the response efforts.

Response Analysts

The response analysts can have various levels of skills and abilities.  Their function is to analyze incoming data for signs of compromise, lateral movement, data exfiltration and so on. Their findings should be documented and communicated to the incident handler so that appropriate actions can be taken.  The people filling this role should also be familiar with the different detection technologies that your org has as well as the ability to create or recommend detection signatures based on their analysis.

Incident Coordinator

This person, unlike all the other’s, does not need to be technical.  The role of this position is to handle all of those administrative tasks that come along with response.  Things such as communication with management regarding incident status, contacting POC’s to help with containment and collection, and maintaining timeline of activity for reporting would all fall under this role.

Reverse Engineer

This position is probably the most difficult to fill.  Harlan Carvey wrote a blog post a few months ago regarding the disconnect between RE and IR which can be found here and describes the reasons why I feel this person needs to be directly involved.  Having an RE person that is directly involved with IR can bridge that disconnect.  There are usually specific questions that need to be answered when you are in the heat of battle, such as how to decode the C2 communication or identifying indicators of malware that can be used in immediate detection.  Having this person directly involved with the response teams can greatly speed up answering those needed questions.

POC’s

This position is not directly related to the response teams, but I feel needs to be mentioned.  You can think of these people as your boots on the ground.  They are directly responsible for collection and containment of assets (depending on the size of your org, they may even be someone on your IR team).  They should be very familiar with your processes and have the ability to act on your team’s behalf.

Please understand that I’m not saying that your org needs to go out and hire multiple people that can fill all of these roles.  I work in an environment that has close to 1 million endpoints so we require a fairly large team.  If your environment is smaller, then I would see people performing many of the above tasks simultaneously.  They are just roles that I think need to be filled for any response team.

Training

Yearly DFIR classes are great.  I have taken a few over the years and can say that they have filled a much needed gap when it comes to learning a specific need.  They can also really get you focused in the right direction, unfortunately the yearly class is not enough in my opinion.  I am a big fan of peer led training and mentoring.  I see this as a must have if you are focused on building teams from within.

As a senior member of our CIRT, I think it’s important that I find out what aspects of response people are interested in.  It may be network analysis, host analysis, reverse engineering to name a few.  Whatever they are interested in I think it’s important not to discourage, but to encourage further development.  That being said, if their passion is red team activities I may try and steer them back to our team needs.  If I or someone else on our team is knowledgeable in that area, working with them can go a long way and can be so much more beneficial then a yearly class as you can focus on methods and tools that your company employs as well as the face time and personal interaction with that junior team member.

I was speaking with a colleague the other day about this.  When I was in the military each unit had a METL (Mission Essential Task List).  This was basically a list of all the essential actions the unit was expected to be able to perform during times of war.  When we trained, our training was always focused according to our METL.  I think response teams can benefit from this methodology.  If you list out the critical tasks that your team needs to be able to perform during response, you can then test your people / team against this list.  This will greatly help you figure out the different aspects in which you should focus your training.

If you are a SME in a certain aspect of response, take the time to develop training in that area.  If the training is good your team will love you for it and can be a great way to get everyone on the same page.  Along those same lines, there is nothing wrong with having junior members develop training to share among peers on your team.  Knowing that you will be teaching someone a certain aspect of response or analysis usually is enough to get that person to do a little deeper research than they may have done before.  I am a firm believer that knowledge not shared is knowledge wasted.

Demand Involvement

As I stated above with regards to the response analyst.  Team members with varying degrees of skill level can often be nervous about analyzing data that may be outside their comfort zone.  As a result, they may shy away and not be as involved as they should be for fear of missing something.  IR is a team sport and needs team participation.  If you notice a team member begin to shy away from analysis, call them on it and get them back in the game.  Mistakes happen and fear of making them should not impact response.  As long as the mistake is recognized and the person has learned from it then I think that’s OK.

Technical Competency

I’ve seen teams that have segmented the different roles on their teams.  This is not always bad as you can get clear focus on different aspects of detection and analysis, but when the s**t hits the fan and you have multiple response activities going on at once you may need to shift people.  If they have no idea how to write a detection rule or look at a memory dump then this may not be so good.  I think that cross training should be a focus and analysts should know the different aspects and technologies.  They also get the added benefit of not being pigeon holed into one area, but can grow and eventually become that senior incident handler if that is their desire.  You also never know, you may find that diamond in the rough that you may never have known about otherwise.

Again, these are just some of my thoughts and I would love to hear what other people think.  You can find me on twitter at @jackcr.