The biggest defining characteristic of an NSM program is the human analyst. The analyst is the individual who interprets alert data, analyzes and investigates that data along with related data, and makes the determination of whether or not the event should be classified as a false positive or if an incident has occurred. Depending on the size and structure of the organization, an analyst may also take part in the incident response process or perform other tasks such as host-based forensics or malware analysis.
The human analyst is the crux of the organization. It is the analyst who is pouring through packet captures looking for a single bit that’s out of place. This same analyst is expected to be up to date on all of the latest tools, tactics, and procedures that the adversary may use to compromise your network. The simple fact of the matter is that that the security of your network depends on the human analysts ability to do their job effectively.
Measuring Analyst Success
Measuring the success or failure of an NSM program is often handled incorrectly. If a compromise occurs, senior management views this as a critical failure of their security team as a whole. If we were still operating under a vulnerability-centric model where prevention is relied upon fully then this might be an appropriate thought pattern. However, once an organization has accepted that prevention eventually fails, they should also expect that compromises will occur. Once this mindset becomes prevalent, you should not measure the effectiveness of an NSM program by whether or not a compromise occurs, but rather, how effectively it is detected, analyzed, and escalated. In the scope of an intrusion, NSM is ultimately responsible for everything that occurs from detection to escalation, with their goal being able to get the appropriate information into the hands of incident responders as quickly as possible once it has been determined that an incident has occurred. Of course, in anything but larger organizations, the NSM team may also be the incident response team, but the functions are still logically separate. Ultimately, instead of asking “why did this happen?”, the questions leadership should be asking your NSM team after a compromise is “how quickly were we able to detect it, how quickly were we able to escalate it to response, and how we can adjust our NSM posture to be better prepared next time?”
The most important part of an NSM program, and the persons who will ultimately be responsible for answering these questions are the human analysts. I’ve had the privilege to work with, lead, and observe several security teams from organizations of all size, and I’ve seen several good programs and several bad programs. There are a lot of ways to create a great NSM team, but all of the organizations that I’ve witnessed that are failing at providing effective security through NSM have the same thing in common: they fail to recognize that the human analyst is the most important facet of the mission.
Rather than investing in their analysts and empowering their efforts, these organizations invest in expensive software packages or unnecessary automation. Two years down the road when a large compromise happens, the stakeholders who made these decisions are left wondering why their SIEM solution and its seven-figure price tag didn’t catch a compromise that started occurring six months prior.
Worse yet, these organizations will attempt to scrimp on staffing to the point where they will only utilize entry-level staff without the required experience or background to perform the task at hand. Although some entry-level staffers are expected, a lack of experienced technical leadership means that your junior level analysts won’t have an appropriate opportunity to grow their expertise. These are often the same organizations that refuse to provide adequate training budgets.
Traits of Successful NSM Teams
While organizations often fail for the same reasons, there are also several traits that I’ve witnessed amongst successful NSM teams. In order to facilitate a successful NSM team, I believe you need to do the some of or all of the following things:
Create a Culture of Learning
NSM thrives on ingenuity and innovation, which are the products of motivation and education. It is one thing to encourage education and provide training opportunities on occasion, but it is a completely different animal to create an entire work culture based upon learning. This means not only allowing for learning, but facilitating, encouraging, and rewarding it.
This type of culture requires overcoming a lot of the resistance associated with a typical workplace. In a traditional workplace, it might be frowned on to walk into an office and see several employees reading books. It also might be looked at negatively to see a group of employees working on personal technical projects that don’t relate to reviewing events or packets. It might even be unfathomable for the majority of the staff to abscond from their desks to discuss the finer points of time travel in front of a whiteboard. The truth of the matter is that these things should be welcomed, as they increase morale and overall happiness, and at the end of the day your analysts go home with an excitement that makes them want to come back with fresh ideas and renewed motivation the next day.
Although some members of the old guard will never be able to accept such a work environment, it’s proven to be very successful. Google is an example of an organization that has created a successful culture of learning, and a large portion of their success is in direct relation to that.
This mantra of a culture of learning can be summed up very simply. In every action an analyst takes, they should either be teaching or learning. No exceptions.
It’s a bit cliché, but the team dynamic ensures mutual success over individual success. This means that team building is a must. Ensuring team cohesiveness starts with hiring the right people. An individual’s capacity to perform is important, but their ability to mesh with existing team members is of equal importance. I’ve seen multiple instances where one bad apple has soured the whole bunch.
At some point, something bad is going to happen and you are going to have to deal with an incident that is going to require an extensive time commitment from all parties involved. Analysts who trust each other and genuinely enjoy spending time with each other are going to be much more effective at ensuring the incident is handled properly. As an added bonus, a cohesive team will help promote a learning culture.
Provide Formalized Opportunities for Professional Growth
One of the biggest fears exhibited by managers is that their staff will become highly trained and certified and then leave the organization for greener pastures. Although this does happen, it shouldn’t steer an organization away from providing opportunities.
In interviewing several NSM analysts who have left various organizations, it’s rarely ever something as simple as a higher salary that has caused them to jump ship. Rather, they almost always cite that they weren’t provided enough opportunity for growth within their organization. Generally, people don’t like change. Changing jobs, especially when it involves relocating, is a big step and something people would generally like to avoid if at all possible. This means that you are likely to keep your staff if you can provide opportunities for professional certifications, advancements in position, or migrations to management roles. Simply having a clearly defined path for this type of advancement can often make the difference. This is one of the reasons why having something like the Level 1/Level 2/Level 3 analyst classification system can benefit an organization.
Information security is notorious for having a culture of people with incredibly large egos. Although there is something to be said for being humble, eventually there comes a point where you can’t change the personal traits that are built into someone and you have to do your best to work with it. If your organization has an employee with a big ego, then turn him into a superstar. People who have an excessive amount of confidence typically desire to succeed in a big way, so if you can make this happen then they will thrive. This is done by challenging them, providing learning opportunities, and instilling responsibility in them. A superstar is rare, so some will flounder when it’s crunch time. If this happens then the reality check often serves to lessen a big ego. If the person continually succeeds, then you’ve found your superstar.
Once you have a superstar, people will want to imitate their success. Their greatness pushes others to be more than they thought they were capable of, and everybody benefits. As long as your superstar isn’t negatively impacting others by being rude, abrasive, or otherwise overbearing, then he is an asset. The difference between Allen Iverson and Kobe Bryant is that Allen Iverson played great, where as Kobe Bryant made everyone around him great. That’s why Iverson’s 76ers didn’t win any championships, and Bryant’s Lakers won 5 under their respective tenures. Make your superstar into a Kobe Bryant.
Positive reinforcement can be a monumental difference maker in morale. If an analyst finds something that nobody else found, everybody should know about it. Furthermore, if an analyst stays late for five hours to follow up on an incident, you should let them know you appreciate their efforts in some way. The mechanism for reward doesn’t particularly matter as long as it is something desirable. I’ve found that gift cards are a simple, cheap, and effective way to say thanks. Make room in your budget for this.
Learn from Failure
Analytical work can get mundane really quickly. This is especially the case in a smaller environment where there simply just aren’t as many events or significant attacks occurring. When this occurs, it becomes very easy for analysts to miss something. Instead of punishing the entire group, take this as another learning opportunity.
One of my favorite ways to promote learning from failure is a concept taken from the medical field. Many times when a patient dies and the death could have been medically prevented, the treating physician and a team of additional physicians will convene for a meeting called Morbidity and Mortality (M&M). In this meeting, the treating physician will present how the patient was cared for and the additional physicians will provide constructive questioning and thoughts on alternative steps that could have been taken. These sessions are often feared, but when moderated effectively and kept positive, they can have the ability to enact a great deal of positive change when similar situations come back around. I’ve written about this concept previously here.
Exercise Servant Leadership
The most successful organizations I’ve had the privilege to work with are those who practice the concept of servant leadership. Servant leadership is something that has been around for quite a while, but is something I was introduced to as a University of Kentucky basketball fan from UK coach John Calipari.
The premise of servant leadership is that rather than establishing leadership based upon a title or some given authority, servant leaders achieve results by giving priority to the needs of their colleagues. This humble mindset is one in which you look to help others achieve their mission such that the organization will prosper. This has the potential to breed an organization that isn’t anchored by one strong leader, but rather, a group of leaders with different strengths and weaknesses working in harmony to achieve a common mission. Although it sounds like a lofty goal, with the right mindset and buy in from all parties involved, this type of environment can become a reality.
Ultimately, the success of the NSM team depends upon management accepting the importance of the analyst role. Once this is done, then some of the items here can be implemented to further enhance that success. Every team is different and brings with it different challenges, but at the end of the day, a cohesive, motivated, and analyst-driven security team is a formidable force to be reckoned with.