In my previous post where I introduced FlowPlotter I showed you some of the basics behind generating data from SiLK and turning it into nice looking visual representation that leveraged Google's Visualization API. That is all wrapped up into an easy to run tool called "FlowPlotter" that runs directly off of data piped from rwfilter to SiLK via stdout. Today I'm pleased to reveal some of the D3 visualizations I promised earlier. Previously I had planned to replicate the Google visualizations with D3 to provide an alternative look, but instead I've gone forward to fill in some of the features that Google Charts is lacking.

D3.js is a popular javascript library that is often used to streamline the way people make visualizations with their data. These visualizations are usually friendly with most browsers and can be augmented to do just about anything you can think of. For the integration of these new graphs, I've went with the same "template" approach as I did with the Google charts, and again with the data being embedded into the html file that is generated. In most D3 visualizations people will host the data on a web server and reference that data directly with d3.js. But, due to how the data is used here, it is more useful for us to be able to generate it for immediate use. In theory you can reference files on the local system, but most browsers (like Chrome) have strict security settings that prohibit this. Don't expect other more forgiving browsers (Firefox) to allow this down the road either. For all of those reasons, FlowPlotter D3 templates have been designed to contain all data within a single HTML file.

Force Directed Link Graphs

So far I've generated two new chart types in D3. The first is a force-directed link graph similar to what is generated with Afterglow. We discussed in the last FlowPlotter post that the purpose behind FlowPlotter is to have a streamlined way of generating visualizations for all of your data and having it easily available to anyone that needs to view it. While Afterglow is great to work with, it doesn't really "flow" with the rest of what we're generating, and streamlining it for wide use can be challenging. The D3 force-directed link graph that I've integrated with SiLK shows the relationship between 2 nodes, with links between of varying opacity based on the value of a 3rd value. Simply put, it is created from a 3 column CSV with a source, target, and value in that order. The overall design was borrowed partly from the work of d3noob.

Force-Directed Link Graph showing all traffic from a specific country

Figure 1: Force-Directed Link Graph showing all traffic to or from a specific country

To create a force-directed link graph directly from rwfilter output, you'll need to run the following:

rwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all --pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 > forcetest.html

You'll notice that there are 4 options after forceopacity. This chart module relies on rwstats to generate this data. The communication you see in Figure 1 represents the top 100 sip-dip pairs for all traffic from South Korea based on total amount of distinct destination port communication attempts. The greater the number, the darker the link between nodes. Similarly you could do something more practical like generating a force-directed link graph between the top 100 sip-dport pairs for outgoing web traffic to South Korea:

rwfilter ../Sampledata/sample.rw --dcc=kr --proto=0- --type=outweb --pass=stdout | ./flowplotter.sh forceopacity sip dport bytes 100 > forceport.html

There is still a lot I would like to do with FlowPlotter's force-directed link graph. For one, it currently does not replace the usefulness of Afterglow as the big benefit of Afterglow lies in being able to specify "rules" about how your data will be displayed, as seen in ANSM. Down the road I would like to be able to create something similar, however it isn't something that is viable from the current stdout data output. Another feature that is currently only in testing is the opacity options of links between nodes. They are representative of the values in the 3rd column for the connection between source and target node, but they could be much more. The combination of these would require a nice legend to read, but would allow for some genuinely useful visualizations.

Automated Asset Discovery

With this FlowPlotter update I'm also showing off the new Asset Discovery addition. In chapter 14 of ANSM we discuss methods of gathering "friendly intelligence" and how the importance of having friendly intelligence can't be overstated. Friendly intelligence is information related to the assets that you are tasked with protecting. In almost all environments, the overall picture of those assets frequently changes, so gathering friendly intelligence is a continuous effort. Since flow data provides are very good high level view of the communications between all devices, it is ideal for creating a general idea of the assets you are monitoring on a given network. It also is easily queried, giving you the ability to keep a constant vigilance on new assets. To generate an asset model from SiLK data, I leverage the friendly intel script provided in chapter 14 of ANSM. To stay with the template-based approach to FlowPlotter, I've made the script a standalone part of FlowPlotter that is referenced within flowplotter.sh. The premise behind doing this is to ensure that changes and additions to the asset model script can be easily made without extreme additions to flowplotter.sh. Like the previous graph, I examined many d3 galleries out there to find the best examples of great ways to visualize parent-child relationships, with the final design borrowing from the collapsible tree layout in the d3 galleries. Whereas the force-directed link graph leveraged a CSV as input, I am using JSON as input for the asset discovery module. This is mainly because extensive parent-child relationships aren't easy to represent with CSV data, but instead I can literally specify the data and fields in a JSON tree, with a good idea of how the data will turn out in the collapsible tree layout results. Also, I like JSON because by name is Jason, and it just feels right.

Auto-generated Asset List based on one hour of traffic.

Figure 2: Auto-generated Asset List based on one hour of traffic.

To create an asset model you can either do it from new data piped in via stdout like other FlowPlotter modules or you can create it based on existing rwfilter files. The important thing to remember is that the dataset that you are looking at should be fairly large, or at least representative of normal network traffic over time. For steady network traffic with production systems, an hour might be fine. For some large enterprise environments, shorter timespans will work.

To generate the data from an rwfilter stdout, do the following:

rwfilter --start-date=2014/02/06 --proto=0- --type=all --pass=stdout | ./flowplotter.sh assetdiscovery > assetlist.html

Alternatively you can create the rwfilter file, then generate the asset model from that:

rwfilter --start-date=2014/02/06 --proto=0- --type=all --pass=sample.rw

cat sample.rw | ./flowplotter.sh assetdiscovery > assetlist.html

The asset model currently attempts to identify various service types (HTTP, DNS, VPN, etc) but is receiving regular updates to include more data that could benefit from regular continuous regeneration. This is currently limited to friendly intelligence, but soon I will be including some additional less friendly detections from some of the statistical approaches discussed in ANSM and using some of the less frequently discussed SiLK features. These will all be wrapped up within FlowPlotter as usual. Be sure to digest the README for FlowPlotter to understand the additions a little better. For instance, you may notice that perhaps you're not seeing EVERY "asset" from your network for a given service. That might be due to thresholding which is set by default to look at the "Servers" that display at least 1% of the total "Server-like" traffic representative of a particular service. This default can be changed easily with options located in the README. As always, you should be aware of how far abstracted from your data you really are when generating visualizations like this.

As usual, I'm open to any and all requests for additions or advice on optimizing anything that currently exists in FlowPlotter.

In Applied NSM we wrote quite a bit about both LogStash and Snorby. Recently, a reader of this blog posed the question if there is a way to pivot from Snorby events to your Bro logs in Logstash. Well, it is actually quite easy.

To start, you'll obviously need to have a functional instance of Snorby and Logstash with Bro logs (or any other relevant parsed PSTR-type data) feeding in to it. In this case, we'll assume that to reach our logstash dashboard, we go to http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json.

If you've played around with Snorby, you've probably noticed the lookup sources when you click on an IP in an event. There are default sources, but you can also add additional sources. Those are configured from the Administration tab at the top right.

Lookup Sources

Figure 1: Lookup Sources Option in Snorby

At this time you're only allowed to use two different variables, ${ip} and ${port}, and from my testing, you can only use them once in a given lookup URL. Normally this isn't an issue if, for instance, you are doing research on an IP from your favorite intel source and you're feeding the IP in as a variable on the URL. However, if for some reason you're needing to feed it in twice, referencing ${ip} will only fill the initial variable, and leave the second blank. This becomes an issue with parsed Bro logs in Logstash.

Though not immediately obvious, Logstash allows for you to control the search from the URL as such:

http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json?query=id.orig_h:192.168.1.75

Test it out!

In Snorby, the lookup source for this would be:

http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json?query=id.orig_h:${ip}

However, lets assume you wanted to find logs where 192.168.1.75 existed as either the source or destination address:

http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json?query=id.orig_h:192.168.1.75%20OR%20id.resp_h:192.168.1.75

That search is perfectly valid for use in Logstash, and the URL functions as expected. However, if you make the lookup source in Snorby to match that by pairing ${ip}, you'll notice that only the initial use of ${ip} is completed, and the use on id.resp.h is left blank. For the reason, I would recommend applying the simpler method of just using the message field (the unanalyzed raw log essentially) in the query. We'll also add in ${port} to narrow down more.

http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json?query=message:(${ip} AND ${port})

That lookup source will look for any instance of the ip and matching port within the log. Word of warning that there is a small chance you'll get an unexpected blip somewhere with this method, as it is literally looking for that IP address and that port number as any unique string within the message, in any order. Hypothetically, you could have an odd log that has the selected IP, but the port actually is the response_body_len or some other integer field, though that would be extremely unlikely.

You'll notice that the lookup source is defaulting to the past 24 hours, and is displaying the entire log by default. If we want to change this, we're going to have to utilize a slightly different method by using a "scripted dashboard". The two Logstash searches below search for traffic that contains "91.189.92.152" and "80". However, there are a few differences.

http://192.168.1.12:9292/index.html#/dashboard/file/logstash.json?query=message:(91.189.95.152%20AND%2080)

http://192.168.1.12:9292/index.html#/dashboard/script/logstash.js?query=message:(91.189.95.152%20AND%2080)&fields=@timestamp,id.orig_h,id.orig_p,id.resp_h,id.resp_p,_type,host,uri

In testing both of these, the difference is immediate. You have the ability to custom output fields, which is essential. You'll also notice that on the second URL we're looking at /dashboard/script/logstash.js instead of /dashboard/file/logstash.json. "Scripted Dashboards" are entirely javascript, thus allowing for full control over the output. While we're adding custom fields, lets go ahead and also say that we want to look at the past 7 days (&from=7d) along with referencing the 7 days based on timestamp collected instead of ingestion time (&timefield=@timestamp).

http://192.168.1.12:9292/index.html#/dashboard/script/logstash.js?query=message:(91.189.92.152%20AND%2080)&from=7d&timefield=@timestamp&fields=@timestamp,id.orig_h,id.orig_p,id.resp_h,id.resp_p,_type,host,uri

Like we did before, lets go ahead and add that as a lookup source with the following URL in snorby:

http://192.168.1.12:9292/index.html#/dashboard/script/logstash.js?query=message:(${ip}%20AND%20${port})&from=7d&timefield=@timestamp&fields=@timestamp,id.orig_h,id.orig_p,id.resp_h,id.resp_p,_type,host,uri

 

Examine the URL for example syntax

Figure 2: Note the Special URL Syntax in this LogStash Example

To summarize, here are some of the possible lookup sources I've mentioned, with the advanced lookup being my recommendation for these purposes:

3 Possible Lookup Sources for Pivoting to Logstash

Figure 3: Three Possible Lookup Sources for Pivoting to Logstash

While a time existed when people believed they could prevent sophisticated attackers from breaching advanced networks, I think most people are finally coming around to the fact that prevention eventually fails. No matter how diligent you are in your prevention effort, eventually a motivated attacker can gain some level of access to your network. This is exactly why rapid detection and response is a critical part of network defense strategy.

As a network defender, I think it can be very easy to get discouraged and to feel like you are fighting a losing battle when dealing with sophisticated attackers. Because of that, I’m always searching for the “big wins” that come from detection and response efforts. For that reason, I was thrilled to read this year’s version of Mandiant's annual M-Trends report. I’d like to share a few excerpts from M-Trends 2014: Beyond the Breach that illustrate why.

Last February we released a report that exposed activity that linked People’s Liberation Army Unit 61398, which we call APT1, with espionage against private US companies. M-Trends 2014 tells us that this resulted in an operational pause for APT1.

Mandiant’s release of the APT1 report coincided with the end of Golden Week, a seven-day government holiday that follows Chinese New Year. APT1 was inactive for 41 days longer than normal following Golden Week and the release of Mandiant’s report compared to patterns of activity from 2010–2012.

When APT1 did become active again, it operated at lower-than-normal levels before returning to consistent intrusion activity nearly 160 days after its exposure.

APT1 2013 Activity (Source: M-Trends 2014)

APT1 2013 Activity (Source: M-Trends 2014)

By exposing the tools, tactics, and procedures used by APT1, this pause in operations indicates that this unit was forced to reassess their operations. Beyond this, we find that associated groups that share similar TTPs, such as APT12, were also forced to temporarily pause or slow their operations.

APT12 briefly resumed operations five days after its exposure in The New York Times article, but did not return to consistent intrusion activity until 81 days later. Even then, APT12 waited until roughly 150 days after the article’s release to resume pre-disclosure levels of activity.

APT12 2013 Activity (Source: M-Trends 2014)

APT12 2013 Activity (Source: M-Trends 2014)

These findings prove that even heavily motivated and highly resourced attackers are susceptible to having their operational tempo slowed when their TTP’s are exposed. While we will probably never stop attackers of this nature, degrading their effectiveness to this degree is one of the big wins that keep me motivated as a defender.

In a world where the bad guy always seems to win and we are constantly working in a reactionary state, the ability to share strategic, operational, and tactical threat intelligence gives us a stronger collective ability to rapidly detect and respond to sophisticated attackers. I think there are a few things that need to happen in order for an organization to be able to do this effectively. First, you have to understand the relationship between the different types of threat intelligence. Next, you should understand the impact on the adversary of successfully detecting specific indicator types. Then, you should be able to collect, organize, and share your threat intelligence effectively. When you can do these things well you will be postured to effectively apply threat intelligence to your detection capabilities so that you can really bring the pain to sophisticated adversaries.

You can find some other interesting data points by reading M-Trends 2014: Beyond the Breach.

While signature-based detection isn’t enough on its own to protect a network against structured attackers, it is one of the cornerstones of a successful network security monitoring capability. If you have ever managed a signature-based detection mechanism then you know that you can’t simply turn the device on and let it work its magic. A signature-based detection mechanism like Snort, Suricata, or various commercial offerings requires careful deployment and tuning of signatures (often called rules) to ensure that you receive reliable high quality alerts. The ultimate goal, while never fully achievable, is that every alert that is generated by these detection mechanisms represents the activity it was designed to detect. This means that every alert will be actionable, and you won’t waste a lot of time chasing down false positives.

A key to achieving high fidelity alerting is to have the ability to answer the question, “Do my signatures effectively detect that activity they are designed to catch?” In order to answer that question, we need a way to track the performance of individual signatures. In the past, organizations have relied on counting false positive alerts in order to determine how effective a signature is. While this is a step in the right direction, I believe that it is one step short of a useful statistic. In this article I will discuss how a statistic called precision can be used to measure the effectiveness of IDS signatures, regardless of the platform that you are using.

Definitions

Before we get started, I think its necessary to describe a few terms that are discussed throughout this article. If we are going to play ball, it helps if we are standing on the same field. This article is meant to be platform-agnostic however, so the equipment you use doesn’t matter.

There are four main data points that are generally referred to when attempting to statistically confirm the effectiveness of a signature: true positives, false positives, true negatives, and false negatives.

 True Positive (TP): An alert that has correctly identified a specific activity. If a signature was designed to detect a certain type of malware, and an alert is generated when that malware is launched on a system, this would be a true positive, which is what we strive for with every deployed signature.

False Positive (FP): An alert has incorrectly identified a specific activity. If a signature was designed to detect a specific type of malware, and an alert is generated for an instance in which that malware was not present, this would be a false positive.

True Negative (TN): An alert has correctly not been generated when a specific activity has not occurred. If a signature was designed to detect a certain type of malware, and no alert is generated without that malware being launched, then this is a true negative, which is also desirable. This is difficult, if not impossible, to quantify in terms of NSM detection.

False Negative (FN): An alert has incorrectly not been generated when a specific activity has occurred. If a signature was designed to detect a certain type of malware, and no alert is generated when that malware is launched on a system, this would be a false negative. A false negative means that we aren’t detecting something we should be detecting, which is the worst-case scenario. False negatives aren’t detectable unless you have a secondary detection mechanism or signature designed to detect that activity that was missed.

 

The Fallacy of Relying Solely on False Positives

Historically, the measure of success for measuring the effectiveness for IDS signatures was the count of false positives over an arbitrary time period compared to a certain threshold. Let’s consider a scenario for this statistic. In this scenario, we’ve deployed signature 100, which is designed to detect the presence of a specific command and control (C2) string in network traffic. Over the course of 24 hours, this signature has generated 500 false positive alerts, which results in an FP rate of 20.8/hour. You have determined that an acceptable threshold for false positives is 0.5/hour, so this signature would be deemed ineffective based upon that threshold.

At face value, this approach may seem effective, but it is flawed. If you remember from a few paragraphs ago, the question we want to answer centers on how well a signature can effectively detect the activity it is designed to catch. When we only consider FP’s, then we are actually only considering how well a signature DOESN’T detect what it is designed to catch. While it sounds similar, detecting whether or not something succeeds is not the same as detecting whether or not it fails.

Earlier, we stated that signature 100 was responsible for 500 false positives over a 24-hour period, meaning that the signature was not effective. However, what if I told you that this signature was also responsible for 5,000 true positives during the same time period? In this case, the FPs were well worth it in order to catch 5,000 other actual infections! This is the key to precision -- taking both false positives and true positives into consideration.

 

Defining Precision

At this point it’s probably important to say that I’m not a statistician. As a matter of fact, I only took the bare minimum required math courses in college, but fortunately precision isn’t too tricky to understand. In short, precision refers to the ability to identify positive results. Often referred to as positive predictive value, precision is shown by determining the proportion of true positives against all positive results (both true positives and false positives) with the formula:

Precision = TP / (TP + FP)

 

This value is expressed as a percentage, and can be used to determine the probability that, given an alert being generated, the activity that has been detected has truly occurred. Therefore, if a signature has a high precision and an alert is generated, then the activity has very likely occurred. On the other hand, if a signature with low precision generates an alert, then it is unlikely that the activity you are attempting to detect has actually occurred. Of course, this is an aggregate statistic, so more data points will generate result in a more reliable precision stat.

In the example we identified in the previous section, we could determine that signature 100, which had 500 FPs and 5000 TPs has a precision of 90.9%.

 

5000 / (5000 + 500) = 90.9

 

That's pretty good!

 

Using Precision in the SOC

Now that you understand precision, it is helpful to think about ways it can be used in a SOC environment. After all, you can have the best statistics in the world but if you don’t use them effectively then they aren’t providing a lot of value. This begins by actually making the commitment to track a precision value for each signature being deployed for a given detection mechanism. Again, it doesn’t matter what detection mechanism you are using. Whenever you deploy a new signature it would be assigned a precision of 0. As analysts review alerts generated by signatures, they will mark the alerts as either a TP or FP. As time goes on, these data points help to determine the precision value, expressed as a percentage.

 

Precision for Signature Tuning

First and foremost, precision provides a mechanism that can be used to track the effectiveness of individual signatures. This provides quite a bit of flexibility that you can tune to the sensitive of your own staffing. If you are a small organization with only a single analyst you can choose to only keep signatures with a precision greater than a high value like >80%. If you have a larger staff and have more resources to devote to alert review, or if you simply have a low risk tolerance, you can choose to keep signatures with lower precision values like >30%. In some instances, you can even define your precision threshold based upon the nature of the threat you are dealing with. Unstructured or commodity signatures could allow for precision >90% but structured or targeted signatures might only be considered effective if they are >20% precision.

There are a few different paths that can be taken when you determine that a signature is ineffective based upon the precision standards you have set. One path would be to spend more time researching whatever it is that you are trying to detect with the signature, and attempt to strengthen it by making your search more specific, or adding more specifics to the content of the signature. Alternatively, you can configure low precision signatures to simply log instead of an alert (an option in a lot of popular IDS software), or you can configure your analysis console (SIEM, etc.) to hide alerts below your acceptable precision threshold.

 

Precision for Signature Comparison

On occasion, you may run into situations where you have more than one signature capable of detecting similar activity. When that happens, precision provides a really useful statistic for comparing those signatures. While you some instances might warrant deployment of both signatures, resources on detection systems can be scarce at times. As such, it might be helpful for performance to only enable to most useful signature. In a scenario where signature 1 has a precision of 85% and signature 2 has a precision of 22%, then signature 1 would be the best candidate for deployment.

 

Precision for Analyst Confidence

Whenever you write a signature, you should consider the analyst who will be reviewing alerts that are generated from that signature. After all, they are the ultimate consumer for the product you are generating, so any context or help you can provide the analyst in relation to the signature is helpful.

One item in particular that can help the analysts is the confidence rating associated with a signature. A lot of detection mechanisms will allow their signature to include some form of confidence rating. In some cases, this is automatically calculated based on magic, which is how a lot of commercial SIEMs operate. In other scenarios, it might be a blank field where you can input a numeric value, or a simple low/medium/high option. I would propose that a good value for this field would be the precision statistic. While precision might not be the only factor that goes into a confidence calculation, I certainly believe it has to be one of them. Another approach might be to have a signature developer subjective confidence rating (human confidence) and a precision-based confidence rating (calculated confidence.)

A confidence rating based on precision is useful for an analyst because it helps them direct their efforts while analyzing alerts. That way, when an alert is generated and the analyst is on the fence regarding whether it might be a TP or FP, they can rely somewhat on that confidence value to help them determine how much investigative effort should be put into attempting to classify the alert. If they aren’t finding much supporting evidence that provides the alert is a TP and it has a precision-based confidence rating of 10%, then it is likely a FP and it might not be worth it to spend too much time on the alert. On the other hand, if it has a rating of 98%, then the analyst should spend a great deal of time and perform thorough due diligence to determine if the alert is indeed a true positive.

 

Conclusion

Signature management can be a tricky business, but it is such a critical part of NSM detection and analysis. Because of that it should be a heavily tracked and constantly improving process. While the precision statistic is by no means a new concept, it is one that I think most people are unaware of as it relates to signature management. Having seen precision used very successfully to help manage IDS signatures in a few varying and larger organizations, I think it is a statistic that could find a home in many SOC environments.

notebook-angle1In Applied NSM, I write about the importance of creating a culture of learning in a SOC. This type of culture goes well beyond simply sending analysts to training or buying a few books here and there. It requires dedication to the concepts of mutual education, shared success, and servant leadership. It’s all about every single moment in a SOC being spent teaching or learning, no exceptions. While most analysts live for the thrill of hunting adversaries, the truth is that the majority of an analysts time will be spent doing less exciting tasks such as reviewing benign alerts, analyzing log data, and building detection signatures. Because of this, it can be difficult to find ways to foster teaching and learning during these times. I’ve struggled with this personally as an analyst and as a technical manager leading analyst teams. In the article, I’m going to talk about an item that I’ve use to successfully enhance the culture of learning in SOC environments I’ve worked in: a spiral notebook.

 

Background

At some point while I was working at the Bowling Green, KY enclave of the Army Research Laboratory I realized that I had a lot of sticky notes laying around. These sticky notes contained items that you might expect analysts to write down during the course of an investigation: IP addresses, domain names, strings, etc. I decided that I should really keep my desk a bit cleaner and organize my notes better in case I needed to go back to them for any reason. I figured the best way to do this was to just put them in a notebook that I kept with me, so I walked to the Dollar General next door and bought a college-ruled spiral notebook for 89 cents. Henceforth, any notes I took while performing analysis stayed in this notebook.

Over time, I began to expand the use of my notebook. Instead of just scribbling down notes, I started writing down more information. This included things like hypotheses related to alerts I was currently investigating and notes about limitations of tools that I experienced during an investigation. I became aware of the value of this notebook pretty quickly. As a senior analyst on staff, one of my responsibilities was to help train our entry-level analysts along with my normal analyst duties. Invariably, these analysts would run into some of the same alerts that I had already looked at. I found that when this happened and these analysts had questions, I could quickly look back at my notebook and explain my investigation of the event as it occurred. The notebook had become an effective teaching tool.

Fast forward a little bit, and I had been promoted to the lead of the intrusion detection team. The first thing I did was to walk down to the Dollar General and buy a couple dozen notebooks for all of my analysts. Let’s talk about a few reasons why.

 

The Analyst Notebook for Learning and Teaching

As an analyst, I am constantly striving for knowledge. I want to learn new things so that I can enhance my skill and refine my processes so that I am better equipped to detect the adversary when they are attacking my network. This isn’t unique to me; it is a quality present in all NSM analysts to some degree. This is so important to some analysts that they will seek new employment if they feel that they aren’t in a learning environment or being given an adequate opportunity to grow their skills. I surveyed 30 of my friends and colleagues who had left an analysis job to pursue a similar job at another employer within the past five years. I asked them what was it that ultimately caused them to leave. The most logical guess would be that the analysts were following a bigger paycheck or a promotion. Believe it or not, that was true for only 23% of respondents. However, an overwhelming 63% of those surveyed cited a lack of educational opportunity as the main reason they left their current analysis job.

 

notebook-figure1

Figure 1: Survey Results for Why Analysts Leave Their Jobs

 

This statistic justifies a need for a culture of learning. I think that the analyst notebook can be a great way to foster that learning environment because I know that it has been a great learning tool for me. This really clicked for me when I started asking a very important question as I was performing analysis.

 

Why?

 

This lead to questions like this:

  • Why does it take so long to determine if a domain is truly malicious?
  • Why do IP addresses in this friendly range always seem to generate these types of alerts?
  • Why do I rarely ever use this data type?
  • Why don’t I have a data type that lets me do this?
  • Why does this detection method never seem to do what it is supposed to di?
  • Why don’t I have any additional intelligence sources that can help with an investigation like this one?
  • Why don’t I have more context with this indicator?
  • Why do I need to keep referencing these old case numbers? Is there a relationship there?
  • Why do I keep seeing this same indicator across multiple attacks? Is this tied to a single adversary?

 

These questions are very broad, but they are all about learning your processes and generating ideas. These ideas can lead to conversations, and those conversations can lead to change that helps you more effectively perform the task at hand. Small scribbles in a notebook can lead to drastic changes in how an organization approaches their collection, detection, and analysis processes.  In the Applied NSM book, I write about two different analysis methods called the Differential Diagnosis and Relational Investigation. These are methods that I use and teach, and they both started from notes in my notebook. As a matter of fact, a lot of the concepts I describe in Applied NSM can be found in a series of analyst notebook that I’ve written in over the years. As an example, Figure 2 shows an old analyst notebook of mine that contains a note that led to the concept of Sensor Visibility Diagrams, which I described in Chapter 3 of Applied NSM and implemented in most every place I’ve worked since then.

 

notebook-samplenote1

Figure 2: A Note that Led to the Development of Sensor Visibility Maps

 

I think the formula is pretty simple. Write down notes as you are doing investigations, regularly question your investigative process by revisiting those notes, and write down the ideas you generate from that questioning. Eventually, you can flesh those ideas out more individually or in a group setting. You will learn more about yourself, your environment, and the process of NSM analysis.

Analyst Notebook Best Practices

If I’ve done a good job so far, then maybe I’ve already convinced you that you need to walk down to the store and buy a bunch of notebooks for you and all of your friends. Before you get started using your notebook, I want to share a few “best practices” for keeping an analyst notebook. Of course, these are based upon my experience and have worked for the kind of culture I’ve wanted to create (and be a part of). Those things might be different for you, so your mileage may vary.

Let’s start with a few ground rules for how the notebook should be used. These are very broad, but I think they hold true to most scenarios for effective use.

  1. The Analyst Notebook should always be at your desk when you are. If it isn’t, then you won’t write in it while you performing analysis, which is the whole point.
  2. The Analyst Notebook should go to every meeting with you. If an analyst is in a meeting then there is a good chance they will have to discuss a specific investigation, their analysis process, or the tools they use. Having the notebook handy is important so that relevant notes can be analyzed.
  3. The Analyst Notebook should never leave the office.  This is for two reasons. First, this tends to result in the notebook being left at home on accident. Second and most important, I believe strongly in a separation of work and home life. There is nothing wrong with putting in a few extra hours here and there, but all work and no play ultimately lead to burnout. This is a serious problem in our industry where it seems as though people are expected to devote 80+ hours a week to their craft. Being an analyst is what I do, but isn’t who I am. The analyst notebook stays at work. When you go home, focus on your family and other hobbies.
  4. Every entry in the Analyst Notebook should be dated. Doing this consistently will ensure that you can piece together items from different dates when you are trying to reconstruct a long-term stream of events. It will also allow you to tie specific notes (whether they are detailed or just scribbles of IP addresses) to case numbers.
  5. An analyst must write something in the notebook every day. In general, the investigative process should yield itself to plenty of notes. If you find that isn’t the case, then start daydreaming a bit. What do you wish one of your tools could do that it can’t? What type of data do you wish you had? How much extra time did you spend on a task because of a process inefficiency? These things can come in handy later when you are trying to justify a request to management or senior analysts. This is hard to get in the groove of at first, but it is a habit that can be developed.
  6. The analyst notebook should be treated as a sensitive document. The notebook will obviously contain information that could cause an issue for you or your constituents if a party with malicious intent obtained it. Accordingly, the notebook should be protected at all times.  This means you shouldn’t forget it on the subway or leave it sitting on the table at Chick-Fil-A while you go to the bathroom.

 

Effectively Using an Analyst Notebook

Finally, let’s look at some strategies for effective analyst notebook use that I think are applicable to people of different experience levels. My goal is for this article to be valuable to new analysts, senior analysts, and analyst managers alike. With that in mind, this section is broken into a section for each group.

 

I’m a New Analyst!

Because new analysts are often overwhelmed by the amount of data and the number of tools they have to work with, I encourage you to write down every step they take during an investigation so you can look back and review the process holistically. While this does take a bit of time, it will eventually result in time savings by making your analysis process more efficient overall. This isn’t meant to describe why you took the actions you took and be overly specific, but should help you replay the what steps you took so you can piece together your process. This might look like Figure 3.

 notebook-Figure3Figure 3: A Note Detailing the Analysis Steps Taken

This exercise becomes more useful when you are paired with more senior analysts so that they can review the investigation that was completed. This provides the opportunity to walk the senior analyst through your thought process and how you arrived at your conclusion. This also provides the senior analyst with the ability to describe what they would have done differently.

This type of pairing is a valuable tool for overcoming some of the initial process hurdles that can trip up new analysts. For instance, I’ve written at length about how most new analysts tend to operate with a philosophy that all network traffic is malicious unless you can prove it is not. As most experienced analysts know, this isn’t a sustainable philosophy, and in truth all network traffic should be treated as inherently good unless you can prove it is malicious. I’ve noticed that by having new analysts take detailed notes and then review those notes and their process with a more experienced analyst, they get over this hump quicker.

 

I’m a Senior Analyst!

As a more experienced analyst, it is likely that you’ve already refined your analysis technique quite a bit. Because of this, in addition to general analysis duties you are likely going to be tasked with bigger picture thinking, such as helping to define how collection, detection, and analysis can be improved. In order to help with this, I recommend writing down items relevant to these processes for later review. This can include things like tool deficiencies, new tool ideas, data collection gaps, and rule/signature tweak suggestions.

As an example, consider a scenario where you are performing analysis of an event and notice that a user workstation that normally acts as a consumer of data has recently become a producer of data. This means that a device that normally downloads much more than it uploads from external hosts has now begun doing the opposite, and is uploading much more than it downloads. This might eventually lead you to find that this host is participating in commodity malware C2 or is being used to exfiltrate data. In this case, you may have stumbled upon this host because of an IDS alert or through manual hunting activities. When the investigation heats up you probably aren’t going to have time to flesh out your notes on how you can identify gaps in your detection capability, but you can quickly use an analyst notebook to jot down a note about how you think there might be room to develop a detection capability associated with detecting changing in producer/consumer (upload/download) ratio.

 notebook-Figure4

Figure 4: A Note Detailing a Potential Detection Scenario

You may not yet realize it but you’ve identified a use case for a new statistical detection capability. Now you can go back later and flesh this idea out and then present it to your peers and superiors for detection planning purposes and possible capability development. This could result in the development of a new script that works off of flow data, a new Bro script that detects this scenario out right, or some other type of statistical detection capability.

 

I’m an Analyst Manager!

As a manager of analysts, you are probably responsible for general analysis duties, helping to refine the SOC processes, and for facilitating training amongst your analysts. While I still recommend keeping an analyst notebook at this level for the reasons already discussed, the real value of the analyst notebook here is your ability to leverage the fact that all of the analysts you manage are keeping their notebooks. In short, it is your responsibility to ensure that the notes your analysts keep in their notebooks become useful by providing them opportunities to share their thoughts. I think there are a couple of ways to do this.

The first way to utilize the notebooks kept by your analysts is through periodic case review meetings. I think there are several ways to do this, but one method I’ve grown to like is to borrow from medical practitioners and have Morbidity and Mortality (M&M) style case reviews. I’ve written about this topic quite extensively, and you can read more about this here (http://chrissanders.org/2012/08/information-security-incident-morbidity-and-mortality/) or in Chapter 15 of the Applied NSM book. These meetings are especially important for junior level analysts who are just getting their feet wet.

Another avenue for leveraging your analysts and their notebooks is through periodic collection and detection planning meetings. In general, organizations tasked with NSM missions should be doing this regularly, and I believe that analysts should be highly involved with the process. This gives your senior level analyst an avenue to share their ideas based upon their work in the trenches. I speak to collection planning and the “Applied Collection Framework” in Chapter 2 of the Applied NSM book, and I speak to detection planning a bit here while discussing ways to effectively use APT1 indicators: http://www.appliednsm.com/making-mandiant-apt1-report-actionable/.

 

Conclusion

I sincerely believe that a simple spiral notebook can be an analyst’s best tool for professional growth. If you are a junior analyst, use it as a tool to develop your analytic technique. If you are a senior analyst, use it as a tool to refine NSM-centric processes in your organization. If you are responsible for leading a team of analysts, ensure that your team is provided the opportunity to use their notebook effectively to better themselves, and your mission. An $0.89 cent notebook can be more powerful than you’d think.

 

 

 

I’ve had the opportunity to directly and indirectly lead teams of talented individuals while working for the Department of Defense in various SOC leadership roles. Anybody who has worked for or with me in those roles knows about my “dirty words” list. Now, these aren’t the typical seven dirty words that the FCC will fine you for if you happen to let one slip on network television, but rather, a series of buzzwords and phrases relevant to information security that tend to be inappropriately applied to certain scenarios or used in the wrong context.

You probably already know about some of these words. For instance, the most revered amongst security practitioners is probably “Advanced Persistent Threat”, which every security appliance vendor on the planet now claims to be able to detect or prevent, even if they can’t clearly define it. Two more favorites are “sophisticated” and “motivated.” These terms are used often to describe attacks, without honoring the fact that the degree of difficulty involved in an attack is very relative to the audience who is analyzing it. While a skilled defender might not consider an attack sophisticated, the attack may still be very advanced for a non-technical person. Furthermore, an attacker is only as sophisticated or motivated as their objective requires. If their tactics allows them to achieve their goals, then the attacker was motivated and sophisticated enough.

Unfortunately, “intelligence” is becoming one of these dirty words. You don’t have to look far to find a company or product that claims to provide “incredible insight through advanced network intelligence” or “the answer to network defense through thorough threat intelligence.” However, even though intelligence has become the latest major buzzword in network defense, I think that it is important when used appropriately. After all, intelligence IS a crucial part of network defense strategy.

So, how do we get away from using “intelligence” as a dirty word? I think the answer lies in carefully identifying what types of intelligence we are producing.

Intelligence has many definitions depending on the application. The definition that most closely aligns to information security is drawn from Department of Defense Joint Publication 1-02, and says that “intelligence is a product resulting from the collection, processing, integration, evaluation, analysis, and interpretation of available information concerning foreign nations, hostile or potentially hostile forces or elements, or areas of actual or potential operations .”

While this definition might not fit perfectly in all instances (particularly the part about information concerning foreign nations since an attacker might be domestic), it does provide the all-important framing required to begin thinking about generating intelligence. The key component of this definition is that intelligence is a product. This doesn’t mean that it is bought or sold for profit, but more specifically, that it is produced from collected data, based upon a specific requirement. This means that an IP address, or the registered owner of that address, or the common characteristics of the network traffic generated by that IP address are not intelligence products. When those things are combined with context through the analysis process and delivered to meet a specific requirement, they become an intelligence product.

In information security, we are generally most concerned with the development of threat intelligence products. These products seek to gather data to support the creation of an intelligence product that can be used to make determinations about the nature of a threat. What is lost on most is that there are actually three major subsets of threat intelligence: strategic, operational, and tactical intelligence.

Strategic Intelligence is information related to the strategy, policy, and plans of an attacker at a high level. Typically, intelligence collection and analysis at this level only occurs by government or military organizations in response to threats from other governments or militaries. With that said, larger organizations are now developing these capabilities, and some of these organizations now sell strategic intelligence as a service. This is focused on the long-term goals of the force supporting the individual attacker or unit. Artifacts of this type of intelligence can include policy documents, war doctrine, position statements, and government, military, or group objectives.

Operational Intelligence is information related to how an attacker or group of attackers plans and supports the operations that support strategic objectives. This is different from strategic intelligence because it focuses on narrower goals, often more timed for short-term objectives that are only a part of the big picture. While this is, once again, usually more within the purview of government or military organizations, it is common that individual organizations will fall victim to attackers who are performing actions aimed at satisfying operational goals. Because of this, some public organizations will have visibility into these attacks, with an ability to generate operational intelligence. Artifacts of this type of intelligence are similar, but often more focused versions of artifacts used for the creation of strategic intelligence.

Tactical Intelligence refers to the information regarding specific actions taken in conducting operations at the mission or task level. This is where we dive into the tools, tactics, and procedures used by an attacker, and where 99% of information security practitioners will focus their efforts. It is here that the individual actions of an attacker or group of attackers are analyzed and collected. This often includes artifacts such as indicators of compromise (IP addresses, file names, text strings) or listings of attacker specific tools. This intelligence is the most transient, and becomes outdated quickly.

The discussion of these types of threat intelligence naturally leads us to another recently popularized dirty word, “attribution.”

Attribution occurs when the actions of an adversary are actually tied back to a physical person or group. The issue with this word arises when information security practitioners attempt to perform attribution as a sole function of intrusion detection without the right resources. It is important to realize that detection and attribution aren’t the same thing, and because of this, detection indicators and attribution indicators aren’t the same thing. Detection involves discovering incidents, where as attribution involves tying those incidents back to an actual person or group. While attribution is most certainly a positive thing, it cannot be done successfully without the correlation of strategic, operational, and tactical threat intelligence data.

Generally speaking, this type of intelligence collection and analysis capability is not present within most private sector organizations without an incredibly large amount of visibility or data sharing from other organizations. The collection of indicators of compromise from multiple network attacks to generate tactical intelligence is an achievable goal. However, collecting and analyzing data from other traditional sources such as human intelligence (HUMINT), signals intelligence (SIGINT), and geospatial intelligence (GEOINT) isn’t within the practical capability of most businesses. Furthermore, even organizations that might have this capability are often limited in their actions by law. Of course, there are some companies that exist who are producing high quality attribution intelligence, so there are exceptions to the rule.

Intelligence is a tremendously valuable thing, and when it is used in the proper context, it shouldn’t have to be a dirty word. The key to not misusing this word in your organization is to ensure that you are focused on intelligence that you actually have the capability to collect, analyze, and utilize.

 

** Note: This content originally appeared on the InGuardians Labs blog. I'm reposting it here since I've changed employment.

TLDR; Check out FlowPlotter on GitHub

If you're like me, it generally makes you happy to see graphs, charts, and tables to aide in network data analysis. This can be useful for analysis of already detected events, or for detection while hunting for evil. You probably also love session (also often called flow) data. Unfortunately, it isn't always easy to generate useful visualizations from flow data. This typically involves multiple steps such as moving data around between different tools, formatting the output of the data to match the format of whatever graphing tool you might be using, and then generate the graph output and making sure it is useful for your goals. Some might turn to the graphing capabilties of spreadsheet applications because of their simplicity, but those can't really handle a large data set like we might see with flow data. With that said, it is still pretty hard to find overly useful network visualizations for NSM detection and analysis.

 

Because of this, I set out to make visualizations from flow data easy and accessible, without the need for several steps between viewing the raw data and having the ready-made chart. The result of this was a tool called FlowPlotter, which we are going to discuss in this article. We will talk about how FlowPlotter came into existence, and its current workflow. FlowPlotter works from NetFlow records viewed with SiLK, so before moving forward, be sure to check out Chris's instructions on setting SiLK up on Security Onion so that you can easily test the tool if you don't already have a SiLK environment available. You can also go ahead and grab FlowPlotter from GitHub so that we can jump into making graphs. To generate graphs, you will only need SiLK and FlowPlotter. If the nitty gritty details bore you and you want to get into FlowPlotter right away with examples, skip this API stuff and scroll down below.

 

Background Mechanics

We will begin by talking about how FlowPlotter works. For the purpose of portability across multiple platforms, it makes sense that we should be viewing this kind of visualization in a web friendly format, rather than just a jpeg or Excel graph. To do this, FlowPlotter uses an implementation of Google Charts. Google charts offers an extensive API that allows for the creation of just about any chart you can imagine. While many of these make more sense to have only a few rows of data (Pie charts for example), some benefit from having as much data as can be shoved in to them (line charts representing bins of data). Google provides thorough explanations on how to create these charts using your own data, but of course, the formatting of the data is still up to you. See their chart gallery for more examples on the structure of these charts.

 

In Applied Network Security Monitoring, we discuss several ways of seeing the "big picture" when it comes to visualizing data of many types. We also go into deep detail on using SiLK to get the most from your flow data. Here I'd like to present a simple method of going from using SiLK to parse data with rwtools, straight to using some bash kung-fu to streamline our way to having an interactive Google Chart full of flow data in your browser. The eventual goal of this exercise is to run rwfilter commands and pipe the binary data straight to a script which will take arguments to generate the data you want. This results in having a tool that we can pass data to that will generate charts using the Google API.  Before we can run though, we need to walk. We'll start by creating a chart that requires fewer data points, such as a bar chart representing the top 10 country codes talking to your network by bytes over the course of a day. The first thing we want to do is generate the SiLK data that we eventually want to plot. Since the goal of this is to make a top-10 lists, we'll use rwstats and rwfilter.

 

rwfilter --start-date=2014/02/06 --proto=0- --type=all --pass=stdout | rwstats --top --count=10 --fields=dcc --value=bytes

Screen Shot 2014-02-13 at 3.29.10 PM

The rwfilter command above states that you wish to filter down to all data that passes the filter --proto=0-255 (shorthand is 0-) that occurred on 2014/02/06. These can be stated in any order, and many people like to type them out as they would verbally say them. For instance, on the rwstats command, we're literally looking for the "top" "10" "destination country codes" by "bytes". It seems that many of the people I teach SiLK to end up having the issue of thinking too rigidly about "what SiLK wants", instead of just writing down what you want, then converting that directly into a query.

 

Now we're going to try and make that rwstats output appear in a bar graph. FlowPlotter, which I talk about in more detail below, works by using templates that allow for the formatted data to be inserted into which then yields a complete chart. Lets look at the most basic template for a column chart. This is taken directly from Google's visualization playground, with their data substituted out for "dataplaceholder" around the middle of the code. You'll even notice that for now, the title is still "Company Performance".

<html>
<head>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = google.visualization.arrayToDataTable([
dataplaceholder
]);

var options = {
title: 'Company Performance',
vAxis: {title: 'Year',  titleTextStyle: {color: 'red'}}
};

var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
chart.draw(data, options);
}
</script>
</head>
<body>
<div id="chart_div" style="width: 900px; height: 500px;"></div>
</body>
</html>

You'll notice the "dataplaceholder" that exists where there should be a table of data. In the place of that, we should be inserting something that looks like the following, which was created from our rwstats command:

['dcc', 'Bytes'],
['--', 26478355345],
['us', 706854881],
['ca', 8665204],
['no', 1893193],
['nl', 1416293],
['bg', 1101223],
['ch', 1092811],
['de', 202948],
['se', 169036],
['gb', 117399]

You'll notice that "- -" is also a country code here representing PRIVATE/EXPERIMENTAL address, which we'll leave for the sake of discussing additional chart features and manipulations later. In the meantime, how did I streamline the creation of that data to replace dataplaceholder? First it is important to add the "--delimited=," option on to rwstats to ease the post processing a bit. After that, I used a mix of cut, sed, and grep to wrap it all into a one liner:

rwfilter --start-date=2014/02/06 --proto=0- --type=all --pass=stdout | rwstats --top --count=10 --fields=dcc --value=bytes --delimited=, | cut -d "," -f1,2 |grep ,| sed "s/\(.*\),\(.*\)/['\1', \2],/g"|sed '$s/,$//'| sed "s/, \([A-Za-z].*\)],/, '\1'],/g" | grep ","

There are probably better ways of generating that data, and I welcome them in comments or variations of FlowPlotter, but for the sake of this particular exercise, this will get you by. The general idea is that the rwstats command output is piped to cut, which strips out the first two columns, then grep is used to filter only data fields and titles. The sed commands that follow sorts everything so that they all have the proper formatting, first by making the table, then by formatting the end line and then the first column identifier line. Now that we have the basic template and some example data, you can manually throw the data into the template  in place of dataplaceholder and change some of the obvious things such as the title so that the HTML looks like the following:

<html>
<head>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = google.visualization.arrayToDataTable([
['dcc', 'Bytes'],
['--', 26478355345],
['us', 706854881],
['ca', 8665204],
['no', 1893193],
['nl', 1416293],
['bg', 1101223],
['ch', 1092811],
['de', 202948],
['se', 169036],
['gb', 117399]
]);

var options = {
title: 'Destination Country Code by Bytes',
vAxis: {title: 'Country Codes', titleTextStyle: {color: 'black'}}
};

var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
chart.draw(data, options);
}
</script>
</head>
<body>
<div id="chart_div" style="width: 900px; height: 500px;"></div>
</body>
</html>

Here you can see the code we've generated.

Screen Shot 2014-02-13 at 8.33.18 PM

Notice that while mouse-over works (at the link) and overall it is a decent looking graph, the scale is being ruined by our large amount of internal destination addresses. There are two options to fix this, one that is obvious but not great, and one that is not obvious, but is by far the better method. Either we can get rid of the internal destinations from the data manually or we can accept it and change the chart to be more forgiving for large outliers. To do that, we need to edit the var options in our code. We're going add in an hAxis option as seen below. This will make the horizontal axis work on a logarithmic scale instead of scaling according to maximum and minimum data values.

<html>
<head>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = google.visualization.arrayToDataTable([
['dcc', 'Bytes'],
['--', 26478355345],
['us', 706854881],
['ca', 8665204],
['no', 1893193],
['nl', 1416293],
['bg', 1101223],
['ch', 1092811],
['de', 202948],
['se', 169036],
['gb', 117399]
]);

var options = {
title: 'Destination Country Code by Bytes',
vAxis: {title: 'Country Codes', titleTextStyle: {color: 'black'}},
hAxis: {logScale: true}
};

var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
chart.draw(data, options);
}
</script>
</head>
<body>
<div id="chart_div" style="width: 900px; height: 500px;"></div>
</body>
</html>

Our new graph looks like this.

Screen Shot 2014-02-13 at 8.33.38 PM

In testing changes like this, I highly recommend playing around in Google Chart's Playground as it can streamline the debugging of small changes.

 

FlowPlotter

Now that you've got an idea of how to manually generate these graphs, we can talk about FlowPlotter in a more official capacity. FlowPlotter is a scripted approach to generating graphs based on SiLK data by using templates so that it is modular enough to accept new graphs with relative ease. In short, it automates everything we just did in the previous example. The only requirement is that you provide an rwfilter command and send that to flowplotter.sh with a chart name and it's independent and dependent variables as options. From there FlowPlotter will make the html page for you, complete with titles and ideal scaling options. For instance, to generate the previous graph, you would simply run the following from the FlowPlotter root directory:

/Flowplotter$ rwfilter --start-date=2014/02/06 --proto=0- --type=all --pass=stdout | ./flowplotter.sh barchart dcc bytes

Here is the current usage page for FlowPlotter:

rwfilter [filter] | flowplotter.sh [charttype] [independent variable] [dependent variable]

Currently you must run a SiLK rwfilter command and pipe it to flowplotter.sh and specify various options as arguments. The following chart types are currently functional

geomap

  • independent variable = Must specify an rwstats compatible field for country type (scc or dcc).
  • dependent variable = Must specify an rwstats compatible value (Records, Packets, Bytes, sIP-Distinct, dIP-Distinct, or Distinct:)

linechart

  • independent variable = Must specify a bin-size that the dependent variable will be calculated by. For example, if you want "Records per Minute", this variable will be 60.
  • dependent variable = Must specify an rwcount compatible value (Records,Packets,Bytes).

treemap

  • independent variable = Must specify an rwstats compatible field.
  • dependent variable = Must specify an rwstats compatible value (Records, Packets, Bytes, sIP-Distinct, dIP-Distinct, or Distinct:)

timeline

  • independent variable = Must specify an rwcut compatible field.
  • dependent variable = Must specify an rwcut compatible field.

piechart

  • independent variable = Must specify an rwstats compatible field.
  • dependent variable = Must specify an rwstats compatible value (Records, Packets, Bytes, sIP-Distinct, dIP-Distinct, or Distinct:)

barchart

  • independent variable = Must specify an rwstats compatible field.
  • dependent variable = Must specify an rwstats compatible value (Records, Packets, Bytes, sIP-Distinct, dIP-Distinct, or Distinct:)

columnchart

  • independent variable = Must specify an rwstats compatible field.
  • dependent variable = Must specify an rwstats compatible value (Records, Packets, Bytes, sIP-Distinct, dIP-Distinct, or Distinct:)

 

As you can see, FlowPlotter doesn't just support bar charts. It currently supports numerous Google Charts. The charts below were all generated using the queries you see accompanying them.

Geomaps

rwfilter --start-date=2013/12/27 --proto=0- --type=all --pass=stdout | ./flowplotter.sh geomap dcc bytes > geomap.html

Screen Shot 2014-02-11 at 5.36.15 PM

 

Linecharts

rwfilter --start-date=2013/12/27 --proto=0- --type=all --pass=stdout | ./flowplotter.sh linechart 60 bytes > linechart.html

Screen Shot 2014-02-11 at 5.47.29 PM

 

Treemaps

rwfilter --start-date=2013/12/27 --sport=1025- --dport=1025- --not-daddress=192.168.1.0/24 --proto=0- --type=all --pass=stdout | ./flowplotter.sh treemap dip records > treemap.html

Screen Shot 2014-02-11 at 5.52.04 PM

 

Timelines

rwfilter --start-date=2013/12/27 --proto=0- --type=out,outweb --dcc=us,-- --fail=stdout | ./flowplotter.sh timeline sip dip > timeline.html

Screen Shot 2014-02-11 at 5.54.16 PM

 

Pie Charts

rwfilter --start-date=2013/12/27 --sport=1025- --dport=1025- --not-daddress=192.168.1.0/24 --proto=0- --type=all --pass=stdout | ./flowplotter.sh piechart dport bytes > piechart.html

Screen Shot 2014-02-11 at 5.55.37 PM

 

Bar Charts

rwfilter --start-date=2013/12/27 --sport=1025- --dport=1025- --not-daddress=192.168.1.0/24 --proto=0- --type=all --pass=stdout | ./flowplotter.sh barchart dport bytes > barchart.html

Screen Shot 2014-02-11 at 6.01.00 PM

 

Column Charts

rwfilter --start-date=2013/12/27 --sport=1025- --dport=1025- --not-daddress=192.168.1.0/24 --proto=0- --type=all --pass=stdout | ./flowplotter.sh columnchart dip bytes > columnchart.html

Screen Shot 2014-02-11 at 6.04.19 PM

Next Steps

FlowPlotter currently only supports charts in the Google Visualizations Chart library, but as time goes by, I'd like to add some sources outside of just google, even if they are duplicates of similar google graphs. I have the project on Github and welcome any comments, ideas, and improvements that you might have. It has examples that you can use, but I encourage the use of any kind of rwfilter input you can think of. If you come across some great visualizations that you think are repeatable by others, post the rwfilter | flowplotter command up as a comment and I'll add it to the examples!

*** Edit 12/30 - The contest is now over, and winners have been notified. Thanks to all of the 100+ folks who entered! ***

We are giving away two FREE signed copies of Applied NSM for the holidays. If you haven't bought a copy yet, you can enter the contest by sending an e-mail to chris@chrissanders.org with "Applied NSM Giveaway" in the subject line. I'll pick winners on December 30th, so you can submit your entry up until midnight the night before. Since I'll be mailing physical copies of the book, so only individuals with US shipping addresses are eligible to win.

I was absolutely thrilled by a special delivery I got yesterday. The print copies of Applied NSM have arrived! I've also seen that a few folks who have preordered have gotten their copies, and it looks like it is now available on the Elsevier bookstore and for Prime shipping from Amazon. A big thanks to all that have preordered and are planning to order!

 book1

It's no secret that I'm a big fan of Wireshark. While it isn't always the best tool for every job, it is the best graphical packet analysis application you will find, and is a must have for NSM analysis. I wanted to share a quick tip that I use nearly every time I'm using Wireshark for analysis.

Most people know that Wireshark will do host name resolution. As a matter of fact, I generally recommend people disable this feature so that your analysis is not causing the generation of additional traffic on the wire when the machine you are running Wireshark from starts generating DNS queries for the hosts in your capture file. However, what a lot of people don't know is that you can actually create a host file just for use by Wireshark so that you can easily identify certain IP addresses.

To do this, let's start with a basic capture file. In Figure 1, there is some traffic being transmitted between a few different hosts.

WS-HostFile-Figure1

Figure 1: Traffic Between A Lot of Hosts

It's pretty common in analysis to be required to examine packet captures that contain traffic from multiple hosts. When this happens, it can be confusing remembering which IP address is what. In this case, let's say that we know that 192.168.3.35 is our friendly host, and 188.124.5.107 is the hostile host we are concerned about. Since there is a lot of other traffic to be found here, it would be nice if we could easily identify these hosts without committing these IP addresses to memory. I don't know about you, but I'm horrible at remembering IP addresses. Especially when I'm having to juggle what may be multiple compromised systems or track down a web of systems involved in a compromise.

Let's remedy this by creating a Wireshark host file. First, we need to tell Wireshark to perform name resolution for IP addresses from a host file. To do this, open Wireshark's preference window (Edit -> Preferences on Windows or Wireshark -> Preferences on OS X). Then make sure that "Resolve network (IP) addresses" and "Only use the profile "hosts" file" are enabled. Also, disable "Use an external name resolver." This is shown on an OS X system (running the latest dev version of Wireshark) in Figure 2.

WS-Hostname-Figure2

Figure 2: Enabling Host File Name Resolution

Now we need to create a host file. This file takes the same form as a Windows or Linux hosts file. In our case, we will create the following hosts file:

192.168.3.35     FRIENDLY
188.124.5.107    HOSTILE

The file should be saved in the following location depending on your architecture:

  • Windows: %USERPROFILE%\Application Data\Wireshark\hosts
  • OS X: /Users/username/.wireshark/hosts
  • Linux: /home/username/.wireshark/hosts

Now, all we have to do is relaunch Wireshark and our capture file is appropriately populated with names for the devices we are examining. This is shown in Figure 3.

WS-Hostname-Figure3

Figure 3: Our Traffic is Easier to Identify

There are a number of strategies you can use for labeling hosts. For instance, you can label hosts by whether they are internal or external to the network as we did here, or you can label them by role (web server 1, dns server 2, known botnet C&C, etc).

This is a pretty simple trick, but it saves me a lot of time and frustration. It also helps the accuracy of my analysis, because I'm less likely to confuse IP addresses this way. You can even create large hosts files that can be used to automatically label known entities on your network.