“How do I find bad stuff on the network?”
The path to knowledge for the practice of NSM typically always begins with that question. It’s because of that question that we refer to NSM as a practice, and someone who is a paid professional in this field as a practitioner of NSM.
Scientists are often referred to as practitioners because of the evolving state of the science. As recently as the mid 1900s, medical science believed that milk was a valid treatment for ulcers. As time progressed, it was found that ulcers were caused by bacteria called helicobacter pylori and that dairy products could actually further aggravate an ulcer. Perceived facts change because although we would like to believe most sciences are exact, they simply aren’t. All scientific knowledge is based upon educated guesses utilizing the best available data at the time. As more data becomes available over time, answers to old questions change, and this redefines things that were once considered facts. This is true for Doctors as practitioners of medical science, and it is true for us as practitioners of NSM.
Unfortunately, when I started practicing NSM there weren’t a lot of reference materials available on the topic. Quite honestly, there still aren’t. Aside from the occasional blog postings of industry pioneers and a few select books, most individuals seeking to learn more about this field are left to their own devices. I feel that it is pertinent to clear up one very important misconception to eliminate potential confusion regarding my previous statement. There are menageries of books available on the topics TCP/IP, packet analysis, and various intrusion detection systems. Although the concepts presented in those texts are important facets of NSM, they don’t constitute the practice of NSM as a whole. That would be like saying a book about wrenches teaches you how to diagnose a car that won’t start.
With that in mind, my co-authors and I are incredibly excited to announce our newest project, a book entitled "Applied Network Security Monitoring". This book is dedicated to the practice of NSM. This means that rather than simply providing an overview of the tools or individuals components of NSM, we will speak to the process of NSM and how those tools and components support the practice.
This book is intended to be a training manual on how to become an NSM analyst. If you’ve never performed NSM analysis, then this book is designed to provide the baseline skills necessary to begin performing these duties. If you are already a practicing analyst, then my hope is that this book will provide a foundation that will allow you to grow your analytic technique in such a way as to make you much more effective at the job you are already doing. We’ve worked with several good analysts who were able to become great analysts because they were able to enhance their effectiveness with some of the techniques presented here.
The effective practice of NSM requires a certain level of adeptness with a variety of tools. As such, the book will discuss several of these tools as well, including the Snort, Bro, and Suricata IDS tools, the SiLK and Argus netflow analysis tool sets, as well as other tools like Snorby, Security Onion, and more.
This book focuses almost entirely on free and open source tools. This is in an effort to appeal to a larger grouping of individuals who may not have the budget to purchase commercial analytic tools such as NetWitness or Arcsight, and also to demonstrate that effective NSM can be achieved without a large budget. Ultimately, talented individuals are what make an NSM program successful. In addition, these open source tools often provide more transparency in how they interact with data, which is also incredibly beneficial to the analyst when working with data at an intimate level.
Table of Contents
Chapter 1: The Applied Practice of Network Security Monitoring
The first chapter is devoted to defining network security monitoring and its relevance in the modern security landscape. It discusses a lot of the core terminology and assumptions that will be used and referenced throughout the book.
Part 1: Collection
Chapter 2: Planning Data Collection
The first chapter in the Collection section of ANSM is an introduction to data collection and an overview of its importance. This chapter will introduce the Applied Collection Framework, which is used for making decisions regarding what data should be collected using a risk-based approach.
Chapter 3: The Sensor Platform
This chapter introduces the most critical piece of hardware in an NSM deployment: the sensor. First, we will look at a brief overview of the various NSM data types, and the types of NSM sensors. This will lead us to discuss important considerations for purchasing and deploying sensors. Finally, we will also cover the placement of NSM sensors on the network, including a primer on creating network visibility maps for analyst use.
Chapter 4: Session Data
This chapter discusses the importance of session data, along with a detailed overview of the SiLK toolset for the collection of NetFlow data. We will also briefly examine the Argus toolset for session data collection and parsing.
Chapter 5: Full Packet Capture Data
This section begins with an overview of the importance of full packet capture data. We will examine several tools that allow for full packet capture of PCAP data, including Netsniff-NG, Daemonlogger, and Dumpcap. This will lead to a discussion of discuss different considerations for the planning of FPC data storage and maintenance of that data, including considerations for trimming down the amount of FPC data stored.
Chapter 6: Packet String Data
This chapter provides an introduction to packet string (PSTR) data and its usefulness inthe NSM analytic process. We will look at several methods for generating PSTR data with tools like Httpry and Justniffer. We will also look at tools that can be used to parse and view PSTR data, including Logstash and Kibana.
Part 2: Detection
Chapter 7: Detection Mechanisms and Indicators of Compromise
This chapter examines the relationship between detection mechanisms and Indicators of Compromise (IOC). We will look at how IOCs can be logically organized, and how they can be effectively managed for incorporation into an NSM program. This will include a system for classifying indicators, as well as metrics for calculating and tracking the precision of indicators that are deployed to various detection mechanisms. We will also look at two different formats for IOC’s, OpenIOC and STIX.
Chapter 8: Reputation-Based Detection
The first specific type of detection that will be discussed is reputation-based detection. We will discuss the fundamental philosophy of reputation-based detection, along with several resources for examining the reputation of devices. This discussion will lean towards solutions that can automate this process, and will demonstrate how to accomplish this with simply BASH scripts, or by using Snort, Suricata, CIF, or Bro.
Chapter 9: Signature-Based Detection with Snort and Suricata
The most traditional form of intrusion detection is signature-based. This chapter will provide a primer on this type of detection and discuss the usage of the Snort and Suricata intrusion detection systems. This will include the usage of Snort and Suricata, and a detailed discussion on the creation of IDS signatures for both platforms.
Chapter 10: Anomaly-Based Detection with Bro IDS
This chapter will cover Bro, one of the more popular anomaly based detection solutions. This will cover a review of the Bro architecture, the Bro language, and several practical use cases that demonstrate the truly awesome power of Bro as an IDS and network logging engine.
Chapter 11: Anomaly-Based Detection with Statistics
This chapter will discuss the use of statistics for identifying anomalies on the network. This will focus on the use of various NetFlow tools like rwstats and rwcount. We will also discuss methods for visualizing statistics by using Gnuplot and the Google Charts API. This chapter will provide several practical examples of useful statistics that can be generated from NSM data.
Chapter 12: Using Canary Honeypots for Detection
Previously only used for research purposes, canary honeypots are a form of operational honeypot that can be used as an effective detection tool. This chapter will provide an overview of the different types of honeypots, and how certain types can be used in an NSM environment. We will look at several popular honeypot applications that can be used for this purpose, including Honeyd, Kippo, and Tom’s Honeypot. We will also briefly discuss the concept of Honeydocs.
Part 3: Analysis
Chapter 13: Packet Analysis for NSM
The most critical skill an NSM analyst can have is the ability to interpret and decipher packet data that represents network communication. In order to do this effectively, it requires a fundamental understanding of how packets are dissected. This chapter provides that fundamental backing and shows how to break down packet fields on a byte by byte basis. We demonstrate these concepts using tcpdump and Wireshark. This chapter will also cover basic to advanced packet filtering techniques using Berkeley Packet Filters and Wireshark Display Filters.
Chapter 14: Friendly and Threat Intelligence
The ability to generate intelligence related to friendly and hostile systems can be the defining factor that makes or breaks an investigation. This chapter begins with an introduction to the traditional intelligence cycle and how it relates to NSM analysis intelligence. Following this, we look at methods for generating friendly intelligence by generating asset data from network scan and leveraging PRADS data. Finally, we examine the types of threat intelligence and discuss some basic methods for researching tactical threat intelligence related to hostile hosts.
Chapter 15: The Analysis Process
The final chapter discusses the analysis process as a whole. This begins with a discussion of the analysis process, and then breaks down into examples of two different analysis processes; relational investigation and differential diagnosis. Following this, the lessons learned process of incident morbidity and mortality is discussed. Finally, we will look at several analysis best practices to conclude the book.
Chris Sanders, Lead Author
Chris Sanders is an information security consultant, author, and researcher originally from Mayfield, Kentucky. That’s thirty miles southwest of a little town called Possum Trot, forty miles southeast of a hole in the wall named Monkey’s Eyebrow, and just north of a bend in the road that really is named Podunk.
Chris is a Threat Intelligence Operations Lead at Mandiant. He has as extensive experience supporting multiple government and military agencies, as well as several Fortune 500 companies. In multiple roles with the US Department of Defense, Chris significantly helped to further to role of the Computer Network Defense Service Provider (CNDSP) model, and helped to create several NSM and intelligence tools currently being used to defend the interests of the nation.
Chris has authored several books and articles, including the international best seller “Practical Packet Analysis” form No Starch Press, currently in its second edition. Chris currently holds several industry certifications, including the SANS GSE and CISSP distinctions.
In 2008, Chris founded the Rural Technology Fund. The RTF is a 501(c)(3) non-profit organization designed to provide scholarship opportunities to students form rural areas pursuing careers in computer technology. The organization also promotes technology advocacy in rural areas through various support programs. The RTF has provided thousands of dollars in scholarships and support to rural students.
When Chris isn’t buried knee-deep in packets, he enjoys watching University of Kentucky Wildcat basketball, being a BBQ Pitmaster, amateur drone building, and spending time at the beach. Chris currently resides in Charleston, South Carolina with his wife Ellen.
Jason Smith, Co-Author
Jason Smith is an intrusion detection analyst by day and junkyard engineer by night. Originally from Bowling Green, Kentucky, Jason started his career mining large data sets and performing finite element analysis as a budding physicist. By dumb luck, his love for data mining led him to information security and network security monitoring where he took up a fascination with data manipulation and automation.
Jason has a long history of assisting state and federal agencies with hardening their defensive perimeters and currently works as a Systems Engineer with Mandiant. As part of his development work, he has created several open source projects, many of which have become “best-practice” tools for the DISA CNDSP program.
Jason regularly spends weekends in the garage building anything from arcade cabinets to open wheel racecars. Other hobbies include home automation, firearms, monopoly, playing guitar, and eating. Jason has a profound love of rural America, a passion for driving, and an unrelenting desire to learn. Jason is currently living in Frankfort, Kentucky.
David J. Bianco, Contributing Author
Before coming to work as a Hunt Team Leader at Mandiant, David spent five years helping to build an intelligence-driven detection & response program for a Fortune 5 company. There, he set detection strategies for a network of nearly 600 NSM sensors in over 160 countries and led response efforts for some of the company¹s the most critical incidents, mainly involving targeted attacks. He stays active in the security community, blogging, speaking and writing.
You can often find David at home watching Doctor Who, playing one of his four sets of bagpipes, or just goofing around with the kids. He enjoys long walks nearly anywhere except the beach.
Liam Randall, Contributing Author
Liam Randall is the Managing Partner with San Francisco based Broala LLC- the Bro Core Teams consulting group. Originally, from Louisville, KY, he worked his way through school as a sysadmin while getting his Bachelors in Computer Science at Xavier University. He first got his start in security writing device drivers and XFS based software for Automated Teller Machines.
Presently he consults on high volume security solutions for the Fortune 50, Research and Education Networks, various branches of the armed service, and other security focused groups. He has spoken at Shmoocon, Derbycon, MIRcon and regularly teaches Bro training classes at security events.
A father and a husband, Liam spends his weekends fermenting wine, working in his garden, restoring gadgets, or making cheese. With a love of the outdoors he and his wife enjoy competing in triathlons, long distance swimming, and enjoying their community.
Applied NSM is scheduled to be released on December 15th, 2013.