It is becoming increasingly common for network devices to handle packets based on the contents of packet payloads. Example applications include intrusion detection, firewalls, web proxies, and layer seven switches. This paper analyzes the problem of intrusion detection and its reliance on fast string matching in packets. We show that the problem can be restructured to allow the use of more efficient string matching algorithms that operate on sets of patterns in parallel. We then introduce and analyze a new string matching algorithm that has average-case performance that is better than the best theoretical algorithm (Aho-Corasick) and much better than the currently deployed algorithm (multiple iterations of Boyer-Moore). Finally, we implement these algorithms in the popular intrusion detection platform Snort and analyze their relative performance on actual packet traces. Our results provide lessons on the structuring of content-based handlers, string matching algorithms in general, and the importance of performance to security.

Pre-2018 CSE ID: CS2001-0670

In this paper we address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. The central difficulty is that count processing must be done in a packet arrival time (8 nsec at OC-768 speeds) and hence must take a small number of memory references to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be more than a million. By contrast, our new algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we used one of our new algorithms to replace the port scan detection component of the popular intrusion detection system Snort. Doing so reduced the memory usage on a ten minute trace from 51 Mbytes to 5.7 Mbytes while maintaining a 99.78% probability of alarming on a scan within 6 seconds of when the large-memory algorithm would alarm. By contrast, the best known prior algorithm (probabilistic counting) takes 4 times more memory on the port scan application and 8 times more memory on a measurement application. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts, or prior knowledge of the likely range of the count.

Pre-2018 CSE ID: CS2002-0705

In this paper we present a family of algorithms that address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. The central difficulty is that count processing must be done within a packet arrival time (8 nsec at OC-768 speeds) and, hence, must require only a small number of memory references to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be more than a million. By contrast, our new algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we used one of our new algorithms to replace the port scan detection component of the popular intrusion detection system Snort. Doing so reduced the memory usage on a ten minute trace from 50 Mbytes to 5.6 Mbytes while maintaining a 99.5% probability of alarming on a scan within 9 seconds of when the large-memory algorithm would alarm. By contrast, the best known prior algorithm (probabilistic counting) takes 4 times more memory on the port scan application and 8 times more memory on a measurement application. Our algorithms also lead to a reduction by a factor of seven in the total memory usage of a traffic analysis application from the CoralReef suite. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts or prior knowledge of the likely range of the count.

Pre-2018 CSE ID: CS2003-0738