A De-compositional Approach to Regular Expression Matching for Network Security Applications

Regular expressions are a very common tool for network security applications because they can match precisely and maintain high matching speed even for many simultaneous patterns. Their core feature is efficient representation as an automaton, where much of the interaction between patterns can be pre-computed and aggregated. Many algorithms have been devised to try and improve this pre-computation to not take exponential space while keeping high performance, but none has met all the requirements of fast, automated construction, small memory image, and high matching speed. We present Match Filtering, a technique for de-composing regular expressions into segments that can be matched independently, while a stateful post-processing engine filters these matches to eliminate those that do not correspond to matches of the original regular expression. Using standard CPU instructions, the post-processing engine can more efficiently represent constructs that would require a multiplicative increase in automaton states. Because the pre-processing is simple, automaton construction can be automated and fast, and because most on-line processing is done by a DFA, its matching speed is close to that of a DFA alone. We demonstrate experimentally 30× smaller, fast (seconds, not minutes) automaton construction and 43% faster matching speeds than state-of-the-art software algorithms.