Existential second-order logic over graphs: charting the tractability frontier

Fagin's (1974) theorem, the first important result of descriptive complexity, asserts that a property of graphs is in NP if and only if it is definable by an existential second-order formula. We study the complexity of evaluating existential second-order formulas that belong to prefix classes of existential second-order logic, where a prefix class is the collection of all existential second-order and the first-order quantifiers obey a certain quantifier pattern. We completely characterize the computation complexity of prefix classes of existential second-order logic in three different contexts: over directed graphs; over undirected graphs with self-loops; and over undirected graphs without self-loops. Our main result is that in each of these three contexts a dichotomy holds, i.e., each prefix class of existential second-order logic either contains sentences that can express NP-complete problems or each of its sentences expresses a polynomial-time solvable problem. Although the boundary of the dichotomy coincides for the first two cases, it changes, as one move to undirected graphs without self-loops.