Chess fortresses, a causal test for state of the art Symbolic [Neuro] architectures

The AI community's growing interest in causality is motivating the need for benchmarks that test the limits of neural network based probabilistic reasoning and state of the art search algorithms. In this paper we present such a benchmark, chess fortresses. To make the benchmarking task as challenging as possible, we compile a test set of positions in which the defender has a unique best move for entering a fortress defense formation. The defense formation can be invariant to adding a certain type of chess piece to the attacking side thereby defying the laws of probabilistic reasoning. The new dataset enables efficient testing of move prediction since every experiment yields a conclusive outcome, which is a significant improvement over traditional methods. We carry out extensive, large scale tests of the convolutional neural networks of Leela Chess Zero [1], an open source chess engine built on the same principles as AlphaZero (AZ), which has passed AZ in chess playing ability according to many rating lists. Our results show that a novel, experimental network, which was intended for efficient endgame play, performed best despite being one of the smallest networks.