Permutation filters are a broad class of nonlinear selection filters that utilize the complete spatial and rank order information of observation samples. This use of joint spatial-rank information has proven useful in numerous applications. The application of permutation filters, however, is limited by the factorial growth in the number of spatial-rank orderings. Although M-permutation filters have been developed to address the growth in orderings, their a priori uniform selection of samples is not appropriate in most cases. Permutation filter implementations based on acyclic connected graphs provide a more general approach that allows the level of ordering information utilized to automatically adjust to the problem at hand. In addition to developing and analyzing graph implementations of permutation filters this paper presents a LNE based optimization of the graph structure and filter operation. Simulation results illustrating the performance of the optimization technique and the advantages of the graph implementation are presented.