Approximate Controllability of Continuity Equation of Transformers

Building on the recent work by Geshkovski et al. (2023) which provides an interacting particle system interpretation of Transformers with a continuous-time evolution, we study the controllability attributes of the corresponding continuity equation across the landscape of probability space curves. In particular, we consider the parameters of the Transformer’s continuous-time evolution as control inputs. We prove that given an absolutely continuous probability measure and a non-local Lipschitz velocity field that satisfy a continuity equation, there exist control inputs such that the measure and the non-local velocity field of the Transformer’s continuous-time evolution approximate them, respectively, in the p-Wasserstein and Lp-sense, where
$1\leq p\lt \infty $
.