This paper deals with finding appropriate hysteresis models that predict the behavior of electro-magnetic lenses used with electron microscopy. The iterative selection procedure consists of experiment design and mathematical analysis of hysteresis models. We will show two examples of pitfalls when the suitability of models is assessed by means of curve-fitting to observations. The first example illustrates the inability of models with local memory to describe the magnetic hysteresis in our application. The second deals with accommodation. With both, the experiments carried out on an off-line setup of a magnetic lens are compared to hysteresis models.