Expression of Fractals Through Neural Network Functions

To help understand the underlying mechanisms of neural networks (NNs), several groups have studied the number of linear regions ℓ of piecewise linear (PwL) functions, generated by deep neural networks (DNN). In particular, they showed that ℓ can grow exponentially with the number of network parameters p, a property often used to explain the advantages of deep over shallow NNs. Nonetheless, a dimension argument shows that DNNs cannot generate all PwL functions with ℓ linear regions when ℓ > p.It is thus natural to seek to characterize specific families of functions with ℓ > p linear regions that can be constructed by DNNs. Iterated Function Systems (IFS) recursively construct a sequence of PwL functions Fk with a number of linear regions which is exponential in k. We show that Fk can be generated by a NN using only O(k) parameters. IFS are used extensively to generate natural-looking landscape textures in artificial images as well as for compression of natural images. The surprisingly good performance of this compression suggests that human visual system may lock in on self-similarities. The combination of this phenomenon with the capacity of DNNs to efficiently approximate IFS may contribute to the success of DNNs, particularly striking for image processing tasks.