Controlled Multi-modal Image Generation for Plant Growth Modeling

Predicting plant development is an important task in precision farming and an essential metric for decision-making by researchers and farmers. In this work, we propose a novel generative modeling technique for plant growth prediction based on conditional generative adversarial networks. We formulate plant growth as an image-to-image translation task and predict the appearance of a plant growth stage as a function of its previous stage. We take into account that plant growth is inherently multi-modal, depending on numerous and highly variable environmental factors, and thus a single input belongs to a distribution of potential outputs. We encode the ambiguity in an interpretable and low-dimensional latent vector space representing the various factors of variation that are influencing plant growth. We use a novel encoder-based data fusion technique and combine information contained in remote sensing imagery of different cropping systems with data containing the factors of variation to adequately model plant growth. This offers several advantages over existing methods: (1) we show that we can model a distribution of potential appearances and simultaneously outperform existing methods in providing more realistic predictions, (2) the complexity of plant growth is more adequately captured, as various factors influencing plant growth can be included, (3) predictions are controllable by being conditioned by an interpretable latent vector representing the factors of variation along with an input image of a previous growth stage.