Future Event Prediction: If and When

We consider the problem of future event prediction in video: if and when a future event will occur. To this end, we propose a number of representations and loss functions tailored to this problem. These include several probabilistic formulations that also model the uncertainty of the prediction. We train and evaluate the approach on two entirely different prediction scenarios: if and when a car will stop in the BDD100k car driving dataset; and if and when a player is going to shoot a basketball towards the basket in the NCAA basketball dataset. We show that (i) we are able to predict events far in the future, up to 10 seconds before they occur; and (ii) using attention, we can determine which areas of the image sequence are responsible for these predictions, and find that they are meaningful, e.g. traffic lights are picked out for predicting when a vehicle will stop.