Inventory Management of the Refrigerator's Produce Bins Using Classification Algorithms and Hand Analysis

Tracking the inventory of one's refrigerator has been a mission for consumers since the advent of the refrigerator. With the improvement of computer vision capabilities, automatic inventory systems are within reach. One inventory area with many potential benefits is the fresh food produce bins. The bins are a unique storage area due to their deep size. A user cannot easily see what is in the bins without opening the drawer. Produce items are also some of the quickest foods in the refrigerator to spoil, despite being temperature and humidity controlled to have the fruits and vegetables last longer. This research addresses the challenges presented by getting a full inventory of all items within the produce bins by observing if the hand can provide useful information. The research proposes that all items must go in or out of the refrigerator by the main door, and by using a single camera to observe the hand-object interactions, a more complete inventory list can be created. As an initial proof-of-concept, two types of produce, an apple and an orange, will be used as a testing ground. The accuracy of the hand analysis (e.g., hand with apple or orange vs. hand empty) was determined by comparing the model predictions to a 301-frame video with ground truth labels. The hand analysis system was 87% accurate for classifying an empty hand, 85% accurate on a hand holding an apple, and 74% accurate on a hand holding an orange. The system was 93% accurate at detecting what was added or removed from the refrigerator.