FoodScan: Food Monitoring App by Scanning the Groceries Receipts

In the mobile device market there is a large number of applications to help people monitor intake or provide suggestions to lose weight and manage a healthy diet. However, the vast majority of these apps consume a lot of time by having to introduce food one by one. This paper presents the work to develop and pilot test a new Android application, FoodScan, aimed at people over 70, specially those from rural environments or with limited technical knowledge, to manage their food from the items that appear on their grocery receipts, avoiding the obligation to introduce one by one those foods, and generating recommendations. To achieve this final objective, specific objectives have been completed as indicated in the methods section. We conducted a review of current calorie control applications to learn about their weaknesses and strengths. Different algorithms were tested to expedite the introduction of food into the application and the most suitable for the FoodScan application was selected. Likewise, several options were taken into account to create the knowledge base of food, taking into account dietary recommendations for people over 70 years. Once developed, a pilot evaluation was carried out with a convenience sample of 109 volunteers in rural areas of Caceres and Valladolid (Spain) and Alentejo (Portugal). They tested FoodScan for a month after which they completed a user satisfaction survey. 93 % (101/109) believed that the app was easy to download and install, 66 % (72/109) thought that it was easy to use, 47 % (51/109) noted that the charts with the recommendations helped them with diet control and 49 % (53/109) indicated that FoodScan helped them improve healthy eating habits. One-month pilot evaluation data suggested that most users found the app somewhat helpful for monitoring food intake, easy to download and easy to use.


I. INTRODUCTION
The guidelines to follow in any healthy diet are summarized in: (1) variety in foods; (2) reduce fat consumption; (3) increase the consumption of complex carbohydrates such as fruits and vitamins; (4) increase the consumption of fiberrich foods; (5) reduce sugar and salt; (6) maintain the intake of vitamin D and calcium; (7) hydrate; (8) moderate alcohol consumption; (9) make 4 or 5 meals a day reducing the amounts in each one; and (10) exercise regularly. Technology is a highly effective tool for controlling diet and physical activity [1], [2].
The associate editor coordinating the review of this manuscript and approving it for publication was Shen Yin.
Overweight, which affects all age groups, has increased to alarming levels as a consequence not only of excessive calorie intake but also of being more sedentary [3], [4]. The increase in the production of processed foods, the rapid urbanization and the change in lifestyles have led to a change in eating habits. More and more people are far from their ideal weight, which is a serious health problem due to related heart diseases, diabetes, stroke, etc. Eating a healthy diet helps protect us from malnutrition in all its forms, as well as the diseases mentioned above. It is necessary to change people's lifestyles, controlling the amount of food we eat and start eating healthier.
In rural areas, with an increasingly aging population and with medical services distant from their homes, it is observed VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ that many elderly people who live alone are suffering from nutritional deficiencies that, in some cases, can cause significant damage to their physical condition. To determine the health of the elderly population and avoid these problems, we should have accurate information about their nutritional patterns to predict their quality of life [5]- [7]. The objective the FoodScan application is to facilitate the control of the elder diet, specially of those living in rural areas. These people have limited technical knowledge and prefer simple devices. In the development of the application, it was taken into account that connectivity can also be a problem. Sometimes the Internet connection does not arrive in optimal conditions to certain rural areas or they do not even have any signal. FoodScan is a simple application that collects the information by scanning the purchase groceries receipts and that does not require an Internet connection for its management. We can store food in an integrated way without having to do it one by one. It has been developed for the Android Operating System, since it has a higher penetration rate than other operating systems.
The steps that we have carried out throughout the work have been: (1) to review existing Calorie Control Apps; (2) to analyse algorithms registering the purchased groceries and select a food collection to compare; (3) to develop the FoodScan App; and (4) to conduct a user evaluation.

II. RELATED WORK
Technology can help people to maintain a healthy diet [8]. A large number of applications have been designed for this purpose [9], [10]. Some manage sports activities or physical exercise plans [11]- [14]. Others focus on food control [15]. Nevertheless, most of those applications unify both aspects: control of food, caloric intake and sports activities.
Generally, applications for food control have the main function of registering and storing meals throughout the day. The user must enter, one by one, the type of food ingested. This controls the calories ingested in a day and the specific nutrients (proteins, carbohydrates, fats, vitamins), and allows comparisons with the data recorded from previous days. Applications can incorporate other functions such as: (1) set goals to lose weight over a period of time; (2) store healthy food recipes; or (3) include a step counter to complement the record of calories ingested with those that have been burned throughout the day doing physical exercises.
Mobile applications have been developed to scan purchase receipts [16], [17]. We found a study to measure the effects of front-of-pack interpretive nutrition labels on grocery shopping using label scanning [18]. In another article, the authors tried to know, by registering receipts, the behavior of buying food in households [19]. In addition, that will be effective if there is a commitment by the user to scan the tickets. However, we have not found any mobile application that photographs and records the groceries receipts to compare them over a period of time, and based on that comparison show recommendations for a balanced diet.

A. WORK FLOW
As a starting point, we carry out a review of mobile applications that offer characteristics similar to those we try to implement in our app. After this comparative review, several algorithms were analysed to speed up the introduction of food into the application. It was also very important to choose the most appropriate food knowledge base, taking into account the food recommendations for people over 70 years old living in rural areas. After achieving these objectives, we developed the application and finally carried out a satisfaction survey among end users. Figure 1 shows the flow chart followed in this work.

B. REVIEW OF CURRENTLY AVAILABLE MOBILE APPS TO CONTROL CALORIES
A search was performed through Google, Scopus and Web of Science. The terms used were: ''best calorie counter apps'', ''apps to control calorie intake'', ''apps to control diet'', and ''apps to control feeding''. After an exhaustive examination of mobile applications, we focused on the three that had the higher number of downloads in order to analyse their functionalities and features: Lose It! [20], Fat Secret [21] and My Fitness Pal [22]. Five people downloaded these three free applications and tested them for a period of approximately two months. They tested them simultaneously by entering the same data in all three applications. It was decided that the evaluators would analyse characteristics of the applications such as ease of downloading, user registration in the application, graphical interface, simplicity of the food registration process, possibility of scanning food barcodes, and using graphs and charts for providing the results. After this trial period, opinions on the functionality of the applications were shared.

C. ANALYSIS OF ALGORITHMS REGISTERING THE PURCHASED GROCERIES AND SELECTION OF A FOOD COLLECTION TO COMPARE
Three options were tested: (1) Web Scraping; (2) object recognition and; (3) Optical Character Recognition (OCR).
Web Scraping allows one to extract large amounts of information from web pages in an automated and dynamic way. In order to prove its effectiveness, online shopping websites were crawled with the aim of analysing and comparing the user's different purchases over a period of time. The technique is not efficient for our purpose because pages have different structures. In addition, data protection problems must be taken into account [23]- [25]. Finally, people over 70 years old living in rural areas do not use online shopping websites.
Image recognition identifies objects in an image or video sequence and facilitates their subsequent classification [26]- [28]. To test the algorithm, photographs of the products of the refrigerator were taken several days apart to see the food that the user consumed during that time interval. Comparing the photographs, and from a healthy diet scheme or template, it can be concluded if the user followed a balanced diet or needs to buy or consume certain foods. Although it is a good strategy to compare images, the amount of comparisons is very large. The vectorization process consumes a large amount of resources, and the methodology to recognize the similarity between these objects is very complex. If we add difficulty in taking a clear and clean photograph of the items in the refrigerator, it is concluded that this technique does not turn out to be the most efficient for our target population.
The Optical Character Recognition (OCR) algorithm consists of the digitalization of texts that are automatically identified from an image, symbols or characters that belong to a certain alphabet [29], [30]. There is a wide range of tools or libraries that make the use of OCR algorithms easier for developers. Asprise OCR SDK provides an API to recognize text and image barcodes. The problem is that the demo has very limited functions [31], [32].
As a search method to select a food collection to compare, we analysed online bibliographic references. It is important to know the recommendations of the specialists, and differentiate the types and groups of food that are classified according to the distribution of the food pyramid for people with the age range above 70 years [33]. And we must keep it in mind to avoid the nutritional disorders of the elderly.

D. APP STRUCTURE AND DEVELOPMENT
FoodScan has been developed for a minimum Android 6.0 version, equivalent to the API level 23, and the version with which the project has been compiled is Oreo 8.1, equivalent to the API level 27. Support for older APIs is provided since the target users of the application tend to own low-end devices that are not frequently updated and the latest data shows that more than 84 % of Android devices run a version 6.0 or higher. Nevertheless, the proposed application has been tested in new and more powerful devices with similar results. The functions allowed by the application are: 1) Scan grocery receipts for food registration.
2) Manually add a food to an already registered grocery receipt, or create a new one. 3) View the record of stored grocery receipts, and search and filter by date. 4) See summaries of statistics, by groups of food consumed in the week or month and those that remain to be consumed, through graphs.
The domain of the FoodScan ontology would be "Food" as the main class. Different food groups are established as subclasses, namely: Legumes, Vegetables, Fruits, Cereals and derivates, Dry fruits, Milk and derivates, Eggs, Meats, and Fish. In the case of meat and fish, new subclasses are established based on the proportion of fat: fatty and lean. Finally, more than 100 individuals, specific products that can be automatically recognized by the application, has been included in the ontology, distributed along the different classes. The relationships that appear in this ontology are only belonging. A food belongs to a certain class, regardless of relationships between classes. We save and export the ontology in a file of the RDF / XML type. The file uses different brands or labels to indicate classes, individuals and relationships. It is possible to modify this file, to add new individuals to some class, because it is not static. For the implementation of the ontology we used the editor ''Protégé'' [34].
The use-cases diagram shows the structure of the application, the actors involved and the different functions performed by those actors. These types of diagrams are designed to show the interaction between the system and the user, or another system. User interactions with the application allows them to: (1) scan a grocery receipt; (2) register a grocery receipt manually; (3) see records; and (4) see recommendations. See in Figure 2.  The activity diagram (see Figure 3) shows the workflow that is performed and details the different routes between the events of the application.
Two tables are created for the design of the relational database and implemented using the SQLite database. The ''grocery receipts'' table contains the grocery receipt data in the app, storing an automatically generated grocery receipt code that identifies it, the date on which this receipt was added to the database and a text with the address where the image is stored of the grocery receipt. The ''food'' table contains in each of its records the name of the recognized food and the code of the grocery receipt to which it belongs.
In Figure 4, one can observe the relationships between the modules that integrate FoodScan: (1) OCR algorithm; (2) food ontology; (3) groceries receipt database; and (4) functions of the result graph.
Since the application data is personal and unique for that application it is not necessary to upload it to a remote server. That is why we have chosen to use SQLite as the internal database for the Android application. Data management is done locally and, therefore, personal data is not sent to a remote server to be analysed, improving the privacy of the proposed solution.
To generate the graphics, we use MPAndroidChart [35]. We established the recommended weekly quantities for each food group, and they were all stored together in a numerical array. Each bar of the graph is composed of two sections. The first section indicates the food bought. The second section of the bar presents the food that remains to be ingested. This amount is the result of the operation: Recommended amount -Ingested amount.

E. END USER EVALUATION
A small group of volunteers visited rural areas from the provinces of Caceres and Valladolid (Spain) and Alentejo (Portugal) to show FoodScan. The population was notified about the investigation by means of posters placed in the civic centers and the town hall. The number of people who attended the first demonstration meeting exceeded 270 among all the rural areas visited. Volunteers gave a practical demonstration of how to download and enter food data into the FoodScan app to get graphs of results.
About a month after the first meeting, volunteers returned to conduct the web based survey among those who used the FoodScan application. A considerably smaller number of users attended this second meeting: 109 Spanish and Portuguese users. All of them tested the application and were interested in answering the satisfaction survey. Please, note that no incentive were offered by the volunteers.
The questions were created by the researchers. It is not based on an established user satisfaction survey. A person in charge of the Privacy Policy Service of the University of Valladolid reviewed the survey and recommended to eliminate the questions that could identify the respondents. After following the recommendations, the survey was approved. In addition, participants were not asked to provide demographic characteristics. We did not consider requesting these data because we did not aim to know the health of the participants. We were only interested in the age of the respondents because the application is aimed to people over 70 years old. All the people recruited for the study were over 50 years old with medium-low technological knowledge.
Although the data collected is anonymous and therefore it is not necessary to include transparency clauses on data protection in the survey, an informed consent was also collected from those who completed the survey. Participants were informed by letter of the content of the survey and asked about their implicit consent. Participants who gave ethical approval were offered the possibility to use tablets to conduct the survey. The tool for the development of the survey was Google Forms due to its simplicity and because a license is not required for its use.
A survey was carried out with 7 questions (see Table 1), accessible through a link provided to the users of the application. We were especially interested in the answers to questions regarding the usability and ease of downloading the application. We decided that the number of questions in the survey should be small, so that people would be encouraged to complete it. Thanks to the simplicity of the questionnaire, all the people who carried out the survey answered all the questions. These results were transferred to an Excel spreadsheet in order to better analyse the results and generate graphs.

A. SUMMARY OF THE REVIEW OF CURRENTLY AVAILABLE APPS
All three applications offer similar features and interfaces. The user can set the ideal weight (s)he wants to achieve and how quickly (s)he hopes to achieve it. Depending on their age, weight, height and sex, the application indicates a daily calorie regime. The user has to record the daily food and physical exercises, can visualize in a graph the calorie and nutrient intake, follow the progress in their marked objectives, modify the quantities or portions of the foods, and see suggestions of healthy recipes. The success of these three applications compared to others is that it is possible to scan the food barcode with the camera of the mobile device and register the food. The scanned item is recognized if it is stored in the database. If the food is not stored, it can be manually included. They are visual applications and have different sections. Graphs show the values of calories and nutrients in quantities and percentages. It also offers a simple and navigable interface, with the possibility of configuring the app in several languages.
But not everything are advantages. The main problem of these applications is that the user must introduce food and physical activities one by one. This can be a tedious and time-consuming task, since they have to provide the food, the quantity and the time of day at which it is ingested. In the long run, there is an increased likelihood of abandonment due to the high burden related to entering foods into apps. Therefore, in order to increase the adherence, users need applications registering the food intake without requiring introducing these foods one by one.

B. ANALYSIS OF ALGORITHMS REGISTERING THE PURCHASED GROCERIES AND SELECTION OF A FOOD COLLECTION TO COMPARE
The one that was finally used in FoodScan is the OCR algorithm. The OCR character recognition services of web pages were tested. Some of those consulted were Newocr.com, Onlineocr.net, Ocr.space, and Tess4j together with tesseract [36]. In the Android Studio development environment, there were difficulties in making the Tess4j library compatible. Finally, Google Vision was used, which allows us to understand the content of an image by encapsulating powerful machine learning models in a simple and easy-to-use way [37]- [39].
Foods are categorized into different groups. It is a data store that allows us to compare the scanned and recognized foods of the grocery receipts using the OCR algorithm. The considered alternatives were: (1) UNESCO Thesaurus [40]; (2) Database; and (3) Ontologies. For FoodScan we use an ontology, because it offers several functions that allow us to build a simple and complete collection. It can be adjusted to the needs of FoodScan, to help and advise people who use it in the diet, offering recommendations based on purchases made within a certain period of time.

C. FOODSCAN APPLICATION
The FoodScan home screen is shown in Figure 5. It offers three options to add or register a grocery receipt in the app's database: (1) take a picture; (2) choose an image from the gallery; and (3) add a new groceries receipt by selecting foods manually. The 3 horizontal lines icon, in the upper left of the screen, takes us to the menu with the functions of the app: (1) back to the beginning; (2) open the record of the groceries receipts stored in the database; and (3) initiate recommendations or statistics. Figure 6 shows the grocery receipts registration screen. It allows one to insert the ''start date'' and the ''end date''. The "Filter" button shows only the grocery receipts between these dates. If the dates are not selected correctly, an error message will be displayed on the screen. If the dates are not selected, all the grocery receipts registered in the database are shown. By clicking on a grocery receipt, a pop-up menu will appear with 3 options: (1) see details, which offers groceries receipt details, the food it contains and an image if it exists; (2) delete, which allows the grocery receipt to be deleted; or (3) cancel, to close the menu without changes. The user can manually add a food to the grocery receipt among those stored in the application. This is done sequentially as can be seen in Figure 7.
Regarding the recommendations of FoodScan based on the recorded data, we decided to provide them using visual techniques such as graphs. FoodScan has the advantage of VOLUME 8, 2020  offering weekly or monthly recommendations. In the case of the weekly recommendation, the user must select the day on which that week begins, and the application will take that day and the next six. If it is monthly, the user must select the month and year. In both cases, the app will search the database for grocery receipts included in this period of time, and if there are grocery receipts on those dates, it will show the graph with the results of the analysis. If there is not any grocery receipt in the database, the process is canceled and the app displays a message to the user warning of the error.
FoodScan offers a visual analysis of the foods of each group consumed in a given time, comparing them with the amounts recommended by specialists, to follow a balanced diet. If there are groceries receipts stored for that period of time, a bar graph is shown for each food group with the amount of food consumed, and those that remain to be consumed according to the established recommended quantities. A possible limitation of FoodScan is that it only tracks the  food bought, not the one consumed. As a criterion of colors, we consider the "traffic light system", whose colors range from green to dark red, and which mark the different levels of risk. A Color Code has been established based on the percentage of food consumed: (1) between 0-25 %, red color; (2) between 25-50 %, orange color; (3) between 50-75 %, yellow color; and (4) between 75-100 %, green color. If the color of the bar for a certain food group is not green, the user should adjust their diet to meet the recommended amounts (in grey color). In Figure 8, we see that only three food groups were added to the application database: (1) vegetables; (2) cereals and derivatives; and (3) milk and derivatives. All in insufficient quantities and that is why the bars are in orange and red. In a balanced diet, Figure 8 would show all the bars in green. Therefore, from the visualization of this graph, the user knows that the consumption of food should be increased, if (s)he wants to follow a healthy diet, according to the recommendations of the dietitians.

D. RESULTS OF THE END USER EVALUATION
The opinion of the users allows us to know if it has a friendly interface. In addition, the information collected will help us to improve the application in the future. Regarding the use of applications for weight loss, more than half users answered that they did not use any before trying FoodScan. Those who had used weight loss applications indicated that the most used were My Fitness Pal, Fat Secret and Lost It! Few users reported having used more than one application. See Figure 9.
Regarding the download and installation of FoodScan, 65 out of 109 participants (60 %) answered that it was easy and 36 out of 109 (33 %) very easy (see Figure 10). As can be seen in Figure 11, regarding ease of use, 4 out of 109 participants (4 %) found it difficult, 33 out of 109 (30 %) average difficulty, 53 out of 109 (49 %) easy and 19 out of 109 (17 %) very easy.
As can be seen in Figure 12, with the exception of 2 of the total of 109 polled (2 %) who do not consider it useful, the public appreciates the recommendations shown in the charts. As shown in Figure 13, users generally agree with the fact that the app helps them improve their healthy eating habits. In Figure 14, one can see the price users would be willing to pay if it were marketed. It is an application that, in general, has liked. 101 out of 109 participants (93 %) would recommend it to other users (see Figure 15).

V. DISCUSSION AND CONCLUSION A. PRINCIPAL RESULTS
In this work, we started from the analysis of mobile applications that offer similar characteristics to those that we try to implement in FoodScan. After this comparative review, various algorithms were analysed to accelerate the introduction of food and the most suitable food knowledge base for people over 70 years old. Next, we developed the application and finally we carried out a satisfaction survey among users who used the application for a month. Below we summarize the objectives achieved: 1) Review the currently available apps. The analysis allowed us to better understand its characteristics, weaknesses and strengths. The main weakness iden-   tified is that the information of the food intake has to be manually provide to the analysed apps. Table 2 compares the features of FoodScan with other apps. VOLUME 8, 2020   Only our application allows you to register the food included in the ticket by scanning. 2) Analysis of algorithms registering purchased foods.
Three options were tested: (1) Web Scraping; (2) object recognition and; (3) Optical Character Recognition (OCR). OCR was finally used because of the number of services available and its accuracy. Also, we selected a food collection to compare. The following alternatives that were considered: (1) UNESCO Thesaurus; (2) Database; and (3) Ontologies. We used ontologies because it allowed us to easily build a complete collection. 3) Development of the FoodScan application. We based on the premise that our target audience is people over 70 years old, with limited technical knowledge and living in rural areas. This population group is generally reluctant to use technologies and, therefore, the need to create a simple, intuitive, friendly application that offers visual results that are very easy to understand is crucial to increase the user experience. In addition, FoodScan works without being connected to the Internet, so it does not require any communication for using it. FoodScan is an Android application that meets the characteristics and functionalities necessary for these users to control their food, in an easy and intuitive way. It allows users to add the food products they purchased by scanning grocery receipts. By means of an OCR algorithm, the grocery receipts are analysed and the food that matches to those stored is added to a grocery receipts register. With these grocery receipts stored, the app offers users a visual and understandable analysis of the product bought in each group during a period of time, comparing them with the amounts recommended by the specialists. 4) End user evaluation. The results of the satisfaction survey, completed by users who tried the FoodScan tool, highlight its ease of installation and handling.

B. LIMITATIONS
The implemented application and this study have some limitation that should be highlighted: 1) The satisfaction survey do not ask for the demographics information of the participants. We did not consider their inclusion because the only requirement for participating in this study was to be over 50 years old. They all met the demand. 2) FoodScan tracks what is purchased, not what is consumed. Therefore, it cannot be used to estimate the daily nutritional intake, since some of the purchased groceries could be shared or thrown away. These drawbacks were explained during the demonstration of the application and the users were given correct guidelines. Also, consideration should be given to whether the person lives alone or not.
3) The user should follow a good practice in using the application, but if (s)he shares, throws away and buys more than necessary, the results will not be realistic. We are not able to specify what a person has eaten at a certain time, but we get a general idea in a period of a week or a month. 4) Further research is required to determine if users are able to persist in app use over the long term and if the app leads to improvement in dietary intake. We are also interested in knowing the lack of adherence to the application. The responses obtained in a short period of use of the application cannot offer reliable data on the improvement in diet habits or weight loss.

C. CONCLUSION
The elderly who are not used to new technologies may find it difficult to use mobile applications to control their diet. The disadvantage of many of these applications is that they require a lot of time to introduce the food consumed throughout the day. From the user's point of view, it is not practical and, in the long run, most of them will abandon this control. Our app is useful and easily manageable and avoids the obligation to introduce one by one those foods. The analysis of the purchased food is offered through a bar graph with different colors, so it is simple and intuitive. The elderly, or the people in their care, can control how they are eating or what food deficiencies they may have. It can be used by all kinds of public, regardless of their technical knowledge. Although it is geared towards serving people over the age of 70, we hope that it will also be used by anyone in their daily lives and that they will be loyal in the long term. Only by adopting healthy habits we can improve our quality of life.
FoodScan is an application that we have just developed and will continue to improve thanks to the contributions of users. We will regularly monitor the 109 people who completed the survey on the regular use of FoodScan to control their diet.

D. FUTURE LINES OF WORK
One of the aspects of the application to consider in the future is to extend it and implement a more complete ontology and a bigger food collection by adding more quantity and variety. Furthermore, we are already working to expand the food ontology and include users of a wider age range. Another important point would be to improve the design of the application based on user ratings. This application is configured in Spanish, Portuguese and English. It is planned to adapt it to other languages to reach a greater number of users.
To make the application more powerful, artificial intelligence should be added, capable of learning what items refer to a specific food, since a direct comparison is currently made. We will have to weigh that it does not affect the performance and/or speed of the groceries receipt analysis, so that the user does not have to wait a long time. Another improvement to be developed is the possibility of making the food analysis more complete by offering, a more specific study of nutrients and calories ingested.