NRP: A Multi-Source, Heterogeneous, Automatic Data Collection System for Infants in Neonatal Intensive Care Units

The incidence of premature births continues to rise each year and premature babies require specialized care in neonatal intensive care units (NICUs). The intermittent monitoring of these babies, however, poses challenges for clinicians in terms of visualizing and detecting meaningful diagnostic trends. In order to address this issue, the adoption of continuous, multi-parameter, electronic medical record-keeping methods is promising in terms of leveraging advanced analytics techniques and potentially improving health outcomes. In this article, we present the design and implementation of the Neonates Recording Platform (NRP), a hardware-software tool to be deployed at the bedside in a real NICU environment. The NRP enables data from various sources to be collected, labelled, processed and stored. We conducted tests involving the acquisition of neonates' physiological parameters, synchronized with video recordings, in addition to real-time analysis of body pose, with the capture of up to 33 reference points, and audio files from both the infant and the environment. In NRP, the collected data is organized hierarchically in a portable format and is automatically cleaned and validated, thereby ensuring its usability for healthcare professionals and data scientists. Additionally, NRP enables medical staff to configure trials and add customized text or tagging events. A significant contribution of the NRP platform is the integration of a unique computer vision algorithm called CardMed, which extracts physiological information (such as heart rate, breath rate, and blood oxygenation) directly from any monitoring device. The development of NRP involved collaboration with medical staff and data scientists, and evaluation took place at the NICU of the Puerta del Mar University Hospital in Cádiz, Spain.


I. INTRODUCTION
P RETERM infants are those born before the 37th gestational week, and they have a greater risk of developing respiratory (e.g.bronchopulmonary dysplasia), cardiological, infectious, and neurodevelopmental pathologies [1].
Clinical evaluation of these infants in NICUs is typically performed through visual observation of the infants' behavior (facial expressions, whimpering, motion), neuro-imaging exploration, and through the use of sensors attached to their bodies to measure physiological parameters (brain activity with EEG, heart and breath rates, oxygen saturation, etc.).
In recent times, there has been a growing interest in exploring the collection and subsequent analysis of video and audio data streams as non-invasive approaches for gathering information that could enhance the clinical monitoring of health status and care [2], [3].For example, motion features and facial analysis from video is being investigated to detect and characterize infant seizures, for evaluating pain [4], and to identify and classify sleep stages [5], [6], [7], [8], for detecting the presence of the baby and the frequency of their manipulation by staff [9], [10], or as a potential non-invasive method to measure cardio-respiratory rates, potentially serving as an alternative to the more aggressive skin sensors [11], [12].Additionally, infants typically exhibit various body movements, some of which seem purposeful, while others appear as primitive reflexes or automatic responses.These spontaneous movements (or general movements as they are known) can be observed in both preterm and term infants during their first months of life and have been shown to be significant predictors of motor and cognitive skills and their development.
However, developing a reliable pattern recognition system for general movements can be particularly challenging since these movements are sporadic and episodic, and require a certain level of expertise from the evaluating clinician.There has, therefore, been a great deal of research in recent years to focus on developing tools for smart, continuous monitoring using automatic video-based analysis of spontaneous movements [12], [13], [14], [15], [16].Meanwhile, a large number of studies have been conducted to explore solutions for analyzing audio signals of infants by combining microphones and advanced signal processing techniques with the aim of creating classifications for infant crying (e.g.hunger, pain, tiredness) [17], [18].
Undoubtedly, the development of reliable devices and software applications for collecting and analyzing a combination of audio and video data could expand the scope of applications in early clinical diagnosis of various pathologies in preterm infants in NICUs.The initial steps towards utilizing audio and video information for continuous health monitoring in real-world scenarios involve the design and calibration of acquisition devices, ensuring secure data collection and storage in addition to tools for real-time and offline annotation of data by experts (clinicians or researchers) with labels or important comments that can be used in subsequent stages of data processing and clinical evaluation.
In this study, we present the Neonates Recording Platform (NRP), a comprehensive hardware-software tool for collecting, labeling, analyzing and storing data from video and audio sources (capturing both the infant's cries and environmental sounds in the NICU) in the real setting of an NICU, along with the physiological parameters of the infants.The NRP incorporates features for processing the video-data streams to generate secondary measurements such as the real-time automatic analysis of infant body pose which in turn produces a sequence of reference points and directly monitors the infant's vital signs from the NICU monitors which are generally placed at the bedside.
In the field of data analytics, data preparation is both crucial and necessary.It offers numerous advantages, such as improving data quality, detecting errors, enhancing scalability, and facilitating data collaboration.Moreover, if the data already has a standardized format or structure, the time spent on data preparation can be reduced, thereby enabling data scientists to invest more time in implementing and deploying intelligent models [19], [20], [21].In many scenarios, raw data from various sources may not initially meet the characteristics required for the data preparation stage.However, the NRP has been specifically designed to provide readable and machine-understandable datasets that are ready to be utilized by data scientists, particularly in health assessment research (e.g.identifying neurological problems, autism disorders, etc.).
This multi-source information can be used by medical staff and scientists to establish better correlations between different biomarkers in order to facilitate real-time health monitoring of infants and facilitate the early detection or identification of potential health issues.
The remainder of the article is organized as follows: Section II presents related work, showcasing existing audio-camera-based  tools for data acquisition and labeling and highlighting any differences with our approach; Section III introduces the NRP acquisition platform, and focuses on both its hardware and software components; Section IV describes not only system integration at the NICU of the Puerta del Mar University Hospital in Cádiz (Spain) but also the validation processes, and then proceeds to discuss the findings; Section VI provides further discussion; and, finally, Section VII outlines our conclusions.

II. RELATED WORK
In this section, we will discuss recent specifically-designed tools for gathering, recording and processing audio and/or video data of infants in cribs or incubators.Table IV presents a detailed comparison of the specifications.
The Audio-Video Infant Recorder (AVIR) system comprises a laptop connected to a high-speed USB video camera, an external audio board, and a microphone [22].It provides a user-friendly interface for managing video and audio recordings, including labeling functionalities to tag events such as the newborn's state Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

TABLE III CARDMED ALGORITHM EVALUATION
(crying, calm, or awake).An improved version of AVIR, called AVIM (Audio-Video Infant Monitoring), was released three years later [23].AVIM enables video and audio to be simultaneously collected from up to 2 webcams and 2 microphones.It can perform semi-automatic analysis of the infant's movements when the baby is lying in a supine position on an open bed with green bed sheets to facilitate video processing.The crying signal can be automatically analyzed after manual suppression of background sounds.During recording, text annotation is possible, and the system offers other features such as configurable front-end for different experiments, cropping and pasting stored files, and storing data in interoperable formats (csv, jpeg, excel, or txt).However, like AVIR, AVIM is designed for offline data analysis and quantification rather than representing a real-time monitoring tool and has not been tested in the crowded conditions of an NICU.Movidea [24] is an offline application for semi-automatic video analysis of the infant's limb movements with the child lying on a green blanket on a bed.This tool does not support audio acquisition or processing.
The Neo and Voxyvi platforms aggregate data streams from different sensors by integrating hardware and/or software and have been specifically designed for use in NICUs 1 [9], [25].Neo collects and aggregates data streams from different devices typically found at the bedside to monitor infants (e.g.physiological parameters, ventilators, temperature, etc.) and also video.It is built on an affordable Internet of Things platform, and sends the data to a cloud-based large data platform where data is stored before analysis.Voxyvi is equipped with a specific device for collecting not only audio but also color and infrared videos and images.It records 30-minute video sequences that can be stored locally or externally, and which are automatically tagged with the presence/non-presence of the infant in the crib or incubator.

III. SYSTEM DESCRIPTION
The goal of the NRP is to integrate and expand existing tool specifications (further information can be found in Related Work) while also introducing new functionalities that are of 1 Although a number of commercial solutions are available for aggregating and transmitting parameters from certain types of medical monitoring devices, these products are not specifically designed for the needs of an NICU, they use specific communication protocols, and the equipment to which they are connected requires suitable communication interfaces.For example, Etiometry, an end-to-end data management software solution for the collection, analysis, visualization, and archiving of ICU clinical datat; and the third-party BedMaster software, which has been designed to extract and store data (SQL Database) from networked General Electric Healthcare Bedside Patient Multiparameter Monitors (https://www.excel-medical.com/)value to both clinicians and data science and AI experts.These new functionalities include real-time AI analysis, online secondary measurements, audio collection from multiple sources, recording automation, a GUI that can be expanded to aggregate information from heterogeneous data sources, while optimizing the structure, cleaning and validation of collected data.The NRP Unit comprises various hardware components that facilitate data collection from the infants, storage (using an external hard drive), and information gathering and exportation (via a WiFi-based CPU).The NRP Unit is placed above the incubator, with the AI camera focused on the infant, and a second camera positioned with a flexible holder which focuses on the monitor for physiological parameters which are measured using sensors attached to the infant's body.Additionally, one microphone is placed inside the incubator close to the infant and another outside the incubator.

A. NRP Architecture
The NRP Software, which runs on the NRP Unit, enables the scheduling of trials to gather information from the various data sources shown in Fig. 2.This includes algorithms for reading/storing/transferring data streams and a graphical user interface (GUI).Nurses, physicians and doctors can use the GUI to specify the total recording time (or select permanent recording), the sampling time (duration per sample), and the sampling monitoring frequency for gathering physiological information.NRP Software organizes the data using the concept of a "trial" (recording session) and creates a hierarchical folder structure to store all the data collected for each trial.The name of the root folder includes the trial ID number, which can either be manually entered by the user or automatically generated by the software.

B. NRP Software and Data Collection
Fig. 2 shows the graphical user interface of the NRP Software.This can be configured to collect data streams from six different sources: 1) Video: The system enables the infant to be recorded using an embedded AI camera2 which includes three onboard image sensors: a 4 K RGB camera and two monochrome binocular cameras for depth sensing (Panel 2 of the GUI).
The camera features a vision processing unit (VPU) for faster AI inference with convolutional neural network (CNN) models.2) Audio: NRP supports Plug&Play detection of any number of audio input devices (e.g.microphones) based on hardwired (e.g.USB, jack connectors) or wireless (e.g.Bluetooth) connectivity.These devices can be located in or outside the incubator (Panel 4 of the GUI).3) Physiological parameters: In many commercial medical systems, the important source of clinical data is not accessible in open format and cannot be recorded.Since Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

TABLE IV COMPARISON OF PLATFORMS FOR THE COLLECTION, LABELLING AND ANALYSIS OF NICU DATA
it can only be monitored through medical displays, the information is then lost.NRP includes an algorithm for the automatic reading of physiological parameters displayed as text on the screen of a medical device, such as a cardiorespiratory monitor.We have named this functionality CardMed and it is described in Sections III-C, IV-A, and V-A (Panel3 of the GUI).The NRP platform incorporates the CardMed algorithm with an adjustable sampling rate, as illustrated in Panel 1 of Fig. 2, with a default value of 60 seconds.This means that every 60 seconds, a frame of the Monitor Cam video (Panel 3) is captured and fed into the algorithm as input, while also saving the timestamp in milliseconds.4) Pose tracking: NRP extracts secondary information from body poses in real time (20-25 fps), consisting of thirtythree 3D reference points per frame corresponding to the x, y, z coordinates of the joints and certain facial features (e.g.left eye, nose, right heel).The pose is marked with yellow lines on Panel 2 of the GUI.For this task, we have used the human pose estimator Mediapipe Blazepose3 .5) Medical data labeling: Nurses, doctors, and neonatologists can add custom labels during the trials to tag events such as abnormal high-rate values or to register the infant's crying (Panel 5 of the GUI).6) Trial metadata: In addition to the collected healthcarerelated information, NRP also records and structures metadata relating to each trial, including the start time, end time, sampling rate, number of physiological parameters gathered, etc.This information is represented using JSON notation (Panel 1 of the GUI).

C. CardMed: An Algorithm to Extract Features From Medical Devices
In recent times, the monitoring of infants' health and maturation often involves the use of wired-connected sensors attached to their skin.These sensors transmit physiological information through multi-parameter patient monitor devices.However, the prevailing approach entails the use of proprietary systems which not only make it challenging to conveniently extract or export the information, but also restrict access to the information displayed on the screen.One of the key features and contributions of the NRP platform is a custom algorithm called CardMed which is used to measure physiological information (e.g.heart rate, breath rate, blood oxygenation) with a camera pointing directly at the display of any multi-parameter patient monitoring device.
CardMed operates on a cardinal, direction-based approach, leveraging the captured frames (images) from the medical device with the camera, and uses a set of predefined value pairs as input.Each value pair consists of two elements.The first element represents the text, label, or tag displayed on the monitor device, and this identifies a specific physiological parameter (e.g.HR for heart rate, SpO2 for oxygen saturation).The second element indicates the cardinal point (N, S, E, W, NE, NW, SW, SE) where the corresponding measurement number is located in relation to the position of the text/label/tag.The output of the CardMed algorithm is a set of value pairs, with each pair comprising the label/tag of the physiological parameter and its corresponding value measured using transfer learning of pre-trained deep  learning models for OCR (Optical Character Recognition), 4  and image pre-processing to improve accuracy.The complete algorithm is presented in pseudocode in Algorithm 1.

IV. METHODS
In this section, we present the experimental setup for evaluating the NRP system by focusing on four key aspects: real-world 4 EasyOCR - [Online].Available: https://github.com/JaidedAI/EasyOCRFor several weeks we conducted multiple tests of the NRP system at the Neonatal Intensive Care Unit (NICU) of the Puerta del Mar University Hospital5 in Cadiz, Spain (see Fig. 3).We obtained informed consent from parents or legal guardians of all the participating infants in accordance with institutional guidelines.
The initial field tests primarily focused on evaluating the NRP Software.We conducted trials with eight infants to assess the functionality of the software, thereby ensuring accurate data Subsequently, in the second phase of field tests, we evaluated the entire NRP system.This involved assessing the optimal positioning of the device, taking into consideration factors such as image conditions for infant recording, the placement of the second camera to capture physiological parameters, and the performance of audio input devices (microphones) positioned at different locations around the incubator.

A. Physiological Feature Extraction
We assessed the reliability of the CardMed algorithm (this is discussed in further detail in Section 1).For this, a total of 785 images were captured with the second USB camera of the NRP Unit directed towards the display of eight different monitors used for monitoring eight infants.The medical devices used were the DrügerInfinity Delta and Infinity Delta XL.These images were manually labeled and resulted in a CSV file containing the image names, heart rate, respiration rate, and oxygen saturation values.CardMed was then applied to each image, and the measured values for heart rate, oxygen saturation, and respiration rate were added to the same CSV file.This final CSV file was then pre-processed to discard inconsistent values.This involved removing heart rate values exceeding 220 beats per minute 6 [27], respiratory rates above 100 breaths per minute (since typical values usually range between 30 and 60 breaths per minute [28]), and oxygen saturation values above 100 (as it is expressed as a percentage).NRP automatically discarded these inconsistent values and requested a new frame (image) to be captured by the camera in order to reapply the CardMed algorithm.
In evaluating the physiological parameters (heart rate, respiration rate, and oxygen saturation), we measured accuracy, false positives, not found values, and the percentage error.Accuracy represents the average similarity between the labeled data and the estimated results provided by the algorithm.False positives refer to values labeled differently from those provided by the algorithm, while not found values indicate values labeled as missing and also identified as missing by the algorithm.Finally, the mean absolute percentage error (MAPE) was estimated using the following equation [29]: Here, Ri represents the real value (labeled manually), and Ei represents the estimated value provided by CardMed.

B. Clinicians Questionnaire
In order to validate NRP, we distributed a user-friendly questionnaire among the medical staff to gather their feedback on the adoption of our solution as a data collection system for infants in the NICU.The questionnaire aimed to assess their opinions and perspectives regarding the usability, effectiveness, and potential benefits of implementing the NRP system in their daily workflow.
The design of the questionnaire was based on an adapted version of the original technology acceptance model (TAM) which had been specifically tailored for the healthcare field [30], [31], [32].The TAM focuses on evaluating and measuring four key variables: perceived usefulness (PU), perceived ease of use (PEU), attitude towards use (Att), which indicates the user's positive or negative feedback about the system, and behavioral intention to use (IU), which measures the user's willingness to continue using the system (in this case, clinicians).
The questionnaire comprises thirteen questions which are listed in Table I and are organized around the four previously mentioned variables (PU, PEU, Att, and IU).Each question uses a 5-point Likert scale, with 1 indicating "Totally disagree" and 5 indicating "Totally agree".The questionnaire also includes additional questions to gather information about the role and years of professional experience of the clinicians participating in the survey.

C. Data Science/AI Experts Questionnaire
A key specification of NRP design is that it collects clean, structured and validated data.This is essential in any AI or data science-related work as a preliminary step before training models or performing data mining tasks.
In order to evaluate the validity of the proposed NRP system, a questionnaire was created and distributed among domain experts.The questionnaire comprises ten questions (these are Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.II) about data collection, preparation, and data scrubbing.Nine of the ten questions follow a five-point Likert scale ranging from strongly disagree (1) to strongly agree (5).A further question (the second question or Q2) is designed as a matrix question to evaluate the data structure of different data sources.

listed in Table
The questionnaire introduces the experts to the NRP system, presenting visual representations such as Figs. 1 and 2.Not only does it describe the various data sources, including data structure information (this is detailed in Section III-B) but there are also additional technical features such as the codec used to compress video files, the sampling rate of audio files, and the accuracy of the CardMed algorithm (Table III).

V. RESULTS
The NRP system was tested for its ability to operate continuously for extended periods of time (up to several hours), thereby ensuring the automatic, uninterrupted collection of data without the need for human intervention.The audio and video components functioned flawlessly with no glitch or distortion.The system also successfully stored pose tracking data, hand-labeled information, physiological parameters, and metadata separately, without any loss of information.
A number of important considerations can be derived from these tests which highlight the requirements of a multi-source, automatic data collection system in the dynamic environment of an NICU: r The gathering of information and the proposed algo- rithms/techniques implemented within NRP remain unaffected by factors such as the infant's gestational age, size, sex, or behavior.
r Two microphones, one placed inside the incubator and one outside, are sufficient for capturing audio data.Ideally, one microphone should be directional, directed towards the baby, while the second microphone can be omnidirectional.
r Lighting conditions can vary significantly throughout the day and among different babies in the NICU.These conditions can include the use of artificial and natural light sources such as ceiling lights and windows, as well as dark environments.The limitation of dark conditions can be addressed by using low-light, infrared cameras, which can operate effectively even in the absence of a proper light source.
r An ongoing challenge arises when infants are partially covered by blankets, pillows, or bed sheets, which can limit the effectiveness of computer vision algorithms.However, in certain scenarios, algorithms for body detection and upper body pose tracking, in addition to face detection and recognition of facial expressions, may still be functional.

A. Physiological Feature Extraction Results
Table III summarizes the outcomes of the evaluation of the CardMed algorithm.Fig. 4 provides a visual representation of the evaluation of the CardMed algorithm's performance by means of Bland-Altman plots and a box plot.These plots enable the comparison of the agreement between two different methods: the real values (labeled data or ground truth) and the estimated values produced by the CardMed algorithm.Each physiological parameter has its corresponding Bland-Altman plot: heart rate (Fig. 4(a)), oxygen saturation (Fig. 4(b)), and respiration rate (Fig. 4(c)).These plots display the difference between the two methods against the average of the measurements.
In addition to the Bland-Altman plots, Fig. 4(d) presents a box plot that compares the overall accuracy for every sample (images analyzed).This plot provides a visual summary of the distribution of accuracy values obtained from the evaluation of the algorithm across the entire dataset of 785 images.

B. Results of the Clinicians Questionnaire
Fig. 5 displays the results of the questionnaire in a stacked bar chart format.The chart provides an overview of the clinicians' perceptions and attitudes towards the NRP system, as reflected in their responses to the questionnaire.It also provides insights into the level of agreement or disagreement among the participants regarding various aspects of the system, enabling us to assess their acceptance and satisfaction.
In total, 24 healthcare professionals from various fields including doctors, nurses, medical researchers, and pediatricians participated in the questionnaire evaluation.Among the participants, 50% (12 individuals) had 5 or more years of experience in their respective roles.The stacked bars depict the distribution of responses for each question, indicating the percentage of participants who selected each response option.
For the variable Att, the mean score obtained was 4.69 with a mode of 5.This indicates that the participants have a positive perception and attitude towards adopting the NRP system in the NICU.A significant proportion (87.5%) of the participants considered the non-invasive data collection system interesting for monitoring and medical diagnoses, while 91.6% agreed that the system is completely safe for the well-being and comfort of the infants.
In terms of the perceived ease of use PEU and the perceived usefulness PU, the mean scores were 4.09 and 4.54, respectively, with a mode of 5 for both.This suggests that the surveyed clinicians perceive the NRP system to be technologically useful in their daily activities, particularly for early diagnosis and monitoring infant health.While most healthcare professionals in the study agreed that not much effort was required to use the system, there was still some apprehension among those without a technological background.This hesitation is common among healthcare professionals when adopting new technologies.Additionally, the intention to use IU variable showed slightly lower statistical values (mean of 3.96 and mode of 4) due to concerns relating to technological knowledge and privacy issues.
In order to assess the suitability of the TAM-based questionnaire, two important measurements need to be calculated: reliability (internal consistency) and validity (accuracy of measurement) of the questionnaire.The results presented in Fig. 5 indicate the frequency distribution of responses for every questionnaire question.Most of the answers for each question scored 4 or 5, suggesting that medical staff were consistently positive about the use of the NRP system in their daily activities.In order to assess questionnaire reliability, the Cronbach's Alpha method was employed, yielding an Alpha value of 0.877.According to research, a value exceeding 0.7 signifies a high reliability index.Hence, the questionnaire demonstrates strong consistency and reliability, establishing it as a valid tool for evaluating the acceptance of the NRP system.

C. Results of Data Science/AI Experts Questionnaire
The survey was conducted with a group of 20 experts consisting of professors and post-doctoral researchers in the fields of AI, machine learning, and data science, who have experience working in the healthcare domain.These experts are affiliated with various universities in Spain, including the University of Granada, University of Cádiz, Polytechnic University of Valencia, and University of Burgos.Among the participants, 11 individuals reported having five or more years of experience in their respective fields.
Fig. 6 presents a summary of the survey results for the nine Likert-based scale questions answered by the experts in AI, machine learning, and data science.The figure demonstrates that the participants hold a highly positive perception of the NRP system, including its data structure, organization, validation, and cleanliness.The majority of participants strongly agree with Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.most of the questions, which serves as validation for the NRP system.
Additionally, the second question (Q2), which is not shown in Table II, is a matrix question where the rows represent different data sources collected by the NRP system (e.g.infant's video recording, environmental sound, pose tracking, medical data labeling), and the columns represent the features of the data.Each element in the matrix is a checkbox item that can be checked or left unchecked.Fig. 7 visualizes the responses to this question using a bubble chart.The x axis represents the data features (e.g.structured data, cleaned data), while the y axis indicates the number of participants who have checked each option.The legend at the top of the chart identifies the different data sources, which are represented by different colors.The size of each bubble corresponds to the number of participants who have checked the respective option.Large bubbles appear at the top of the chart, indicating that the majority of participants have checked those options, thereby confirming the benefits of the NRP system in terms of data collection, structuring, cleaning, and validation.
While the overall outcomes and conclusions are clearly positive, many participants expressed the need for a pre-processing stage for the infant's audio and video recordings.This suggests that the raw recording data may not be in the optimal format, and this is further discussed in Section VI.
In addition to the previously described questions, the questionnaire also includes optional sections where participants can provide information about their knowledge and skills in data science, machine learning, and AI, the number of years they have been involved in healthcare projects, and three comment sections for suggesting improvements or highlighting drawbacks relating to the NRP system.These valuable suggestions are addressed in Section VI as areas for further work.

VI. DISCUSSION
The NRP system introduces several features, including: r Trial-oriented system: All the data collected in a trial are well identified with a unique identifier, structured in a hierarchical folder-based format, represented in portable and machine-readable formats such as CSV and JSON, cleaned (no null values or empty rows), and validated automatically by the software without human intervention.
r Trial scheduling: The system enables the scheduling of trials, allowing for continuous infant monitoring.
r Plug&play audio input sources: The system allows for the integration of multiple audio input sources (microphones), enabling sound recording from different areas in the NICU.
r Real-time secondary measurements: The system provides the capability to visualize and store infant's poses and movements.
r CardMed: This feature facilitates the automatic detection and measurement of infant physiological parameters from any commercial medical device monitor.
r Cross-platform compatibility: The NRP software is im- plemented in a cross-platform programming language, enabling deployment on any system regardless of the platform.
r Real-time medical data labeling: The NRP software al- lows for real-time medical data labeling, which is included in the trial information.This feature has significant benefits for training models in future tasks conducted by data scientists.When compared with other proposed solutions (as detailed in Section II), the advantages of NRP mainly stem from its hardware/software architecture with edge computing capabilities, the possibility of collecting physiological information from any display device, and the ability to automate trial conduction and to collect well-structured, cleaned, and validated data.A summary of the comparison between NRP and other notable systems is included in Table IV.
One of the main contributions of the NRP system is the CardMed algorithm, which can be used to read text information from medical device displays using computer vision techniques.While the algorithm has been designed to work with any medical device display, it has only been tested with two different displays so far.Further studies are, therefore, necessary to ensure the algorithm's robustness.Additionally, the algorithm requires adjustment to define the input, and, more specifically, the physiological parameters to be extracted and their respective locations on the screen.This adjustment involves modifying the algorithm's source code.In order to address this drawback, it is important to include a high-level, user-friendly front-end in the NRP software, which will enable end-users to easily adjust CardMed.
Moreoever, the NRP system has undergone two validation processes based on two separate, independent questionnaires that were distributed among clinicians and data science, AI and machine learning experts.The feedback from both questionnaires has been undeniably positive (further details can be found in Section V-B and Section V-C).However, participants have provided a number of suggestions for desired features and applications, and conveyed their thoughts about the NRP system.
A number of clinicians mentioned the following: r Continuous monitoring of crying and movement patterns can undoubtedly assist medical staff perform informal evaluations that are currently sporadic or incomplete because of lack of time or experience.
r Physiological parameters could be used to anticipate decision-making and identify infant complications.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.r Clinicians spend a significant amount of time near the incubator and their conversations are also recorded by the NRP microphones, which may conflict with their need for free speech.A number of data science or AI experts highlighted the following tips and improvements: r Provide a portable, machine-readable file with audio fea- tures (sampling rate, frequency, amplitude, etc.) for each audio file.
r Use real-time audio recording to detect infant crying on- the-fly, once features have been extracted.This, combined with medical data labeling, could lead to a classification system.r Utilize the system for automatic sleep analysis, where both motion and sounds are important behavioral descriptors.
r Enhance the NRP software to include personal informa- tion about the infant (age, weight, sex, pathology, etc.) as part of the provided data.
r Enrich the NRP software to incorporate clinical informa- tion, such as blood test results and genetic test outcomes.
r Address the issue of hand-typing medical data labels, as this can lead to an inconsistent set of labels.It would be beneficial to have a closed-set of labels included in the software natively or defined by healthcare professionals and shared with other experts.Although the NRP system has shown promising outcomes in the in-the-wild tests conducted, and the feedback from both healthcare professionals and data scientists/AI/machine learning experts has been positive, it is important to note that further development of the NRP system is necessary.This includes improving existing features with the inclusion of additional ones, conducting further studies, and ultimately building a more comprehensive tool.

VII. CONCLUSION AND FURTHER WORK
In NICUs, healthcare professionals rely heavily on personal note-taking and sporadic infant observation.However, there is a lack of consistent recording of vital information, such as continuously measured physiological parameters in current practices.Additionally, the consistent visual observation of infants, including the frequency and types of motions, which could provide valuable clinical information, is often impractical to maintain.
NRP automates the process of collecting data from various sources, including video, audio, clinical data, and also for processing secondary measurements.The NRP Unit includes various components such as AI-based, regular cameras, microphones, an external hard drive, and a WiFi-based CPU.Physiological information is collected directly from medical devices, with a regular camera and the CardMed algorithm, which has been specifically designed for this study.This information can be used not only to provide enhanced medical information for healthcare professionals to monitor infant maturation, detect abnormal behavior, and conduct health assessments, but also as a dataset that can be employed by professionals with expertise in data science, AI, or machine learning.Data in the NRP system is organized hierarchically in portable, machine-readable formats.It is automatically cleaned and validated, thereby ensuring usability for healthcare professionals and data scientists.This automatic, heterogeneous, multi-source data collection system can, therefore, be used by both medical staff or health researchers and data scientists.
Trials conducted at the Puerta del Mar University Hospital in Cádiz, Spain, demonstrated the effectiveness of the NRP.The CardMed algorithm was evaluated using approximately 800 images, thereby proving its reliability.
Validation from clinicians and AI and data science experts yielded positive results.The clinicians accepted the NRP system for daily use and recognized the medical value of the collected data.AI experts, on the other hand, approved of the data structure, cleanliness, and validation, confirming its suitability for AI techniques.
Although the NRP system is primarily designed for infants, it can be adapted for other healthcare application areas with similar characteristics or different population groups, such as regular hospitalized patients and bedridden patients.
Future work involves incorporating user-proposed functionalities, such as fixed labels and audio file featurization.Additional sources, such as personal details and blood test results, will also be included.Further privacy-preserving measures need to be implemented and explored in order to focus not only on anonymizing videos but also on protecting infants' crying and physiological sequences to prevent any potential re-identification of individuals.Finally, the NRP system will undergo further testing at other hospitals and locations as part of the PARENT project 7 so that it can be improved and enhanced before it is released as open-source software.

VII. ETHICS APPROVAL
The study was approved by the Puertal del Mar University Hospital 8 (Cadiz, Spain) (reference number 125.22) and after obtaining the informed consent of the parents.All research data are de-identified and securely stored.Data access is limited to approved study personnel.

Fig. 1
Fig.1displays the NRP architecture.It consists of two core elements: the NRP Unit, an edge-based, all-in-one embedded hardware device, and the NRP Software.The NRP Unit comprises various hardware components that facilitate data collection from the infants, storage (using an external hard drive), and information gathering and exportation (via a WiFi-based CPU).The NRP Unit is placed above the incubator, with the AI camera focused on the infant, and a second camera positioned with a flexible holder which focuses on the monitor for physiological parameters which are measured using sensors attached to the infant's body.Additionally, one microphone is placed inside the incubator close to the infant and another outside the incubator.The NRP Software, which runs on the NRP Unit, enables the scheduling of trials to gather information from the various data sources shown in Fig.2.This includes algorithms for reading/storing/transferring data streams and a graphical user interface (GUI).Nurses, physicians and doctors can use the GUI to specify the total recording time (or select permanent recording), the sampling time (duration per sample), and the sampling monitoring frequency for gathering physiological information.NRP Software organizes the data using the concept of a "trial" (recording session) and creates a hierarchical folder structure to store all the data collected for each trial.The name of the root folder includes the trial ID number, which can either be manually entered by the user or automatically generated by the software.

Fig. 7 .
Fig. 7. Answers by data scientists to Q2 on data structure and validation.

TABLE I TAM
QUESTIONNAIRE FOR USER-ACCEPTANCE VALIDATION OF NRP AMONG CLINICIANS Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.