Handling Sparse Longitudinal Data with Irregular Missing Data - Analysis of Fecal Coliform Bacteria Data

Fecal coliform bacteria are commonly used as an indicator to reflect the fecal contamination level in the water. To manage a healthy and thriving aquaculture industry, the Canadian Shellfish Sanitation Program (CSSP) was established in 1948. As a part of the CSSP mandate, fecal coliform bacteria levels in shellfish growing habitat have been monitored at nearly 15, 000 shellfish harvesting sites across the six coastal provinces of Canada over 40 years (1980–2019). The irregular sparseness in the measurement data presented a critical challenge for reliable analysis of fecal contamination patterns along Canada's coastline. This paper illustrates a preprocessing approach to handle the irregular sparseness in the measurement data of fecal coliform bacteria levels and demonstrates the effectiveness of data filtering, pooling, binning, partitioning, and the application of functional principal component analysis. We managed to transform the irregularly sparse measurements at a site into surrogate variables without missing data that represent the differences in the site's contamination amplitude and seasonal variation from the average. The surrogate variables will be used for downstream analyses, such as associating contamination with climate change.