FlySeg: an Automated Volumetric Instance Segmentation Algorithm for Dense Cell Populations in Drosophila Melanogaster Nervous System

The Drosophila Melanogaster (fruit fly) nervous system serves as a highly convenient model for investigating the underpinnings of diverse biological processes at molecular, cellular, and circuit levels. Advances in molecular genetics, in conjunction with state-of-the-art imaging techniques, now enable the integration of molecular profiles of individual neurons with their physiological and morphological properties. This integration facilitates a comprehensive understanding of the nervous system, bridging the gap between development, structure, and function. Spatial transcriptomics, a technique leveraging fluorescent probes attached to specific DNA or RNA sequences, plays a critical role in achieving this integration. However, the accurate segmentation of neurons is crucial for quantifying the fluorescent signatures (i.e., individual transcripts) within each cell. In this context, we introduce FlySeg, an automated volumetric instance segmentation algorithm that employs a combination of scale space analysis, Voronoi tessellation, and energy functional minimization to identify the boundary between the nucleus and the surrounding cyto-plasm. The resulting masks serve as a basis for quantifying gene expression across individual neurons, achieved by counting the fluorescent signatures within each segmented nucleus. Therefore, FlySeg emerges as a promising and innovative tool that effectively address contemporary questions in molecular neurobiology.