PocketNet: A Smaller Neural Network for Medical Image Analysis

Medical imaging deep learning models are often large and complex, requiring specialized hardware to train and evaluate these models. To address such issues, we propose the PocketNet paradigm to reduce the size of deep learning models by throttling the growth of the number of channels in convolutional neural networks. We demonstrate that, for a range of segmentation and classification tasks, PocketNet architectures produce results comparable to that of conventional neural networks while reducing the number of parameters by multiple orders of magnitude, using up to 90% less GPU memory, and speeding up training times by up to 40%, thereby allowing such models to be trained and deployed in resource-constrained settings.

fully automated segmentation can substantially reduce the time required for target volume delineation and produce more consistent segmentation masks [2], [3].Over the last several years, deep learning methods have produced impressive results for segmentation tasks such as labeling tumors or various anatomical structures [4]- [7].
However, the performance of deep learning methods comes at an enormous computational and monetary cost, independent of concerns about data quality.Training networks to convergence can take several days or weeks using specialized computing equipment with sufficient computing capacity and available memory to handle large imaging datasets.The cost of a workstation with suitable hardware specifications for training large deep learning models ranges from roughly $5,700 to $49,000, whereas a dedicated deep learning-enabled server blade can range from $31,500 to $134,000 [8].Cloud-based solutions offer a more economical option for training models by allowing users to pay for time on accelerated computing instances.However, the cost of such instances ranges from $3 to $32 per hour, and additional measures must be implemented to protect patient privacy [9].The latter may involve an institution entering into a service agreement with a cloudcomputing resource provider.The cost of such an agreement is another consideration that must be taken into account.
We wish to reduce the costs -in model size, training time, and memory requirements -associated with training deep learning models while preserving their performance.To do so, we propose the PocketNet paradigm for deep learning models, a straightforward modification to existing architectures that substantially reduces the number of parameters and maintains the same performance as the original architecture.This modification questions the long-held assumption that doubling the number of features after each downsampling operation (i.e., pooling or convolution) is necessary for convolutional neural networks [4]- [7], [10]- [15].Our work demonstrates the effectiveness of our PocketNets in several 3D segmentation tasks and a classification problem.We also show that the PocketNet paradigm carries several practical benefits -less GPU memory required for training, less time required for training, and faster inference times.

A. Previous work
The last several years have seen several attempts to decrease the number of parameters in deep learning architectures in medical imaging.Broadly, these attempts fall into two categories: post-processing tools and architecture design strategies.

arXiv:2104.10745v4 [eess.IV] 18 Sep 2022
More specifically, pruning (post-processing tool), depth-wise separable (DS) convolutions (architecture design strategy), and filter reductions (architecture design strategy) are known to help mitigate the overparameterization of deep convolutional neural networks (CNNs) for 3D medical image segmentation [16]- [18].These methods, alone and combined, give rise to many of the novel, efficient deep learning architectures that are currently popular.We briefly survey these network reduction strategies below.
Introduced by LeCun et al., the purpose of pruning is to remove the redundant connections within a neural network [16].Pruning starts with a large pre-trained model and involves deleting weights and iteratively retraining the model until a significant drop in performance occurs.Pruning in medical image analysis reduces the required inference time and GPU memory while maintaining model performance [19]- [21].However, network pruning is a post-processing step that is applied to an existing pre-trained network; although useful, pruning does not solve the demands of training large, overparameterized models, which requires a great deal of memory and high-end computing hardware.
Network architecture design strategies for reducing the number of parameters in deep neural networks for medical image analysis include DS convolutions and reduction of the number of feature maps at each layer.DS convolutions are well known for reducing the number of parameters associated with convolution [17], [22], [23].DS convolution factorizes the standard convolution operation into two distinct steps: depth-wise convolution followed by a point-wise convolution.A normal convolution layer has a number of parameters that is quadratic in the number of channels.In contrast, DS convolutions have a number of parameters that is linear in the number of channels.In practice, recently developed medical neural network architectures take advantage of DS convolution to reduce the number of parameters in their networks by up to a factor of 5 while achieving results comparable with those of conventional convolution [24], [25].However, 3D DS convolution is not supported in standard deep learning packages and requires more memory than standard convolution during training due to the storage of an extra gradient layer.
By convention, the number of feature maps doubles after each downsampling operation in a CNN.This growth in the number of feature maps accounts for a large portion of the number of training parameters.With this in mind, perhaps the simplest way to reduce overparameterization in a deep learning model is to reduce the number of feature maps.Van der Putten et al. explore this idea in [18] by dividing the number of feature maps in the decoder portion of a U-Net by a constant factor r.For increasing values of r, the number of parameters in the decoder is reduced by up to a factor of 100, and segmentation performance remains the same.However, this approach introduces another hyperparameter r that needs to be tuned, does not reduce the number of parameters in the encoder branch of the U-Net architecture, and is only applicable to segmentation models.

B. Novel Contributions
This paper makes the following novel contributions:

A. The PocketNet Paradigm
We propose a modification to existing network architectures that dramatically reduces the number of parameters while also retaining performance.Many common network architectures for imaging tasks rely on manipulating images (or image features) at multiple scales because natural images encapsulate data at multiple resolutions.As a result, most CNN architectures -including many popular state-of-the-art methods such as nnUNet [14] and HRNet [13] -follow a pattern of downsampling and upsampling, following the intuition of the original U-Net paper [4] that popularized this approach.In the architecture first presented there, the number of feature maps (i.e., channels) in each convolution operator is doubled each time the resolution of the images decreases; the justification being that the increased number of feature maps offsets the loss of information from downsampling.This idea, of compensating for lost information by increasing the number of features, can be traced before U-Net, to the original ImageNet paper [11] and earlier.In all of these architectures, from ImageNet to U-Net to the state-of-the-art architectures today, the recurring refrain is that training is limited by compute availability due to the network's size, since the number of parameters in all of these architectures grows exponentially as the number of channels is doubled.
However, other classical methods that manipulate images (or other signals) at multiple scales assume a hierarchy of scales i.e. that information can be decomposed into coarse-scale and fine-scale features independently; most prominent among such methods are those based on wavelets [26] and on multigrid methods [27].Intrinsic to these methods is the construction of a series of grids of appropriate resolution.At each resolution, the constructed grid allows for the specific frequencies to be resolved, without aliasing.Because of this hierarchy of scales, and that specific sets of frequencies are resolved by specific grids at specific scales, the information capacity of coarser scales is guaranteed to be less than that of finer scales, and as a result, fewer operations (for multigrid solvers) and memory (for wavelets) are required: information "lost" by downsampling into a smaller, coarser subspace is accounted for at a grid of different resolution, and when images are downsampled to a coarser resolution, it is not necessary to double the number of channels or dimensions to preserve the information capacity at each downsampling instance.As a result, we propose that the doubling of the channels at each resolution in CNN architectures like U-Net is not intrinsically necessary, since each depth in these architectures corresponds to features at a different scale.
A generic U-Net framework is fully written out in Algorithm 1.In the Block procedure of this algorithm, each convolution kernel K i has some number of channels-in and channels-out that depend on the layer's depth in the 'U' of the architecture.The overall depth D used in this architecture is commonly set to D = 3 to D = 6 [4], [5].If the convolutions have c in channels-in and c out channels-out at the network's finest resolution, the convolutions in the next layer will have 2c in channels-in and 2c out channels-out.Subsequently, at resolution depth d, there are 2 d c in channels-in and 2 d c out channels-out Algorithm 2 Multigrid V-Cycle, adapted from [28] Input: for the convolutions.As a result, the number of parameters in a CNN grows exponentially with increasing depth, and this exponential growth is the driving factor for the large size of image segmentation networks.
We remark that Algorithm 1 is nearly identical to a single V-cycle of a geometric multigrid solver for solving the linear system of equations Au = f , where the linear system A and the unknown variable u relate to a geometric grid involving multiple resolutions [27], [28].An algorithm for a V-cycle is shown in Algorithm 2.
The PocketNet paradigm, defined in Definition 2.1, exploits this similarity between multigrid methods and U-Net-like architectures.This paradigm proposes that the number of feature maps used at the finest resolution is sufficiently rich to capture the relevant information for the imaging task at hand and that doubling the number of channels is unnecessary.Instead of doubling the number of feature maps at every level of a CNN, we keep them constant, substantially reducing the number of parameters in our models in the process.We designate network architectures that keep the number of feature maps constant as Pocket Networks, or PocketNets for short, in the sense that these networks are "small enough to fit in one's back pocket".For example, "Pocket U-Net" refers to a U-Net architecture where we apply our proposed modification.Figure 1 provides a visual representation of the PocketNet paradigm applied to a U-Net.We present a formalized definition of the PocketNet paradigm in Definition 2.1.

Definition 2.1:
A network architecture obeys the PocketNet paradigm if the range of all convolution operators (except the final output layer) present in the network is a subset of R h×w×cout , where c out is fixed.Such a network is called a Pocket Network, or PocketNet for short.
While the relationship between multigrid algorithms and U-Nets has been explored (see [29] and references therein), the PocketNet paradigm is applicable to any CNN architecture, regardless of whether or not the architecture is similar in overall structure to a U-Net.
The number of parameters saved in the PocketNet architectures is substantial.Assume that a PocketNet and its corresponding full-sized architecture are identical apart from the doubling of the number of channels in standard convolutions.For simplicity's sake, we further assume that the number of convolutions performed at each resolution is the same, denoted by C. Also for simplicity's sake, we assume that the window of each convolution kernel is isotropic, with a stencil width of k in each dimension.We denote the maximum resolution depth, i.e. the number of downsampling operations in the network architecture, as D. In a full-sized network, a convolution kernel at resolution depth d operating on an nD image (for n = 2 or n = 3) with a stencil width k has k n c in 2 d c out 2 d parameters.Therefore, the full-sized network has the following number of parameters (1) On the other hand, the convolution kernels in a PocketNet architecture have k n c in c out parameters at each resolution depth, resulting in the following number of parameters Therefore, the PocketNet paradigm reduces the number of parameters in a network by up to a factor of This analysis highlights that the growth of parameters with increasing depth is exponential for full networks but only linear for PocketNets.

B. Experiments 1) Data:
We test PocketNet on a range of recent public medical imaging challenge datasets.These datasets encompass three segmentation problems and one classification task, all with various data set sizes, complexity, and modalities.Two of our segmentation tasks -binary liver segmentation in the MICCAI Liver and Tumor Segmentation (LiTS) Challenge 2017 dataset [30] and single-contrast brain extraction in the Neurofeedback Skull-stripped (NFBS) repository [31] -are comparatively simple.We use these datasets for baseline comparisons, much like the MNIST Fashion and CIFAR-10 datasets are used to evaluate newly proposed classification architectures.The third segmentation task -multilabel tumor segmentation in the MICCAI Brain Tumor Segmentation (BraTS) Challenge 2020 dataset [32]- [34] -is commonly viewed as a more complex and technical segmentation problem than LiTS or NFBS segmentation.Therefore, we use BraTS data for our performance benchmarks (see Section III-B).Finally, our classification task -binary classification in the COVIDx8B dataset [35] -evaluates the PocketNet paradigm with a much larger dataset, demonstrating its appropriateness for pertinent problems other than image segmentation.These datasets and their related pre-processing and post-processing methods are described below.
a) LiTS Data: For the LiTS dataset, we perform binary liver segmentation.This dataset consists of the 131 CT scans from the MICCAI 2017 Challenge's multi-institutional training set.These scans vary significantly in the number of slices in the axial direction and voxel resolution, although all axial slices are at 512×512 resolution.As a result, we use the preprocessing steps proposed by nnUNet to handle this variability [14].We resample each image to the median resolution of the training data in the x and y-directions and use the 90th percentile resolution in the z-direction.For intensity normalization, we window each image according to the foreground voxels' 0.5 and 99.5 percentile intensity values across all of the training data.This scheme results in windowing from -17 to 201 HU.We also apply z-score normalization according to the foreground voxels' mean and standard deviation.The LiTS dataset is available for download https://competitions. codalab.org/competitions/17094#learn the details-overview.
b) NFBS Data: The segmentation task for the NFBS dataset is extraction (i.e., segmentation) of the brain from MR data.The NFBS dataset consists of 125 T1-weighted MR images with manually labeled ground truth masks.All images are provided with an isotropic voxel resolution of 1×1×1 mm 3 and are of 256×256×192 resolution.For pre-processing, we apply z-score intensity normalization.The NFBS dataset is available for download at http: //preprocessed-connectomes-project.org/NFBskullstripped/.
c) BraTS Data: The BraTS training set contains 369 multimodal scans from 19 institutions.Each set of scans includes a T1-weighted, post-contrast T1-weighted, T2weighted, and T2 Fluid Attenuated Inversion Recovery volume and a multilabel ground truth segmentation.We merge the labels in each ground truth segmentation and perform whole tumor segmentation for our analysis.All volumes are provided at an isotropic voxel resolution of 1×1×1 mm 3 , co-registered to one another, and skull stripped, with a size of 240×240×155.We crop each image according to the brainmask (i.e., non-zero voxels) and apply z-score intensity normalization on only non-zero voxels for pre-processing.The BraTS training dataset is available for download at https://www.med.upenn.edu/cbica/brats2020/registration.html.2) Network Architectures: For each of our tasks and datasets, we compare the performance of various full-sized architectures with their PocketNet counterparts.
a) Segmentation Architectures: We examine the effects of our proposed modification strategy using five segmentation architectures -U-Net [5], ResNet [7], DenseNet [6], HRNet [13], and nnUNet [14].The U-Net, ResNet, and DenseNet architectures are similar.They use the same U-Net backbone with variations in their convolution blocks (see Figure 2) but are highly prevalent architectures in the literature for 3D medical image segmentation [15].HRNet is not as prevalent in the literature as U-Net and its variants but does represent a fundamentally different deep learning architecture for 3D segmentation.Rather than using a "U" shape, where we continually coarsen (i.e., downsample via pooling) high-resolution features, HRNet preserves features at each resolution level.The nnUNet architecture provides a state-of-the-art baseline for our analysis.
b) COVIDx8B Architectures: Our classification architecture for the COVIDx8B dataset is a U-Net encoder with four downsampling layers.The final layer of the U-Net encoder is flattened via global max-pooling and passed to a fully connected layer for the network's final output.Visually, this architecture is represented by the left half of the Pocket U-Net architecture in Figure 1. 3) Training Protocols and Hyperparameters: For each task (e.g., segmentation and classification), we initialize the first layer in each network with 32 feature maps and use the Adam optimizer [36].The learning rate is set to 0.0003.Training for all segmentation tasks uses a batch size of two and a batch size of 32 for COVIDx8B classification.For all segmentation tasks, we use a patch size of 128 × 128 × 128 and apply the same random augmentation described in [14].To evaluate a predicted segmentation mask's validity, we use the Sorensen-Dice Similarity Coefficient (Dice), the 95 th percentile Hausdorff distance, and the average surface distance.Implementations of these metrics are available through the SimpleITK Python package [37]- [39].For the segmentation tasks, our loss function is calculated as an L 2 relaxation of the Dice score; for a true segmentation Y true and a predicted segmentation Y pred , our L 2 -Dice loss function is taken from [40] and is given as For the nnUNet architecture, we add binary cross entropy to the Dice loss shown above.
For the classification tasks, we use categorical cross-entropy as our loss function, with outputs being of two classes (COVID-19 positive and negative).We use the receiver operating characteristic area under the curve (AUC) metric to evaluate each classification model's validity.This metric is available via the scikit-learn Python package.Our models are implemented in Python using TensorFlow (v2.8.0) and trained on an NVIDIA Quadro RTX 8000 GPU [41].All network weights are initialized using the default initializers from TensorFlow.All other hyperparameters are left at their default values.The code for each network architecture is available at github.com/aecelaya/pocketnet.
4) Inference and Post-Processing: We perform inference on test images using a sliding window approach for each segmentation task, where the window size equals the patch size set during training.After each window prediction, we slide the window by half the size of the patch.Additionally, we apply a Gaussian importance weighting (σ = 0.125) to each window prediction [14].
For post-processing, we try each of the following strategies for each segmentation task and network: 1) Apply morphological clean-up -i.e., erode by two voxels, retain the largest connected component, dilate by one voxel, and fill holes.
2) Only retains the largest connected component.
3) Do nothing.We select the strategy that yields the highest mean Dice as our final post-processing strategy for a given task and network.
We note that the inference and post-processing steps described above are applied to all segmentation tasks for every network we test.

A. Comparable Accuracy
We use the training parameters described in Section II-B3 to train the architectures listed in Section II-B2.For the segmentation tasks described in Section II-B1, we employ a five-fold cross-validation scheme to obtain predictions for each dataset and architecture.For classification on the COVIDx8B dataset, we train each model with the training set and generate predictions on the test set.These results of these experiments are shown in Tables I and II.
For the BraTS and LiTS datasets, we generally do not see significant performance differences (p < 0.05 [Wilcoxon signed-rank test]) between the full and pocket architectures.In the cases where we see a significant difference in performance, the differences are small, with median Dice scores differing by less than 1% and median Hausdorff distances differing by less than a fraction of a millimeter.We also do not see a clear pattern in significant performance differences between each pocket and full architecture.Given this and the fact that these differences are generally small, we believe that the outperformance of a full-sized network by its pocket counterpart or vice versa can be explained by the stochastic nature of training each network.
For the NFBS dataset, we generally see statistically significant differences in performance.However, we again note that these differences are small, and there is no clear pattern for which architecture (i.e., full or pocket) outperforms.These differences are so small (less than 0.0001 for Dice) that they bear no clinical significance.Such minor differences, in this case, are imperceptible and meaningless in a practical setting.
For all three segmentation tasks, these insignificant or minor differences in performance indicate a reduction in the number of parameters by more than an order of magnitude.

B. Performance Analysis
Using the training parameters described in Section II-B3, we profile the training performance of a full U-Net and a Pocket U-Net using the BraTS dataset.Namely, we measure peak GPU memory utilization during training and the average time per training step for varying batch sizes for each network using the TensorFlow Profiler [42].To ensure accurate comparisons of performance, we conduct these experiments on a Google Colaboratory notebook with a dedicated NVIDIA Tesla T4 GPU with 16 GB of available memory.
The GPU memory usage and training time per step for this experiment are shown in Figure 3   28.3% and 87.7%, with smaller batch sizes resulting in greater savings.This relationship is possibly due to the increasing portion of GPU memory allocated for storing data as the batch size increases.The PocketNet models improve the average time per training step by between 25.0% and 43.2%, with larger batches yielding greater time savings.This behavior may be due to larger batch sizes taking advantage of the computational parallelism of modern GPUs.
In addition to training performance, we benchmark the inference speed of full-sized networks and their PocketNet counterparts.We choose throughput in terms of the number of images predicted per second as our metric for inference speed.Throughput is a popular way of benchmarking the inference speed of a network [43]- [46].Higher throughput means that a network can perform faster inference in a given computing environment.We measure the throughput of the full and pocket variants for several networks and compute the percent speed up of the PocketNet for several different image sizes.The percent speed up is given by the following: Table III shows the inference throughput of each architecture for various image sizes.For architectures with a standard U-Net backbone (i.e., U-Net, ResNet, and DenseNet), we see modest improvements in inference speed for the PocketNet variants that range from 1% to 12%.A possible explanation for these minor improvements in inference speed is the highly parallelized computation of convolution operators on modern GPUs.For HRNet, we see more significant improvements in inference throughput ranging from 12% to 17%.Within the HRNet architecture, we see more non-convolution operations like upsampling, downsampling, and addition than in U-shaped architectures, which may explain the more significant increases in throughput for the Pocket HRNet architecture.

C. Ablation Study
To assess the effects of feature map doubling in U-shaped architectures, we perform an ablation study on a standard U-   net using the LiTS dataset.In the first iteration of the ablation study, we start with a Full U-Net where we double the number of feature maps at every resolution level.In the next iteration, we construct a U-Net where we stop doubling feature maps after the second-to-last resolution level (i.e., 8 × 8 × 8 at d = 1).We continue until we arrive at the Pocket U-Net.
By performing this ablation study, we can determine at what resolution does feature map doubling become important for the network's accuracy.
For each of these networks, we perform a five-fold crossvalidation using training and inference parameters described in Sections II-B3 and II-B4.Table IV shows the results of each iteration.In every case, we see small differences in the distribution of the resulting Dice scores.This small difference in performance among iterations suggests that doubling the number of feature maps at each resolution level might be unnecessary.

D. Feature Activation Analysis
We conjecture that the comparable performance between the PocketNet and full architectures is due to both networks having similar representation capabilities, that ultimately both networks build similar representations of the image data as they compute the final segmentations.To test whether the networks learn similar features, we look at the mean of the feature map activations in the final layer before voxel-wise classification in full and Pocket U-Nets trained on BraTS data.In this way, we can determine if both networks are capturing similar features and information.For each image in the test set, we examine the output of the activation functions from the final layer before classification and measure the size of the response towards the convolution feature, by taking the channel-wise average of the resulting feature maps of the activation output.This process is repeated for the entire test set.More precisely, let F i be the average of the i th feature map resulting from activation outputs from the final layer before classification over the entire test set.Let f j i be the i th feature  map resulting from the activation functions for test patch j.
For each i = 1, . . ., 32, we have where N test is the number patches in the test set, V j is the volume of test patch j, and f j i k is the intensity of the k th voxel in f j i .Figure 4 shows the averages of the resulting feature maps.We see a similar number of features being activated with similar intensities for both cases.This similarity suggests that the full and Pocket U-Nets learn similar latent feature representations used for the final voxel-wise classification.Note that we sort the mean feature activations from highest to lowest for visual purposes.This order does not matter because the indexing in any hidden layer can always be permuted by the next layer.

E. Model Saturation
A possible concern is that PocketNet models, due to their reduced parameter count, could saturate earlier during training than do full-sized architectures, which could result in the comparable performance we observe in our results in Section III-A.To test this, we repeat the experiments described above for the COVIDx8B and BraTS data challenges using successively less data in the training set.For every iteration, we keep a fixed validation and test set.For the COVIDx8B dataset, we fix 10% of the training data as a validation set and use the original test set.Similarly, for the BraTS data, we take 20% of the training data as a test set and use 10% of the remaining patients as a validation set.Additionally, we do not use data augmentation for this particular experiment.The results of this are shown in Figure 5a and Figure 5b.
In Figure 5a, we see that the AUC values increase for both the PocketNet and full architectures as the size of the training set increases.Furthermore, we observe that these AUC values plateau at 1.0 (i.e., perfect prediction), and the PocketNet classifier saturates sooner than its full-sized counterpart.These observations suggest that the reduced architecture resulting from the PocketNet paradigm learns faster with fewer data points than its full-sized counterpart.Similarly, Figures 5b,  5c, and 5d show that the segmentation accuracy of the full and pocket U-Net architectures improves as the training set size increases.Both architecture types show the expected improvement in performance with each increase in the dataset size, plateauing to similar distributions.

IV. DISCUSSION
Our results show that large numbers of parameters (millions or tens of millions) may not be necessary for deep learning in medical image analysis, as comparable performance is achievable with substantially smaller networks using the same architectures but without doubling the number of channels at coarser resolutions.This suggests that overparameterization, which is increasingly regarded as a key reason why neural networks learn efficiently, might not be as critical as previously suggested [47]- [49].However, we note that our PocketNets may still be are overparameterized, and the combination of our proposed PocketNet paradigm with other model reduction techniques should be explored.For example, replacing the traditional convolution layers with DS convolution layers in our Pocket ResNet for LiTS liver segmentation further reduces the number of parameters to roughly 10,000.Pruning an already trained PocketNet model may also potentially yield further parameter reductions.
The deep learning tasks presented in this study are all single-label segmentation or binary classification.The goal of ongoing and future work using the PocketNet paradigm is to test this approach on more complex domains such as BraTS multi-class tumor segmentation and LiTS tumor segmentation.Figure 6 shows an example of a multiclass segmentation prediction mask produced by a Pocket DenseNet.Our results for PocketNet architectures applied to the BraTS multi-class segmentation task are available at https: //www.cbica.upenn.edu/BraTS20/lboardValidation.htmlunder the team name "aecmda" and will be updated periodically.
Regarding Section III-E and the results shown in Figure 5, we note that the results for each task using 1.5% of the data look surprisingly good.For the segmentation task (i.e., BraTS tumor segmentation), a closer analysis reveals that while the full and pocket U-Nets can identify brain tumors in the BraTS dataset with a small percentage of the available training data, they tend to produce rather noisy predictions.Figure 7 illustrates an example of these predictions.In both cases, the networks erroneously predict brighter areas of the MR images as tumors.We hypothesize that the images in the BraTS dataset are similar enough that even with a small percentage of available data, each network can learn to predict rough segmentations of the target.This behavior, however, supports our claim that the pocket and full-sized U-Nets behave similarly with varying dataset sizes.Again, Figure 7 illustrates this by showing an example prediction from each network for small and large training sets.In both cases, we see similar behavior, with both architectures producing noisy predictions with fewer data and more accurate predictions as we increase the dataset size.
Additionally, the Dice coefficient may be too forgiving of a metric for this type of analysis.Even for noisy predictions, we can achieve Dice scores of about 0.7.To mitigate this, we compute the Hausdorff 95 and average surface distances for each set of predictions in Figures 5c and 5d, respectively.With the Hausdorff 95 metric, we see that predictions from training on small datasets generally result in significant errors with substantial variation.However, for larger datasets, we see that for each network variant, the distribution of Hausdorff 95 distances converges to smaller values with reduced variance.We again see similar behavior with the average surface distance.We see significant errors with considerable variance for each network variant for smaller datasets.These distributions plateau to smaller values with less variance with increased training set size for both full and pocket networks.These cases support our initial claim that both networks exhibit similar behavior with different dataset sizes.
For the classification results in Figure 5, we again argue that the images in the dataset are similar enough to each other that even with a small percentage taken as a training set, each architecture can roughly discriminate between the two classes.
When we employ PocketNet models, we achieve similar performance to full-sized networks while enjoying the advantages of faster training times and lower memory requirements.The smaller models produced by our proposed PocketNet paradigm can potentially lower the entry costs (computational and monetary) of training deep learning models in resourceconstrained environments without access to specialized computing equipment.With less GPU memory required for training, cheaper hardware can be purchased, or less expensive cloud computing instances can be used to train deep learning models for medical image analysis.The faster training times for PocketNets can also reduce costs by reducing the number of hours spent training models on cloud computing instances.
As to why the PocketNet models perform at least comparable to their counterpart full models, the similarity in intensities of each signal in the final layer activation maps suggests that the models ultimately learn the same representations.Despite the reduced number of parameters, the approximation space   that the Pocket architectures can represent is comparable to the approximation space that the full models can achieve.This conjecture is supported by the ablation study results in Table IV: the median Dice scores for e.g.U-Net is nearly identical, regardless of the depth that the feature map stops doubling.In this study, the additional features supplied at depths where the doubling continued did not improve the model's performance, as the approximation spaces are all similar regardless of whether the number of features was doubled at that depth.This ablation study suggests that, since there is no increased benefit of having larger models with the number of features doubling per layer, that for these medical imaging problems, doubling the number of channels is unnecessary and PocketNet models can be used to achieve comparable accuracy instead.Since these smaller models are just as expressive (and capable) as their full counterparts, these models can be trained (and later, deployed) with cheaper hardware or by provisioning smaller cloud instances, saving time, money, and effort by institutions performing deep learning medical image analysis.Fig. 7: Example predictions for full and pocket U-Net on BraTS dataset with varying percentages of the dataset used for training.In both cases, we see similar behavior, with both architectures producing noisy predictions with fewer data and more accurate predictions as we increase the dataset size.
d) COVIDx8B Data: The classification task for the COVIDx8B dataset is COVID-19 detection on 2D chest xrays.The COVIDx8B dataset consists of a training set with

Fig. 1 :
Fig. 1: (Left) Full U-Net (top) and ResNet (bottom) segmentation architectures where for each downsampling step, the number of feature maps doubles after each convolution layer.(Right) Pocket U-Net (top) and ResNet (bottom) segmentation architecture.In this PocketNet architecture, the number of feature maps resulting from each convolution layer remains constant regardless of spatial resolution, resulting in substantially fewer parameters overall.

Fig. 3 :
Fig. 3: (Left) Peak memory usage during training on the BraTS dataset for Pocket U-Net vs Full U-Net for varying batch sizes.The PocketNet architecture results in memory savings of between 28.3% and 87.7%.(Right) Average time per training step on BraTS dataset for Pocket U-Net vs Full U-Net for varying batch sizes.The PocketNet models speed up the average time per training step by between 25.0% and 43.2%.

Fig. 4 :
Fig. 4: Feature map activations in final layer before voxel-wise classification in a Pocket U-Net (top) and full U-Net (bottom) for the BraTS dataset.In both cases, we see a similar number of features being activated with roughly the same intensities.
(a) Classification performance on COVIDx8B data for PocketNet vs. Standard U-Net encoder classifier, trained using various subsets of the training data.(b) Distributions of Dice scores on BraTS test set for a full U-Net and a pocket U-Net for varying training set sizes.(c) Distributions of Hausdorff 95 distances on BraTS test set for a full U-Net and a pocket U-Net for varying training set sizes.(d) Distributions of average surface distances on BraTS test set for a full U-Net and a pocket U-Net for varying training set sizes.

Fig. 5 :
Fig. 5: Testing results from using small subsets of data for PocketNet vs. full architectures on COVID19x8B classification and BraTS segmentation challenges.
; we see that our PocketNet architecture reduces memory usage and speeds up training time for every batch size.Specifically, the Pocket U-Net reduces the peak memory usage for training by between

TABLE I :
Accuracy of deep learning models on a set of medical imaging tasks for Pocket vs. full-sized architectures.The PocketNet models' accuracy scores are generally comparable (within 1% or less) of the full models' accuracy scores.Accuracy scores for segmentation tasks (LiTS, NFBS, BraTS) are evaluated using Dice Similarity Coefficient, Hausdorff 95 distance, and average surface distance.

TABLE II :
Accuracy of deep learning models on the COVIDx classification task for Pocket vs. full-sized architectures.The PocketNet models' accuracy scores are generally comparable (within 1% or less) to the full models' accuracy scores.Accuracy scores for the COVIDx classification task are evaluated using AUC, and hence do not have standard deviations or pvalues.

TABLE IV :
Accuracy of U-Net architectures on LiTS dataset where feature map doubling stops after a given depth d.In every case, we see small differences in the distribution of the resulting Dice scores.