Separating and labeling each nuclear instance (instance-aware segmentation) is the
key challenge in nuclear image segmentation. Deep Convolutional Neural Networks have
been demonstrated to solve nuclear image segmentation tasks across different imaging
modalities, but a systematic comparison on complex immunofluorescence images has not
been performed. Deep learning based segmentation requires annotated datasets for training,
but annotated fluorescence nuclear image datasets are rare and of limited size and
complexity. In this work, we evaluate and compare the segmentation effectiveness of
multiple deep learning architectures (U-Net, U-Net ResNet, Cellpose, Mask R-CNN, KG
instance segmentation) and two conventional algorithms (Iterative h-min based watershed,
Attributed relational graphs) on complex fluorescence nuclear images of various types.
We propose and evaluate a novel strategy to create artificial images to extend the
training set. Results show that instance-aware segmentation architectures and Cellpose
outperform the U-Net architectures and conventional methods on complex images in terms
of F1 scores, while the U-Net architectures achieve overall higher mean Dice scores.
Training with additional artificially generated images improves recall and F1 scores
for complex images, thereby leading to top F1 scores for three out of five sample
preparation types. Mask R-CNN trained on artificial images achieves the overall highest
F1 score on complex images of similar conditions to the training set images while
Cellpose achieves the overall highest F1 score on complex images of new imaging conditions.
We provide quantitative results demonstrating that images annotated by under-graduates
are sufficient for training instance-aware segmentation architectures to efficiently
segment complex fluorescence nuclear images.