Unsupervised segmentation on microscopy images

Supervisor: Dmytro Fishman

Example of prediction by U-Net trained on Otsu on its best epoch (IOU = 0.7834). From top to bottom: original image, network prediction, ground truth.

Our goal

Introduction

Data

Model

U-Net architecture. Source: [1]

We have implemented U-Net and data loading and preprocessing pipeline following a great tutorial by Aladdin Persson:

Setup

Baseline

Example of Otsu’s prediction. We observe that darker nuclei are not being detected.

Methods and approaches

Otsu as a ground truth

Example of prediction of U-Net trained on Otsu’s method as a ground truth. We observe that the network can generalize beyond imperfect ground truth, however, it quickly starts overfitting Otsu’s prediction.
We observe that IOU metric is unstable when we use Otsu’s output as a ground truth, however on some epochs we outperform baseline Otsu, hence the network can generalize beyond the imperfect ground truth. We will use this idea later.

ACWE as a loss for CNN

One of the widely used methods in medical imaging segmentation is Active Contours Without Edges² introduced back in 2001.

Energy functional and corresponding system of differential equations in Active Contours Without Edges method. Source: [2]

In deep learning, loss function plays a vital role, since the network extracts information based on what we define as an objective. In the case of the unsupervised setting, we do not have a ground truth to compensate for the lack of a strong objective function. Hence, one of the approaches could be to have a fresh look at the traditional methods in the field and parametrize proven-to-work energy functions as a loss function for the deep neural network.

One of the works³ that parametrizes ACWE energy functional as a loss for CNN was recently presented at MIDL 2020 conference. There, the authors solve a task of unsupervised bone segmentation from synthetic CT-scan images. We have tried the corresponding loss function on our data set.

Example of the prediction by U-Net trained with ACWE loss. On the right-most plot, one can see the difference between a true mask and a predicted mask. True positive predictions are depicted with green, false negative — with red, and false positive — with blue. We can observe that our network predicts nuclei that is not present on GT, however, is visible on the input image. Our GT is imperfect — it does not contain all annotations. Metrics: IOU: 0.8578, Accuracy’: 0.985, Recall: 0.985, Precision: 0.868, Specificity: 0.985, F1 (Dice) score: 0.923.
Histogram of probability map for U-Net trained with ACWE loss. One needs to look at the histogram of intensities to select a correct threshold to obtain a binary mask. Histogram is consistent across samples. The corresponding threshold was 0.05.

We have also used masks generated by U-Net trained with ACWE loss as ground truth for training U-net with BCE, and it helped improve the performance.

RFCM as a loss for CNN

RFCM as a loss function for CNN. Source: [4]

As one can observe, RFCM loss has 2 hyperparameters: fuzzy factor q and regularization weight beta. q = 1 corresponds to hard clustering, while the larger values result in soft clustering. The regularization parameter penalizes “changes in the value of the membership functions in local neighborhoods.”⁴ We have experimented with different values of q and beta, and the best results were obtained for q = 1 and beta = 0.

Example of prediction by U-Net trained with RFCM loss. On the right-most plot, one can see the difference between a true mask and a predicted mask. True positive predictions are depicted with green, false negative — with red, and false positive — with blue. We observe false positive nuclei on the “Difference” plot, however, our ground truth does not contain all nuclei, and we observe that corresponding predictions are present on the input image.
Histogram of probability map for U-Net trained with RFCM loss. One needs to look at the histogram of intensities to select a correct threshold to obtain a binary mask. Histogram is consistent across samples. The corresponding threshold was 0.05.

Similar to the ACWE loss experiments, we have also used masks generated by U-Net trained with RFCM loss as a ground truth. However, we have also added post-processing of the generated masks, specifically, we have applied erosion⁵ procedure (with 2 by 2 kernel) from the OpenCV library to reduce false positive margin around the predicted nuclei.

Results

Metrics on the training set. * means U-Net trained with BCE on masks generated by U-Net trained with RFCM loss (we have also applied cv2.erode() to generated masks to reduce false positive boundary around predictions). ** means U-Net trained with BCE on masks generated by U-Net trained with ACWE loss
Metrics on validation set. * means U-Net trained with BCE on masks generated by U-Net trained with RFCM loss (we have also applied cv2.erode() to generated masks to reduce false positive boundary around predictions). ** means U-Net trained with BCE on masks generated by U-Net trained with ACWE loss

We observe that we have outperformed the baseline method. Noticeably, we have obtained significantly larger Recall,

Conclusions

Future plans

Example of brightfield and fluorescence modalities for one sample.

The project has been done as a part of the Neural Networks course at the University of Tartu.

References

[2] T. F. Chan and L. A. Vese, “Active contours without edges,” in IEEE Transactions on Image Processing, vol. 10, no. 2, pp. 266–277, Feb. 2001, doi: 10.1109/83.902291.

[3] J. Chen and E. C. Frey. Medical image segmentation via unsupervised convolutional neural net-work. InMedical Imaging with Deep Learning, 2020. https://2020.midl.io/papers/chen20.html

[4] J. Chen, Y. Li, L. P. Luna, H. Chung, S. Rowe, Y. Du, L. Solnes, and E. Frey. Learning fuzzy clustering for spect/ct segmentation via convolutional neural networks.Medical physics, 2021.

[5] https://docs.opencv.org/master/d9/d61/tutorial_py_morphological_ops.html

[6] Pham DL. Spatial models for fuzzy clustering. Comput Vis Image Underst. 2001;84:285–297

[7] N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” in IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62–66, Jan. 1979, doi: 10.1109/TSMC.1979.4310076.