Optimization of the architecture and hyperparameters of the U-Net model to improve the quality of biological objects segmentation

Main Article Content

Anton Kovalenko
Valerii Severyn

Abstract

The paper addresses the optimization of the U-Net architecture and hyperparameters to improve the segmentation accuracy of biological objects in microscopic images. Image segmentation is formalized as the optimization of a parametric mapping that minimizes a compound loss function combining binary cross-entropy and the Dice coefficient, ensuring better boundary detection and robustness to class imbalance. A modified U-Net with three encoding and decoding levels was developed to balance computational efficiency and segmentation quality. During preprocessing, segmentation masks were refined by filling object contours, which improved model stability and mask accuracy. Experiments on microscopic cell images showed that the combined loss function achieved a mean Dice index of 0.9037, outperforming binary cross-entropy alone. The Adam optimizer provided better convergence and stability than RMSProp, confirming its effectiveness for small datasets. The proposed approach can be applied to automated cell analysis and biomedical diagnostic systems.

Article Details

Section

Articles

References

1. Kovalenko, S. M., Kutsenko, O. S., Kovalenko, S. V., Kovalenko, A. S. Approach to the Automatic Creation of an Annotated Dataset for the Detection, Localization and Classification of Blood Cells in an Image // Radio Electronics, Computer Science, Control. 2024. (1), 128. https://doi.org/10.15588/1607-3274-2024-1-12

2. Kovalenko S., Kovalenko S., Kutsenko A., Godlevskyi M., Severin V. Kovalenko A. Methodology for Creating Annotated Datasets of Biological Objects in Microscopic Images // 2024 IEEE 5th KhPI Week on Advanced Technology (KhPIWeek), Kharkiv, Ukraine, 2024, pp. 1-6, doi: 10.1109/KhPIWeek61434.2024.10878016.

3. Kovalenko S., Kovalenko S., Mikhnova O., Kovalenko A., Pelikh D., Severin V. An Approach to Blood Cell Classification Based on Object Segmentation and Machine Learning // 2023 IEEE 4th KhPI Week on Advanced Technology (KhPIWeek), Kharkiv, Ukraine, 2023, pp. 1-6, doi: 10.1109/KhPIWeek61412.2023.10312903.

4. Jin, S., Yu, S., Peng, J. et al. A novel medical image segmentation approach by using multi-branch segmentation network based on local and global information synchronous learning // Sci Rep 13, 6762. 2023. https://doi.org/10.1038/s41598-023-33357-y

5. Totosko O., Stukhliak D., Stukhliak P. Usage of neural networks for analysis and processing of experimental research of composite materials // Scientific Journal of TNTU (Tern.). 2025. vol 118, no 2, pp. 42–55.

6. Stefanyshyn V., Stefanyshyn I., Pastukh O., Kulikov S. Comparison of the accuracy of machine learning algorithms for brain-computer interaction based on high-performance computing technologies // Scientific Journal of TNTU (Tern.). 2024, vol 115, no 3, pp. 82–90.

7 Jha S., Son L.H., Kumar R., Priyadarshini I., Smarandache F., Long H.V.,Neutrosophic image segmentation with Dice Coefficients // Measurement. 2019. vol. 134. pp. 762-772, ISSN 0263-2241. https://doi.org/10.1016/j.measurement.2018.11.006.

8. Müller, D., Soto-Rey, I., Kramer, F. Towards a guideline for evaluation metrics in medical image segmentation // BMC Res Notes. 2022. 15. 210. https://doi.org/10.1186/s13104-022-06096-y.

9. Ronneberger, O., Fischer, P., Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. // In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science(), vol 9351. 2015. Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28

10. Wang Y., Kong J., Zhang H. U‐Net: A Smart Application with Multidimensional Attention Network for Remote Sensing Images // Scientific Programming 2022.1 2022. 1603273. https://doi.org/10.48550/arXiv.2401.07654

11 Jena B., Jain S., Nayak G.K., Saxena S. Analysis of depth variation of U-NET architecture for brain tumor segmentation // Multimedia Tools Appl. 82, 7, Mar 2023, 10723–10743. https://doi.org/10.1007/s11042-022-13730-1

12 Gkologkinas G.D.; Ntouros K.; Protopapadakis E.; Rallis I. A Comparative Analysis of U-Net Architectures with Dimensionality Reduction for Agricultural Crop Classification Using Hyperspectral Data // Algorithms 2025, 18, 588. https://doi.org/10.3390/a18090588

13 Chinese Hamster Ovary Cells dataset// https://bbbc.broadinstitute.org/BBBC030

14. Furtado P. Testing Segmentation Popular Loss and Variations in Three Multiclass Medical Imaging Problems // J Imaging. 2021 Jan 27;7(2):16. doi: 10.3390/jimaging7020016. PMID: 34460615; PMCID: PMC8321275.

15 Kartowisastro I.H., Latupapua, J.. A comparison of Adaptive Moment Estimation (Adam) and RMSProp optimisation techniques for wildlife animal classification using convolutional neural networks // Revue d'Intelligence Artificielle. 2023. vol. 37, No. 4, pp. 1023-1030. https://doi.org/10.18280/ria.370424