Large-scale performance evaluation of galaxy image classification models trained on small datasets

Abstract

Many studies have utilized convolutional neural networks to classify galaxies into spiral and elliptical, achieving near-perfect results. However, the authors used relatively small datasets (up to 104,787) that often consist of higher-quality images, making it difficult to extrapolate model large-scale performance. In contrast, we combine data from Galaxy Zoo projects (1, 2, Hubble, CANDELS, DECaLS), preserving as many galaxies as possible. To achieve that, we propose a novel methodology – spiral certainty index (SCI), which allows to extend binary classification by introducing uncertain classes for galaxies where volunteers do not fully agree on the class. Using SCI, we bin galaxies into 2, 3, and 5 classes, resulting in three large datasets of 719,133–800,448 galaxies. We then quantitatively evaluate large-scale performance of Cavanagh, ResNet50, and EfficientNetV2S architecture models by training them using small fractions (0.4–46.5%) of the datasets and validating them on the rest. Experiments were repeated with and without standard augmentations (rotation, flipping, zoom, noise). We conclude that for future research using Galaxy Zoo data, the training dataset of at least 71,913 images (10% of 5-class dataset) is recommended to reach stable accuracy and F1 metrics for all three architectures studied. Below this threshold, Cavanagh, and ResNet50 models either overfit or display high variability of metrics. Moreover, at this threshold or above, the usage of augmentations is equivalent to doubling or even quadrupling (in case of ResNet50 for all classes) the training dataset size. The most stable architecture was EfficientNetV2S, which only overfitted with 2-class dataset, when trained on 3,715 images (0.4% of the dataset), without augmentations.