Home / Current Issue / Paper 1724322
Beyond Clean Benchmarks: A Reproducible Severity -Sweep Evaluation of AI Generated Image Detector Robustness
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence, Computer Vision
Abstract
AI-generated image detectors are used often for verifying news, moderating content and handling evidence. However most tests for these detectors use images. Images that circulate in the world are often recompressed, resized, cropped or filtered. Large studies already show that detector accuracy drops when images are degraded. This study does not claim to discover that. Instead it offers a severity-sweep test. This test asks whether a detector that works well on images will still work well when the images are intentionally degraded. Two detectors that are publicly available were tested. CNNSpot is a network trained with ProGAN. UnivFD is a CLIP linear probe. The detectors were not retrained. They were tested on 2,000 images: 1,000 images and 1,000 synthetic images from Stable Diffusion v1.5, ADM, Midjourney and BigGAN. These images come from a community mirror of GenImage. Five image transformations were applied at levels of severity: JPEG compression, resizing, centre cropping, blur and Gaussian noise. Two chains of transformations were also used. On images UnivFD performed better than CNNSpot with AUROC values of 0.741 versus 0.662. All conditions showed the pattern. Gaussian blur was the transformation that consistently lowered AUROC for both detectors. For CNNSpot AUROC fell from 0.661 to 0.627 when sigma increased from 1 to 4. For UnivFD AUROC fell from 0.733 to 0.688 over the range. JPEG compression, resizing and noise produced non-monotonic changes. Cropping unexpectedly improved AUROC for both detectors. No single transformation reduced AUROC by than 0.10. The chains of transformations were usually no more harmful than the steps. Two of four planned detectors were tested. The dataset is small. Contains a mix of generators. Therefore the findings apply to these two detectors in these conditions. The results indicate that the reliability of detection tools should be checked step by step for each transformation than relying on a clean-image benchmark score. This has implications for how organisations validate, deploy and govern tools.
Keywords
generated image detection; trustworthy AI; robustness evaluation; digital forensics; responsible AI
References
[1] Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
[2] Choi S, Lee H, Lee M (2025). Training-free detection of AI-generated images via cropping robustness. In: Advances in Neural Information Processing Systems (NeurIPS 2025). https://arxiv.org/abs/2511.14030
[3] Li C, et al. (2025). Bridging the gap between real-world evaluation: Benchmarking image detection in challenging scenarios. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20379–20389. https://arxiv.org/abs/2509.09172
[4] Li K, Ren W, Wang W, Cao X (2026). Detecting compressed AI-generated images via phase spectrum robustness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[5] Ojha U, Li Y, Lee YJ (2023). Towards universal fake image detectors that generalize across generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24480–24489. https://arxiv.org/abs/2302.10174
[6] Pellegrini L, et al. (2025). AI-GenBench: A new ongoing benchmark for AI-generated image detection. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/IJCNN64981.2025.11228377
[7] TrueMedia.org (n.d.). Distil-DIRE: A lightweight distilled variant of DIRE for image detection [Computer software]. GitHub. https://github.com/truemediaorg/ml-models
[8] Wang SY, Wang O, Zhang R, Owens A, Efros AA (2020). CNN-generated images are surprisingly easy to spot… for now. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8695–8704.
[9] Wang Z, et al. (2023). DIRE for diffusion-generated image detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22445–22455.
[10] Yan S, et al. (2025). A sanity check for AI-generated image detection. In: Proceedings of the International Conference on Learning Representations (ICLR). https://github.com/shilinyan99/AIDE
[11] Zhu M, et al. (2023). GenImage: A million-scale benchmark for detecting AI-generated image. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. https://arxiv.org/abs/2306.08571
How to cite this paper
@article{1724322,
author = {Sunidhi, Ragini Rai, Pranay Tehra, Sumit Mahajan},
title = {Beyond Clean Benchmarks: A Reproducible Severity -Sweep Evaluation of AI Generated Image Detector Robustness},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {10},
number = {4},
pages = {3854-3863},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1724322.pdf},
abstract = {AI-generated image detectors are used often for verifying news, moderating content and handling evidence. However most tests for these detectors use images. Images that circulate in the world are often recompressed, resized, cropped or filtered. Large studies already show that detector accuracy drops when images are degraded. This study does not claim to discover that. Instead it offers a severity-sweep test. This test asks whether a detector that works well on images will still work well when the images are intentionally degraded. Two detectors that are publicly available were tested. CNNSpot is a network trained with ProGAN. UnivFD is a CLIP linear probe. The detectors were not retrained. They were tested on 2,000 images: 1,000 images and 1,000 synthetic images from Stable Diffusion v1.5, ADM, Midjourney and BigGAN. These images come from a community mirror of GenImage. Five image transformations were applied at levels of severity: JPEG compression, resizing, centre cropping, blur and Gaussian noise. Two chains of transformations were also used. On images UnivFD performed better than CNNSpot with AUROC values of 0.741 versus 0.662. All conditions showed the pattern. Gaussian blur was the transformation that consistently lowered AUROC for both detectors. For CNNSpot AUROC fell from 0.661 to 0.627 when sigma increased from 1 to 4. For UnivFD AUROC fell from 0.733 to 0.688 over the range. JPEG compression, resizing and noise produced non-monotonic changes. Cropping unexpectedly improved AUROC for both detectors. No single transformation reduced AUROC by than 0.10. The chains of transformations were usually no more harmful than the steps. Two of four planned detectors were tested. The dataset is small. Contains a mix of generators. Therefore the findings apply to these two detectors in these conditions. The results indicate that the reliability of detection tools should be checked step by step for each transformation than relying on a clean-image benchmark score. This has implications for how organisations validate, deploy and govern tools.},
keywords = {generated image detection; trustworthy AI; robustness evaluation; digital forensics; responsible AI},
month = {October},
}