Improving Bug Detection in LLM-Generated Unit Tests: Revisiting Test-Oracle Reliability Across Modern Large Language Models

Zeyad Farooq Lutfi

doi:10.71229/b40nsd71

Authors

Zeyad Farooq Lutfi Computer Science Department, Collage of Science, Mustansiriyah University, Baghdad-Iraq

DOI:

https://doi.org/10.71229/b40nsd71

Keywords:

software testing, , large language models, , automated test generation, , est oracle, , bug detection, , empirical study.

Abstract

The widespread adoption of Large Language Models (LLMs) has given rise to a new generation of automated unit-test generation tools, such as Codium CoverAgent, CoverUp, and GitHub Copilot, promising to reduce the manual burden of writing tests while preserving bug-detection quality. However, a recent study revealed a fundamental issue: these tools may generate tests that are effectively "designed to pass" rather than designed to catch bugs, because the test oracle is frequently derived from the very code under test rather than from an independent specification of correct behavior. This paper presents a Replication and Extension Study that re-examines this issue using a newer generation of large language models (e.g., Claude Sonnet 4.6, GPT-5, and open-source models such as Qwen3-Coder), using the Refactory dataset, which contains 1,783 real buggy Python programs written by actual university students alongside 2,442 correct reference solutions. This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests: McNemar test and Wilson confidence interval to ensure that the results are rigorous. The paper sets out to address a direct research question: has the effectiveness of bug detection increased with the capabilities of LLMs, or does it stem from a structural problem that is not dependent on the underlying model? The Bug-Revealing Rate increases from 22.9% (GPT-4) to 39.1% (Qwen3-Coder) for all four models, reflecting an improvement in model ability to reveal bugs; however, the ability of the strongest model (Qwen3-Coder) to validate bugs decreases from 41.1% to 26.2%, suggesting that the test-oracle design problem is significantly reduced but not eliminated by model strength.

References

[1] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, "Software Testing With Large Language Models: Survey, Landscape, and Vision," IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, 2024. https://arxiv.org/abs/2307.07221 DOI: https://doi.org/10.1109/TSE.2024.3368208

[2] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, "An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation," IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024. DOI: https://doi.org/10.1109/TSE.2023.3334955

DOI: 10.1109/TSE.2023.3332574

[3] Z. Yuan et al., "Evaluating and Improving ChatGPT for Unit Test Generation," Proceedings of the ACM on Software Engineering, vol. 1, FSE, Art. 76, pp. 1703–1726, 2024. https://dl.acm.org/doi/10.1145/3660783 DOI: https://doi.org/10.1145/3660783

[4] N. Alshahwan et al., "Automated Unit Test Improvement using Large Language Models at Meta," Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion), 2024. https://doi.org/10.1145/3663529.3663839 DOI: https://doi.org/10.1145/3663529.3663839

[5] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, "CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models," Proceedings of ICSE, 2023, pp. 919–931. DOI: 10.1109/ICSE48619.2023.00085 DOI: https://doi.org/10.1109/ICSE48619.2023.00085

[6] N. S. Mathews and M. Nagappan, "Design Choices Made by LLM-Based Test Generators Prevent Them From Finding Bugs," arXiv:2412.14137, 2024. https://doi.org/10.48550/arXiv.2412.14137

[7] F. Molina, A. Gorla, and M. d'Amorim, "Test Oracle Automation in the Era of LLMs," ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, Art. 150, 2025. https://doi.org/10.1145/3715107 DOI: https://doi.org/10.1145/3715107

[8] C. Augusto et al., "Large Language Models for Software Testing: A Research Roadmap," arXiv:2509.25043, 2025. https://doi.org/10.48550/arXiv.2509.25043

[9] I. K. Schieferdecker, "Navigating the Growing Field of Research on AI for Software Testing," arXiv:2506.14640, 2025.

https://doi.org/10.48550/arXiv.2506.14640

[10] A. Celik and Q. H. Mahmoud, "A Review of Large Language Models for Automated Test Case Generation," Machine Learning and Knowledge Extraction, vol. 7, no. 3, Art. 97, 2025.DOI: 10.3390/make7030097 DOI: https://doi.org/10.3390/make7030097

[11] M. D. Santana, C. Magalhaes, and R. de Souza Santos, "Software Testing with Large Language Models: An Interview Study with Practitioners," arXiv:2510.17164, 2025. DOI: https://doi.org/10.1109/AIware69974.2025.00018

[12] W. C. Ouédraogo, A. K. Kaboré, Y. Li, H. Tian, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyandé, "Prompt Engineering in LLMs for Automated Unit Test Generation: A Large-Scale Study," Empir. Softw. Eng., vol. 31, no. 4, Art. 103, 2026. DOI: https://doi.org/10.1007/s10664-026-10840-4

[13] G. Wang, Q. Xu, L. C. Briand, Z. Guo, and K. Liu, "Call-Chain-Aware LLM-Based Test Generation for Java Projects," arXiv:2604.22046, 2026.

[14] S. Gu, N. Nashid, and A. Mesbah, "LLM Test Generation via Iterative Hybrid Program Analysis," arXiv:2503.13580, 2025.

[15] S. Haroon, M. T. Khan, and M. A. Gulzar, "Evaluating LLM-Based Test Generation Under Software Evolution," arXiv:2603.23443, 2026.

[16] E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, "TOGA: A Neural Method for Test Oracle Generation," in Proc. ICSE, 2022, pp. 2130–2141. DOI: https://doi.org/10.1145/3510003.3510141

[17] M. Jalili Torkamani, A. Sharma, N. Mehrotra, and R. Purandare, "ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code," arXiv:2411.16927, 2024.

[18] A. Bodicoat, G. Jahangirova, and V. Terragni, "Understanding LLM-Driven Test Oracle Generation," arXiv:2601.05542, 2026. DOI: https://doi.org/10.1109/AIware69974.2025.00011

[19] J. C. Alonso, S. Segura, and A. Ruiz-Cortés, "AGORA: Automated Generation of Test Oracles for REST APIs," in Proc. 32nd ACM SIGSOFT Int. Symp. Softw. Testing and Analysis (ISSTA), 2023, pp. 1018–1030. DOI: https://doi.org/10.1145/3597926.3598114

[20] Y. Hu, U. Z. Ahmed, S. Mechtaev, B. Leong, and A. Roychoudhury, "Re-factoring based Program Repair applied to Programming Assignments," in Proc. 34th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE), 2019, pp. 388–398. DOI: https://doi.org/10.1109/ASE.2019.00044

[21] N. S. Mathews and M. Nagappan, "Is Your Automated Software Engineer Trustworthy?," arXiv:2506.17812, 2025.

[22] L. Khatib, N. S. Mathews, and M. Nagappan, "AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests," arXiv:2507.17542, 2025.

[23] H. Taherkhani, A. DaghighFarsoodeh, M. Chowdhury, H. V. Pham, and H. Hemmati, "Consistency Meets Verification: Enhancing Test Generation Quality in LLMs Without Ground-Truth Solutions," arXiv:2602.10522, 2026. DOI: https://doi.org/10.1109/ICST69053.2026.00039

[24] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, "Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models," in Proc. ISSTA, 2023, pp. 423–435.

[25] Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, "Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT," arXiv:2304.02014, 2023. DOI: https://doi.org/10.1145/3597926.3598067

[26] R. Cheng, M. Tufano, J. Cambronero, R. Wei, S. Shi, G. Uy, P. Rondon, and F. Ivančić, "Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair," arXiv:2601.19066, 2026. DOI: https://doi.org/10.1145/3803437.3805237

[27] P. Straubinger, M. Kreis, S. Lukasczyk, and G. Fraser, "Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging," arXiv:2503.08182, 2025. DOI: https://doi.org/10.1109/ICSTW64639.2025.10962485

[28] S. Li, L. Bu, G. Bai, F. Xie, K. Chen, and C. Yue, "Model-Enhanced LLM-Driven VUI Testing of VPA Apps," arXiv:2407.02791, 2024.