CHATGPT VS GITHUB COPILOT – A REQUIREMENT-BASED TEST CASES GENERATION CAPABILITIES EVALUATION

Codrina-Victoria LISARU, Claudiu V. KIFOR

Abstract


With the rapid development of AI-based tools, such as ChatGPT and GitHub Copilot, the desire to explore them and evaluate their usability is also developing, to use their capabilities to make human work more efficient. The purpose of this paper is to assess whether ChatGPT and GitHub Copilot can generate precise, correct, and enough test cases based on requirements, to help and encourage the software testing community to use these AI-based tools. This paper compares the answers of both AI-based engines and offers a brief overview of the test cases generation capabilities.

Full Text:

PDF

References


Agnia S., Yaroslav G., Timofey B., Iftekhar A., Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward, Information and Software Technology 178, ISSN 0950-5849, DOI DOI https://doi.org/10.1016/j.infsof.2024.107610, 2025

Hussam H., Ahmad H., Mohammad L., The Impact of Artificial intelligence on Software Testing, 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), IEEE, Amman, Jordan, April 2019, pp. 565 – 570, DOI: 1109/JEEIT.2019.8717439.

Lucas L., Ron V., Generative Artificial Intelligence and the Future of Software Testing, Computer, Volume 57, Issue 1, IEEE, pp. 27-32, January 2024, DOI https://doi.org/10.1109/MC.2023.3306998 2024

Rui L., Antonio Miguel Rosado da C., Jorge R., Artificial intelligence Applied to Software Testing: A Literature Review, 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), Seville, Spain, June 2020, pp. 1-6, ISBN: 978-989-54659-0-3,DOI:10.23919/CISTI49556.2020.91411242020.

Minimol Anil J., Automating and Optimizing Software Testing Using Artificial Intelligence Techniques, (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12 (5), , pp. 594 – 602, 2021

Kohani K. M., Nurezayana Z., Mohd Zanes S., Test Case Prioritization Using Swarm Intelligence Algorithm to Improve Fault Detection and Time for Web Application, Journal of Soft Computing and Data Mining, 4(2), pp. 59-66, October 2023

Shahbaa I. K., Raghda A., A review paper: optimal test cases for regression testing using artificial intelligent techniques, International Journal of Electrical and Computer Engineering (IJECE), Vol. 13, No. 2, April, pp. 1803 – 1816, ISSN: 2088 - 8708, 2023

Davide T., Studying the Quality of Source Code Generated by Different AI Generative Engines: An Empirical Evaluation, Future Internet 16, no 6: 188, DOI https://doi.org/10.3390/fi16060188 2024

Tanha M., Hong Z., User Centric Evaluation of Code Generation Tools, 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), IEEE, Shanghai, China, July 2024, pp 109-119, 2024

Sajed J., Suzzana R., Thomas D. L., Kevin M., Wing L., ChatGPT and Software Testing Education: Promises & Perils, 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, Dublin, Ireland, April 2023, pp. 4130-4137, 2023

Yihao L., Pan L., Haiyang W., Jie C., W. Eric W., Evaluating large language models for software testing, Computer Standards & Interfaces, Volume 93, Elsevier, April 2025, ISSN 0920-5489, 2025

Andrew M., Christiane B., Coding with AI as an assistant: Can AI generate concise computer code?, Journal of Information Technology Education: Innovations in Practice, Volume 23, 2024

Miroslaw S., Silvia A., Gregory G., Alexander S., Testing, Debugging, and Log Analysis with modern AI tools, IEEE Software, Volume 41(2), March - April 2024, pp. 99-102, 2024

Shreya B., Tarushi G., Dhruv K., Pankaj J., Unit Test Generation using Generative AI: A Comparative Performance Analysis of Autogeneration Tools, in Proceedings of the 1st International Workshop on Large Language Models for Code, ACM, Lisbon, Portugal, April 2024, pp.54-61, 2024

Jacob H., Demian F., Generating Software Tests for Mobile Applications Using Fine-Tuned Large Language Models, 2024 IEEE/ACM International Conference on Automation of Software Test (AST), IEEE, Lisbon, Portugal, April 2024, pp 76-77, ISBN 979-8-4007-0588-5/24/04, 2024

Maurizio L., Filippo R., Hafiz Zeeshan Y., Boni G., AI-Generated Test Scripts for Web E2E Testing with ChatGPT and Copilot: A Preliminary Study, 28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024), ACM, Salerno, Italy, June 2024, pp. 339-344, ISBN 979-8-4007-1701-7/24/06, 2024

OpenAI, ChatGPT: https://chatgpt.com/ accessed in 2025

GitHub, GitHub Copilot: https://github.com/copilot accessed in 2025

Werner H., P-99: Ninety-Nine Prolog Problems, https://www.ic.unicamp.br/~meidanis/courses/mc336/2009s2/prolog/problemas/ accessed in 2025


Refbacks

  • There are currently no refbacks.


JOURNAL INDEXED IN :