Measuring the Impact of AI Coding Assistants (e.g., GitHub Copilot, ChatGPT) on Programming Productivity among BSCS Students

Dua Nadeem; Nosheen Asif; Mariam Tanveer; Kinza Javed

doi:10.71146/kjmr877

Authors

Dua Nadeem BSCS scholar Air University Multan Campus, Islamabad Author
Nosheen Asif BSCS scholar Air University Multan Campus, Islamabad Author
Mariam Tanveer BSCS scholar Air University Multan Campus, Islamabad Author
Kinza Javed BSCS scholar Air University Multan Campus, Islamabad Author

DOI:

https://doi.org/10.71146/kjmr877

Keywords:

AI coding assistants, Programming productivity, BSCS students, Code quality, Debugging efficiency, Dependency risk, Student perception

Abstract

This study examines the impact of AI coding assistants on programming productivity among undergraduate computing students, with a focus on BSCS learners. The rapid adoption of tools such as GitHub Copilot and ChatGPT has transformed how students approach coding tasks, raising questions about whether these tools enhance productivity or alter learning behavior. Existing research largely emphasizes efficiency gains in controlled or professional settings by creating a gap in understanding their real-world academic impact on students. A quantitative, cross-sectional survey design was employed, collecting data from 127 participants through a structured questionnaire. The study measures productivity as a multi-dimensional construct, including task completion time, code quality, debugging efficiency, and perceived problem-solving independence. The findings suggest that AI coding assistants contribute to improved task efficiency and support learning processes, particularly in understanding programming concepts and completing coding tasks more effectively. Students reported benefits in terms of ease of coding and assistance during debugging, while also acknowledging concerns regarding the reliability of AI-generated outputs. Additionally, the use of these tools appears to influence students’ approach to problem-solving and their level of independence. In conclusion, AI coding assistants offer notable benefits in supporting programming tasks and learning experiences, but they also introduce challenges related to over-reliance. Their overall impact depends on how they are used, highlighting the importance of guided and balanced integration within programming education to ensure that productivity gains do not come at the expense of independent skill development.

Downloads

Download data is not yet available.

References

Barke, S., James, M. B., & Polikarpova, N. (2023). Grounded Copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages,

7(OOPSLA1), 1–27.https://doi.org/10.1145/3586030

Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023). Programming is hard — or at least it used to be: Educational opportunities and challenges of AI code generation. Proceedings of the 54th ACM Technical Symposium on Computer Science Education.https://doi.org/10.1145/3545945.3569759

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.https://doi.org/10.1037/0033-2909.112.1.155

Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge.

Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE Publications.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340.

Denny, P., Kumar, V., &Giacaman, N. (2023). Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. Proceedings of the 54th ACM Technical Symposium on Computer Science Education (SIGCSE 2023) (pp. 1136–1142). https://doi.org/10.1145/3545945.3569823

Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE Publications.

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., &Hüllermeier, E. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274.https://doi.org/10.1016/j.lindif.2023.102274

Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., & Zhou, Y. (2022). CodeGen: An open large language model for code with multi-turn program synthesis. arXiv.https://doi.org/10.48550/arXiv.2203.13474

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). McGraw-Hill.

Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2025). Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. Communications of the ACM, 68(2), 96–105.https://doi.org/10.1145/3610721

Peng, S., Kalliamvakou, E., Cihon, P., &Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv.https://doi.org/10.48550/arXiv.2302.06590

Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., & Finnie-Ansley, J. (2023). “It’s Weird That It Knows What I Want”: Usability and interactions of novice programmers with GitHub Copilot. ACM Transactions on Computer-Human

Interaction, 31(1), 1–31.https://doi.org/10.1145/3617367

Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2023). Lost at C: A user study on the security implications of large language model code assistants. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security 2023) (pp. 2205–2222). USENIX

Association.https://www.usenix.org/system/files/usenixsecurity23-sandoval.pdf

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.

Zhang, B., Liang, P., Zhou, X., Ahmad, A., & Waseem, M. (2023). Practices and challenges of using GitHub Copilot: An empirical study. arXiv.https://doi.org/10.48550/arXiv.2303.08733