Authors: Sayan Chatterjee, Ching Louis Liu, Gareth Rowland, Tim Hogarth

Published on: February 08, 2024

Impact Score: 8.3

Arxiv code: Arxiv:2402.05636

Summary

• What is new: This paper presents one of the earliest large-scale studies on the effectiveness of GitHub Copilot in real-world software engineering tasks within a large organization, ANZ Bank.
• Why this is important: The integration of AI tools like Large Language Models into software engineering practices and their real-world effectiveness is not well-documented, especially in large organizations.
• What the research proposes: Conducted a controlled experiment with about 1000 engineers at ANZ Bank using GitHub Copilot to evaluate its impact on productivity, code quality, and security.
• Results: Notable boost in productivity and code quality was observed, with positive responses from participants, though the impact on code security was inconclusive.

Technical Details

Technological frameworks used: GitHub Copilot

Models used: Large Language Models (LLMs)

Data used: Participant sentiment, productivity metrics, code quality and security evaluations

Potential Impact

Software Engineering, AI tool development companies, banking and financial services sectors could be significantly impacted or benefit from these insights.

Want to implement this idea in a business?

We have generated a startup concept here: CodeFlow.