A recent experiment conducted by Accenture, a global professional services company, aimed to measure the impact of GitHub Copilot on software developers’ productivity. The study, which was later abandoned due to layoffs, included 204 developers from Southeast Asia offices. Despite the challenges, researchers were able to analyze the data and found that Copilot’s effect on productivity was negative, although not statistically significant. The experiment’s results showed a 39.18% decrease in the number of tasks completed by developers using Copilot.
However, the estimates had a high degree of statistical uncertainty. Microsoft, which owns GitHub, did not log all Copilot usage data for this experiment, leading to potential biases in the analysis. Accenture laid off 19,000 employees, including 42% of the developers participating in the experiment, but the attrition was balanced across treatment and control groups. The study’s findings are presented in an appendix, highlighting the challenges of conducting experiments in real-world settings.
The balance table shows that the treatment and control groups are well-balanced across various measures, including Pull Requests, Commits, Builds, and Build Success Rate. The mean differences between the two groups are small, and the p-values associated with these differences are all above 0.05, indicating no significant differences between the groups.
The results of the first Accenture experiment show a negative point estimate of -39.18% for the effect of GitHub Copilot adoption on the number of tasks completed (Pull Requests). However, this estimate has a high degree of statistical uncertainty, with a standard error of 36.78%. The estimates for Commits and Builds are positive, but also not statistically significant.
The authors note that there is a data quality concern due to the lack of adoption data for the control group until October 2023. This means that any analysis may be potentially biased, and the treatment effect estimates may be conservative. The authors address this concern by imputing that nobody in the Control group adopts Copilot until October 2023.
The cumulative adoption rates for the first Accenture experiment show a clear difference between the treatment and control groups, with the treatment group adopting Copilot at a faster rate. However, the authors note that this data may be biased due to the lack of adoption data for the control group until October 2023.
In conclusion, while the results of the first Accenture experiment suggest a negative effect of GitHub Copilot adoption on productivity, the estimates are not statistically significant and have a high degree of uncertainty. Additionally, there are concerns about the quality of the data due to the lack of adoption data for the control group until October 2023. Therefore, these results should be interpreted with caution.
External Link: Click Here For More
