Thesis Defence: Understanding the Perceptions and Challenges of Large Language Models through Social and Technical Lenses

Manaal Basha, supervised by Dr. Gema Rodriguez-Perez, will defend their thesis titled “Understanding the Perceptions and Challenges of Large Language Models through Social and Technical Lenses” in partial fulfillment of the requirements for the degree of Master of Science in Computer Science.

An abstract for Manaal Basha’s thesis is included below.

Defences are open to all members of the campus community as well as the general public. Registration is not required for in-person defences.

---

ABSTRACT

AI-based code generation tools have been revolutionizing the software development process and industry over the last couple of years. With many uncertain of what the industry will look like in the near future due to this automation, understanding user perceptions and experiences of Large Language Models (LLMs) and Code Generation Tools (CGTs) becomes imperative. In this thesis, we comprehensively investigate user sentiments and experiences with code generation tools from Twitter, Stack Overflow, and surveys. Our goals encompass identifying key themes and patterns of discussion on Twitter and Stack Overflow, determining common user questions and challenges, and providing recommendations for improving CGTs. Additionally, we explore the impact of CGTs on developers’ programming abilities and their subjective experiences, including reasons for continued use through surveys.

Leveraging NLP techniques such as BERTopic and RoBERTa for sentiment and topic modelling, we conduct thematic analysis through a multiphased approach with a specific taxonomy developed using Stack Overflow. We use Twitter to understand general user opinions, Stack Overflow to extract developer experiences, and surveys to gain insights into intermediate programmers’ perceptions. Through this comprehensive methodology, we aim to offer valuable insights into user perceptions and experiences with CGTs and inform future developments in this field. Our findings reveal overall positivity with geographic variations in sentiment and intermediate users intending to continue their use of CGTs. Primary sources of negativity revolve around difficulty interpreting tool output, prompting, integration issues, and concerns regarding legal and ethical implications. We propose actionable recommendations aimed at improving the usability of these tools and effectively mitigating potential risks associated with their usage. This thesis contributes to the field by offering a detailed analysis of user perceptions, identifying key challenges, and providing actionable insights for the future development of CGTs.