The growing complexity, the ever-ending demands for new features, and the need to become faster to remain competitive force software development organizations to rethink their development and value delivery practices. While continuous delivery has become more popular, it still relies mainly on internal metrics, ad-hoc data, and expert opinions. As a result, software organizations stumble to find the balance between improving internal system quality and delivering external value. In fact, understanding and measuring customer value is on itself essential. In this PhD project, we aim for a better understanding of customer value and develop measurement instruments to be integrated with internal perspectives to drive proactive and continuous internal improvement while delivering relevant customer value. 

Context: The evidence for the prevalence of test smells at the unit testing level has relied on the accuracy of detection tools, which have seen intense research in the last two decades. The Eager Test smell, one of the most prevalent, is often identified using simplified detection rules that practitioners find inadequate.

Objective: We aim to improve the rules for detecting the Eager Test smell.

Method: We reviewed the literature on test smells to analyze the definitions and detection rules of the Eager Test smell. We proposed a novel, unambiguous definition of the test smell and a heuristic to address the limitations of the existing rules. We evaluated our heuristic against existing detection rules by manually applying it to 300 unit test cases in Java.

Results: Our review identified 56 relevant studies. We found that inadequate interpretations of original definitions of the Eager Test smell led to imprecise detection rules, resulting in a high level of disagreement in detection outcomes. Also, our heuristic detected patterns of eager and non-eager tests that existing rules missed.

Conclusion: Our heuristic captures the essence of the Eager Test smell more precisely; hence, it may address practitioners’ concerns regarding the adequacy of existing detection rules.

It is commonly accepted that the quality of requirements specifications impacts subsequent software engineering activities. However, we still lack empirical evidence to support organizations in deciding whether their requirements are good enough or impede subsequent activities. We aim to contribute empirical evidence to the effect that requirements quality defects have on a software engineering activity that depends on this requirement. We conduct a controlled experiment in which 25 participants from industry and university generate domain models from four natural language requirements containing different quality defects. We evaluate the resulting models using both frequentist and Bayesian data analysis. Contrary to our expectations, our results show that the use of passive voice only has a minor impact on the resulting domain models. The use of ambiguous pronouns, however, shows a strong effect on various properties of the resulting domain models. Most notably, ambiguous pronouns lead to incorrect associations in domain models. Despite being equally advised against by literature and frequentist methods, the Bayesian data analysis shows that the two investigated quality defects have vastly different impacts on software engineering activities and, hence, deserve different levels of attention. Our employed method can be further utilized by researchers to improve reliable, detailed empirical evidence on requirements quality. © The Author(s) 2024.

Software Bills of Material (SBOMs) are becoming a consolidated-and often enforced by governmental regulations-way to describe software composition. However, based on recent studies, SBOMs suffer from limited support for their consumption and lack information beyond simple dependencies, especially regarding software vulnerabilities. This paper reports the results of a preliminary study in which we augmented SBOMs of 40 open-source projects with information about Common Vulnerabilities and Exposures (CVE) exposed by project dependencies. Our augmented SBOMs have been evaluated by submitting pull requests and by asking project owners to answer a survey. Although, in most cases, augmented SBOMs were not directly accepted because owners required a continuous SBOM update, the received feedback shows the usefulness of the suggested SBOM augmentation.

Background: Modern software systems are often too large and complex for an individual developer to fully oversee, making it difficult to understand the implications of changes. Therefore, most collaborative software projects rely on code review as communication network to foster asynchronous discussions about changes before they are merged. Although prior qualitative studies have revealed that practitioners view code review as a communication network, no formal theory or empirical validation exists. Without formalization and confirmatory evidence, the theory remains uncertain, limiting its credibility, practical relevance, and future development.

Objective: In this thesis, our objective is to (1) formalize the theory of code review as a communication network, (2) empirically evaluate the theory across varied perspectives, contexts, and conditions by quantifying the capability of code review to diffuse information among its participants, (3) demonstrate its practical relevance by applying the theory to the domain of tax compliance in collaborative software engineering, and (4) examine how the role of code review as a communication network for collaborative software engineering may evolve in the future.

Methods: To formalize the theory of code review as a communication network, we developed and validated a simulation model that operationalizes its core propositions about information diffusion among participants. To empirically evaluate the theory, we employed two complementary research approaches. First, we used the simulation model to conduct in silico experiments with closed-source code review systems from Microsoft, Spotify, and Trivago, as well as open-source code review systems from Android, Visual Studio Code, and React, to estimate the upper bound of information diffusion in code review. Second, through an observational study, we quantified the diffusion of information in code review across social, organizational, and architectural boundaries at Spotify. To demonstrate the practical relevance of the theory, we analyzed the code review system of a multinational enterprise as a communication network to reveal the latent collaboration structure among developers across borders, which is taxable. To explore the future of code review as a communication network, we conducted a questionnaire survey with 92 practitioners to gather their expectations and discuss how these anticipated changes may reshape our understanding of code review.

Results: By formalizing the theory of code review as a communication network modelled as a time-varying hypergraph, we were able to empirically demonstrate that traditional time-agnostic models substantially overestimate information diffusion in code review. Throughout our empirical studies, we found substential evidence supporting the theory of code review as a communication network: We confirmed that code review is capable of diffusing information quickly and widely among participants, even at a large scale. We also observed extensive information diffusion across social, organizational, and architectural boundaries at Spotify corroborating our theory. However, we also found that information diffusion patterns in open-source code review systems differ significantly, suggesting that findings from open-source environments may not directly apply to closed-source contexts. Through applying the theory of code review as a communication network in the domain of tax compliance, we were able to uncover the significant and previously unrecognized tax risks associated with collaborative software engineering within multinational enterprises. While practitioners consider code review also in the future a core practice in collaborative software engineering, we identify a potential risk that generative AI may undermine code review’s role as a human communication network.

Conclusion: Our work on understanding code review as a communication network contributes not only to theory-driven, empirical software engineering research but also lays the groundwork for practical applications, particularly in the context of tax compliance. Future research is needed to explore the evolving role of code review as a communication network.

Background: Modern software systems are large and complex, requiring collaboration among developers with diverse skills to manage this complexity. Code review is an essential collaborative software engineering practice in which changes are discussed before they are integrated into the codebase, enhancing code quality and promoting knowledge sharing. GUI-based testing is a technique that verifies and validates a system’s behavior through its GUI by simulating user interactions. Like production code, it requires collaboration, as tests are created, reviewed, and maintained alongside production code. Code review practices for tests differ from those for production code. Omitting reviews of tests can lower their quality and increase maintenance costs. However, practices for reviewing GUI-based tests are not well understood, as academic literature mainly focuses on production code and low-level tests.

Objective: We aim to advance the understanding and practice of code review of GUI-based tests by (a) identifying code review guidelines; (b) investigating the specific practices, challenges, and information needs; (c) finding empirical evidence supporting the proposed guidelines; and (d) providing an outlook on how code review may evolve in the future.

Methods: First, we conduct a literature review of white and gray literature to identify guidelines for source and test code, and synthesize them for GUI-based tests. Next, we perform qualitative interviews with software testing professionals to identify practices, challenges, and information needs when reviewing GUI-based tests. To find empirical evidence for the proposed guidelines, we mine open-source software repositories. Finally, we conduct a questionnaire survey to gather practitioners’ expectations about the future importance of code reviews.

Results: We synthesized 33 guidelines for GUI-based tests from literature sources. In analyzing code review comments from open-source repositories, we found empirical evidence supporting 25 out of the 33 proposed guidelines. Practitioners acknowledge the importance of code reviews, but lack defined practices for reviewing GUI-based tests. We identified four practices, six challenges, and four information needs related to reviewing GUI-based tests. The survey results indicate that code review will remain an essential practice with an anticipated increase in code review activities, including those for GUI-based tests.

Conclusion: This thesis advances the understanding and practice of code review for GUI-based tests to improve both review effectiveness and the quality of the tests underreview. We present a set of empirically grounded guidelines derived from literature and refined through the analysis of code review comments of open-source repositories. Our research investigates current practices for reviewing GUI-based tests, highlighting the specific challenges and information needs that distinguish these reviews from those for production code. Finally, we highlight the relevance of our research for the future.

Federated learning (FL), a decentralized machine learning framework that allows edge devices (i.e., clients) to train a global model while preserving data/client privacy, has become increasingly popular recently. In FL, a shared global model is built by aggregating the updated parameters in a distributed manner. To incentivize data owners to participate in FL, it is essential for service providers to fairly evaluate the contribution of each data owner to the shared model during the learning process. To the best of our knowledge, most existing solutions are resource-demanding and usually run as an additional evaluation procedure. The latter produces an expensive computational cost for large data owners. In this paper, we present simple and effective FL solutions that show how the clients’ behavior can be evaluated during the training process with respect to reliability, and this is demonstrated for two existing FL models, Cluster Analysis-based Federated Learning (CA-FL) and Group-Personalized FL (GP-FL), respectively. In the former model, CA-FL, the frequency of each client to be selected as a cluster representative and in that way to be involved in the building of the shared model is assessed. This can eventually be considered as a measure of the respective client data reliability. In the latter model, GP-FL, we calculate how many times each client changes a cluster it belongs to during FL training, which can be interpreted as a measure of the client's unstable behavior, i.e., it can be considered as not very reliable. We validate our FL approaches on three LEAF datasets and benchmark their performance to two baseline contribution evaluation approaches. The experimental results demonstrate that by applying the two FL models we are able to get robust evaluations of clients’ behavior during the training process. These evaluations can be used for further studying, comparing, understanding, and eventually predicting clients’ contributions to the shared global model.

Automated testing, particularly for GUI-based systems, remains a costly and labor-intensive process and prone to errors. Despite advancements in automation, manual testing still dominates in industrial practice, resulting in delays, higher costs, and increased error rates. Large Language Models (LLMs) have shown great potential to automate tasks traditionally requiring human intervention, leveraging their cognitive-like abilities for test generation and evaluation. In this study, we present PathFinder, a Multi-Agent LLM (MALLM) framework that incorporates four agents responsible for (a) perception and summarization, (b) decision-making, (c) input handling and extraction, and (d) validation, which work collaboratively to automate exploratory web-based GUI testing. The goal of this study is to assess how different LLMs, applied to different agents, affect the efficacy of automated exploratory GUI testing. We evaluate PathFinder with three models, Mistral-Nemo, Gemma2, and Llama3.1, on four e-commerce websites. Thus, 27 permutations of the LLMs, across three agents (excluding the validation agent), to test the hypothesis that a solution with multiple agents, each using different LLMs, is more efficacious (efficient and effective) than a multi-agent solution where all agents use the same LLM. The results indicate that the choice of LLM constellation (combination of LLMs) significantly impacts efficacy, suggesting that a single LLM across agents may yield the best balance of efficacy (measured by F1-score). Hypothesis to explain this result include, but are not limited to: improved decision-making consistency and reduced task coordination discrepancies. The contributions of this study are an architecture for MALLM-based GUI testing, empirical results on its performance, and novel insights into how LLM selection impacts the efficacy of automated testing. 

Large Language Models (LLMs) offer promising capabilities for information retrieval and processing. However, the LLM deployment for querying proprietary enterprise data poses unique challenges, particularly for companies with strict data security policies. This study shares our experience in setting up a secure LLM environment within a FinTech company and utilizing it for enterprise information retrieval while adhering to data privacy protocols. 

We conducted three workshops and 30 interviews with industrial engineers to gather data and requirements. The interviews further enriched the insights collected from the workshops. We report the steps to deploy an LLM solution in an industrial sandboxed environment and lessons learned from the experience. These lessons contain LLM configuration (e.g., chunk_size and top_k settings), local document ingestion, and evaluating LLM outputs.

Our lessons learned serve as a practical guide for practitioners seeking to use private data with LLMs to achieve better usability, improve user experiences, or explore new business opportunities. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Background: Software companies must balance fast delivery and quality, a trade-off that often introduces technical debt and wastes developer's time. Technical debt tends to increase as software evolves, which is assumed to slow down development and maintenance activities. However, the potential relationship between technical debt and lead time lacks empirical evidence.

Objective: This paper reports an empirical study to explore the potential relationship between technical debt and lead time in resolving Jira tickets. We further aim to measure the extent to which technical debt can explain the variation in lead time.

Method: We conducted an industrial case study to explore this relationship in six components, each of which was analyzed individually. Technical debt was measured using SonarQube and normalized with the component's size. Lead times to resolve Jira tickets were collected from Jira and averaged monthly.

Results: The study found little to no correlation between technical debt and lead time to resolve Jira tickets in five components, with technical debt explaining a variation in lead time ranging from 0% to 41%. However, it is less than 30% in most of the components.

Conclusion: Technical debt alone does not fully explain the variation in lead time. There should be some other confounding variables (e.g., size and complexity of the changes, number of teams involved, priorities, component ownership) affecting lead time or a residual effect, i.e., interest, that might manifest later. Further investigation into those confounding variables is essential.