Analyzing Root Causes and Smells of Test Flakiness by Simulating Resource Usage: A study about how system resource limitations can induce flaky behavior
2025 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesisAlternative title
Att Analysera Rotorsaker och Lukter av Testinstabilitet genom att Simulera Resursanvändning : En studie om hur resursbegränsningar i system kan orsaka testinstabilitet (Swedish)
Abstract [en]
Flaky tests, which intermittently pass or fail when no changes have been made to the code, are a significant challenge in regression testing. Such flakiness affects the reliability of test outcomes, forcing developers to debug false alerts. Current research suggests that some test flakiness is associated with computational resource availability. In this thesis, we investigate the impact of computational resource availability, specifically CPU time, on the frequency of test flakiness in an industrial C++ codebase. We conduct experiments by rerunning tests under simulated CPU usage to quantify this impact. Furthermore, we analyze test source code characteristics and known flakiness root causes to provide insight.
The findings reveal that CPU usage significantly influences certain types of flakiness. While concurrency-related flakiness is varyingly affected, flakiness associated with waiting asynchronously is shown to be completely dependent on CPU usage. Additionally, the study suggests that maintainability and thread sleep-related software characteristics could serve as predictors of resource-affected flakiness. These insights highlight the importance of considering computational resources in test design and suggest potential areas for improving test reliability through resource management and code analysis practices.
Place, publisher, year, edition, pages
2025. , p. 67
Keywords [en]
Regression Testing, Flaky Tests, Test Smells, Flaky Test Detection, Test Flakiness Root Causes, Stress Testing, Computational Resources, Resource Availability, Code Metrics
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:liu:diva-212252ISRN: LIU-IDA/LITH-EX-A--25/001--SEOAI: oai:DiVA.org:liu-212252DiVA, id: diva2:1955392
External cooperation
Ericsson
Subject / course
Computer Engineering
Presentation
2025-01-17, Alan Turing, Linköping, 13:15 (English)
Supervisors
Examiners
2025-04-302025-04-302025-04-30Bibliographically approved