Test automation primarily aims to reduce manual testing effort, improve test coverage, and achieve an efficient testing process. For an automated test, typically, the outcome is a set of logs that will include all of the information from the test execution. When the automated test fails, these logs are useful to identify the root cause. However, manually analyzing logs for each failed test will significantly increase the feedback loop, which can practically nullify all the importance of automation. The focus of the thesis is on quick identification of the root cause in order to maintain an efficient flow of delivery of functionality.
This study presents a data-oriented, automated log analysis and root cause identification pipeline, integrated with text similarity search algorithms which we can refer to as a recommendation system. The pipeline utilizes a knowledge base created with historical data which helps in the recommendation of the root cause along with the corrective action. The main steps of the pipeline are: data pre-processing, failure message clustering, text similarity searching against the knowledge base and as a result visualizing the similarity percentage with the root cause and the corrective action.
Consequently, this study employs five different text similarity algorithms; Cosine, LCS, Jaccard, Jaro-Winkler and Matching Coefficient to execute the implemented pipeline. Combining the text similarity algorithm within the context of automated log analysis is a novel aspect. As a contribution, an experiment is performed to confirm the accuracy of these selected text similarity search algorithms in providing relevant recommendations. The conclusion of this thesis is that the proposed data-oriented pipeline provides the most accurate results with LCS, Cosine and Jaro-Winkler text similarity algorithms.