CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Regression Testing Challenges and Solutions: An Industry-Academia Perspective
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering. (SERL)ORCID iD: 0000-0001-8177-4355
2019 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Background: Software quality assurance (QA) is an essential activity in the software development lifecycle. Among the different QA activities, regression testing is a challenging task for large-scale software development. Regression testing is a well-researched area, and a large number of techniques have been proposed to fulfill the needs of industry. Despite the extensive research, the adoption of proposed regression testing techniques in the industry is limited. Studies show that there is a visible gap between research and practice.

Objective: This work aims at reducing the gap between industry and academia in regression testing. To fulfill this aim we have the following objectives:

1) Understanding the practitioners' goals regarding regression testing.

2) Understanding the current state of regression testing practice and challenges in the industry.

3) Investigating the testing research applicable in an industrial context.

Method: We conducted multiple studies using different methods.

To explore the industry perspective on regression testing we used focus group and interview-based studies. To explore solutions from the literature, we used the systematic literature review and systematic mapping study.

Results: This thesis presents the practitioners' specific regression testing goals. The identified goals are confidence, controlled fault slippage, effectiveness, efficiency, and customer satisfaction. The challenges identified in the thesis are of two categories, 1) management related challenges and 2) technical challenges. Technical challenges relate to test suite maintenance, test case selection, test case prioritization, evaluation of regression testing.

We have mapped 26 empirically evaluated regression testing techniques to the context, effect, and information taxonomies, and provided a guide to the practitioners regarding the adoption of the techniques in an industrial setting. We have also classified 56 model-based test case generation techniques regarding their strengths/limitations, input/intermediate models used, and relevance to the industrial context.

Conclusions: The challenges identified in this study are not new for research and practice. There could be two reasons regarding the presence of recurring challenges: 1) regression testing techniques proposed in the literature do not fit the companies’ context, 2) or, companies are not aware of the availability of the techniques that could be suitable for their context. To support the adoption of existing research on regression testing in the industry, we have presented three taxonomies. These taxonomies, allow the characterization of regression testing techniques and enable to determine which of these techniques might be suitable in a given context. Furthermore, the identification of information needs for these techniques would be helpful to learn the implications regarding the cost of adoption. Regarding the support in test case generation, we conclude that current research on interaction model-based test case generation techniques did not illustrate the use of rigorous methodology, and currently, model-based test case generation techniques have low relevance for the industrial problems.

Place, publisher, year, edition, pages
Karlskrona, Sweden: Blekinge Tekniska Högskola, 2019. , p. 146
Series
Blekinge Institute of Technology Licentiate Dissertation Series, ISSN 1650-2140 ; 2
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-17381ISBN: 978-91-7295-365-9 (print)OAI: oai:DiVA.org:bth-17381DiVA, id: diva2:1268462
Presentation
2019-01-08, Blekinge Institute of Technology, Karlskrona, 14:00 (English)
Opponent
Supervisors
Funder
VINNOVA, 2015-03235Available from: 2018-12-06 Created: 2018-12-05 Last updated: 2018-12-06Bibliographically approved
List of papers
1. Regression testing goals: View of practitioners and researchers
Open this publication in new window or tab >>Regression testing goals: View of practitioners and researchers
2017 (English)In: 24th Asia-Pacific Software Engineering Conference Workshops (APSECW), IEEE, 2017, p. 25-32Conference paper, Published paper (Refereed)
Abstract [en]

Context: Regression testing is a well-researched area. However, the majority regression testing techniques proposed by the researchers are not getting the attention of the practitioners. Communication gaps between industry and academia and disparity in the regression testing goals are the main reasons. Close collaboration can help in bridging the communication gaps and resolving the disparities.Objective: The study aims at exploring the views of academics and practitioners about the goals of regression testing. The purpose is to investigate the commonalities and differences in their viewpoints and defining some common goals for the success of regression testing.Method: We conducted a focus group study, with 7 testing experts from industry and academia. 4 testing practitioners from 2companies and 3 researchers from 2 universities participated in the study. We followed GQM approach, to elicit the regression testing goals, information needs, and measures.Results: 43 regression testing goals were identified by the participants, which were reduced to 10 on the basis of similarity among the identified goals. Later during the priority assignment process, 5 goals were discarded, because the priority assigned to these goals was very low. Participants identified 47 information needs/questions required to evaluate the success of regression testing with reference to goal G5 (confidence). Which were then reduced to10 on the basis of similarity. Finally, we identified measures to gauge those information needs/questions, which were corresponding to the goal (G5).Conclusions: We observed that participation level of practitioners and researchers during the elicitation of goals and questions was same. We found a certain level of agreement between the participants regarding the regression testing definitions and goals.But there was some level of disagreement regarding the priorities of the goals. We also identified the need to implement a regression testing evaluation framework in the participating companies.

Place, publisher, year, edition, pages
IEEE, 2017
Keywords
Regression testing, Regression testing goals, GQM, Focus group
National Category
Computer Systems
Identifiers
urn:nbn:se:bth-16009 (URN)10.1109/APSECW.2017.23 (DOI)000428319200008 ()978-1-5386-2649-8 (ISBN)
Conference
24th Asia-Pacific Software Engineering Conference, Nanjing
Projects
EASE (Embedded Applications Software Engineering, ease.cs.lth.se)
Available from: 2018-03-22 Created: 2018-03-22 Last updated: 2018-12-05Bibliographically approved
2. Regression testing for large-scale embedded software development: Exploring the state of practice
Open this publication in new window or tab >>Regression testing for large-scale embedded software development: Exploring the state of practice
2018 (English)Manuscript (preprint) (Other academic)
Abstract [en]

Context: A majority of the regression testing techniques proposed by the research have not been adopted in industry. To increase adoption rates, we need to better understand the practitioners' perspectives on regression testing.

Objective: This study aims at exploring the regression testing state of practice in the large-scale embedded software development. The study has two objectives, 1) to highlight the potential challenges in practice, and 2) to identify the industry-relevant research areas regarding regression testing.

Method: We conducted a qualitative study in two large-scale embedded software development companies, where we carried out semi-structured interviews with representatives from five software testing teams. We did conduct the detailed review of the process documentation of the companies to complement/validate the findings of the interviews.

Results: Mostly, the practitioners run regression testing with a selected scope, the selection of scope depends upon the size, complexity, and location of the change. Test cases are prioritized on the basis of risk and critical functionality. The practitioners rely on their knowledge and experience for the decision making regarding selection and prioritization of test cases.The companies are using both automated and manual regression testing, and mainly they rely on in-house developed tools for test automation. The challenges identified in the companies are: time to test, information management, test suite maintenance, lack of communication, test selection/prioritization, lack of assessment, etc. The proposed improvements are in line with the identified challenges. Regression testing goals identified in this study are customer satisfaction, critical defect detection, confidence, effectiveness, efficiency, and controlled slip through of faults.

Conclusions: Considering the current state of practice and identified challenges we conclude that there is a need to reconsider the regression test strategy in the companies. Researchers need to analyze the industry perspective while proposing new regression testing techniques. The industry-academia collaboration projects would be a good platform in this regard.

Keywords
Regression testing, practices, challenges, goals, multi-case study
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-17362 (URN)
Projects
EASE - Embedded Applications Software Engineering
Funder
VINNOVA, 2015-03235
Available from: 2018-11-30 Created: 2018-11-30 Last updated: 2018-12-05Bibliographically approved
3. On the search for industry-relevant regression testing research
Open this publication in new window or tab >>On the search for industry-relevant regression testing research
Show others...
(English)In: Journal of Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616Article in journal (Refereed) Accepted
Abstract [en]

Regression testing is a means to assure that a change in the software, or

its execution environment, does not introduce new defects. It involves the expensive

undertaking of rerunning test cases. Several techniques have been proposed

to reduce the number of test cases to execute in regression testing, however, there

is no research on how to assess industrial relevance and applicability of such techniques.

We conducted a systematic literature review with the following two goals:

rstly, to enable researchers to design and present regression testing research with

a focus on industrial relevance and applicability and secondly, to facilitate the industrial

adoption of such research by addressing the attributes of concern from the

practitioners' perspective. Using a reference-based search approach, we identied

1068 papers on regression testing. We then reduced the scope to only include papers

with explicit discussions about relevance and applicability (i.e. mainly studies

involving industrial stakeholders). Uniquely in this literature review, practitioners

were consulted at several steps to increase the likelihood of achieving our aim of

identifying factors important for relevance and applicability. We have summarised

the results of these consultations and an analysis of the literature in three taxonomies,

which capture aspects of industrial-relevance regarding the regression

testing techniques. Based on these taxonomies, we mapped 38 papers reporting

the evaluation of 26 regression testing techniques in industrial settings.

National Category
Software Engineering
Identifiers
urn:nbn:se:bth-17364 (URN)
Available from: 2018-11-29 Created: 2018-11-29 Last updated: 2018-12-06Bibliographically approved
4. A Systematic Mapping of Test Case Generation Techniques Using UML Interaction Diagram
Open this publication in new window or tab >>A Systematic Mapping of Test Case Generation Techniques Using UML Interaction Diagram
2018 (English)In: Journal of Software: Evolution and Process, ISSN 2047-7473, E-ISSN 2047-7481Article in journal (Refereed) Submitted
Abstract [en]

Testing plays a vital role for assuring software quality. Among the activities performed during testing process, test cases generation is a challenging and labor intensive task. Test case generation techniques based on UML models are getting the attention of researchers and practitioners. This study provides a systematic mapping of test case generation techniques based on interaction diagrams. The study compares the test case generation techniques, regarding their capabilities and limitations, and it also assesses the reporting quality of the primary studies. It has been revealed that UML interaction diagrams based techniques are mainly used for integration testing. The majority of the techniques are using sequence diagrams as input models, while some are using collaboration. A notable number of techniques are using interaction diagram along with some other UML diagram for test case generation. These techniques are mainly focusing on interaction, scenario, operational, concurrency, synchronization and deadlock related faults.

From the results of this study, we can conclude that the studies presenting test case generation techniques using UML interaction diagrams failed to illustrate the use of rigorous methodology, and these techniques did not demonstrate the empirical evaluation in an industrial context. Our study revealed the need for tool support to facilitate the transfer of solutions to industry.

Keywords
Software testing, Test case generation, Interaction diagrams, Model based testing, Systematic mapping
National Category
Computer Systems
Identifiers
urn:nbn:se:bth-17363 (URN)
Projects
EASE - Embedded Applications Software Engineering
Funder
VINNOVA, 2015-03235
Available from: 2018-11-30 Created: 2018-11-30 Last updated: 2018-12-05Bibliographically approved

Open Access in DiVA

draft_181206(973 kB)19 downloads
File information
File name FULLTEXT01.pdfFile size 973 kBChecksum SHA-512
5dc485e918a0c781b163ec6a0d28295a622da494e0082d71db5ff4711bab0c3df24ac179b2746584bf9fc321f9fe74a2afa2a8803d3678119668a8fe8d7420ed
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Minhas, Nasir Mehmood
By organisation
Department of Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 19 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 134 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf