Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Decision Support System for Integration Test Selection
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-8724-9049
2016 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software testing generally suffers from time and budget limitations. Indiscriminately executing all available test cases leads to sub-optimal exploitation of testing resources. Selecting too few test cases for execution on the other hand might leave a large number of faults undiscovered. Test case selection and prioritization techniques can lead to more efficient usage of testing resources and also early detection of faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Test case prioritization schedules test cases for execution in an order to increase their effectiveness at achieving some performance goals such as: earlier fault detection, optimal allocation of testing resources and reducing overall testing effort. In practice, prioritized selection of test cases requires the evaluation of different test case criteria, and therefore, this problem can be formulated as a multi-criteria decision making problem. As the number of decision criteria grows, application of a systematic decision making solution becomes a necessity. In this thesis, we propose a tool-supported framework using a decision support system, for prioritizing and selecting integration test cases in embedded system development. The framework provides a complete loop for selecting the best candidate test case for execution based on a finite set of criteria. The results of multiple case studies, done on a train control management subsystem from Bombardier Transportation AB in Sweden, demonstrate how our approach helps to select test cases in a systematic way. This can lead to early detection of faults while respecting various criteria. Also, we have evaluated a customized return on investment metric to quantify the economic benefits in optimizing system integration testing using our framework.

Place, publisher, year, edition, pages
Västerås: Mälardalen University , 2016.
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 242
National Category
Computer Systems
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-33118ISBN: 978-91-7485-282-0 (print)OAI: oai:DiVA.org:mdh-33118DiVA: diva2:967586
Presentation
2016-10-25, Omega, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2016-09-12 Created: 2016-09-09 Last updated: 2016-10-04Bibliographically approved
List of papers
1. Multi-Criteria Test Case Prioritization Using Fuzzy Analytic Hierarchy Process
Open this publication in new window or tab >>Multi-Criteria Test Case Prioritization Using Fuzzy Analytic Hierarchy Process
2015 (English)In: ICSEA 2015: The Tenth International Conference on Software Engineering Advances, 2015, 290-296 p.Conference paper, Published paper (Refereed)
Abstract [en]

One of the key challenges in software testing today is prioritizing and evaluating test cases. The decision of which test cases to design, select and execute first is of great importance, in particular considering that testing is often done late in the implementation process, and therefore needs to be done within tight resource constraints on time and budget. In practice, prioritized selection of test cases requires the evaluation of different test case criteria, and therefore, test case prioritization can be formulated as a multi-criteria decision making problem. As the number of decision criteria grows, application of a systematic decision making solution becomes a necessity. In this paper we propose an approach for prioritized selection of test cases by using the Analytic Hierarchy Process (AHP) technique. To improve the practicality of the approach in real world scenarios, we apply AHP in fuzzy environment so that criteria values can be specified using fuzzy variables than requiring precise quantified values. One of the advantages of the decision making process it that the defined criteria with the biggest and most critical role in priority ranking of test cases is identified. We have applied our approach on an example case in which several test cases for testing non-functional requirements in a systems are defined.

Keyword
Software testing, Test case prioritization, MCDM, Fuzzy logic, NFR, AHP, Fault detection
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-29634 (URN)978-1-61208-438-1 (ISBN)
Conference
The Tenth International Conference on Software Engineering Advances ICSEA 2015, 15-20 Nov 2015, Barcelona, Spain
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)IMPRINT - Innovative Model-Based Product Integration Testing (Vinnova)
Available from: 2015-12-10 Created: 2015-11-26 Last updated: 2016-12-27Bibliographically approved
2. Towards Earlier Fault Detection by Value-Driven Prioritization of Test Cases Using Fuzzy TOPSIS
Open this publication in new window or tab >>Towards Earlier Fault Detection by Value-Driven Prioritization of Test Cases Using Fuzzy TOPSIS
Show others...
2016 (English)In: Information Technology: New Generations, vol. 440, Las Vegas, United States, 2016, 745-759 p.Conference paper, Published paper (Refereed)
Abstract [en]

In industrial software testing, development projects typically set up and maintain test suites containing large numbers of test cases. Executing a large number of test cases can be expensive in terms of effort and wall-clock time. Moreover, indiscriminate execution of all available test cases typically lead to sub-optimal use of testing resources. On the other hand, selecting too few test cases for execution might leave a large number of faults undiscovered. Limiting factors such as allocated budget and time constraints for testing further emphasizes the importance of test case prioritization in order to identify test cases that enable earlier detection of faults while respecting such constraints. In this paper, we propose a multi-criteria decision making approach for prioritizing test cases in order to detect faults earlier. This is achieved by applying the TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) decision making technique combined with fuzzy principles. Our solution is based on important criteria such as fault detection probability, execution time, complexity, and other test case properties. By applying the approach on a train control management subsystem from Bombardier Transportation in Sweden, we demonstrate how it helps, in a systematic way, to identify test cases that can lead to early detection of faults while respecting various criteria.

Place, publisher, year, edition, pages
Las Vegas, United States: , 2016
Keyword
Software Testing, Fault Detection, Test Cases Prioritization, Optimization, Fuzzy Logic, MCDM, TOPSIS, Failure Rate
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-30517 (URN)10.1007/978-3-319-32467-8_65 (DOI)000385289400065 ()2-s2.0-84962655573 (Scopus ID)978-3-319-32466-1 (ISBN)
Conference
13th International Conference on Information Technology : New Generations (ITNG 2016) ITNG'16, 11-13 Apr 2016, Las Vegas, United States
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)IMPRINT - Innovative Model-Based Product Integration Testing (Vinnova)
Available from: 2015-12-22 Created: 2015-12-21 Last updated: 2016-12-27Bibliographically approved
3. Dynamic Integration Test Selection Based on Test Case Dependencies
Open this publication in new window or tab >>Dynamic Integration Test Selection Based on Test Case Dependencies
Show others...
2016 (English)In: 2016 IEEE NINTH INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS (ICSTW), Chicago, United States, 2016, 277-286 p.Conference paper, Published paper (Refereed)
Abstract [en]

Prioritization, selection and minimization of test cases are well-known problems in software testing. Test case prioritization deals with the problem of ordering an existing set of test cases, typically with respect to the estimated likelihood of detecting faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Most existing approaches for test case prioritization and selection suffer from one or several drawbacks. For example, they to a large extent utilize static analysis of code for that purpose, making them unfit for higher levels of testing such as integration testing. Moreover, they do not exploit the possibility of dynamically changing the prioritization or selection of test cases based on the execution results of prior test cases. Such dynamic analysis allows for discarding test cases that do not need to be executed and are thus redundant. This paper proposes a generic method for prioritization and selection of test cases in integration testing that addresses the above issues. We also present the results of an industrial case study where initial evidence suggests the potential usefulness of our approach in testing a safety-critical train control management subsystem.

Place, publisher, year, edition, pages
Chicago, United States: , 2016
Keyword
Software testing, Integration testing, Test selection, Test prioritization, Fuzzy, AHP, Optimization
National Category
Engineering and Technology Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-33116 (URN)10.1109/ICSTW.2016.14 (DOI)000382490200038 ()2-s2.0-84992215253 (Scopus ID)978-1-5090-3674-5 (ISBN)
Conference
9th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)IMPRINT - Innovative Model-Based Product Integration Testing (Vinnova)
Available from: 2016-09-08 Created: 2016-09-08 Last updated: 2017-10-16Bibliographically approved
4. Cost-Benefit Analysis of Using Dependency Knowledge at Integration Testing
Open this publication in new window or tab >>Cost-Benefit Analysis of Using Dependency Knowledge at Integration Testing
Show others...
2016 (English)In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016, Vol. 10027, 268-284 p.Conference paper, Published paper (Refereed)
Abstract [en]

In software system development, testing can take considerable time and resources, and there are numerous examples in the literature of how to improve the testing process. In particular, methods for selection and prioritization of test cases can play a critical role in efficient use of testing resources. This paper focuses on the problem of selection and ordering of integration-level test cases. Integration testing is performed to evaluate the correctness of several units in composition. Further, for reasons of both effectiveness and safety, many embedded systems are still tested manually. To this end, we propose a process, supported by an online decision support system, for ordering and selection of test cases based on the test result of previously executed test cases. To analyze the economic efficiency of such a system, a customized return on investment (ROI) metric tailored for system integration testing is introduced. Using data collected from the development process of a large-scale safety-critical embedded system, we perform Monte Carlo simulations to evaluate the expected ROI of three variants of the proposed new process. The results show that our proposed decision support system is beneficial in terms of ROI at system integration testing and thus qualifies as an important element in improving the integration testing process.

Keyword
Process improvement, Software testing, Decision support system, Integration testing, Test case selection, Prioritization, Optimization, Return on investment
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-32887 (URN)10.1007/978-3-319-49094-6_17 (DOI)2-s2.0-84998880972 (Scopus ID)
Conference
THE 17TH INTERNATIONAL CONFERENCE ON PRODUCT-FOCUSED SOFTWARE PROCESS IMPROVEMENT PROFES'16, 22-24 Nov 2016, TRONDHEIM, Norway
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)IMPRINT - Innovative Model-Based Product Integration Testing (Vinnova)
Available from: 2016-08-29 Created: 2016-08-24 Last updated: 2016-12-15Bibliographically approved

Open Access in DiVA

fulltext(334 kB)209 downloads
File information
File name FULLTEXT02.pdfFile size 334 kBChecksum SHA-512
105d7d100d0d9bc6e4c1d809787031c3522a21360c2d367abd5e8190004b387ae9dca7c8d0f8aaaad75daeb39964eaf98144d1939d0d1ad7591e298ed6c8df3f
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Tahvili, Sahar
By organisation
Embedded Systems
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 209 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 394 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf