Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Automated Test Generation using Model-Checking: An Industrial Evaluation
Mälardalens högskola, Akademin för innovation, design och teknik, Inbyggda system.ORCID-id: 0000-0003-2416-4205
Mälardalens högskola, Akademin för innovation, design och teknik, Inbyggda system.ORCID-id: 0000-0001-8009-9052
Software Engineering Research Consultant, Västerås, Sweden.
Mälardalens högskola, Akademin för innovation, design och teknik, Inbyggda system.ORCID-id: 0000-0002-1660-199X
Visa övriga samt affilieringar
2016 (Engelska)Ingår i: International Journal on Software Tools for Technology Transfer (STTT), ISSN 1433-2779, E-ISSN 1433-2787, Vol. 18, nr 3, s. 335-353Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.

Ort, förlag, år, upplaga, sidor
Germany: Springer , 2016. Vol. 18, nr 3, s. 335-353
Nyckelord [en]
test generation, model checking, automated software testing, function block diagram, safety-critical software, industrial study
Nationell ämneskategori
Elektroteknik och elektronik
Identifikatorer
URN: urn:nbn:se:mdh:diva-28100DOI: 10.1007/s10009-014-0355-9ISI: 000374974200007Scopus ID: 2-s2.0-84911400022ISBN: 1433-2779 (tryckt)OAI: oai:DiVA.org:mdh-28100DiVA, id: diva2:818214
Projekt
ATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova)ITS-EASY Post Graduate School for Embedded Software and SystemsTillgänglig från: 2015-06-08 Skapad: 2015-06-08 Senast uppdaterad: 2018-10-31Bibliografiskt granskad
Ingår i avhandling
1. Automatic test generation for industrial control software
Öppna denna publikation i ny flik eller fönster >>Automatic test generation for industrial control software
2016 (Engelska)Doktorsavhandling, sammanläggning (Övrigt vetenskapligt)
Abstract [en]

Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.

In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. 

The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.

However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.

In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.

Ort, förlag, år, upplaga, sidor
Västerås: Mälardalen University, 2016
Serie
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 214
Nyckelord
automatic test generation, software testing, automated test generation
Nationell ämneskategori
Inbäddad systemteknik
Forskningsämne
datavetenskap
Identifikatorer
urn:nbn:se:mdh:diva-33364 (URN)978-91-7485-291-2 (ISBN)
Disputation
2016-11-22, Pi, Mälardalens högskola, Västerås, 13:15 (Engelska)
Opponent
Handledare
Tillgänglig från: 2016-10-07 Skapad: 2016-10-07 Senast uppdaterad: 2016-11-02Bibliografiskt granskad

Open Access i DiVA

fulltext(764 kB)86 nedladdningar
Filinformation
Filnamn FULLTEXT01.pdfFilstorlek 764 kBChecksumma SHA-512
46bf120976920b6751c50ee3d4718ddbf63bbeac12604eec4b744be0a81be70c32c215c8344613fa91e5ab263b396304ed0c81464696cd71e9cf4901f587f87b
Typ fulltextMimetyp application/pdf

Övriga länkar

Förlagets fulltextScopus

Sök vidare i DiVA

Av författaren/redaktören
Enoiu, Eduard PaulCausevic, AdnanWeyuker, ElaineSundmark, DanielPettersson, Paul
Av organisationen
Inbyggda system
I samma tidskrift
International Journal on Software Tools for Technology Transfer (STTT)
Elektroteknik och elektronik

Sök vidare utanför DiVA

GoogleGoogle Scholar
Totalt: 86 nedladdningar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

doi
isbn
urn-nbn

Altmetricpoäng

doi
isbn
urn-nbn
Totalt: 269 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf