Change search
ReferencesLink to record
Permanent link

Direct link
Automatic test generation for industrial control software
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. (Software Testing Laboratory)ORCID iD: 0000-0003-2416-4205
2016 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.

In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. 

The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.

However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.

In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.

Place, publisher, year, edition, pages
Västerås: Mälardalen University , 2016.
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 214
Keyword [en]
automatic test generation, software testing, automated test generation
National Category
Embedded Systems
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-33364ISBN: 978-91-7485-291-2OAI: oai:DiVA.org:mdh-33364DiVA: diva2:1033600
Public defence
2016-11-22, Pi, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2016-10-07 Created: 2016-10-07 Last updated: 2016-11-02Bibliographically approved
List of papers
1. Using Logic Coverage to Improve Testing Function Block Diagrams
Open this publication in new window or tab >>Using Logic Coverage to Improve Testing Function Block Diagrams
2013 (English)In: Testing Software and Systems: Lecture Notes in Computer Science, Volume 8254, Springer Berlin Heidelberg , 2013, 1-16 p.Chapter in book (Refereed)
Abstract [en]

In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams.

Place, publisher, year, edition, pages
Springer Berlin Heidelberg, 2013
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 8254
Keyword
logic coveragemodel-based testingfunction block diagramtimed automataIEC 61131-3test automation
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-22882 (URN)10.1007/978-3-642-41707-8_1 (DOI)2-s2.0-84893417933 (ScopusID)978-3-642-41706-1 (ISBN)
Projects
ATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova)
Note

25th IFIP WG 6.1 International Conference, ICTSS 2013, Istanbul, Turkey, November 13-15, 2013, Proceedings

Available from: 2013-11-20 Created: 2013-11-19 Last updated: 2016-10-07Bibliographically approved
2. Automated Test Generation using Model-Checking: An Industrial Evaluation
Open this publication in new window or tab >>Automated Test Generation using Model-Checking: An Industrial Evaluation
Show others...
2016 (English)In: International Journal on Software Tools for Technology Transfer STTT, Vol. 18, no 3, 335-353 p.Article in journal (Refereed) Published
Abstract [en]

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.

Place, publisher, year, edition, pages
Germany: Springer, 2016
Keyword
test generation, model checking, automated software testing, function block diagram, safety-critical software, industrial study
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-28100 (URN)10.1007/s10009-014-0355-9 (DOI)CCC:000374974200007 ()2-s2.0-84911400022 (ScopusID)1433-2779 (ISBN)
Projects
ATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova)ITS-EASY Post Graduate School for Embedded Software and Systems
Available from: 2015-06-08 Created: 2015-06-08 Last updated: 2016-10-07Bibliographically approved
3. A Controlled Experiment in Testing of Safety-Critical Embedded Software
Open this publication in new window or tab >>A Controlled Experiment in Testing of Safety-Critical Embedded Software
2016 (English)In: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, 2016, 1-11 p.Conference paper (Refereed)
Abstract [en]

In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification-and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.

Keyword
automated test generation, controlled experiment, embedded software, manual testing, safety-critical systems, specification-based testing, Automatic test pattern generation, Automation, Embedded systems, Fault detection, Safety engineering, Safety testing, Security systems, Software engineering, Specifications, Standards, Verification, Automated test generations, Safety critical systems, Specification Based Testing, Software testing
National Category
Computer and Information Science
Identifiers
urn:nbn:se:mdh:diva-33113 (URN)10.1109/ICST.2016.15 (DOI)2-s2.0-84983326093 (ScopusID)9781509018260 (ISBN)
External cooperation:
Conference
9th IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, 10 April 2016 through 15 April 2016
Available from: 2016-09-08 Created: 2016-09-08 Last updated: 2016-10-07Bibliographically approved
4. A Comparative Study of Manual and Automated Testing in Industrial Embedded Software
Open this publication in new window or tab >>A Comparative Study of Manual and Automated Testing in Industrial Embedded Software
(English)Manuscript (preprint) (Other (popular science, discussion, etc.))
Abstract [en]

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-33389 (URN)
Conference
The International Conference on Testing Software and Systems ICTSS, 17-19 Oct 2016, Graz, Sweden
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)AGENTS - Automated Generation of Tests for Simulated Software Systems (KKS)
Available from: 2016-10-11 Created: 2016-10-11 Last updated: 2016-10-11Bibliographically approved
5. Mutation-Based Test Generation for PLC Embedded Software using Model Checking
Open this publication in new window or tab >>Mutation-Based Test Generation for PLC Embedded Software using Model Checking
Show others...
2016 (English)In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016, Vol. 9976, 155-171 p.Conference paper (Refereed)
Abstract [en]

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 9976
Keyword
Engineering controlled terms: Accident prevention; Application programs; Automation; Embedded software; Fault detection; Java programming language; Model checking; Programmable logic controllers; Safety engineering; Safety testing; Standards Automated test generations; Decision coverage; Detection ability; Engineering software; Experimental evidence; Mutation analysis; Mutation operators; Uppaal model checkers Engineering main heading: Software testing
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-32886 (URN)10.1007/978-3-319-47443-4_10 (DOI)2-s2.0-84992445107 (ScopusID)978-331947442-7 (ISBN)
Conference
28th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2016; Graz; Austria; 17 October 2016 through 19 October 2016; Code 185379
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)AGENTS - Automated Generation of Tests for Simulated Software Systems (KKS)
Available from: 2016-08-29 Created: 2016-08-24 Last updated: 2016-11-10Bibliographically approved

Open Access in DiVA

fulltext(402 kB)24 downloads
File information
File name FULLTEXT02.pdfFile size 402 kBChecksum SHA-512
637474989db999784acfe0296feae87e419d00bdd331372ca36f8942f389fde44c6c86577787b986c411d96145effc898d485d6c9ecc5db2c5faed8e73387ef1
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Enoiu, Eduard
By organisation
Embedded Systems
Embedded Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 24 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 300 hits
ReferencesLink to record
Permanent link

Direct link