Change search
ReferencesLink to record
Permanent link

Direct link
An ALARP Stop-Test Decision for the Worst-Case Timing Characteristics of Safety-Critical Systems
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0003-4127-5839
2016 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Safety-critical systems are those in which failure can lead to loss of people’s lives, or catastrophic damage to the environment. Timeliness is an important requirement in safety-critical systems, which relates to the notion of response time, i.e., the time a system takes to respond to stimuli from the environment. If the response time exceeds a specified time interval, a catastrophe might occur.

 

Stringent timing requirements make testing a necessary and important process with which not only the correct system functionality has to be verified but also the system timing behaviour. However, a key issue for testers is to determine when to stop testing, as stopping too early may result in defects remaining in the system, or a catastrophe due to high severity level of undiscovered defects; and stopping too late will result in waste of time and resources. To date, researchers and practitioners have mainly focused on the design and application of diverse testing strategies, leaving the critical stop-test decision a largely open issue, especially with respect to timeliness.

 

In the first part of this thesis, we propose a novel approach to make a stop-test decision in the context of testing the worst-case timing characteristics of systems. More specifically, we propose a convergence algorithm that informs the tester whether further testing would reveal significant new insight into the timing behaviour of the system, and if not, it suggests testing to be stopped. The convergence algorithm looks into the observed response times achieved by testing, and examines whether the Maximum Observed Response Time (MORT) has recently increased, and when this is no longer the case, it investigates if the distribution of response times has changed significantly. When no significant new information about the system is revealed during a given period of time it is concluded, with some statistical confidence, that more testing of the same nature is not going to be useful. However, some other testing techniques may still achieve significant new findings.

 

Furthermore, the convergence algorithm is evaluated based on the As Low As Reasonably Practicable (ALARP)  principle which is an underpinning concept in most safety standards. ALARP involves weighting benefit against the associated cost. In order to evaluate the convergence algorithm, it is shown that the sacrifice, here testing time, would be grossly disproportionate compared to the benefit attained, which in this context is any further significant increase in the MORT after stopping the test.

 

Our algorithm includes a set of tunable parameters. The second part of this work is to improve the algorithm performance and scalability through the following steps: firstly, it is determined whether the parameters do affect the algorithm. Secondly, the most influential parameters are identified and tuned. This process is based on the Design of Experiment (DoE)  approach.

 

Moreover, the algorithm is required to be robust, which in this context is defined “the algorithm provides valid stop-test decisions across a required range of task sets”. For example, if the system’s number of tasks varies from 10 to 50 tasks and the tasks’ periods change from the range [200 μ s, 400 μ s] to the range [200 μ s, 1000 μ s], the algorithm performance would not be adversely affected. In order to achieve robustness, firstly, the most influential task set parameters on the algorithm performance are identified by the Analysis of Variance (ANOVA)  approach. Secondly, it is examined whether the algorithm is sound over some required ranges of those parameters, and if not, the situations in which the algorithm’s performance significantly degrades are identified. Then, these situations will be used in our future work to stress test the algorithm and to tune it so that it becomes robust across the required ranges.

 

Finally, the convergence algorithm was shown to be successful while being applied on task sets having similar characteristics. However, we observe some experiments in which the algorithm could not suggest a proper stop-test decision in compliance to the ALARP principle, e.g., it stops sooner than expected. Therefore, we examine whether the algorithm itself can be further improved focusing on the statistical test it uses and if another test would perform better.

Place, publisher, year, edition, pages
Västerås: Mälardalen University , 2016.
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 238
National Category
Computer and Information Science
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-32588ISBN: 978-91-7485-279-0OAI: oai:DiVA.org:mdh-32588DiVA: diva2:953847
Presentation
2016-09-19, Gamma, Mälardalens högskola, Västerås, 13:00 (English)
Opponent
Supervisors
Available from: 2016-08-19 Created: 2016-08-18 Last updated: 2016-09-01Bibliographically approved
List of papers
1. Using Design of Experiments to Optimise a Decision of Sufficient Testing
Open this publication in new window or tab >>Using Design of Experiments to Optimise a Decision of Sufficient Testing
2015 (English)In: The 41st Euromicro Conference on Software Engineering and Advanced Applications SEAA'15, 2015, 53-60 p.Conference paper (Refereed)
Abstract [en]

Testing of safety-critical embedded systems is an important and costly endeavor. To date researchers and practitioners have been mainly focusing on the design and application of diverse testing strategies, but leaving the test stopping criteria as an ad hoc decision and an open research issue. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of tasks and hence should be stopped. This algorithm was shown to be successful but its trial and error tuning of parameters was an issue. In this paper, we use the Design of Experiment (DOE) approach to optimise the algorithm's performance and to improve its scalability. During our experimental evaluations the optimised algorithm showed improved performance by achieving relatively the same results with 42% less testing cost as compared to our previous work. The algorithm also has better scalability and opens up a new path towards achieving cost effective non-functional testing of real-time embedded systems.

National Category
Embedded Systems
Identifiers
urn:nbn:se:mdh:diva-30473 (URN)10.1109/SEAA.2015.79 (DOI)000380478300008 ()2-s2.0-84958238553 (ScopusID)978-1-4673-7585-6 (ISBN)
External cooperation:
Conference
The 41st Euromicro Conference on Software Engineering and Advanced Applications SEAA'15, 26-28 Aug 2015, Funchal, Madeira, Portugal
Projects
SYNOPSIS - Safety Analysis for Predictable Software Intensive Systems
Available from: 2015-12-21 Created: 2015-12-21 Last updated: 2016-09-01Bibliographically approved
2. Improving the Stop-Test Decision When Testing Data are Slow to Converge
Open this publication in new window or tab >>Improving the Stop-Test Decision When Testing Data are Slow to Converge
2016 (English)Report (Other academic)
Place, publisher, year, edition, pages
Sweden: Mälardalen Real-Time Research Centre, Mälardalen University, 2016
Series
, MRTC Reports, ISSN 1404-3041
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-32583 (URN)MDH-MRTC-310/2016-1-SE (ISRN)
Projects
SYNOPSIS - Safety Analysis for Predictable Software Intensive Systems
Available from: 2016-08-18 Created: 2016-08-18 Last updated: 2016-08-19
3. Influential Nuisance Factors on a Decision of Sufficient Testing
Open this publication in new window or tab >>Influential Nuisance Factors on a Decision of Sufficient Testing
2015 (English)In: Algorithms and Architectures for Parallel Processing: ICA3PP International Workshops and Symposiums, Zhangjiajie, China, November 18–20, 2015, Proceedings, 2015, 819-828 p.Conference paper (Refereed)
Abstract [en]

Testing of safety-critical embedded systems is an important and costly endeavor. To date work has been mainly focusing on the design and application of diverse testing strategies. However, they have left an open research issue of when to stop testing a system. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of system tasks, hence, should be stopped. This algorithm was shown to be successful while being applied across task sets having similar characteristics. For the convergence algorithm to become robust, it is important that it holds even if the task set characteristics here called nuisance factors, vary. Generally speaking, there might be either the main factors under analysis, called design factors, or nuisance factors that influence the performance of a process or system. Nuisance factors are not typically of interest in the context of the analysis. However, they vary from system to system and may have large effects on the performance, hence, being very important to be accounted for. Consequently, the current paper looks into a set of nuisance factors that affect our proposed convergence algorithm performance. More specifically, it is interested in situations when the convergence algorithm performance significantly degrades influencing its reliability. The work systematically analyzes each nuisance factor effect using a well-known statistical method, further, derives the most influential factors.

Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 9352
Keyword
Testing, Safety, ALARP, Nuisance factor, Real-time system, ANOVA, Analysis of variance
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-30474 (URN)10.1007/978-3-319-27161-3_75 (DOI)000373630000075 ()2-s2.0-84951948384 (ScopusID)978-3-319-27160-6 (ISBN)
Conference
The 15th International Conference on Algorithms and Architectures for Parallel Processing ICA3PP'15, 18-20 Nov 2015, Zhangjiajie, China
Projects
SYNOPSIS - Safety Analysis for Predictable Software Intensive Systems
Available from: 2015-12-21 Created: 2015-12-21 Last updated: 2016-08-19Bibliographically approved
4. Making an ALARP Decision of Sufficient Testing
Open this publication in new window or tab >>Making an ALARP Decision of Sufficient Testing
2014 (English)In: Proceedings - 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering, HASE 2014, Miami, United States, 2014, 57-64 p.Conference paper (Refereed)
Abstract [en]

ALARP is an important concept in many safety standards. It helps in making a decision about how tolerable a risk is. A tolerable risk should be reduced to a point that is As Low As Reasonably Practicable (ALARP) which implies further risk-reduction is grossly inappropriate compared to the benefit attained. To date work has considered the process, safety arguments, and influencing factors of how to make an ALARP decision but not shown how to make a quantified judgement for it. In this paper a method for making an ALARP judgement decision is proposed in the context of testing the worst-case timing properties of systems. The method is based around a convergence algorithm that informs the tester when it is believed that testing for longer will not reveal sufficiently important new findings, i.e. any significant increase in observed worst-case timing needs a disproportionate amount of testing time.

Place, publisher, year, edition, pages
Miami, United States: , 2014
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-23335 (URN)10.1109/HASE.2014.17 (DOI)000351728000008 ()2-s2.0-84898623634 (ScopusID)978-1-4799-3465-2 (ISBN)
Conference
HASE 2014: 15th IEEE International Symposium on High Assurance Systems Engineering, "Towards 21st Century High Assurance System's Engineering", Miami, Florida, USA: January 9 - 11, 2014
Projects
SYNOPSIS - Safety Analysis for Predictable Software Intensive Systems
Available from: 2013-12-13 Created: 2013-12-10 Last updated: 2016-08-19Bibliographically approved

Open Access in DiVA

fulltext(3297 kB)10 downloads
File information
File name FULLTEXT02.pdfFile size 3297 kBChecksum SHA-512
fb134e01419c0aecad9419597eafeb99856722caac81bf1987338d3f343fd4c9853a280fed81cf31d83204ca784d839cd7ca189f0683da18965b6486d94210a2
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Malekzadeh, Mahnaz
By organisation
Embedded Systems
Computer and Information Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 10 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 59 hits
ReferencesLink to record
Permanent link

Direct link