Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Paving the Way for Self-driving Cars - Software Testing for Safety-critical Systems Based on Machine Learning: A Systematic Mapping Study and a Survey
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.
2017 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Context: With the development of artificial intelligence, autonomous vehicles are becoming more and more feasible and the safety of Automated Driving (AD) system should be assured. This creates a need to analyze the feasibility of verification and validation approaches when testing safety-critical system that contains machine learning (ML) elements. There are many studies published in the context of verification and validation (V&V) research area related to safety-critical components. However, there are still blind spots of research to identify which test methods can be used to test components with deep learning elements for AD system. Therefore, research should focus on researching the relation of test methods and safety-critical components, also need to find more feasible V&V testing methods for AD system with deep learning structure.

Objectives: The main objectives of this thesis is to understand the challenges and solution proposals related to V&V of safety-critical systems that rely on machine learning and provide recommendations for future V&V of AD based on deep learning, both for research and practice.

Methods: We performed a Systematic Literature Review (SLR) through a snowballing method, based on the guidelines from Wohlin [1], to identify research on V&V methods development for machine learning. A web-based survey was used to complement the result of literature review and evaluate the V&V challenge and methods for machine learning system. We identified 64 peer-reviewed papers and analysed the methods and challenges of V&V for testing machine learning components. We conducted an industrial survey that was answered by 63 subjects. We analyzed the survey results with the help of descriptive statistics and Chi-squared tests.

Result: Through the SLR we identified two peaks for research on V&V of machine learning. Early research focused on the aerospace field and in recent years the research has been more active in other fields like automotive and robotics. 21 challenges during V&V safety-critical systems have been described and 32 solution proposals are addressing the challenges have been identified. To find the relationship between challenges and methods, a classification has been done that seven different type of challenges and five different type of solution proposals have been identified. The classification and mapping of challenges and solution methods are included in the survey questionnaire. From the survey, it was observed that some solution proposals which have attracted much research are not considered as particularly promising by practitioners. On the other hand, some new solution methods like simulated test cases are extremely promising to support V&V for safety-critical systems. Six suggestions are provided to both researchers and practitioners.

Conclusion: To conclude the thesis, our study presented a classification of challenges and solution methods for V&V of safety-critical ML-based systems. We also provide a mapping for helping practitioners understand the different kinds of challenges the respective solution methods address. Based on our findings, we provide suggestions to both researchers and practitioners. Thus, through the analysis, we have given the most concern on types of challenges and solution proposals for AD systems that use deep learning, which provides certain help to design processes for V&V of safety-critical ML-based systems in the future.

Place, publisher, year, edition, pages
2017.
Keywords [en]
Safety-critical system, Machine learning, Software testing
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-15681OAI: oai:DiVA.org:bth-15681DiVA, id: diva2:1168896
Subject / course
PA2534 Master's Thesis (120 credits) in Software Engineering
Educational program
PAAXA Master of Science Programme in Software Engineering
Supervisors
Examiners
Available from: 2017-12-22 Created: 2017-12-21 Last updated: 2018-01-13Bibliographically approved

Open Access in DiVA

fulltext(4208 kB)165 downloads
File information
File name FULLTEXT01.pdfFile size 4208 kBChecksum SHA-512
23c0390ef00a22d2b6d7332ce0194a42a37ed3d696e620038900e5627ebe19ba22626d259b4b150006963622cd1bce78ff7be5d65ed68cfd8fd0388021f53799
Type fulltextMimetype application/pdf

By organisation
Department of Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 165 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 352 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf