Digitala Vetenskapliga Arkivet

Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
On Evaluating Test Techniques in an Industrial Setting
Mälardalens högskola, Institutionen för datavetenskap och elektronik.ORCID-id: 0000-0002-5070-9312
2007 (engelsk)Licentiatavhandling, med artikler (Annet vitenskapelig)
Abstract [en]

Testing is a costly and an important activity in the software industry today. The systems are becoming more complex and the amount of code is constantly increasing. The majority of systems need to rely on its testing to show that it works, is reliable, and performs according to user expectations and specifications.

Testing is performed in a multitude of ways, using different test approaches. How testing is conducted becomes essential, when time is limited, since exhaustive testing is not an option in large complex systems, Therefore, the design of the individual test case – and what part and aspect of the system it exercises, is the main focus of testing. Not only do we need to create, and execute test cases efficiently, but we also want them to expose important faults in the system. This main topic of testing has long been a focus of practitioners in industry, and there exists over 70 test techniques that aim to describe how to design a test case. Unfortunately, despite the industrial needs, research on test techniques are seldom performed in large complex systems.

The main purpose of this licentiate thesis is to create an environment and framework where it is possible to evaluate test techniques. Our overall goal is to investigate suitable test techniques for different levels, (e.g. component, integration and system level) and to provide guidelines to industry on what is effective, efficient and applicable to test, based on knowledge of failure-fault distribution in a particular domain. In this thesis, our research has been described through four papers that start from a broad overview of typical industrial systems and arrive at a specific focus on how to set up a controlled experiment in an industrial environment. Our initial paper has stated the status of testing in industry, and aided in identifying specific issues as well as underlined the need for further research. We then made experiments with component test improvements, by simple utilization of known approaches (e.g. static analysis, code reviews and statement coverage). This resulted in a substantial cost-reduction and increased quality, and provided us better understanding of the difficulties in deploying known test techniques in reality, which are described in our second paper. These works lead us to our third paper, which describes the framework and process for evaluating test techniques. The first sub-process in this framework deals with how to prepare the experiment with a known set of faults. We aimed to investigate fault classifications to get a useful set of faults of different types to inject. In addition, we investigated real faults reported in an industrial system, performed controlled experiments, and the results were published in our fourth paper.

The main contributions of this Licentiate thesis are the valuable insights in the context of evaluation of test techniques, specifically the problems of creating a useful experiment in an industrial setting, in addition to the survey of the state of practice of software testing in Industry. We want to better understand what needs to be done to create efficient evaluations of test techniques, and secondly what is the relation between faults/failures and test techniques. Though our experiments have not yet been able to create ‘the ultimate’ classification for such an aim, the results indicate the appropriateness of this approach. With these valuable insights, we believe that we will be able to direct our future research, to make better evaluations that have a larger potential to generalize and scale.

sted, utgiver, år, opplag, sider
Institutionen för datavetenskap och elektronik , 2007. , s. 116
Serie
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 78
Emneord [en]
Fault, Failure, Fault injection, Test Techniques,
HSV kategori
Forskningsprogram
Datavetenskap
Identifikatorer
URN: urn:nbn:se:mdh:diva-470ISBN: 978-91-85485-68-0 (tryckt)OAI: oai:DiVA.org:mdh-470DiVA, id: diva2:120997
Presentation
2007-12-18, Delta, Mälardalens Högskola, Rosenhill, Högskoleplan 1, Västerås, 14:00
Opponent
Veileder
Tilgjengelig fra: 2007-12-04 Laget: 2007-12-04 Sist oppdatert: 2018-01-13
Delarbeid
1. How to Save on Quality Assurance – Challenges in Software Testing
Åpne denne publikasjonen i ny fane eller vindu >>How to Save on Quality Assurance – Challenges in Software Testing
2006 (engelsk)Inngår i: Jornadas sobre Testeo de SoftwareArtikkel i tidsskrift (Fagfellevurdert) Published
Identifikatorer
urn:nbn:se:mdh:diva-4143 (URN)
Tilgjengelig fra: 2007-12-04 Laget: 2007-12-04 Sist oppdatert: 2015-03-05bibliografisk kontrollert
2. Experiments with Component Test to Improve Software Quality
Åpne denne publikasjonen i ny fane eller vindu >>Experiments with Component Test to Improve Software Quality
(engelsk)Manuskript (preprint) (Annet vitenskapelig)
HSV kategori
Identifikatorer
urn:nbn:se:mdh:diva-4144 (URN)
Tilgjengelig fra: 2007-12-04 Laget: 2007-12-04 Sist oppdatert: 2018-01-13bibliografisk kontrollert
3. Framework for Comparing Efficiency, Effectiveness and Applicability of Software Testing Techniques
Åpne denne publikasjonen i ny fane eller vindu >>Framework for Comparing Efficiency, Effectiveness and Applicability of Software Testing Techniques
Vise andre…
2006 (engelsk)Inngår i: Proceedings - Testing: Academic and Industrial Conference - Practice and Research Techniques, TAIC PART 2006, 2006, s. 159-170, artikkel-id 1691683Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

Software testing is expensive for the industry, and always constrained by time and effort. Although there is a multitude of test techniques, there are currently no scientifically based guidelines for the selection of appropriate techniques of different domains and contexts. For large complex systems, some techniques are more efficient in finding failures than others and some are easier to apply than others are. From an industrial perspective, it is important to find the most effective and efficient test design technique that is possible to automate and apply. In this paper, we propose an experimental framework for comparison of test techniques with respect to efficiency, effectiveness and applicability. We also plan to evaluate ease of automation, which has not been addressed by previous studies. We highlight some of the problems of evaluating or comparingtest techniques in an objective manner. We describe our planned process for this multi-phase experimental study. This includes presentation of some of the important measurements to be collected with the dual goals of analyzing the properties of the test technique, as well as validating our experimental framework.

HSV kategori
Identifikatorer
urn:nbn:se:mdh:diva-4145 (URN)10.1109/TAIC-PART.2006.1 (DOI)2-s2.0-80053523364 (Scopus ID)
Konferanse
1st Testing: Academic and Industrial Conference - Practice and Research Techniques, TAIC PART 2006; Windsor; United Kingdom; 29 August 2006 through 31 August 2006
Tilgjengelig fra: 2007-12-04 Laget: 2007-12-04 Sist oppdatert: 2015-06-01bibliografisk kontrollert
4. Component Testing is Not Enough - A Study of Software Faults in Telecom Middleware
Åpne denne publikasjonen i ny fane eller vindu >>Component Testing is Not Enough - A Study of Software Faults in Telecom Middleware
2007 (engelsk)Inngår i: Lecture Notes in Computer Science, vol. 4581, Springer, 2007, s. 74-89Kapittel i bok, del av antologi (Fagfellevurdert)
Abstract [en]

The interrelationship between software faults and failures is quite intricate and obtaining a meaningful characterization of it would definitely help the testing community in deciding on efficient and effective test strategies. Towards this objective, we have investigated and classified failures observed in a large complex telecommunication industry middleware system during 2003-2006. In this paper, we describe the process used in our study for tracking faults from failures along with the details of failure data. We present the distribution and frequency of the failures along with some interesting findings unravelled while analyzing the origins of these failures. Firstly, though "simple" faults happen, together they account for only less than 10%. The majority of faults come from either missing code or path, or superfluous code, which are all faults that manifest themselves for the first time at integration/system level; not at component level. These faults are more frequent in the early versions of the software, and could very well be attributed to the difficulties in comprehending and specifying the context (and adjacent code) and its dependencies well enough, in a large complex system with time to market pressures. This exposes the limitations of component testing in such complex systems and underlines the need for allocating more resources for higher level integration and system testing.

sted, utgiver, år, opplag, sider
Springer, 2007
Serie
Lecture Notes in Computer Science, ISSN 0302-9743 ; 4581
HSV kategori
Identifikatorer
urn:nbn:se:mdh:diva-4146 (URN)10.1007/978-3-540-73066-8_6 (DOI)978-3-540-73065-1 (ISBN)
Merknad

19th IFIP TC6/WG6.1 International Conference on Testing of Communicating Systems, Testcom 2007, and 7th International Workshop on Formal Approaches to Testing Software, FATES 2007; Tallinn ;26 June 2007 through 29 June 2007

Tilgjengelig fra: 2007-12-04 Laget: 2007-12-04 Sist oppdatert: 2015-02-03bibliografisk kontrollert

Open Access i DiVA

Fulltekst mangler i DiVA

Søk i DiVA

Av forfatter/redaktør
Eldh, Sigrid
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar

isbn
urn-nbn

Altmetric

isbn
urn-nbn
Totalt: 870 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf