Torx : Automated Model Based Testing

Abstract
Systematic testing is very important for assessing and improving the quality of software
systems. Yet, testing turns out to be expensive, laborious, time-consuming and error-prone.
The Dutch research and development project C^ote de Resyste worked on methods, tech-
niques and tools for automating specication based testing using formal methods. The
main achievement of the project isa test tool, baptized TorX, which integrates automatic
test generation, test execution, and test analysis in an on-the-
y manner. On the one hand,
TorX is based on well-dened theory, viz. the ioco-test theory, which has its roots in the
theory of testing- and refusal-equivalences for transition systems. On the other hand, the
applicability of TorX has been demonstrated by testing severalacademic and industrial
case studies. This paper summarizes the main results of the project C^ote de Resyste.
Keywords: model-based testing, specication-based testing, formal methods, test au-
tomation, test generation.
1 Introduction
Software Testing Software quality is an issue that currently attracts a lot of attention. Software
invades everywhere in our society and lives, and we areincreasingly dependent on it. Moreover,
the complexity of software is still growing. Consequently, the quality, functional correctness
and reliability of software is an issue of increasing importance and growing concern. Systematic
testing of software plays an important role in the quest for improved quality.
Despite its importance, testing is often an under-exposed phase in the softwaredevelopment
process. Moreover, testing has turned out to be expensive, dicult, and problematic. One source
of problems is that specications are usually imprecise, incomplete and ambiguous, so that a good
basis for testing is lacking. Another source is that testing is often a manual and laborious process
without eective automation, so it is error-prone and consumes many resources. The testing
phaseoften gets jammed between moving code delivery dates and xed custom delivery dates.
Besides, research and development in testing have been rather immature. Testing methodology
is mostly ad hoc and governed by heuristics.
Fortunately, this situation is gradually improving. Triggered by the quest for improved quality
and imposed by increased product liability, testing is considered increasinglyimportant and
treated more seriously. Research in software testing is growing, the testing phase is more seriously
planned and managed, and being a software tester is starting to be a true profession.
C^ote de Resyste The research and development project C^ote de Resyste{ Conformance
Testing of Reactive Systems { aimed at improving the testing process by using formal methods.
C^ote de Resystewas supported by the Dutch Technology Foundation STW in the context of
the \Progress" programme. The project is a cooperation between Philips Research Laboratories
Eindhoven, Lucent Technologies R&D Centre Twente, Eindhoven University of Technology, and
the University of Twente, while close relationships existed with CMG (now LogicaCMG) and
Interpay. The 23 man-year project started in 1998and ended in 2002.
Model Based Testing C^ote de Resyste worked on improving the software testing process by
enabling automatic testing of software systems based on formal models of these systems. In doing
so, C^ote de Resyste concentrated on specication based, functional testing of reactive systems.
Reactive systems are mostly technical, event-driven software systems in which stimulus /response
behaviour is very important. Examples are embedded systems, communication protocols,
and process control software. Administrative systems are typically not reactive systems.
Testing involves checking the correctness of a reactive system by performing experiments in a
systematic and controlled way. Functional testing involves checking whether the system behaves
correctly: does the system...