Vectores

Towards Incremental Component Compatibility Testing
Ilchul Yoon, Alan Sussman, Atif Memon, Adam Porter
Dept. of Computer Science, University of Maryland, College Park, MD, 20742 USA

{iyoon,als,atif,aporter}@cs.umd.edu

ABSTRACT
Software components are increasingly assembled from other components. Each component may further depend on others, and each may have multiple active versions. Thetotal number of configurations—combinations of components and their versions—deployed by end users can be very large. Component developers, therefore, spend considerable time and effort doing compatibility testing – determining whether their components can be built correctly for all deployed configurations. In previous work we developed Rachet to support large-scale compatibility testing ofcomponents. In this paper, we describe and evaluate methods to enable Rachet to perform incremental compatibility testing. We describe algorithms to compute differences in component compatibilities between current and previous component builds, a formal test adequacy criterion based on covering the differences, and cache-aware configuration sampling and testing methods that attempt to reuse effort fromprevious testing sessions. We evaluate our approach using the 5-year evolution history of a scientific middleware component. Our results show significant performance improvements over Rachet’s previous retest-all approach, making the process of compatibility testing practical for evolving components.

Categories and Subject Descriptors
D.2.5 [Software Engineering]: Testing and Debugging— Testing toolsGeneral Terms
Design, Experimentation

Keywords
incremental testing, software component, compatibility

1. INTRODUCTION
Testing modern software components is extremely difficult. One particular challenge is that components may be

built in a great many configurations. Consider, for example, InterComm [10], a component used to support coupled parallel scientific simulations. InterComm hascomplex dependencies on multiple third-party components, each of which in turn depend on other components, and every component has multiple active versions. Each possible combination of components and their versions is a configuration that might contain unique errors. To make matters worse, each component may evolve independently, and each configuration may need to be rebuilt and retested after eachchange. In past work, we have addressed some of these challenges by creating Rachet [14, 15, 16], a process and infrastructure for testing whether a component can be built correctly for all its configurations. Rachet includes a formal, graphbased representation for encoding a component’s configuration space – the set of all possible configurations. Using this representation, developers specify thecomponents and versions, component dependencies, and constraints. Rachet then automatically computes the component’s configuration space. Rachet also defines a test adequacy criterion and algorithms for generating a set of configurations that satisfy it. Finally, Rachet efficiently tests the selected configurations, distributing the build effort across a grid of computers. However, Rachet did notaccommodate component evolution; it simply retested all configurations, including the ones that had already been tested in previous test sessions. In this paper, we describe methods to enhance Rachet that test component compatibilities incrementally, taking into account various types of component changes. We evaluate the effectiveness of our approach on 20 actual builds for the InterComm component, developedover a 5-year period. Our results show that the incremental approach is more efficient than the previous retest-all approach. We also present optimization techniques to reduce testing time even further by reusing test artifacts and results inherited from previous test sessions. More specifically, the work described in this paper makes the following contributions: • An incremental compatibility...