A Methodology For

PROCESS IMPROVEMENT EXPERIMENT
FINAL REPORT



A Methodology for
Preventing Requirements Issues
from Becoming Defects
(PRIDE)

ESSI Project No. 21167




Otto Vinter

Sound & Vibration Measurement A/S
(Contractor)

Søren Lauesen, Jan Pries-Heje
Copenhagen Business School
(Subcontractor)



1999-05-31
Version No. 3

TABLE OF CONTENTS
1. EXECUTIVE SUMMARY 32. BACKGROUND INFORMATION 4
2.1. OBJECTIVES 4
2.2. INVOLVED COMPANIES AND THEIR ROLES 4
2.3. STARTING SCENARIO 4
2.4. EXPECTED OUTCOMES 5
3. WORK PERFORMED 6
3.1. ORGANISATION 6
3.2. TECHNICAL ENVIRONMENT 6
3.3. TRAINING 7
3.4. ROLE OF THE CONSULTANTS 7
3.5. PHASES OF THE EXPERIMENT 7
3.6. DISSEMINATION 10
4. RESULTS AND ANALYSIS 11
4.1. TECHNICAL 11
4.2.BUSINESS 17
4.3. ORGANISATION 18
4.4. CULTURE 18
4.5. SKILLS 18
5. KEY LESSONS 18
5.1. TECHNOLOGICAL POINT OF VIEW 18
5.2. BUSINESS POINT OF VIEW 19
5.3. STRENGTHS AND WEAKNESSES OF THE EXPERIMENT 19
6. CONCLUSIONS AND FUTURE ACTIONS 20
7. EPILOGUE 20
8. GLOSSARY 20
9. REFERENCES 21
COMPANY LOCATIONS AND CONTACT POINTS 22
ANNEX A REQUIREMENTS RELATED BUG CATEGORIES 23ANNEX B ERROR SOURCES 25
ANNEX C QUALITY FACTORS 27
ANNEX D PREVENTION TECHNIQUES 28
ANNEX E THE TECHNIQUES OF THE METHODOLOGY 31
E1. SCENARIOS (101) 31
E2. USABILITY TEST, SCREEN MOCKUP, DAILY TASKS (220) 32
E3. LET PRODUCT EXPERT REVIEW SCREENS (280) 32
E4. EXTERNAL SOFTWARE STRESS TEST (301) 33
E5. ORTHOGONALITY CHECK (721) 33
E6. PERFORMANCE SPECIFICATIONS (820) 33

1. ExecutiveSummary
Through a rigorous analysis of problem reports from previous projects Brüel & Kjær has achieved a major change in the requirements engineering process. We have developed and validated an experience based requirements engineering methodology, which has proven to give quantitative as well as qualitative improvements to our products and business.

Through our analysis of error reports in aprevious process improvement experiment aimed at improving the efficiency of our testing process [10][11], we have found that requirements related bugs are the major cause of bugs. We also found that the vast majority of these problems (85%) were due to missing or changed requirements; and misunderstood requirements by developers and testers.

In both experiments we have categorised bugsaccording to a taxonomy described by Boris Beizer [2]. For the current PRIDE experiment, however, we have limited the study to those bugs which can be related to requirements issues. We found that requirements related bugs represented 51% of all the bugs analysed.

Furthermore we have found that requirements issues are not what is expected from the literature. Usability issues dominate (64%). Problemswith understanding and co-operating with 3rd party software packages and circumventing their errors are also very frequent (28%). Functionality issues that we (and others) originally thought were the major requirements problems only represent a smaller part (22%). Other issues account for 13%. The sum of these figures adds up to more than 100% because one bug may involve more than one issue.

Acloser study of the bugs suggested a number of techniques that could prevent a significant number of requirements related bugs.

Since usability issues dominate, the corner stone of the new requirements engineering methodology was to introduce usability tests [7][8] on very early prototypes. These were aimed at validating the development team’s understanding of common use situations (scenarios)[4][9] for the new product. Focused readings and specific actions to verify the resulting requirements specification were also introduced.

The validation techniques of the methodology were introduced on a real-life development project. On this project we achieved an almost 3 times increase in developer productivity on the user interface, and usability issues per new screen were reduced by...