The Metrics: An Industry-Wide Survey Into The Use And Effectiveness Of Internal Software Prediction Metrics. Thank you for taking part in the first, industry-wide investigative survey into the application and use of internal software prediction metrics in the decision to stop testing software. Much has been written about the decision to stop testing and the use and effectiveness of internal software prediction metrics but never before has a true picture of where the industry viewpoint stands on the subject matter been presented. It is the intent of this research to investigate this area and distribute the findings within the academic body of knowledge, and your participation is a key part of this. As a thank you for your help, the first 50 valid respondents will receive UK£0.50p to their PayPal accounts (you must already have a PayPal account or activate a new PayPal account). Additionally, you will also receive a free PDF version of the final paper entitled “The Metrics – An Industrial Evaluation of the Relevance and Application of Internal Software Prediction Metrics in Determining when to Stop Testing”. To receive your free copy of this paper and your PayPal contribution, please ensure that you enter your email address below. This will be sent in October 2007. The email address will be used for these purposes only and no follow up or junk emails will be received as a result of this. In addition to this, email addresses will not be stored by the author for future reference and any personally identifiable information will not be used as part of the research. There will be no follow-up commitment required on your part. Survey responses are to be submitted no later than September 1st 2007. Once again, many thanks for your time and effort and please feel free to pass this survey along to anyone you may feel might benefit from this paper / who might wish to participate in this research. A. Personal Information 1 - E-Mail Address (optional, required to receive your free PDF report): 2 - Country Of Location (optional): 3 - Which type of industry do you operate? (optional): 4 – Do you wish for express acknowledgement of your participation in this research? Yes No If Yes, please enter your name and organisation: Name : Organisation : 5 - What professional category best suits your occupation? Academic (lecturer / writer / theorist) Manager (project / organisational) Tester (Black Box / White Box / Unit / Integration / System) Developer (requirements / analyst / engineer / programmer) Quality (Assurance / Control) Other (please specify) 6 – Considering the criticality of software produced by your organisation, would you consider the majority of projects to be: Critical Non-Critical Unclassified 7 - When thinking about Software Testing, in which area do you consider your primary strategy to be? (if you are not in control strategic management, consider the area in which you feel would be most appropriate if you were) Maximising Customer Satisfaction Minimising Engineer Effort and Schedule Minimising Defects B.The Metrics 8 - Does your organisation keep internal software prediction metrics for use during and after the development process? Yes No If Yes, please give as much detail regarding the methods employed by yourself / the organization. 9 – How many internal software prediction metrics do you / the organization tend to use on a standard project? less than 7 between 7-15 greater than 15 10 – Do those metrics form part of any of the following?: Principal Components Factor Analysis Relative Complexity Metrics None of the above 11 – Considering the success of the application of internal software prediction metrics, please rate their overall effectiveness in your organisation? (1 being highest) 1 2 3 4 5 12 – Some common, standard internal software prediction methods are available in industry, which of the following such methods have you heard of? Size: Akiyama’s Data Fitting Model Lines Of Code Delivered Source Statements Statement Counts Halstead’s Metrics Bytes Characters per line Object counts Method Size Flesch-kincaid readability Albrecht's function points Demarco's Bang (Specification Design Weight, Fan Out) Levitin's Token Count Cocomo 2.0 Execution times Structure: McCabe’s Cyclomatic Complexity metric Percent coverage Modularity Morphology Tree impurity Henry and Kafura's information flow fan-in fan-out complexity measure Harrison's complexity Object Orientated: Weighted Methods per class Depth Of Inheritance Tree Number Of Children Coupling Between Class Objects Response For Class Lack Of Cohesion If other, please specify 13 – Which methods have you used? Size: Akiyama’s Data Fitting Model Lines Of Code Delivered Source Statements Statement Counts Halstead’s Metrics Bytes Characters per line Object counts Method Size Flesch-kincaid readability Albrecht's function points Demarco's Bang (Specification Design Weight, Fan Out) Levitin's Token Count Cocomo 2.0 Execution times Structure: McCabe’s Cyclomatic Complexity metric Percent coverage Modularity Morphology Tree impurity Henry and Kafura's information flow fan-in fan-out complexity measure Harrison's complexity Object Orientated: Weighted Methods per class Depth Of Inheritance Tree Number Of Children Coupling Between Class Objects Response For Class Lack Of Cohesion If other, please specify 14- Do you believe such methods would be of any use in the domain of software testing? Yes No If No, please detail: 15- Are you familiar with any of the following? Bayesian Belief Networks Belief Networks Causal Probalistic Networks Causal Nets Graphical Probability Networks Probabilistic Cause – Effect Models Probabilistic Influence Diagrams 16 - Do you use any software testing metrics software? Yes No If Yes, please advise the software and manufacturer name (e.g. McCabe’s McCabe IQ): C. The Decision To Stop Testing Software 17 - The following criteria are used in the industry, usually in combination, when deciding the right time to terminate a testing process. When thinking about the decision to stop testing, in which order of importance do you consider the following criteria: Please order the following between 1-15 (one being the highest - e.g. if you feel Test Time Expiring being most important, rank it "1", then rank the next one you feel important as "2", and so on, until you get to the one you feel least important which will be ranked "15"). When Test Time Expires Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Monetary Budget Is Exhausted Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When All Faults Removed Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Required Test Coverage Achieved Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When All Test Cases Exhausted Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When No New Errors Revealed By Continued Testing Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Common Programming Errors Accounted For Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Beta Testing Employed Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Competition A Business Concern Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Minimum Number Of Faults Detected And Corrected Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Error Seeding Employed Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Capture – Recapture Approach Employed Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Metrics Employed Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Faults Found Drop Below Threshold Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 When Acceptance Testing Is Signed Off Select 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 If other, please specify 18 – Which of the criteria are used in your organisation, by yourself or other employees. When Test Time Expires When Monetary Budget Is Exhausted When All Faults Removed When Required Test Coverage Achieved When All Test Cases Exhausted When No New Errors Revealed By Continued Testing When Common Programming Errors Accounted For When Beta Testing Employed When Competition A Business Concern When Minimum Number Of Faults Detected And Corrected When Error Seeding Employed When Capture – Recapture Approach Employed When Metrics Employed When Faults Found Drop Below Threshold When Acceptance Testing Is Signed Off If other, please specify 19 - Does your organisation keep internal software prediction metrics that are used specifically for use in the decision of deciding when software testing should stop? Yes No 20 – If a standardized simple measure to gather internal software prediction metrics were available, do you believe that they would be useful in the decision to stop testing in the software development process? Yes No 21 - Implementing internal software prediction metrics can save time in testing, increase the quality of software products, and reduce costs involved. With this in mind, how would you rate your opinion of such metrics in the decision to stop testing software? (1 being highest) 1 2 3 4 5 22 - Do you believe that your organisation should further integrate the decision to stop testing with the use of internal software prediction metrics? Yes No Not Applicable D. Training 23 - Have you had any formal software testing metrics based training? Yes No (if no, go to question 26) 24 - If yes, in which format did the training take? Class room training On-The-Job training Other (please specify): 25 – How would you rate the training you have received (1 being highest) 1 2 3 4 5 26 – Would you be willing to further engage in internal software prediction metrics based training activities? Yes No Thank you for taking part in this survey. Please be sure to submit your response as soon as you can before the 1st September 2007. If you have any further comments, please feel free to write your contributions in the box provided below. E. Further Comments :
