CSMN-647-1121: Independant Verification and Validation



David R. Mapes

Date: 04/27/1995


This paper identifies, and analyzes methods an independent verification and validation (IV&V) effort can use to identify, assess and monitor risk in a software development project with a goal of defining a set of effective methods and practices that can be scaled to the effort. This information is key to efficient use of IV&V's limited resources. By focusing IV&V (and customer) attention on those areas of the project with the greatest potential for problems, risk identification, assessment and monitoring help IV&V provide the most benefit for the customer's money.

A literature search was conducted to formulate a definition of risk as it pertains to software projects, identify typical key types of risk for software development, and find techniques to allow IV&V to identify, assess and monitor these risks. Once these key pieces of information were gathered, an analysis was conducted to evaluate these techniques with regard to effort and effectiveness.

As a result of this research risk was defined as any possible occurrence or set of circumstances that could cause a project to fail to meet any one or more of four key goals: budget, schedule, function, and quality. Various types of risk associated with software development projects were identified and categorized. Methods of identifying and assessing risk were reviewed. A metrics approach was defined for tracking risk factors identified and selected during the assessment process.

While the types of reviews, testing and oversight that IV&V performs are largely a function to the type of software development life cycle being use by the developer, IV&V can take a proactive role in determining the direction of a portion of the validation and verification effort by performing a risk assessment as part of the initial review process. The results of this additional review help to focus IV&V's attention and additional efforts on areas of the project where the most benefit can be realized.


Independent Verification & Validation (IV&V), in general, receives between 5 and 20 percent of a development project's funds (U. S. Environmental Protection Agency (EPA) - Systems Development Center (SDC), SDC Policy: Independent Product Assurance (IPA), personal communication, January 13, 1995) (usually 10% (Lewis, 1992, p. 234). IV&V must spend this limited funding in a way that will yield the most benefit to the project in terms of meeting cost, schedule, functionality, and quality goals. One way in which IV&V can better allocate its limited resources is by concentrating its oversight on those areas of the project where the highest likelihood and/or impact of a problem exists. Risk assessment methods allow IV&V to identify potential problem areas, determine the likelihood of their occurring, predict their probable impact on the project and provide clues on how to avoid, mitigate or manage them. From an IV&V standpoint risk avoidance, mitigation and management mean identifying those areas of the project that are most in need of extra scrutiny and support to ensure that they do not become a problem, or if they do, that it is caught early enough to keep the project on track. From one perspective all of IV&V's efforts in evaluating the development process, products and progress can be seen as risk management and mitigation in that they remove some of the uncertainty and opacity from the development process for the customer, and, one hopes, the developer. IV&V does this by maintaining traceability for the customer through the development process from requirements to the final system, and by validating that final system against those requirements (Roetzheim, 1991, pp. 141-144). Risk assessment, then, is a way for IV&V to identify those areas in its' standard repertoire that may need added attention and other items that may need to be added (or deleted) for a particular project.

A Definition of Risk

In identifying, assessing and managing software development risk the place to start is with a good working definition of risk. The Webster's New Twentieth Century Dictionary (unabridged) defines risk as (in part) "... the chance of injury, damage, or loss; a dangerous chance; a hazard...." While this definition is, perhaps, a little general for our purposes, it does serve to focus attention on what constitutes a risk in the software development arena. That is, the chance that something will occur to jeopardize the development effort. A definition more applicable to software development in general, and IV&V in particular is that "...a risk is a potential problem..." that could cause a project to not meet schedule, cost, functionality, or quality goals (Fairly, 1994, p. 57). More specifically, a risk is the probability that a potential problem will materialize and have some measurable impact (in terms of time lost, money spent over budget, or functionality or performance requirements not met). Ultimately a risk, that has become a problem that was not dealt with successfully, could lead to the failure of the development effort.

Identifying Risk

Below are listed some possible areas of risk that might apply to any development effort. This is, however, by no means meant to be a comprehensive listing. As can be seen by comparing the generalized "top-down view" and "bottom-up view" (Down, Coleman & Absolon, 1994) lists with that of the U.S. Environmental Protection Agency (EPA) Systems Development Center (SDC), specific organizations, projects and environments greatly effect what is identified as a risk, and, by extension, what its perceived impact is. Because of this situational dependency of risk definition, generalized checklists can only serve as a guide and starting point for risk identification and assessment. They are not a substitute for experience, investigation, analysis and creative thinking (Down, et al., 1994, p. 50).

Types of Risk, Three views:

A top-down view (Down, et al. 1994, p. 51)
A bottom-up view (Down, et al., 1994, p. 51-52)
U. S. EPA - SDC - IPA Policy: Approach to Assessing Project Risk for IPA Level of Effort (personal communication, January 13, 1995)
With this in mind, some areas to begin looking to identify project risk are the concept of operations (if one exists), statement of work, system requirements specification and development contract documents. A careful reading of these documents will help flesh-out how the resources available to the developing agency (scheduled time, allocated funding, etc...) compared to what is expected in the eventual system. This information coupled with an assessment of developer experience and ability (from winning proposal documents, CMM assessment or other sources) will begin to point out various project areas upon which a more detailed risk assessment should focus (i.e., "Project size...", "Experience with technology...", and "Project structure..." (Boehm, 1989/1993, p. 195)).

Additional sources of risk information that may be especially revealing are any risk assessments performed by the customer and/or the developer. These will provide a valuable insight into the areas of concern to the two primary parties to development. To identify risk, in the form of developing agency ability, Boehm (1989/1993, pp. 143-144) suggests conducting a Capability Maturity Model (CMM) assessment. In many cases this may be a little extreme, but a good (or not so good) risk assessment done by the developer should give IV&V some idea of how the developer views itself, and of how competent and experienced the developer is.

For IV&V, identifying risk in a software project could become much like an anthropological field investigation with members of the IV&V team acting as "participant-observers" (Cohen & Eames, 1982, p. 24) of the development effort. And, like anthropological field work, this could get both time consuming and expensive. Because of this IV&V should be very careful not to become totally engrossed in the risk identification process, but rather, try to quickly identify those items that are of most concern to the specific effort in terms of likelihood and potential impact. One method of identifying (and removing) risk from requirements is by using an automated requirements analysis tool like that described by Atlee (1992). Such a tool can quickly identify inconsistencies and incompleteness in a set of "Software Cost Reduction (SCR) project [Hen80, AFB+88,HL]" (1992, p. 4) formatted requirements. While work in this area is proceeding (Chechik & Gannon, 1994), it is not yet ready for general application (Chechik & Gannon, 1994, p. 14). One major shortcoming of this method is the need to reformat requirements to the SCR tabular syntax. For a system of any size this would be both a time consuming and an error prone undertaking. On a more promising note the results obtained by Briand, Thomas and Hetmanski (1993, p. 57) using Optimized Set Reduction to analyze Ada design constructs show that for formalized requirements and design specifications machine driven risk analysis can work. Perhaps, the best approach for identifying risk, for the moment, is to rely on a careful, formal review of available project documents with a "weather eye" to omissions, inconsistencies and inadequacies both within and between them. This review coupled with a qualitative assessment of the developer and customer with regard to technical competence and communicative ability should allow IV&V to identify most of the risks inherent in the effort.

Assessing Risk

Once a set of risks have been identified for a project, it is important to define how likely they are to become actual problems and how they would impact the project. When assessing the likelihood of a risk, historical data for the industry, the specific developer, or the project type can provide a means of quantifying its probability (Fairley, 1994, p. 57-58). Absent this, probability may be defined in terms of qualitative measures like high, medium and low (Klein, Powell & Chapman, 1994, p. 751- 752) based upon the assessor's experience, judgement, and knowledge gained while identifying the risks and performing other IV&V functions.

In gauging risk probability from a quantitative standpoint one can calculate, based on historical data, the likelihood of staff turnover becoming a problem as a function of schedule compression (a risk in and of itself). Because work pressure rises as developers are expected to do more in less time, there is an increased probability that some may leave. Given an estimate of system size in function points (whether Albrecht's or Mk II (Symons, 1991, pp. 18-33)), a delivery schedule and proposed staffing level, IV&V can derive the proposed work load on team members in terms of function points per day (FPPD). Armed with historical data from similar projects, this FPPD figure can be plotted on a line of best fit for the historical distribution of FPPD versus staff turnover over the course of the project. Where our calculated FPPD falls on that line is a good indication of the level of staffing problems likely for this project. Note, however, that this is a simplified case; staff turnover is subject to many other work environment variables that would need to be taken into account to more accurately place the project in our distribution. But, from the standpoint of identifying a potential problem area (e.g., staff turnover), the level of effort required is a good gauge.

In the absence of easily quantifiable information IV&V may want to make a subjective evaluation of a risk. For example, after a review of available requirements documents, the IV&V analyst determines that, due to a lack of sufficient detail and specificity, the system's requirements are likely to be subjected to a high degree of volatility. This moving target syndrome can be a major source of rework, and by extension, cost and schedule variances.

Having determined the likelihood of a problem, it remains to decide how bad it would be if it transpired. Again, historical data can be used to try to predict the impact of a problem in terms of time lost or money spent, or subjective valuations can be made based on the perceived criticality to the system of the specific risk area. Returning to the staff turn over example, above, historical data might indicate, on average, how long it takes to find a qualified replacement and what the learning curve is for a position on a similar project. This data would allow us to calculate the impact of having to replace a given team member in terms of days of productive work lost. Reverting to our FPPD metric, the number of undeveloped function points can be calculated and assessed for their impact on the project schedule.

One method of assessing risk that is of particular interest in an environment where control of risk is being exercised by the use of the incremental build approach to the software development life cycle (Martin, 1988, pp. 220-221) is that proposed by Klein et al., Project Risk Analysis Based on Prototype Activities (1994, p. 751). This method of assessing risk assumes that for a given set of iterative development activities a detailed risk assessment need only be done once and then simply tailor it to later rounds "by identifying influencing circumstances, and activities in which such circumstances exist or are likely to exist, a set of rules may be developed which indicates how the risk analysis for the prototype activity may be converted into analyses for the actual activities." (1994, p. 751)

Whether assessing risk with quantitative methods, or by setting up a qualitative ranking of risks based on low, medium and high probability and criticality, it is important to be consistent in the methods used so that comparisons can be drawn between various risks as to probability and severity. This will allow IV&V to selectively increase oversight in the areas identified as having the highest overall potential for trouble. Once some form of ranking has been established for the identified risks, those posing the greatest threat to the success of the project can be singled out by IV&V for extra attention in the course of the development effort.

Risk Tracking/Metrics Collection and Evaluation

Having identified areas of concern additional to its basic review and oversight responsibilities, IV&V must decide how, within the scope of the project, it is going to monitor these areas. The collection and evaluation of various software development metrics represents an avenue for this monitoring. Whether we assessed the risk severity of these areas through quantitative regression modeling or qualitative valuation does not really matter. Having selected them, by whatever method, quantitative measures should be used to track them.

First, for each risk area that IV&V is tracking appropriate metrics should be selected and trigger values established that determine when risks have become problems. Then IV&V should decide on what course of action they will take, for each selected risk area, should that problem materialize. Next, responsibility for collecting and tracking each selected risk area metric should be assigned to an individual IV&V team member or small sub-team. Finally, the customer should be made aware of the identified risks, their likelihood and expected severity, and IV&V's monitoring plans. If IV&V has identified any ways to eliminate, mitigate or ameliorate these potential problems, it should also communicate these to the customer, just as it would in a standard problem report (Lewis, 1992, p. 181) (Boehm, 1989/1993, p. 15).

Returning to the staff turnover example, two metrics of interest would be development staff turnover and function points implemented. Having established an expected level of effort in terms of FPPD, is it being met? What is the development staff's actual turnover rate? Is turnover tracking at or below the predicted level or is it starting to exceed that level? This is a good area for one member of the IV&V team to concentrate on. Indeed it is recommended by Whitten (1990, p. 150) that each risk be assigned one owner. In the context of IV&V this would devolve to a team member. From the stand point of work load, having one person track development staff turnover and FPPD is acceptable, but they should not be required to monitor any more than those two additional metrics. IV&V team members have plenty to do outside of the risk monitoring task so following two additional metrics could put them beyond the recommended per person maximum of five metrics (Moller & Paulish, 1993, p. 62).

Summary and Conclusions

Within any given software development life cycle IV&V has a well-defined set of standard activities. While these review, oversight and testing activities are IV&V's principal concerns, there is still room for IV&V to improve its effectiveness. One way for IV&V to make better use of its limited resources is to perform a risk assessment as an integral part of the standard review process. This risk assessment should be focused on identifying areas, specific to the project, that have an increased potential for developing high impact problems. After risks have been identified and assessed through quantitative or qualitative methods, the risks that represent the greatest threat to the success of the project can be selected for closer, ongoing scrutiny by the IV&V staff.

In assigning monitoring responsibility the number and type of metrics each IV&V staff member needs to follow must be kept in mind. No one member of the IV&V team should have to keep tabs on more that five metrics including their regular review, oversight and testing duties. Levels of acceptable deviation should be determined and a threshold set for each risk as to when it will be considered to have become a problem.

While a full blown risk assessment effort might require more resources than are available to the IV&V team, spending some time as a part of regular IV&V review activity identifying and evaluating risk early in the project will allow for the timely detection of problems later. For IV&V, risk assessment can be a key way of answering the question "What else should we do?" in addition to the standard reviews, testing and oversight that make up the IV&V activity. Assessing risk can identify those areas of concern to IV&V where extra emphasis and effort may do the most in assuring the success of the project. Keeping in mind the scale of the project, a relatively small investment in risk assessment at the out-set with ongoing reevaluation of specific risks identified and any new ones that may be detected in the course of the project will be rewarded by a greater chance of project success.


IV&V should, as a regular part of its initial review process, assess the project for risk. It should use a careful evaluation of available documents (i.e., requirements specification, contract, development schedule, development contractor and customer risk assessments, etc...), to identify potential problems. The risks should be evaluated in a consistent manner to allow them to be compared relative to one another in terms of threats to system cost, schedule, functionality or quality goals. Risks that top this threat list should be assigned to individual IV&V team members or subteams for continued monitoring through appropriate metrics. Care should be taken, however, not to overload any one IV&V team member or subteam. Threshold values for these risk tracking metrics should be established as a part of the risk assessment process to allow responsible IV&V team members to determine when a risk has become a problem.

Additionally, a proactive attitude should be fostered among the IV&V team members. A few words of wisdom quoted by Boehm (1989/1993, p. 15) from Gilb (1988) serve to define how software development risk should be approached:

While these sentiments are expressed from the developer's point of view, they nonetheless place an emphasis on actively seeking risk information and taking quick and decisive action when a risk becomes a problem. Note, also that the second item on this list is something that we should always be on the lookout for from all parties to the development effort. Experience has shown that making verbal agreements on requirements can lead to the dreaded moving target system that never holds still long enough to complete. Risks should be dealt with in an open and forthright manner. When they begin to transition into problems this must be noted and dealt with at the earliest possible moment. Putting things in writing (be they requirements or problem reports) is one of the best methods of risk mitigation.


Atlee, J. M. (1992).  Automated analysis of software
     requirements.  Unpublished doctoral dissertation, University
     of Maryland College Park.

Boehm, W. B. (1989/1993).  Software risk management.  Los 
     Alamitos, CA: IEEE Computer Society Press.

Briand, L. C., Thomas, W. M., & Hetmanski, C. J. (1993). 
     Modeling and managing risk early in software development.
     Proceedings of the 15th International Conference on Software
     Engineering. (pp. 55-65).  Los Alamitos, CA: IEEE Computer
     Society Press.

Chechik, M., & Gannon, J. (1994). Automatic verification of 
     requirements implementation. In T. Ostrand (Ed.),  Proceedings
     of the 1994 International Symposium on Software Testing and
     Analysis (ISSTA). (pp. 1-14).  New York: The Association for
     Computing Machinery.

Cohen, E. N., Eames, E. (1982).  Cultural anthropology.  Boston, 
     MA: Little, Brown and Company.

Down, A., Coleman, M., & Absolon, P. (1994).  Risk management 
     for software projects.  London: McGraw-Hill Book Company.

Fairley, R. (1994, May). Risk management for software 
     projects.  IEEE Software, 11, 57-67.

Klein, J. H., Powell, P. L., Chapman, C. B. (1994).  Project 
     risk analysis based on prototype activities.  Journal of the
     operational research society, 45, (7), 749-757.

Lewis, R. O. (1992).  Independent verification and validation: A
     life cycle engineering process for quality software.  New
     York: John Wiley & Sons, Inc.

Martin, C. F. (1988).  User centered requirements analysis. 
     Englewood Cliffs, NJ: Prentice Hall, Inc.

Moller, K. H., Paulish, D. J. (1993).  Software metrics: A
     practitioner's guide to improved product development.
     Piscataway, NJ: IEEE Press.

Roetzheim, W. H. (1991).  Developing software to government 
     standards.  Englewood Cliffs, NJ: Prentice-Hall, Inc.

Symons, C. R. (1991).  Software sizing and estimating: Mk II
     FPA (Function Point Analysis).  Chichester, England: John
     Wiley & Sons.

Whitten, N. (1990).  Managing software development projects:
     Formula for success.  New York: John Wiley & Sons, Inc.