THE UNIVERSITY OF MARYLAND UNIVERSITY COLLEGE GRADUATE SCHOOL
OF MANAGEMENT AND TECHNOLOGY
A STUDENT RESEARCH REPORT
for
CSMN-647-1121: Independant Verification and Validation
SOFTWARE PROJECT RISK ASSESSMENT FOR
INDEPENDANT VERIFICATION & VALIDATION
by
David R. Mapes
Date: 04/27/1995
Abstract
This paper identifies, and analyzes methods an independent
verification and validation (IV&V) effort can use to identify,
assess and monitor risk in a software development project with a
goal of defining a set of effective methods and practices that
can be scaled to the effort. This information is key to
efficient use of IV&V's limited resources. By focusing IV&V (and
customer) attention on those areas of the project with the
greatest potential for problems, risk identification, assessment
and monitoring help IV&V provide the most benefit for the
customer's money.
A literature search was conducted to formulate a definition
of risk as it pertains to software projects, identify typical key
types of risk for software development, and find techniques to
allow IV&V to identify, assess and monitor these risks. Once
these key pieces of information were gathered, an analysis was
conducted to evaluate these techniques with regard to effort and
effectiveness.
As a result of this research risk was defined as any
possible occurrence or set of circumstances that could cause a
project to fail to meet any one or more of four key goals:
budget, schedule, function, and quality. Various types of risk
associated with software development projects were identified and
categorized. Methods of identifying and assessing risk were
reviewed. A metrics approach was defined for tracking risk
factors identified and selected during the assessment process.
While the types of reviews, testing and oversight that IV&V
performs are largely a function to the type of software
development life cycle being use by the developer, IV&V can take
a proactive role in determining the direction of a portion of the
validation and verification effort by performing a risk
assessment as part of the initial review process. The results of
this additional review help to focus IV&V's attention and
additional efforts on areas of the project where the most benefit
can be realized.
Introduction
Independent Verification & Validation (IV&V), in general,
receives between 5 and 20 percent of a development project's
funds (U. S. Environmental Protection Agency (EPA) - Systems
Development Center (SDC), SDC Policy: Independent Product
Assurance (IPA), personal communication, January 13, 1995)
(usually 10% (Lewis, 1992, p. 234). IV&V must spend this limited
funding in a way that will yield the most benefit to the project
in terms of meeting cost, schedule, functionality, and quality
goals. One way in which IV&V can better allocate its limited
resources is by concentrating its oversight on those areas of the
project where the highest likelihood and/or impact of a problem
exists. Risk assessment methods allow IV&V to identify
potential problem areas, determine the likelihood of their
occurring, predict their probable impact on the project and
provide clues on how to avoid, mitigate or manage them. From an
IV&V standpoint risk avoidance, mitigation and management mean
identifying those areas of the project that are most in need of
extra scrutiny and support to ensure that they do not become a
problem, or if they do, that it is caught early enough to keep
the project on track. From one perspective all of IV&V's efforts
in evaluating the development process, products and progress can
be seen as risk management and mitigation in that they remove
some of the uncertainty and opacity from the development process
for the customer, and, one hopes, the developer. IV&V does this
by maintaining traceability for the customer through the
development process from requirements to the final system, and by
validating that final system against those requirements
(Roetzheim, 1991, pp. 141-144). Risk assessment, then, is a way
for IV&V to identify those areas in its' standard repertoire that
may need added attention and other items that may need to be
added (or deleted) for a particular project.
A Definition of Risk
In identifying, assessing and managing software development
risk the place to start is with a good working definition of
risk. The Webster's New Twentieth Century Dictionary
(unabridged) defines risk as (in part) "... the chance of injury,
damage, or loss; a dangerous chance; a hazard...." While this
definition is, perhaps, a little general for our purposes, it
does serve to focus attention on what constitutes a risk in the
software development arena. That is, the chance that something
will occur to jeopardize the development effort. A definition
more applicable to software development in general, and IV&V in
particular is that "...a risk is a potential problem..." that
could cause a project to not meet schedule, cost, functionality,
or quality goals (Fairly, 1994, p. 57). More specifically, a
risk is the probability that a potential problem will materialize
and have some measurable impact (in terms of time lost, money
spent over budget, or functionality or performance requirements
not met). Ultimately a risk, that has become a problem that was
not dealt with successfully, could lead to the failure of the
development effort.
Identifying Risk
Below are listed some possible areas of risk that might
apply to any development effort. This is, however, by no means
meant to be a comprehensive listing. As can be seen by comparing
the generalized "top-down view" and "bottom-up view" (Down,
Coleman & Absolon, 1994) lists with that of the U.S.
Environmental Protection Agency (EPA) Systems Development Center
(SDC), specific organizations, projects and environments greatly
effect what is identified as a risk, and, by extension, what its
perceived impact is. Because of this situational dependency of
risk definition, generalized checklists can only serve as a guide
and starting point for risk identification and assessment. They
are not a substitute for experience, investigation, analysis and
creative thinking (Down, et al., 1994, p. 50).
Types of Risk, Three views:
A top-down view (Down, et al. 1994, p. 51)
- Business risk
- The market for product is not as expected.
- The product cannot be brought to market in time to
capitalize on demand.
- The product cannot be developed to a price that
the customer is prepared to pay.
- The resources needed--people, equipment,
infrastructure--are beyond our capacity.
- Competitors are already further advanced.
- User acceptance risk
- The function the customer requires cannot be
delivered.
- A demand for the function in the product cannot be
created.
- The requirements may not have been established
correctly.
- The customer wants more than can be delivered.
- Technical risk
- The required function is beyond current
technological capability.
- The required quality--performance, usability,
etc., is beyond current technological capability.
- There are dependencies on corequisite product ship
dates.
- All external standards may not be met.
- Implementation risk
- There are not the required skills/abilities to do
the job.
- The most appropriate tools, methods, etc., are not
available.
- The people do not understand the process/there is
not enough support to implement the process
properly.
- Ditto with regard to the programming language to
be used, development tools, etc.
- Adequate project management skills are not
available.
- External risk
- The product is dependant on the successful
development of another product/component.
- Some of the work needs to subcontracted to a third
party developer.
- A third party developer cannot deliver on time.
- Deliverables required by a third party developer
cannot be delivered to them on time.
A bottom-up view (Down, et al., 1994, p. 51-52)
- Schedule Will target dates be met for each
development phase? For producing deliverables
needed by another organization, either internal or
external? For announcing the product that we
eventually want to ship?
- Function Can we deliver all committed line items?
Will they satisfy targets with respect to
usability? performance? reliability?
maintainability? Will the documentation be
complete?
- Quality Can we assess our code quality? Is it
satisfactory? What are our expectations for
defects remaining in the shipped product?
- Process Do we have plans? Are they valid? Are
we tracking to them? Is the development process
defined, and is it being followed?
- Business Is the business case for our product
still valid? are we tracking to budget? What are
the competition doing--are our prices likely to be
undercut?
- Resource Do we have enough people? Do they have
the right skills? Can we give them the equipment
they need? Pay them enough? What is the expected
attrition rate during the development of our
product and is it going to cause problems?
- Third parties/vendors Are we using a vendor when
we should not? Not using one when we should? Are
our vendor management procedures adequate? How
confident are we that we will get deliverables on
time? Are they crucial to the successful
development of the product?
- Dependencies Is there anything that our product
is dependant on from another organization? Are
our requirements understood? Is the other
organization tracking to schedule? Do we have an
organization in place to manage these
dependencies?
- Constraints Are there any factors which are
totally beyond our control? Might global
standards change to which our product must adhere?
Are there inextricable links between our ship/
announcement dates to those of another, higher
priority product?
U. S. EPA - SDC - IPA Policy: Approach to Assessing
Project Risk for IPA Level of Effort (personal
communication, January 13, 1995)
- A project is high risk if two or more of following
apply (20% of labor hours devoted to Independent
Product Assurance (IPA))
- Unique application
- Lack of up-to-date documentation
- Inexperienced development staff
- No schedule slack
- Uncertain requirements
- Multiple customers
- Customer came unwillingly to SDC
- Subcontractor labor hours at least 50% of total
development labor hours (associated risks: less
management flexibility, subcontractor rate
increases)
- Software failure results in financial loss
- A project is medium risk if two or more of the
following apply (10% of labor hours devoted to IPA)
- Application domain not well understood
- Documentation not up-to-date
- Some inexperienced development staff
- Little schedule slack
- Some major requirements uncertain
- Customer new to SDC
- Subcontractor labor hours between 25% and 50%
of total development labor hours
- Software failure results in high visibility
within EPA
- A project is low risk if three or more of the following
apply (5% of labor hours devoted to IPA)
- Application domain well understood
- Up-to-date documentation
- Experienced development staff
- Flexible schedule
- Requirements certain
- Customer not new to the SDC
- Customer came willingly to the SDC
- Software failure does not result in financial
loss
With this in mind, some areas to begin looking to identify
project risk are the concept of operations (if one exists),
statement of work, system requirements specification and
development contract documents. A careful reading of these
documents will help flesh-out how the resources available to the
developing agency (scheduled time, allocated funding, etc...)
compared to what is expected in the eventual system. This
information coupled with an assessment of developer experience
and ability (from winning proposal documents, CMM assessment or
other sources) will begin to point out various project areas upon
which a more detailed risk assessment should focus (i.e.,
"Project size...", "Experience with technology...", and "Project
structure..." (Boehm, 1989/1993, p. 195)).
Additional sources of risk information that may be
especially revealing are any risk assessments performed by the
customer and/or the developer. These will provide a valuable
insight into the areas of concern to the two primary parties to
development. To identify risk, in the form of developing agency
ability, Boehm (1989/1993, pp. 143-144) suggests conducting a
Capability Maturity Model (CMM) assessment. In many cases this
may be a little extreme, but a good (or not so good) risk
assessment done by the developer should give IV&V some idea of
how the developer views itself, and of how competent and
experienced the developer is.
For IV&V, identifying risk in a software project could
become much like an anthropological field investigation with
members of the IV&V team acting as "participant-observers" (Cohen
& Eames, 1982, p. 24) of the development effort. And, like
anthropological field work, this could get both time consuming
and expensive. Because of this IV&V should be very careful not
to become totally engrossed in the risk identification process,
but rather, try to quickly identify those items that are of most
concern to the specific effort in terms of likelihood and
potential impact. One method of identifying (and removing) risk
from requirements is by using an automated requirements analysis
tool like that described by Atlee (1992). Such a tool can
quickly identify inconsistencies and incompleteness in a set of
"Software Cost Reduction (SCR) project [Hen80, AFB+88,HL]" (1992,
p. 4) formatted requirements. While work in this area is
proceeding (Chechik & Gannon, 1994), it is not yet ready for
general application (Chechik & Gannon, 1994, p. 14). One major
shortcoming of this method is the need to reformat requirements
to the SCR tabular syntax. For a system of any size this would
be both a time consuming and an error prone undertaking. On a
more promising note the results obtained by Briand, Thomas and
Hetmanski (1993, p. 57) using Optimized Set Reduction to analyze
Ada design constructs show that for formalized requirements and
design specifications machine driven risk analysis can work.
Perhaps, the best approach for identifying risk, for the moment,
is to rely on a careful, formal review of available project
documents with a "weather eye" to omissions, inconsistencies and
inadequacies both within and between them. This review coupled
with a qualitative assessment of the developer and customer with
regard to technical competence and communicative ability should
allow IV&V to identify most of the risks inherent in the effort.
Assessing Risk
Once a set of risks have been identified for a project, it
is important to define how likely they are to become actual
problems and how they would impact the project. When assessing
the likelihood of a risk, historical data for the industry, the
specific developer, or the project type can provide a means of
quantifying its probability (Fairley, 1994, p. 57-58). Absent
this, probability may be defined in terms of qualitative measures
like high, medium and low (Klein, Powell & Chapman, 1994, p. 751-
752) based upon the assessor's experience, judgement, and
knowledge gained while identifying the risks and performing other
IV&V functions.
In gauging risk probability from a quantitative standpoint
one can calculate, based on historical data, the likelihood of
staff turnover becoming a problem as a function of schedule
compression (a risk in and of itself). Because work pressure
rises as developers are expected to do more in less time, there
is an increased probability that some may leave. Given an
estimate of system size in function points (whether Albrecht's or
Mk II (Symons, 1991, pp. 18-33)), a delivery schedule and
proposed staffing level, IV&V can derive the proposed work load
on team members in terms of function points per day (FPPD).
Armed with historical data from similar projects, this FPPD
figure can be plotted on a line of best fit for the historical
distribution of FPPD versus staff turnover over the course of the
project. Where our calculated FPPD falls on that line is a good
indication of the level of staffing problems likely for this
project. Note, however, that this is a simplified case; staff
turnover is subject to many other work environment variables that
would need to be taken into account to more accurately place the
project in our distribution. But, from the standpoint of
identifying a potential problem area (e.g., staff turnover), the
level of effort required is a good gauge.
In the absence of easily quantifiable information IV&V may
want to make a subjective evaluation of a risk. For example,
after a review of available requirements documents, the IV&V
analyst determines that, due to a lack of sufficient detail and
specificity, the system's requirements are likely to be subjected
to a high degree of volatility. This moving target syndrome can
be a major source of rework, and by extension, cost and schedule
variances.
Having determined the likelihood of a problem, it remains to
decide how bad it would be if it transpired. Again, historical
data can be used to try to predict the impact of a problem in
terms of time lost or money spent, or subjective valuations can
be made based on the perceived criticality to the system of the
specific risk area. Returning to the staff turn over example,
above, historical data might indicate, on average, how long it
takes to find a qualified replacement and what the learning curve
is for a position on a similar project. This data would allow us
to calculate the impact of having to replace a given team member
in terms of days of productive work lost. Reverting to our FPPD
metric, the number of undeveloped function points can be
calculated and assessed for their impact on the project schedule.
One method of assessing risk that is of particular interest
in an environment where control of risk is being exercised by the
use of the incremental build approach to the software development
life cycle (Martin, 1988, pp. 220-221) is that proposed by Klein
et al., Project Risk Analysis Based on Prototype Activities
(1994, p. 751). This method of assessing risk assumes that for a
given set of iterative development activities a detailed risk
assessment need only be done once and then simply tailor it to
later rounds "by identifying influencing circumstances, and
activities in which such circumstances exist or are likely to
exist, a set of rules may be developed which indicates how the
risk analysis for the prototype activity may be converted into
analyses for the actual activities." (1994, p. 751)
Whether assessing risk with quantitative methods, or by
setting up a qualitative ranking of risks based on low, medium
and high probability and criticality, it is important to be
consistent in the methods used so that comparisons can be drawn
between various risks as to probability and severity. This will
allow IV&V to selectively increase oversight in the areas
identified as having the highest overall potential for trouble.
Once some form of ranking has been established for the identified
risks, those posing the greatest threat to the success of the
project can be singled out by IV&V for extra attention in the
course of the development effort.
Risk Tracking/Metrics Collection and Evaluation
Having identified areas of concern additional to its basic review
and oversight responsibilities, IV&V must decide how, within the
scope of the project, it is going to monitor these areas. The
collection and evaluation of various software development metrics
represents an avenue for this monitoring. Whether we assessed
the risk severity of these areas through quantitative regression
modeling or qualitative valuation does not really matter. Having
selected them, by whatever method, quantitative measures should
be used to track them.
First, for each risk area that IV&V is tracking appropriate
metrics should be selected and trigger values established that
determine when risks have become problems. Then IV&V should
decide on what course of action they will take, for each selected
risk area, should that problem materialize. Next, responsibility
for collecting and tracking each selected risk area metric should
be assigned to an individual IV&V team member or small sub-team.
Finally, the customer should be made aware of the identified
risks, their likelihood and expected severity, and IV&V's
monitoring plans. If IV&V has identified any ways to eliminate,
mitigate or ameliorate these potential problems, it should also
communicate these to the customer, just as it would in a standard
problem report (Lewis, 1992, p. 181) (Boehm, 1989/1993, p. 15).
Returning to the staff turnover example, two metrics of
interest would be development staff turnover and function points
implemented. Having established an expected level of effort in
terms of FPPD, is it being met? What is the development staff's
actual turnover rate? Is turnover tracking at or below the
predicted level or is it starting to exceed that level? This is
a good area for one member of the IV&V team to concentrate on.
Indeed it is recommended by Whitten (1990, p. 150) that each risk
be assigned one owner. In the context of IV&V this would devolve
to a team member. From the stand point of work load, having one
person track development staff turnover and FPPD is acceptable,
but they should not be required to monitor any more than those
two additional metrics. IV&V team members have plenty to do
outside of the risk monitoring task so following two additional
metrics could put them beyond the recommended per person maximum
of five metrics (Moller & Paulish, 1993, p. 62).
Summary and Conclusions
Within any given software development life cycle IV&V has a
well-defined set of standard activities. While these review,
oversight and testing activities are IV&V's principal concerns,
there is still room for IV&V to improve its effectiveness. One
way for IV&V to make better use of its limited resources is to
perform a risk assessment as an integral part of the standard
review process. This risk assessment should be focused on
identifying areas, specific to the project, that have an
increased potential for developing high impact problems. After
risks have been identified and assessed through quantitative or
qualitative methods, the risks that represent the greatest threat
to the success of the project can be selected for closer, ongoing
scrutiny by the IV&V staff.
In assigning monitoring responsibility the number and type
of metrics each IV&V staff member needs to follow must be kept in
mind. No one member of the IV&V team should have to keep tabs on
more that five metrics including their regular review, oversight
and testing duties. Levels of acceptable deviation should be
determined and a threshold set for each risk as to when it will
be considered to have become a problem.
While a full blown risk assessment effort might require more
resources than are available to the IV&V team, spending some time
as a part of regular IV&V review activity identifying and
evaluating risk early in the project will allow for the timely
detection of problems later. For IV&V, risk assessment can be a
key way of answering the question "What else should we do?" in
addition to the standard reviews, testing and oversight that make
up the IV&V activity. Assessing risk can identify those areas of
concern to IV&V where extra emphasis and effort may do the most
in assuring the success of the project. Keeping in mind the
scale of the project, a relatively small investment in risk
assessment at the out-set with ongoing reevaluation of specific
risks identified and any new ones that may be detected in the
course of the project will be rewarded by a greater chance of
project success.
Recommendations
IV&V should, as a regular part of its initial review
process, assess the project for risk. It should use a careful
evaluation of available documents (i.e., requirements
specification, contract, development schedule, development
contractor and customer risk assessments, etc...), to identify
potential problems. The risks should be evaluated in a
consistent manner to allow them to be compared relative to one
another in terms of threats to system cost, schedule,
functionality or quality goals. Risks that top this threat list
should be assigned to individual IV&V team members or subteams
for continued monitoring through appropriate metrics. Care
should be taken, however, not to overload any one IV&V team
member or subteam. Threshold values for these risk tracking
metrics should be established as a part of the risk assessment
process to allow responsible IV&V team members to determine when
a risk has become a problem.
Additionally, a proactive attitude should be fostered among
the IV&V team members. A few words of wisdom quoted by Boehm
(1989/1993, p. 15) from Gilb (1988) serve to define how software
development risk should be approached:
- If you don't actively attack the risks, they will
attack you
- Never make promises you cannot keep, no matter what the
pressure
- When something happens during the project that you did
not foresee, which increases deviation from the planned
risk, immediately raise the issue , in writing, with
your constructive suggestions as to how to deal with it
- If you don't ask for risk information, you are asking
for trouble
While these sentiments are expressed from the developer's point
of view, they nonetheless place an emphasis on actively seeking
risk information and taking quick and decisive action when a risk
becomes a problem. Note, also that the second item on this list
is something that we should always be on the lookout for from all
parties to the development effort. Experience has shown that
making verbal agreements on requirements can lead to the dreaded
moving target system that never holds still long enough to
complete. Risks should be dealt with in an open and forthright
manner. When they begin to transition into problems this must be
noted and dealt with at the earliest possible moment. Putting
things in writing (be they requirements or problem reports) is
one of the best methods of risk mitigation.
References
Atlee, J. M. (1992). Automated analysis of software
requirements. Unpublished doctoral dissertation, University
of Maryland College Park.
Boehm, W. B. (1989/1993). Software risk management. Los
Alamitos, CA: IEEE Computer Society Press.
Briand, L. C., Thomas, W. M., & Hetmanski, C. J. (1993).
Modeling and managing risk early in software development.
Proceedings of the 15th International Conference on Software
Engineering. (pp. 55-65). Los Alamitos, CA: IEEE Computer
Society Press.
Chechik, M., & Gannon, J. (1994). Automatic verification of
requirements implementation. In T. Ostrand (Ed.), Proceedings
of the 1994 International Symposium on Software Testing and
Analysis (ISSTA). (pp. 1-14). New York: The Association for
Computing Machinery.
Cohen, E. N., Eames, E. (1982). Cultural anthropology. Boston,
MA: Little, Brown and Company.
Down, A., Coleman, M., & Absolon, P. (1994). Risk management
for software projects. London: McGraw-Hill Book Company.
Fairley, R. (1994, May). Risk management for software
projects. IEEE Software, 11, 57-67.
Klein, J. H., Powell, P. L., Chapman, C. B. (1994). Project
risk analysis based on prototype activities. Journal of the
operational research society, 45, (7), 749-757.
Lewis, R. O. (1992). Independent verification and validation: A
life cycle engineering process for quality software. New
York: John Wiley & Sons, Inc.
Martin, C. F. (1988). User centered requirements analysis.
Englewood Cliffs, NJ: Prentice Hall, Inc.
Moller, K. H., Paulish, D. J. (1993). Software metrics: A
practitioner's guide to improved product development.
Piscataway, NJ: IEEE Press.
Roetzheim, W. H. (1991). Developing software to government
standards. Englewood Cliffs, NJ: Prentice-Hall, Inc.
Symons, C. R. (1991). Software sizing and estimating: Mk II
FPA (Function Point Analysis). Chichester, England: John
Wiley & Sons.
Whitten, N. (1990). Managing software development projects:
Formula for success. New York: John Wiley & Sons, Inc.