CSMN 601-1111: Issues, Trends, and
Strategies for Computer Systems







David R. Mapes



Training is a key factor in maintaining and enhancing a software development and support organization's (SDSO) competitiveness. In business, as in any facet of modern life, change is the only constant. This is especially so in the software industry, where new tools for, and approaches to, dealing with the complexities of software development and support appear almost daily. Additionally, the cost and uncertainty of hiring staff to apply these new languages, techniques and tools provide a strong incentive to develop current staff through training rather than acquiring new capability by hiring it. Therefore top SDSOs must, not only seek to identify useful new tools and techniques, but also maintain a well-organized and effective means of integrating them into the SDSO's way of doing business. This means instituting an effective and efficient training program. This paper deals with the initial establishment of a training program and its evaluation for the purposes of continual improvement. In the course of this investigation four additional questions pose themselves:

        a)     Is there a pay-off for the organization in
               terms of competitive advantage from
               supporting an on-going training/education
               program, and can it be measured?

        b)     What types of training should be supported
               and what are the best ways to provide them
               (in-house, third party, trade/technical
               schools, colleges and universities)?

        c)     What are the costs in terms of technical/
               administrative staff time and cash outlay?

        d)     What can the organization do to reduce the
               chance that newly trained staff members will
               leave, taking their new/updated skills with

Answering these questions should help to define an outline for an affordable program of technical/ managerial staff development that is scalable to organizations of varying size and type.

Analyzing The Current Situation: Defining Training

SDSOs range in size from single-person consulting ventures to corporations with many thousand employees. Existing training efforts tend to be scaled to the size of the business they support, with large companies having dedicated human resource development (HRD) departments and smaller organizations having smaller, less formal, training programs. Indeed, some smaller businesses may not even be aware that they have a training program. To develop a training program (or any other system), the first step is to identify what is already in place. For this to be accomplished there needs to be some definition of exactly what a training program is.

In general training can be defined as any instruction, formal or informal, provided to individuals to impart "skill, knowledge or attitude (SKA)" (Brinkerhoff, 1987, p. 10). The SKAs may be new or refreshed. The instruction may be in the form of interpersonal communication, computer assisted instruction, reading books and manuals, or other practice activities. Interpersonal communication might consist of something as formal as lecture in a classroom setting to something as informal as a brief exchange at the water cooler. Computer assisted instruction can vary from elaborate instruction systems like those on the Department of Health and Human Services' Parklawn Computer Center (PCC) mainframe or various compact disk driven PC systems to the simple use (for practice and experimentation purposes) of available development tools. Finally, written material can range from custom purpose training manuals to a quick note of instruction in answer to a question or in response to a perceived need.

Given this broad definition of training, it becomes obvious that all software organizations from the smallest and most static to the largest and most dynamic have some form of training program. The effectiveness of these programs, however, is difficult to demonstrate. Unless considerable thought and effort is expended on identifying training needs, designing the program and assessing results, the organization may not be realizing any benefit from the training effort. Also the program will not have any empirical data upon which to base and track improvement efforts and benefits.

Identifying Training Requirements: Needs Assessment

The purpose of any training program is to answer some need or exploit some opportunity within an organization and its people. Examples of needs include "reduced waste, improved morale, greater productivity" (Brinkerhoff, 1987, p. 17), and, ultimately, increased competitiveness. Opportunities might include such things as "a training grant, a resource newly available, or a new program" (Brinkerhoff, 1987, p. 17). Other opportunities might be that "a new retail outlet is to be opened, or a new product or service is about to be launched" (Robinson, D. G., Robinson, J. C., 1989, p. 15). Within the milieu of an SDSO, opportunities could include the integration of new software development tools like compilers or CASE tools, or new development management paradigms like the Software Engineering Institute's Capability Maturity Model (CMM) or Total Quality Management (TQM).

Each of these needs or opportunities may be brought to light in any one of several ways depending upon the size of the organization and the position, within that organization, of the person responsible for developing training programs. In a very small organization, where the person who must identify and act on these needs and opportunities is likely to be the business owner or chief executive, in addition to other duties having to do with running the business and doing development work, he/she must keep an eye open to new needs within the organization and new sources of opportunity wherever they arise. While a manager in this position is likely to have an intimate knowledge of the organization's strengths and weaknesses, they are less likely to have the leisure to act upon that knowledge. Because of this limited attention from upper management, small firms are apt to have very limited and simple training programs consisting, largely of a shelf full of manuals and a knowledgeable staff member or two to ask questions of over lunch. Organizations of this size would benefit from a more proactive approach to applying what the owner/CEO already knows and providing a more active and formal effort to identifying and acting upon business training needs and opportunities.

Within the atmosphere of a larger concern, where one or more people have as their primary responsibility the ongoing development of staff capabilities, these needs and opportunities may be identified through a wider array of channels, and additional incentives are present for this recognition. On the one hand full-time HRD professionals are more likely to be skilled at gauging an organization's needs, and on the other they may be less dialed into the business activities and competitive environment of the organization. For the HRD professional the identification of training needs and opportunities is key to both enhancing the competitiveness of the organization and continued employment. The channels through which an HRD professional may identify these needs and opportunities include requests from management for training programs, direct observation of ongoing work, interviews with staff at all levels, study of industry and market trends, and analysis of other organization's training programs.

Whatever the size of an organization, or the position of training staff within it, the key is to identify and assess the needs and opportunities for training. To aid in focusing this process Jack Phillips (1991, p. 64) poses a number of questions:

        -  Is there a performance problem?

        -  What is the gap between desired and actual

        -  How important is the problem?

        -  Is there a lack of skill contributing to
           the problem?

        -  Do employee attitudes need to be improved?

        -  Is this program designed to meet some
           outside requirement or satisfy an external
           influence? If so, what is it/are they?

        -  Which employees will need the training?

        -  Are there alternative ways to satisfy the

        -  To what extent will other departments be

        -  Are there natural barriers to accomplishing
           the program?

        -  Who supports this new program?

The application of the answers to these questions will help to determine what needs and opportunities are addressed by training, how the training is administered and whether it succeeds in imparting useful new SKAs. The last of the above questions is the most important; if any program does not have the support of management, it is likely to fail.

Developing the Training Program: A Structured Approach

Developing a training program is much like creating a computer software system or a new car. First a need or opportunity is identified: "we need a new inventory control system," "if we build a better minivan for less money, we'll clean-up," or "if we train our staff to be expert in this new integrated CASE tool we can eat the competition's lunch." Next, the situation is analyzed to study the feasibility, verify the approach, and identify the requirements. Then, if the project is deemed feasible, the approach is found correct and most of the requirements have been identified and described, the project can proceed to design and implementation.

Once a need or opportunity for improvement is established, an analysis, that will help answer some of the questions listed in the previous section should be performed. This process breaks into three steps: organizational analysis, task analysis and trainee analysis (Gordon, S. E., 1994, pp. 11-13). Organizational analysis looks at the target organization of a possible training program to ensure that instilling new or better SKAs in the personnel is the correct approach rather than providing, say, better equipment or more window offices. Task analysis looks at the specific need or opportunity to identify what, exactly, is to be taught, and whether, within the context of the organization, it is adequately defined. Trainee analysis seeks to describe the target population for a training program with respect to age, gender, numbers, professional experience, relevant level of knowledge and experience in the area to be covered, what the trainees themselves consider important, and what their attitude is towards training programs in the subject area. These analyses should confirm the need for the training program, identify and define many of its requirements, describe its audience, and provide some indication of its probable success.

With the initial analysis complete, a few questions can be answered. Should we proceed with developing a training program? How should the training be provided? How can the results of the training program be assessed? These are the first steps in the design process. The first question is a simple go/no-go decision. If the perceived benefit to the organization from the new/improved SKAs of its members is outweighed by the cost of training them, then why proceed? If, however, the benefits are sufficiently greater than the required expenditure of time, effort and money, implementing the training program can become an organizational priority. Alternatively, a training program may be mandated by the nature of the business or by some external entity (i.e. government), in which case cost/benefit calculations become irrelevant because the training must be undertaken regardless of estimated costs and benefits (Phillips, J. J., 1991, p. 70).

Deciding how to provide the training, informally through provision of books, other reading material, on-line help/instruction systems, or ready access to expert advice and mentoring; or more formally in a classroom/"instructor- led" (Kimmerling, 1993, p. 5) setting provided by the company internally, through a manufacturer or "third party" training firm, from a trade/technical school, or college/university, is dependent upon a number of factors: the type of SKA to be imparted, the audience to be trained, the resources available (time, money, equipment, and personnel) to the organization, the cultural biases of the organization for and against providing training, and various outside influences.

If the SKA to be imparted is in a specific area for which the organization has existing, available, expertise, then producing the program internally may make sense. If, however, the SKA is totally new to the organization (a new CASE tool perhaps), or beyond the scope of most internal training departments (instilling the knowledge and skills required for analysts to become program managers for example), then finding an outside provider of training or education is indicated. Assuming that resources are available, and that the scale and content of the program provides a fit with organizational training ability, then the training program design and implementation process can continue. Note, however, that even if training is to be out-sourced, an abbreviated design and implementation process needs to take place so that the external assets used may be evaluated and their future use/selection improved.

Finally, evaluation of training program results should be geared to measure the impact of the training program on those areas of the business for which the instigating need or opportunity was identified. This evaluation should be designed to focus on the specific objectives to be met by the training effort. Measures should be defined that assess both overall objectives for the program and specific milestones along the way. Keeping in mind that the ultimate goal of all training programs is to improve an organization's competitiveness, then evaluation should be able to show a positive tie-in between the effort and business factors that top management identifies as important to competitiveness.

Whether the training program is to be developed internally, or purchased from an outside supplier, the program design process should continue with the statement of clear, testable objectives that can be related to the identified need or opportunity. Some method for obtaining a baseline assessment of these objectives should be established. In some cases this might take the form of productivity or profitability measurements; in others a simple pretest of the material to be trained. The key is to get the baseline so the impact of the training program can be gauged. Once objectives are established and a method of baselining determined, the process of selecting a vendor whose program best meets the needs expressed in the objectives, or of designing a custom training program can continue. The process of selecting an outside vendor, and/or designing a training program is beyond the scope of this paper. Suffice it to say that the primary concern in selecting a vendor is that the program does a quality job of meeting the training objectives within the resource limitations of the organization. In the case of actually developing the training program in-house, the process entails identifying the skills, information and/or attitudes required to meet the established objectives, arranging the material and/or practice in a logical, understandable order at an appropriate level of detail, and creating presentation materials, lesson plans and exercises to teach the SKAs. And it is important that the training program, even a one- time effort, be evaluated to see that it has performed as intended.

Evaluation: Assessing and Improving Training

Given that we are to undertake a training program, the profit motive in business, the push towards efficiency in government, and the desire on the part of the trainer that the program be effective mandate the need for evaluation. The purpose of evaluation, in this context, is to develop a measure of the quality of the training program with respect to training objective accomplishment, cost-efficiency, and program improvement/update. To meet these evaluation goals four criteria areas should be investigated: trainee reaction; learning of principles, facts,techniques and attitudes; trainee behavior relevant to job performance; and results of the training program related to organizational objectives (Gordon, S. E., 1994, pp. 375- 377).

Trainee reaction is primarily a measure of whether participants liked the program (Phillips, J. J., 1991 p. 47). This data can be key for HRD because how much the trainees liked the program may influence the degree to which they accept the material, and whether it will continue to be offered. Trainees may also provide suggestions on how the program can be improved.

Was there a successful transfer of knowledge? Did trainees learn the principles, facts,techniques and attitudes that comprised the material and objectives for the program? Measuring this is key to understanding whether the program is providing the needed information in the right format for learning to occur. If not, trainees will never get to apply the SKAs on the job (Robinson, D. G., and Robinson, J. C., 1989, p. 187).

The observation of behavior relevant to job performance is key to assessing whether SKAs taught in the training program are being successfully transferred to the work environment. Essentially, can the trainees be observed to use new skills and knowledge acquired from training, or can their work products and performance be shown to have been impacted by the training (Gordon, S. E., 1994, p. 376)?

Does the training program result in improvement in/ progress toward organizational objectives? Over time do measures of productivity, morale and bottom line performance show a positive correlation to the level and type of training efforts undertaken? To measure these criteria there must be some baseline to which to compare post training values over time (Philips, J. J., 1991, p. 44).

How, when, and where should the these criteria be measured? Trainee reaction and initial learning should be measured at the site of instruction with tests and evaluations. And later, after a period of time suitable to the frequency with which the SKA is used, additional checks of reaction and learning should (ideally) be carried out on the job (Gordon, S. E., 1994, pp. 377-378).

Post training checks on job behavior and business results take place at work site. Ideally these measures should be of an on-going nature, both proceeding the training event and following it. Observation and interviews should be used prior to training to help determine how SKAs may best be added to or enhanced, and after training to see if the program has had an impact on behavior (Phillips, J. J., 1991, pp. 106-107). Similarly, business performance and process metrics should be collected and analyzed over time to identify how and if the organization is benefitting from training programs (Gordon, S. E., 1994, p. 377).

Two key points to keep in mind when evaluating training program results are the validity of the experiment design and the degree to which individuals taking part in the program are likely to benefit. In the case of experiment design training programs should, ideally, have experimental and control groups for comparison so that the effects of on the job learning can be sorted out from the impact of training (Gordon, S. E., 1994, pp. 373-375). Three experimental models are diagrammed below, each uses an experimental group and a control group selected at random from among potential trainees. The models A)-C) show progressively greater levels of criteria evaluation over time: A) provides only single post event checks of criteria; B) looks to see how the criteria vary over some period following the training; and C) adds the gathering pre-training data about the criteria.

        Experiment design, best models from behavioral science:

               (R)     X      T
        A)     ----------------
               (R)            T

               (R)     X      T1      T2... Tn
        B)     -------------------------------
               (R)            T1      T2... Tn

               (R)     T1     X       T2     T3... Tn
        C)     --------------------------------------
               (R)     T1             T2     T3... Tn

               (R)            = random personnel assignment
               X              = training event
               T(1,2, n)      = criterion measures taken

               (Gordon, S. E., 1994, pp. 373-374)

The other factor that may confound results is the case where individuals within the study group have reached, what W. Edwards Deming termed, a "state of statistical control" (1986, pp. 250-251). Deming used two golfers as an example: one has little experience and thus few ingrained bad habits but lacks any consistency in his scores; the other has been playing for years and while he is not satisfied with his performance his scores tend to be grouped fairly tightly around his average. In Deming's terms the experienced golfer has reached a state of statistical control. After training the beginner shows improvement as he incorporates what he has been taught in his game, while the experienced golfer's scores actually show a greater variance as a result making changes to his technique. Over time the experienced golfer may eventually benefit from lessons but, in general Deming thinks not.

        Importance of selecting the right people to train:

        Deming on golf lessons:

              Beginning Golfer       Experienced Golfer
              BT       AT            BT        AT

        ULC   -----*------------     -------------*---------
                 *   *                              *
                       *               *       *        *
        Mean  ------------*---*-     ----*---*--------*---*-
                            *              *     * 
                *       * 
        LLC   ------*-----------     -----------------------

                       BT   =  Before Training
                       AT   =  After Training
                       ULC  =  Upper Control Limit
                       LLC  =  Lower Control Limit

                   (Deming, W. E., 1986, pp. 252, 254)

Deming's view of the impact of training on the SKAs of workers whose behavior, within the area addressed by the training, is consistent, is that it will have little value. This puts a premium on using an experimental model like C), above that provides for pre-training checks.

Conclusions and Recommendations: Questions Answered

In the introduction of this paper four questions were posed:

        a)     Is there a pay-off for the organization in
               terms of competitive advantage from
               supporting an on-going training/education
               program, and can it be measured?

        b)     What types of training should be supported
               and what are the best ways to provide them
               (in-house, third party, trade/technical
               schools, colleges and universities)?

        c)     What are the costs in terms of technical/
               administrative staff time and cash outlay?

        d)     What can the organization do to reduce the
               chance that newly trained staff members will
               leave, taking their new/updated skills with

Providing generalizable answers to these questions is now possible at least in part.

Does training pay off in terms of competitive advantage, and can it be measured? Yes, a well-planned and executed training and education program, aimed at specific business needs and opportunities can lead to an advantage in the market place as it expands overall corporate capability. And, yes it can, with good design, be measured.

What types of training should be supported? The answer to this depends upon the priorities set for the needs and opportunities identified as potential areas for training. Each method of training is best suited to a different set of circumstances. Material to cover, resources available, training audience, and numerous other factors need to be considered when selecting from among these options.

What are the costs? While a detailed investigation of training costs has proven to be beyond the scope of this paper, among organizations reporting to the American Society for Training and Development's Benchmarking Forum Kimmerling reports a "typical value of 3.2 percent" (1993, p. 8) of payroll is spent on training. One should note, however, that these are companies who place enough of an emphasis on training and educating their staff that they have made a two year, $15,000 (Kimmerling, G., 1993, p. 1) commitment to the Forum.

How does one retain staff once they have been trained? This is another question beyond the scope of this paper, but it seems that an organization need only provide fair and competitive compensation, appreciation of hard work, interesting work tasks, and a good work environment to keep most employees happy.

To initiate a successful training program it is recommended that HRD staff do more than just provide training on demand. Training programs must be focused on the real training needs and opportunities of the organization. Emphasis must be placed on analysis of the program's objectives, audience, resources, and evaluation so that training efforts can be mapped to business results.


Brinkerhoff, R. O. (1987).  Achieving results from
        training: How to evaluate human resource
        development to strengthen programs and increase
        impact.  San Francisco, CA: Jossey-Bass, Inc.

Deming, W. E. (1986).  Out of the crisis.  Cambridge,
        MA:  Massachusetts Institute of Technology,
        Center for Advanced Engineering Study.

Gordon, S. E. (1994).  Systematic training program
        design: Maximizing effectiveness and minimizing
        liability.  Englewood Cliffs, NJ: P T R Prentice
        Hall, Prentice-Hall, Inc.

Kimmerling, G. (1993, Sept).  Gathering best practices.
        Training & development. 28-? (8 pages (10 in on-line
        text format)).

Phillips, J. J. (1991).  Handbook of training
        evaluation and measurement methods.
        Houston, TX: Gulf Publishing Company.

Robinson, D. G., and Robinson, J. C. (1989).  Training
        for impact.  San Francisco, CA: Jossey-Bass, Inc.