7. HSI Implementation of COTS NDI

Overview

DoD 5000.2R directs that, when acquiring COTS software products or other commercial items, the PM shall implement a spiral development process. In this context, integration may encompass the amalgamation of multiple COTS components into one deployable system (or block of a system) or the assimilation of a single COTS product (such as an enterprise resource planning system). In either case, the PM shall ensure that the system co-evolves with essential changes to doctrine (for combat systems) or reengineered business processes (for combat support and IT systems). The PM shall apply commercial item best practices.

No matter how much of a system is provided by commercial items, the PM shall engineer, develop, integrate, test, evaluate, deliver, sustain, and manage the overall system. Using commercial items offers significant opportunities for reduced cycle time, faster insertion of new technology, lower life cycle costs, greater reliability and availability, and support from a more robust industrial base. The keys to success involve thinking and acting as an informed consumer; planning for continuous evolution of the system; and maintaining a flexible posture throughout the life of the program. The use of commercial items often requires changes in the way systems are conceived, acquired, and sustained, to include:

• When purchasing a commercial item, the PM shall adopt commercial business practice(s). The extent to which the DoD business practices match the business practices supported by commercial items determines the likelihood that the items will meet DoD needs. It is likely, however, that a gap will exist-and the gap may be large. Negotiation, flexibility and communication on the part of the stakeholders, the commercial vendors, and the program manager are required.
• The PM shall plan for robust evaluations to assist in fully identifying commercial capabilities, to choose between alternate architectures and designs, to determine whether new releases continue to meet requirements, and to ensure that the commercial items function as expected when linked to other system components. In addition, evaluation provides the critical source of information about the tradeoffs that must be made between the capabilities of the system to be fielded and the system architecture and design that makes best use of commercial capabilities. Evaluating commercial items requires a focus on mission accomplishment, and matching the commercial item to system requirements.
• The PM shall remain aware of and influence product enhancements with key commercial item vendors to the extent practical and in compliance with FACA. Vendors are different from contractors and subcontractors; different practices and relationships are needed. Vendors react to the marketplace, not the unique needs of DoD programs. To successfully work with vendors, the PM shall adopt practices and expectations that are similar to other buyers in the marketplace. Traditional DoD acquisition and business models are not sufficient for programs acquiring commercial items, as they do not take into account the marketplace factors that motivate vendors.
• The PM shall engineer the system architecture and establish a rigorous change management process for life-cycle support. Systems that integrate multiple commercial items require extensive engineering to facilitate the insertion of planned new commercial technology. This is not a "one time" activity because unanticipated changes may drive reconsideration of engineering decisions throughout the life of the program. Failure to address changes in commercial items and the marketplace will potentially result in a system that cannot be maintained as vendors drop support for obsolete commercial items.
• The PM shall develop an appropriate T&E strategy for commercial items to include evaluating potential commercial items in a system test bed, when practical; focusing test beds on high-risk items; and testing commercial-item upgrades for unanticipated side effects in areas such as security, safety, reliability, and performance.
• Programs are encouraged to use code-scanning tools, within the scope and limitations of the licensing agreements, to ensure both COTS and GOTS software do not pose any information assurance or security risks.

NDI has been defined as:

• Any item available in the commercial marketplace;
• Any previously developed item in use by a Federal, State, or local agency of the U.S. or a foreign government with which the U.S. has a mutual defense cooperation agreement;
• Any item that requires only minor modification to meet the requirements of the
• Any item currently being produced that does not meet the requirements of sections above, solely because the item is not yet in use or is not yet available in the commercial marketplace.

A commercial product is defined as a nondevelopmental item that has been produced for sale in the commercial marketplace. To have established market acceptability means that a product has been successfully marketed in substantial quantities to either the private sector or the Government.

• Prototypes, models, or experimental production runs generally do not qualify.
• It may be appropriate for some items to make provision for products currently in production, without sales history, that are slightly modified or improved versions of the items previously sold.

Material requirements shall be satisfied to the maximum practicable extent through the use of NDI when such products will meet the user's needs and are cost-effective over the entire life cycle.

Material requirements will be stated to the extent practicable in terms of required function, performance, or physical characteristics.

• Non-Government standards and commercial item descriptions will be used in reference to Federal and military specifications and standards whenever practicable.
• The use of NDI should be incorporated in the design and development process consistent with operational requirements.
• Market research and analysis should be conducted to determine the suitability and availability of any item prior to the commencement of a developmental effort.

NDI will be evaluated for operational use by considering all aspects of the items' suitability for the intended purpose.

• Suitability criteria should include technical performance, safety, reliability, maintainability, interoperability, logistics support, expected operational environment, survivability, and intended life cycle.
• The suitability analysis should consider that unmodified NDIare preferred. However, items requiring minor modifications may be used when cost, performance, and support benefits warrant.
• Prudent risks should be taken to evaluate and field NDI.
• Test and evaluation of NDI will be conducted to, at a minimum, verify integration and interoperability with other system elements. All NDI modifications necessary to adapt them to the weapon system environment will also be subject to test and evaluation. As appropriate, test and evaluation should be conducted for other aspects of NDI to evaluate and control risk.

Significant consideration must be given to logistics support when acquiring NDI.

• Programs using commercial system or equipment should make maximum use of existing commercial logistics support and data. Development of new organic logistics elements will be based on critical mission need or substantial cost savings.
• It may be necessary to modify existing logistics support procedures, varying from established practices, to allow for maximum use of NDI. This may involve innovative logistics concepts to support accelerated logistics support schedules and require acquisition techniques such as buyouts, warranties, and data rights escrow. The use of these techniques and concepts is preferred to developmental effort.
• Manufacturer or supply source distribution channels should be used in supplying commercial products and other NDIto operational users when it is economically advantageous, and the impact on military readiness and wartime sustainability is acceptable.

The acquisition strategy should be tailored to the extent feasible to employ commercial practices when purchasing commercial products or other nondevelopmental items. Such practices include, but are not limited to:

• Seeking the greatest benefit to the Government in terms of overall cost, product quality, timeliness of delivery, and supportability (past performance should be a significant factor in making such determinations);
• Accepting commercial operational, maintenance, and safety data and commercial logistics support, consistent with the user's operational needs;
• Using commercial marketing, preservation, and packaging to the maximum extent consistent with user needs; and
• Requiring that a product solicited using a commercial item description have established market acceptability.
  

Relationship to the systems acquisition process

Inputs

• Decisions to employ NDI
• Information on human-machine interfaces for alternative NDI
• Information on training packages/programs for alternative NDI
• Information on safety/health hazards for alternative NDI approaches
• Information on manpower and personnel requirements for alternative NDI approaches
• Information on human performance requirements for alternative NDI
• Information on supportability requirements for alternative NDI
• Information on cost constraints for alternative NDI approaches
• Information on survivability requirements for alternative NDI
  

Outputs

• HSI inputs to Commercial Item Descriptions;
• HSI inputs to NDI Acquisition Program Requirements and documents, including procurement documents
  

Substeps/Activities/Guidelines

The process for applying HSI to the acquisition of NDI is depicted above. The steps of this figure are described below.

Step 1. Identify HSI Inputs to NDI Concepts and Issues

1.1 Determine the extent to which the NDI must meet users' needs in the users' environment.

1.2 Determine HSI issues in NDI operational requirements.

• human performance issues
• human safety and health issues
• human quality of life - habitability issues
• personnel management issues
• workload and manning issues
• training issues

1.3 Provide inputs to ensure that the developer is responsive to legitimate needs but is also conscious of technical risks and affordability constraints.

1.4 Provide inputs to ensure that the user is realistic in stating needs and considering trade-offs. After weighing the benefits of proven capability and more rapid deployment against any performance limitations, the user must determine whether the trade-offs are acceptable.

1.5 Tailor the acquisition process for NDI

1.6 Provide inputs to NDI effectiveness trade-offs.

1.6.1 Assess NDI reliability vis-a-vis the total system reliability.

1.6.2 Provide inputs to evaluate producers' processes, production methods, and production control procedures.

1.7 Provide HSI inputs to the determination of Life Cycle Cost. Determine how to identify which NDI approach has the:

1.7.1 lowest projected life-cycle cost, within acceptable risks, and meets essential requirements, including human performance and safety requirements.

1.7.2 lowest human workload and manning requirements for operations and maintenance.

1.7.3 most effective training program

1.7.4 least safety and health hazards

1.7.5 best mean time to repair

1.7.6 best overall availability

1.7.7 best overall supportability

1.8 Provide HSI inputs to the determination of ILS requirements

1.9 Identify Safety and Environment Issues.

1.9.1 Determine the extent to which NDIs must comply with military standards.

1.9.2 Determine how to identify hazards

1.10 Determine Manpower and Personnel Issues.

1.10.1 Determine how to identify manning requirements

1.10.2 Determine how to identify skill requirements

1.10.3 Determine how to identify workload requirements

1.11 Determine Training Issues.

1.11.1 Determine how to identify training requirements

1.11.2 The acquisition of NDI may affect both formal classroom and on-the-job training.

1.11.3 Determine how to identify training device requirements

1.11.4 Training plans should consider not only the impact on training but also the possibility that the item may require new, additional, or modified training devices.

1.12 Determine Human Performance Issues

1.12.1 Human error potential

1.12.2 Time to respond

1.12.3 Time to perform

1.12.4 Workloads

1.13 Determine HSI Survivability Issues

1.13.1 Requirements for Cold weather gear

1.13.2 Requirements for NBC suits

1.13.3 Requirements for collective protection

1.13.4 Requirements for individual protection

1.14 Determine HSI Test and Evaluation Issues

1.14.1 HSI T&E requirements

1.14.2 HSI T&E methods, measures, criteria and procedures

1.15 Determine HSI issues for NDI Modification.

1.15.1 Determine what modifications are required to the human- machine interfaces, operating environments, maintenance access provisions, maintenance workspace, manning levels, skill requirements, training, and safety provisions.

1.15.2 Determine HSI inputs to evaluation of the total effect of modifications, particularly in the area of logistics support.

5

Step 2. Provide HSI Inputs to market surveys

2.1 Provide HSI Inputs to Market Surveillance

2.1.1 Review technology requirements

2.1.2 Identify technology available

2.1.3 Assess technologies

2.2 Provide HSI Inputs to Market Investigation Plan

2.2.1 Finalize the operational requirement;

2.2.2 Develop a form, fit and function description to obtain competition;

2.2.3 Determine logistic support requirements;

2.2.4 Determine what additional testing is required.

2.3 Provide HSI Inputs to Market Investigations.

2.3.1 Collect data to support a definitive NDI decision.

2.3.2 Provide Input to Market Investigation Requirements.

2.3.2.1 Identify System Performance Requirements:

• Identify detailed operating parameters for hardware and software
• Identify environmental conditions
• Define usage (fixed, airborne, tactically deployable, etc)
• Identify system interface, integration requirements
• For computers: identify required software language, speed, throughput, ports, memory and expansion potential
• Identify human operator performance requirements

2.3.2.2 Identify Reliability, Maintainability, and Survivability Requirements

• Identify human reliability requirements - error rates
• Identify human maintainability requirements

2.3.2.3 Identify Logistics Support Requirements

2.3.4 Analyze data from market investigations

2.3.4.1 Product quality, electromagnetic compatibility, reliability, human performance, training, and maintainability experience of similar users.
2.3.4.2 Are modifications to the NDI needed?
2.3.4.3 Stability of current configuration and technology.
2.3.4.4 Packaging, handling, storage, and transportation practices.
2.3.4.5 Commercial market acceptability-related data
2.3.4.6 Need for any pre-production or production qualification testing and special quality assurance requirements.
2.3.4.7 Hardware, software, and manpower interface issues such as human factors and product safety as experienced by similar users.
2.3.4.8 Repair parts availability and lead times, documentation, pricing, and distribution systems.
2.3.4.9 Customer service, installation, checkout, and user maintenance instructions.
2.3.4.10 Requirements and provisions for manpower and personnel.
2.3.4.11 Competitive or sole source repair and support base.
2.3.4.12 Training and training support requirements.
2.3.4.13 Requirements for and availability of tools, test equipment, computer support resources, calibration procedures, operations, and maintenance manuals.
2.3.4.14 Warranty procedures and commercial repair capabilities.
2.3.4.15 Degree of technical data package availability/adequacy.

5

Step 3. Identify human performance requirements for NDI

3.1 determine human error potential

3.1.1 determine human error potential in NDI operation

• likelihood of error occurrence
• likelihood of error detection
• likelihood of error correction
• impact of errors on performance and safety
• data on error rates and effects
• identify the extent to which the error potential can be reduced through design modifications

3.1.2 determine human error potential in NDI maintenance

• likelihood of error occurrence
• likelihood of error detection
• likelihood of error correction
• impact of errors on performance and safety
• data on error rates and effects
• identify the extent to which the error potential can be reduced through design modifications

3.1.3 Determine potential problems with time to respond/perform

• determine potential problems with reaction time
• determine potential problems with time to perform
• identify the extent to which the timing problems can be alleviated through design modifications

3.1.4 Determine the extent to which NDI tasks are unduly complex

3.1.4.1 Determine the potential for task simplification to reduce workload/manning.
3.1.4.2 Identify the potential for reducing physical task demands
3.1.4.3 Identify the potential for reducing cognitive task demands
3.1.4.4 Identify the potential for reducing perceptual-motor task demands
3.1.4.5 Identify Task Simplification approaches

• Reduce the amount of information to be processed,
• Reduce the complexity of the information processing,
• Reduce the number of decisions and options to be handled,
• Reduce the complexity of actions,
• Reduce the needs for interactions with other operators,
• Reduce the extent and complexity of communications,
• Reduce the task performance accuracy required,
• Reduce the special skills and knowledge required,
• Reduce the levels of skills such as reading comprehension,
• Reduce the level of stress associated with the performance of tasks under representative mission conditions,
• Reduce the time constraints.

3.1.4.6 Identify how task simplification will modify task sequences and/or reduce the likelihood of human error

3.2 Establish criteria which address the adequacy of the system design for fightability

• time to detect and range at detection
• first round accuracy subsequent round accuracy projected rate of fire
• expected target engagement performance
• expected performance throughout graceful degradation
• expected performance of battlefield management activities
• expected performance of command and control activities
• expected performance of surveillance activities
• expected performance of damage assessment
• adequacy of the design for fightability

3.3 Establish criteria which address the adequacy of the system design for operability

• adequacy of communications
• workloads associated with operations
• projected error occurrence rates
• expected effects of errors
• projected error recovery rates
• expected information handling performance
• human-machine interface design compliance with standards
• effects of sustained performance
• adequacy of the design for operability

3.4 Establish criteria which address the adequacy of the system design for maintainability

• expected fault detection performance
• expected fault isolation - troubleshooting performance
• expected accessibility design
• adequacy of the design for maintainability
  

5

Step 4. Identify HSI supportability requirements for NDI

4.1 Identify procedures/documentation requirements

4.2 Evaluate procedures/documentation

4.3 Identify impacts of HSI on life cycle costs

4.3.1 Determine which NDI approach has the lowest projected life-cycle cost, within acceptable risks, and meets essential requirements, including human performance and safety requirements.

4.3.2 Determine which NDI approach has the lowest manning requirements including operations and maintenance.

4.3.3 Determine which NDI approach has the most effective training program

4.3.4 Determine which NDI approach has the least safety and health hazards

4.3.5 Determine which NDI approach has the best mean time to repair

4.3.6 Determine which NDI approach has the best overall availability

4.3.7 Determine NDI approach has the best overall supportability

4.3.8 Determine NDI approach has the least overall risk

4.3.9 Determine NDI approach has the best overall affordability

5

Step 5. Identify safety and health requirements for NDI

5.1 Determine hazards due to materials

5.2 Determine hazards due to sharp edges/corners

5.3 Determine hazards due to lasers

5.4 Determine hazards due to RF

5.5 Determine hazards due to breakage

5.6 Determine hazards due to explosion

5.7 Determine hazards due to electric shock

5.8 Determine hazards due to environmental factors

5.9 Determine hazards due to man-machine interface design

5

Step 6. Identify manpower & personnel requirements for NDI

6.1 Identify manpower requirements from system acquisition documentation

6.2 Identify manpower requirements from ILS/LSAR documentation

6.3 Identify manpower requirements from lessons learned

6.3.1 Identify problems in existing systems

• excessive workloads
• unbalanced or highly variable workloads
• understaffing problems
• overstaffing problems
• unavailability of personnel with requisite skills
• workload distribution in existing systems
• functional allocation to human and automated performance in existing systems
• constraints on personnel availability
• results of attempt to reduce system manning

6.4 Identify manpower requirements from HSI issues

6.5 Identify manpower requirements from function/task analysis data

6.6 Identify manpower requirements from manning/workload studies

6.7 Identify requirements for reduced manning

6.7.1 investigate alternate manpower/skill reduction options, such as:

• cross training
• automation of manual operations
• improved design
• changes in policy and doctrine

6.7.2 identify task sequences where manpower/skill reduction is feasible and potentially beneficial

6.7.3 conduct workload analyses of alternate manpower reduction approaches

5

Step 7. Identify training requirements for NDI

7.1 Identify training requirements from system acquisition documentation

7.2 Identify training requirements from lessons learned

7.2.1 Identify training problems in commercial systems

7.2.2 Identify training effectiveness in commercial systems

7.2.3 Identify training devices and equipment used in commercial systems

7.2.4 Identify applications of embedded training technology

7.2.5 Identify applications of onboard training technology

7.2.6 Identify requirements for special skills

7.3 Identify training requirements from HSI issues

7.4 Identify training requirements from function/task analysis data

7.5 Integrate training requirements

7.5.1 Identify skill requirements by tasks

7.5.2 integrate manning and skill requirements across task sequences

7.5.3 identify training system requirements in terms of

• training objectives
• training method requirements by training objectives
• training measurements requirements by training objectives
• training material requirements by training objectives
• training media requirements by training objectives
• training management requirements

7.5.4 identify training device design requirements

• training objectives addressed
• information reception media
• skills acquisition media
• fidelity levels
• display formats
• range of conditions
• use of augmented feedback
• use of prompting and cueing
• use of branching /programming
• use of CAI/CMI
• degree of flexibility to different system configurations
• instructor interfaces
• data acquisition and recording requirements
• embedded training requirements

7.5.5 review training devices approaches implemented in predecessor/baseline systems

7.5.6 identify problems with existing training devices for selected training objectives

5

Step 8. Identify survivability requirements for NDI

8.1 Assess availability/adequacy of protection systems and devices

8.2 Assess expected human performance wearing protective ensembles

8.3 Assess adequacy of countermeasures

8.4 Assess adequacy of the design for survivability

5

Step 9. Identify test and evaluation requirements for NDI

9.1 Identify extent to which T&E data will influence selection of design concepts, and incorporate into T&E planning

9.2 Identify the extent to which HSI T&E will be accomplished under operationally realistic conditions using personnel deemed to be typical users

9.3 Identify extent to which results of evaluations of predecessor systems will input HSI T&E exercises

9.4 Identify NDI test and Evaluation issues

9.4.1 Identify HSI inputs to T&E

9.4.2 Identify categories of HSI T&E required

9.4.3 Identify overall T&E objectives

9.4.4 Identify overall objectives of HSI T&E

9.4.5 Integrate HSI T&E objectives with system-level T&E objectives

9.5 Identify system features to be evaluated

9.6 Identify test conditions

9.6.1 Identify sources of information on test conditions

9.6.2 Identify tactical conditions to be evaluated or controlled

0. Identify conditions of readiness to be evaluated or controlled -
1. Identify workload conditions to be evaluated or controlled -
2. Identify personnel conditions to be evaluated or controlled -
3. Identify operational conditions to be evaluated or controlled -
4. Identify environmental conditions to be evaluated or controlled -
5. Integrate information on test conditions

9.7 Identify T&E data requirements

9.7.1 Identify test data validity requirements

9.7.2 Identify test data reliability requirements

9.7.3 Identify test data usability requirements

9.7.4 Identify test data accuracy requirements

9.8 Identify T&E methods

9.8.1 Identify candidate methods

9.8.2 Identify constraints on method selection

9.8.3 Identify considerations in method selection

9.8.4 Identify advantages/disadvantages of alternate methods

9.8.5 Select T&E methods

9.9 Identify T&E measures

9.9.1 Identify HSI T&E data requirements

9.9.2 Define HSI measures of effectiveness

9.9.3 Identify data to be acquired on each measure

9.9.4 Identify data acquisition and recording requirements

9.9.5 Identify data analysis and integration requirements

9.10 Identify T&E criteria

9.10.1 Identify T&E criteria based on system acquisition/development requirements

9.10.2 Identify T&E criteria based on lessons learned

9.10.3 Identify criteria based on HSI principles/design criteria

9.11 Develop test plan

9.11.1 Collect HSI T&E requirements and approaches

9.11.2 Develop experiment design

9.11.3 Identify requirements for mock-ups, models, facilities and instrumentation

9.11.4 Describe individual tests and evaluation procedures

9.11.5 Describe test procedures for experimenter and subjects

9.11.6 Integrate tests

9.12 Conduct HSI T&E

9.12.1 Select and prepare test subjects

9.12.2 Acquire and prepare test materials

9.12.3 Acquire and prepare test instrumentation

9.12.4 Identify test initial conditions

9.12.5 Initiate tests

9.12.6 Monitor the progress of tests

9.12.7 Terminate testing

9.13 Prepare test reports

9.6.13.1 prepare reports on HSI T&E describing major findings and implications for system design and operation

9.6.13.2 integrate HSI T&E with logistics and system level T&E

5

Step 10. Identify ILS requirements for NDI

10.1 identify criteria which address the adequacy of the system design for supportability

10.2 determine the adequacy of system documentation

10.3 determine the adequacy of spares access

10.4 determine the adequacy of the design for supportability

5

Step 11. Provide HSI input to the Procurement Documents

11.1 Provide HSI inputs to the Statement of Work.

11.1.1 Identify non-specification requirements for contractor efforts

11.1.2 Identify requirements for reliability programs,

11.1.3 Identify requirements for maintainability programs,

11.1.4 Identify requirements for reliability prediction,

11.1.5 Identify requirements for HSI design reviews

11.1.6 Establish tasks and identify the work effort to be performed expressed as minimal needs.

11.1.7 Statements of work should incorporate requirements from military specifications and standards only when absolutely necessary.

11.2 Provide HSI inputs to the Contract Data Requirements List.

11.2.1 Identify applicable HFE DIDs

11.2.2 Identify applicable MPT DIDs

11.3 Provide HSI inputs to the Contract Specification.

11.4 Provide HSI inputs to the selection of Requirements Documents

11.4.1 Ensure that stated requirements accurately reflect the user's operational and supportability requirements.

11.4.2 Establish criteria by which proposed items will be evaluated and best value determined.

11.5 Develop an Effective Requirements Document

11.5.1 Tailor the requirements documents

11.5.2 Identify tools and techniques for developing a requirements document.

11.5.3 Communicate with the User.

11.5.3.1 understand the operational requirement

11.5.3.2 establish communication between the user and the acquisition activity

11.6 identify new products, technologies, materials, etc. being used by industry. This information allows the user to evaluate operational requirement. The user also can validate the acceptability of new products, technologies, or materials being considered.

11.7 Establish flexibility in user's needs to allow consideration of a broader range of alternatives . Requirements document should reflect the user flexibility by stating requirements in terms of acceptable ranges or thresholds to be met rather than exact requirements.

11.8 Focus on Performance-Oriented Requirements. State requirements in performance rather than "how-to" terms allowing suppliers to offer existing products which meet the requirements.

11.9 Provide inputs to Application and Tailoring which allows the preparer to consider the pertinence and cost effectiveness of requirements imposed through standardization documents.

11.10 Tailoring is the process by which individual requirements of applicable specifications, standards, or related documents are evaluated to determine the extent to which they are most suitable for a specific acquisition and the modification of these requirements to ensure that each achieves an optimal balance between operational need and cost.

5

Step 12. Provide HSI input to the Commercial Item Description

12.1 Establish requirements for the Commercial Item Description (CID) - a simplified specification that describes, by salient functional or performance characteristics, the available, acceptable commercial or commercial-type products that will satisfy the Government's needs. It is a type of Federal specification to prepare technical documents that are easier for suppliers to use and that allow manufacturers to provide products to DoD from their standard production line.

12.2 the item to provide potential suppliers and users a brief description of the item(s), which will allow suppliers to decide if they may be able to supply the item and users to decide if the CID is appropriate for use in meeting their requirements. The abstract may also be used in the synopsis of the procurement in the Commerce Business Daily.

12.3 Identify Salient Characteristics:

12.3.1 Describe salient characteristics which capture technical aspects of the item and provide a basis for its acceptance or rejection. User requirements and research and analysis of comparable items available commercially provide the basis for preparing the salient characteristics section of a commercial item description. Referencing non-government test methods and standards is the preferred method for incorporating technical characteristics, materials, and testing procedures. For HSI applied to maritime systems, the reference standards should be ASTM 1166 and the American Bureau of Shipping (ABS) Human Factors Standard. Commonly used commercial test methods and units of measurement should be used. If a non-government standard is not available for the entire item or sufficiently definitive, consider using non-government standards in part or as a basis. Determining the appropriate salient characteristics is an item specific, technically demanding task which is based on the judgment of the technical expert or engineer. However, user input and feedback, market research, and industry comments are essential considerations.

12.3.2 The following tools and techniques are available to the specification preparer in developing salient characteristics initially and in keeping them current, in response to changing user requirements and technology.

• Test and evaluation of product samples.
• Industry publications such as catalogs and product data sheets.
• Technical journals.
• Previous Government contract performance.
• Discussions with manufacturers and users.
• Industry references
• Trade shows
• Foreign military data exchange programs
• Standardization organizations

12.4 Identify Quality assurance provisions:

12.4.1 Manufacturer's Standard Quality Assurance Program - The quality assurance of products described using a CID should rely entirely or primarily on the manufacturer's standard quality assurance program in providing products for the commercial market. Quality assurance provisions should be directed toward determining compliance with the salient characteristics of the CID, Government acceptance of the product, and compliance human factors engineering standards such as MIL-STD-1472, ASTM 1166, and ABS.

12.4.2 Market Acceptability - Quality assurance provisions in a CID may be stated in terms of market acceptability. Market acceptability must be based on a reasonable assessment of the Government's minimum needs and should be developed considering the specific product and its market. Flexibility is encouraged in constructing market acceptability requirements to fit the product and may vary, i.e. volume of product, time on the market, etc.

12.4.3 Assess the extent to which the item meets requirements associated with achieving interoperability and integration of a family of systems. According to DoD 5000.1, interoperability is the ability of systems, units, or forces to provide data, information, materiel, and services to and accept the same from other systems, units, or forces, and to use the data, information, materiel, and services so exchanged to enable them to operate effectively together. Interoperability within and among United States forces and U.S. coalition partners is a key goal that must be addressed satisfactorily for all Defense systems so that the DoD has the ability to conduct joint and combined operations successfully. The use of standardized data shall be considered to facilitate interoperability and information sharing. To the extent possible, systems and software shall be designed to permit use in a multi-national environment.

• standards for common displays, display formats, feedback, alerts and alarms
• standards for common and consistent procedures and methods for system operation and maintenance
• standards for common protocols and architectures for information retrieval, integration, and dissemination
• standards for common verbal and written communications message structure, format, syntax, and semantics
• techniques for visualization of system architecture
  • to depict system structure
  • to depict the system environment
  • to facilitate navigation through the architecture
• standards for common tactical frames of reference
  • to enhance communications
  • to support decision making
  • to facilitate maintaining situation awareness and tactical perspective

12.4.4 Assess the item in terms of usability ­ a major requirement for ensuring quality of a commercial item is that it must be usable by the sailor irrespective of his or her level of computer literacy. Usability requires that user expectancies are met, that operations are intuitive, that displays are readable, that presented information and communications are immediately meaningful , and that human interfaces are designed in conformity with human engineering design standards. The activities involved in assessing the usability of technology are as follows:

• identify and characterize operator/user problems with use of the technology
• assess user productivity
• determine the extent to which human-computer interfaces have been designed in accordance with user cognitive, perceptual, and short-term memory capabilities and limitations
• determine the degree of transparency of software command modes to the user
• determine the degree to which displays and display formats are standardized and are easily read and interpreted
• determine the extent to which the user is always aware of where he or she is in a program, and what action is required next
• determine the extent to which the user understands the capabilities of the system and the procedures required to exercise these capabilities
• determine the extent to which procedures are logically consistent
• determine the extent to which user documentation is clear, easily accessed, and readable
• determine the extent to which on-line help is available and responsive
• determine the extent to which the user is only provided with that information needed when it is needed
• determine the extent to which the user understands how to navigate through a program and retrieve needed information

5

Step 13. Evaluate application of HSI to NDI

13.1 Identify equipment to be assessed for HSI problems or positive aspects and determine where (on what systems or other platforms) the equipment has had operational experience.

13.2 identify technique for collecting data on potential HSI problems/positive aspects for NDI

• Manpower , Personnel and Training analysis - go to step 13.3
• Safety & Health evaluation - go to step 13.4
• Human engineering evaluation - go to step 13.5
• 3-M data evaluation - go to step 13.6
• test and evaluation - go to step 13.7
• training audits - go to step 13.8
• incident reports at TYCOM or NAVSEA - go to step 13.9
• casualty reports (CASREPS) exist - go to step 13.10
• INSURV data - go to step 13.11
• FLTEX data - go to step 13.12
• Commanding Officer Narrative Reports (CONARs) - to step 13.13
• JAG investigation reports - go to step 13.14
• message traffic - go to step 13.15
• ship personnel interviews - go to step 13.16
• HSI observations/evaluations - go to step 13.17

13.3 Determine if a Manpower , Personnel and Training (MPT) Analysis has been conducted for the equipment

13.3.1 If no - go to step 8.0

13.3.2 If yes - assess the results of the Manpower , Personnel and Training Analysis

13.3.2.1 Assess the Manpower Requirements Analysis

• Identify the qualitative and quantitative manpower requirements
• Determine the manpower required to operate and maintain the system
• Determine maintenance workload for each equipment item
• Determine standard-position, supervisory, and policy-driven manpower requirements

13.3.2.2 Assess the Personnel Pipeline Analysis

• Assess the estimates of the personnel required

13.3.2.3 Assess the Training Resource Requirements Analysis

• Assess the adequacy of training data
• Assess the adequacy of task comparability analysis
• Assess the adequacy of unit training products determination
• Assess the adequacy of course requirements analysis
• Assess the adequacy of course material requirements determination
• Assess the adequacy of course cost and resources determination

13.3.2.4 Assess manpower requirements

• Identify manpower requirements from system acquisition documentation
• Identify manpower requirements from ILS/LSAR documentation
• Identify manpower requirements from lessons learned
• Identify problems in existing systems with excessive workloads
• Identify problems in existing systems with understaffing/overstaffing
• Identify problems in existing systems with unavailability of skilled personnel
• Identify problems in existing systems with workload distribution
• Identify requirements for reduced manning

13.3.2.5 Assess training provisions

• determine that training devices are available
• determine that training devices conform to requirements
• determine that training facilities are adequate for the student throughput
• determine that critical issues have been identified for training manuals
• Assess the applications of embedded training technology
• Identify requirements for special skills

13.4 Determine if a Safety Hazard Analysis has been conducted for the equipment

13.4.1 If no - proceed to step 13.7

13.4.2 If yes - evaluate the results of the analysis

13.4.2.1 Review the Preliminary Hazard Analysis (PHA) to identify safety critical areas, evaluate hazards, and identify the safety design criteria to be used.

• Determine the potential for hazardous fuels/propellants
• Determine the potential for hazardous lasers
• Determine the potential for hazardous explosives
• Determine the potential for hazardous substances
• Determine the potential for hazardous materials
• Determine the potential for hazardous pressure systems
• Determine the potential for hazardous energy sources
• Determine the potential for systemic poisons
• Determine the potential for asphyxiants
• Determine the potential for respiratory irritants
• Determine the potential for noise hazards
• Determine the potential for heat or cold stress hazards
• Determine the potential for radiation hazards
• Determine the potential for hazards due to sharp edges/corners
• Determine the potential for hazards due to other environmental factors
• Assess the adequacy of safety interlocks
• Assess the adequacy of system redundancy
• Assess the adequacy of hardware or software final safety design
• Assess the adequacy of fire suppression systems
• Assess the adequacy of personal protective equipment
• Assess the adequacy of ventilation
• Assess the adequacy of noise or radiation barriers
• Assess protective clothing

13.5 Determine if a human engineering assessment has been conducted for the equipment

13.5.1 If no - proceed to step 13.7

13.5.2 If yes - identify human engineering/human performance problems

• Determine human error potential for equipment operation and use
• Determine human error potential in equipment maintenance
• Determine potential problems with time to respond/perform
• Determine the extent to which equipment tasks are complex
• Determine problems with the amount of information
• Determine problems with the complexity of the information processing,
• Determine problems with the number of decisions and options to be handled,
• Determine problems with the extent and complexity of communications,
• Determine problems with task performance accuracy required,
• Determine problems with special skills and knowledge required,
• Determine problems with the levels of skills
• Determine positive aspects of human engineering design approach

13.6 Determine if 3-M data exist for the equipment

13.6.1 If no - proceed to step 13.7

13.6.2 If yes - review 3-M data as follows

13.6.2.1 determine what EIC code is associated with the equipment

13.6.2.2 retrieve 2-K reports for the equipment

13.6.2.3 review block 8 cause codes

• is cause 2 - manufacturer/installation defects, cited?
• is cause 3 - lack of knowledge or skill, cited?
• is cause 4 - communications problems, cited?
• is cause 5 - inadequate instruction/procedure, cited?
• is cause 6 - inadequate design, cited?

13.6.2.4 review block 9 deferral codes

• is reason 3 - no formal training, cited?
• is reason 4 - formal training inadequate, cited?
• is reason 5 - inadequate school practical training, cited?

13.6.2.5 review block 15 - safety hazard identified?

13.7 Determine if test and evaluation data exist at OPTEVFOR or NAVSEA

13.7.1 If no - proceed to step 13.8, 13.9, 13.10, 13.11, 13.12, 13.13, 13.14, or 13.15

13.7.2 If yes - review sections of T&E reports that address design issues

13.7.2.1 assess system/equipment design for operability

13.7.2.2 assess system/equipment design for maintainability

13.7.2.3 assess system/equipment design for safety

13.7.2.4 assess system/equipment design for survivability

13.7.2.5 assess system/equipment design for supportability

13.7.2.6 assess system/equipment design for usability

13.7.2.7 assess system/equipment design for habitability

13.7.2.8 assess system/equipment design for installability/erectability

12.7.2.9 assess system/equipment design for affordability

13.7.2.10 assess system/equipment design for interoperability

13.7.3 Review sections of T&E reports that address MPT

13.7.3.1 Assess the training device design

• Adequacy of the extent to which training objectives are addressed
• Adequacy of the information reception media
• Adequacy of the skills acquisition media
• Adequacy of fidelity levels of training equipment and simulation systems
• Adequacy of display formats of training equipment
• Adequacy of the range of conditions implemented
• Adequacy of the use of augmented feedback
• Adequacy of the use of prompting and cueing
• Adequacy of programming
• Adequacy of the use of computer based instruction
• Adequacy of instructor interfaces
• Adequacy of data acquisition and recording
• Adequacy of embedded training provisions

13.8 Determine if training audits exist

63.8.1 If no - proceed to step 13.16

13.8.2 If yes - review results of audits

13.8.2.1 Assess Training Effectiveness

• Assess the training for completeness - verify that the training addresses operators and maintainers; and that the training addresses all requirements for training devices
• Assess the training for accuracy - verify that the training requirements are based on job requirements and that training objectives are met
• Assess the training for consistency - verify that the training requirements for the new system are consistent with the requirements for similar systems; and that the training requirements are consistent with required skills and knowledge
• Assess the training for timeliness - verify that the training requirements were identified in time to allow for development of new facilities/devices
• Assess the quality of the training program - verify that trainees possess required skills and knowledge at the termination of training

13.8.2.2 Assess training devices

• verify that training device requirements include specific skills to be acquired
• verify that training device requirements include criteria for judging skills are learned and quantifying the level of skill
• verify that training device requirements include performance measures
• verify that training devices meet required levels of fidelity to actual systems
• verify that training devices control extraneous factors
• verify that each device relates to devices already in use
• verify that device requirements include device supportability/maintainability
• verify that device requirements include estimated life cycle costs
• verify that training device requirements include range of conditions (i.e., maintenance conditions) expected

13.9 Determine if incident reports exist at TYCOM or SYSCOMs which address HSI issues

13.9.1 If no - proceed to step 13.10

13.9.2 If yes - review incident reports

13.9.2.1 Identify where incidents were caused by inadequate human performance

13.9.2.2 Identify where incidents were caused by inadequate manning

13.9.2.3 Identify where incidents were caused by inadequate training

13.9.2.4 Identify where incidents were caused by inadequate procedures

13.9.2.5 Identify where incidents were caused by inadequate design

13.9.2.6 Identify where incidents were caused by inadequate organization

13.10 Determine if any casualty reports (CASREPS) exist

13.10.1 If no - proceed to step 13.11

13.10.2 If yes - review casualty reports

13.10.2.1 Identify where casualties were caused by human performance

13.10.2.2 Identify where casualties were caused by inadequate manning

13.10.2.3 Identify where casualties were caused by inadequate training

13.10.2.4 Identify where casualties were caused by inadequate procedures

13.10.2.5 Identify where casualties were caused by inadequate design

13.10.2.6 Identify where casualties were caused by inadequate organization

13.11 Determine if INSURV data exist which indicate HSI problems

13.11.1 If no - proceed to step 13.12

13.11.2 If yes - review casualty reports

13.11.2.1 Identify where problems were caused by human performance

13.11.2.2 Identify where problems were caused by inadequate manning

13.11.2.3 Identify where problems were caused by inadequate training

13.11.2.4 Identify where problems were caused by inadequate procedures

13.11.2.5 Identify where problems were caused by inadequate design

13.11.2.6 Identify where problems were caused by inadequate organization

13.12 Determine if FLTEX data exist which indicate HSI problems

13.12.1 If no - proceed to step 13.13

13.12.2 If yes - review casualty reports

13.12.2.1 Identify where problems were caused by human performance

13.12.2.2 Identify where problems were caused by inadequate manning

13.12.2.3 Identify where problems were caused by inadequate training

13.12.2.4 Identify where problems were caused by inadequate procedures

13.12.2.5 Identify where problems were caused by inadequate design

13.12.2.6 Identify where problems were caused by inadequate organization

13.13 Determine if Commanding Officer Narrative Reports (CONARs) exist which indicate HSI problems

13.6.13.1 If no - proceed to step 13.14

13.6.13.2 If yes - review casualty reports

13.6.13.2.1 Identify where problems were caused by human performance

13.6.13.2.2 Identify where problems were caused by inadequate manning

13.6.13.2.3 Identify where problems were caused by inadequate training

13.6.13.2.4 Identify where problems were caused by inadequate procedures

13.6.13.2.5 Identify where problems were caused by inadequate design

13.6.13.2.6 Identify where problems were caused by inadequate organization

13.14 Determine if any JAG investigation reports exist which indicate HSI problems

13.14.1 If no - proceed to step 13.15

13.14.2 If yes - review casualty reports

13.14.2.1 Identify where problems were caused by human performance

13.14.2.2 Identify where problems were caused by inadequate manning

13.14.2.3 Identify where problems were caused by inadequate training

13.14.2.4 Identify where problems were caused by inadequate procedures

13.14.2.5 Identify where problems were caused by inadequate design

13.14.2.6 Identify where problems were caused by inadequate organization

13.15 Determine if message traffic indicates problems or positive aspects of the equipment

13.15.1 If no - proceed to step 13.16

13.15.2 If yes - review casualty reports

13.15.2.1 Identify where problems were caused by human performance

13.15.2.2 Identify where problems were caused by inadequate manning

13.15.2.3 Identify where problems were caused by inadequate training

13.15.2.4 Identify where problems were caused by inadequate procedures

13.15.2.5 Identify where problems were caused by inadequate design

13.15.2.6 Identify where problems were caused by inadequate organization

13.16 Conduct interviews of ship personnel concerning HSI issues with the equipment

13.16.1 Determine if interviews are warranted, feasible, and required

13.16.1.1 If no - proceed to step 13.17

13.16.1.2 If yes - plan and conduct interviews

• identify situations where human error has been a problem
• identify situations where human performance has been a problem
• identify situations where team performance has been a problem
• identify situations where manning has been a problem
• identify situations where skill level has been a problem
• identify situations where training has been a problem
• identify situations where documentation has been a problem
• identify situations where information management has been a problem
• identify situations where organizational factors have been a problem
• identify situations where equipment design has been a problem
• identify situations where workplace layout has been a problem
• identify situations where arrangements have been a problem
• identify situations where equipment installation has been a problem
• identify situations where environmental factors have been a problem
• identify positive aspects of HSI design approach

13.17 Conduct HSI observations/evaluations at the worksite

13.17.1 Determine if observations and on-site evaluations are warranted, feasible, and required

13.17.1 If no - proceed to step 13.4, 13.5, 13.6, 13.7, 13.8, 13. 9, 13.10, 13.11, 13.12, 13.13, 13.14, 13.15, or 13.16

13.17.2 If yes - plan and conduct observations/evaluations

• identify situations where human error could be a problem
• identify situations where human performance could be a problem
• identify situations where team performance could be a problem
• identify situations where manning could be a problem
• identify situations where skill level could be a problem
• identify where training could be a problem
• identify where documentation could be a problem
• identify where information management could be a problem
• identify where organizational factors could be a problem
• identify situations where equipment design has been a problem
• identify situations where workplace layout has been a problem
• identify situations where arrangements have been a problem
• identify where equipment installation has been a problem
• identify where environmental factors have been a problem
• identify positive aspects of HSI design approach

5